SOURCE-LINKED INTELLIGENCE

Clustering-Based Balanced Sampling and Allocation with Data Parallelism for High-Performance Fine-Tuning

arXiv · AI, language, vision and robotics · article · Sep 11, 2026 · UTC

Instruction-tuning datasets for large language models (LLMs) are often large, redundant, and imbalanced, limiting efficient adaptation. Naive large-batch fine-tuning repeatedly includes overrepresented sample groups while weakly covering underrepresented but informative ones, especially under data parallelism (DP) across multiple GPUs. We propose CluSTER, a Cluster-aware balanced Sampling framework for Training Efficient data Reduction in DP instruction tuning. CluSTER curates a representative reduced dataset through gradient-space clustering and DP-aware balanced allocation, ensuring dual-lev

Read original source ↗ Open in workspace

recordType

paper

region

Global