SOURCE-LINKED INTELLIGENCE

Good Pretraining, Bad SFT: Checkpoint Quality Across the Training Stack

arXiv · AI, language, vision and robotics · article · Sep 8, 2026 · UTC

Language-model checkpoints are commonly selected by pretraining loss or benchmark scores, assuming that the highest-scoring checkpoint will remain the best starting point for subsequent training. We show that this assumption can fail in a full 30B mixture-of-experts training pipeline. The checkpoints that perform better after the full downstream training stack also have higher solution density, i.e., retain downstream performance under local weight perturbations.

Read original source ↗ Open in workspace

recordType

paper

region

Global