SOURCE-LINKED INTELLIGENCE

Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall

arXiv · AI, language, vision and robotics · article · Sep 1, 2026 · UTC

Logit-based knowledge distillation (KD) is used to train smaller language models (LMs) via supervision from stronger teachers, but whether its benefits are consistent across training stages remains unclear. Through controlled experiments, we find that forward Kullback-Leibler (KL) distillation--the standard KD formulation--with post-trained teachers behaves fundamentally differently during mid-training, an intermediate phase of self-supervised learning on curated corpora. Surprisingly, while forward KD simultaneously improves reasoning and factual recall during pre-training relative to standar

Read original source ↗ Open in workspace

recordType

paper

region

Global