SOURCE-LINKED INTELLIGENCE

A Token-Level Analysis of Sampled-Token Reverse-KL On-Policy Distillation

arXiv · AI, language, vision and robotics · article · Aug 26, 2026 · UTC

On-policy distillation (OPD) supervises a student on its own trajectories with token-level signals from a frozen teacher, yet how a sampled loss allocates updates across tokens remains poorly understood. We analyze the gradient of the per-token K2 estimator of reverse KL with respect to the student logits. The $\ell_1$ norm of this gradient factorizes into the absolute teacher--student log-probability gap and a student-side softmax factor that grows as the sampled token becomes less likely under the student. In our math-distillation runs, these per-token norms are highly non-uniform: low-stude

Read original source ↗ Open in workspace

recordType

paper

region

Global