SOURCE-LINKED INTELLIGENCE

What Does Privileged Information Add to On-Policy Self-Distillation?

arXiv · AI, language, vision and robotics · article · Sep 17, 2026 · UTC

On-policy self-distillation (OPSD) lets a language model learn from a frozen copy of itself that sees an answer or a worked solution. Giving the teacher this extra information seems to offer the student more to learn, but how much does it add beyond distillation itself? To isolate that contribution, we construct AMPLE-Math, a reusable suite of 5,319 mathematical problems with six reasoning views that share the same answer, and compare each view with matched reference-free distillation. With a thinking-enabled teacher supervising direct-response rollouts, reference-free distillation accounts fo

Read original source ↗ Open in workspace

recordType

paper

region

Global