SOURCE-LINKED INTELLIGENCE

Score Centering Stabilizes Off-policy Reinforcement Learning

arXiv · AI, language, vision and robotics · article · Sep 17, 2026 · UTC

Reinforcement learning (RL) of large language models is notoriously sensitive to small differences between training and inference engines, often referred to as the training-inference mismatch (TIM). However, completely eliminating TIM is impractical, as it would come at a major cost to rollout efficiency. In this paper, we show that the instability of RL under TIM is primarily caused by drift: a persistent bias between training and inference engines that accumulates with every training step. We derive an additive "score centering" correction term that stabilizes RL under TIM by canceling drift

Read original source ↗ Open in workspace

recordType

paper

region

Global