SOURCE-LINKED INTELLIGENCE

Disentangling Optimization Scale from Preference Scale in DPO

arXiv · AI, language, vision and robotics · article · Aug 27, 2026 · UTC

Direct Preference Optimization (DPO) is a widely used objective for aligning language models from preference data, with the coefficient $β$ commonly interpreted as controlling the KL constraint to a reference policy. We show that $β$ entangles two distinct roles: it governs the effective inverse preference-noise scale and simultaneously rescales the optimization dynamics, coupling this scale with the effective step size. As a consequence, at a fixed learning rate the achieved policy deviation is non-monotone in $β$: it vanishes in a dead zone at small $β$, reaches a peak at an intermediate val

Read original source ↗ Open in workspace

recordType

paper

region

Global