SOURCE-LINKED INTELLIGENCE

Inducing Emergent Misalignment from Reward Hacks with Iterative DPO

arXiv · AI, language, vision and robotics · article · Sep 6, 2026 · UTC

Reward hacking during reinforcement learning from verifiable rewards (RLVR) can induce reward seeking and broad misalignment in language models. Studying this misgeneralization is important for developing better threat models and countermeasures, but is often infeasible due to the cost of RL on large models. As an alternative, we propose studying emergent misalignment from iterative DPO, which preserves important properties of RLVR while reducing costs and enabling training on popular finetuning APIs. In practice, we find that training GPT-4.1 with iterative DPO on a single-turn reward hacking

Read original source ↗ Open in workspace

recordType

paper

region

Global