SOURCE-LINKED INTELLIGENCE

Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking

arXiv · AI, language, vision and robotics · article · Sep 14, 2026 · UTC

Recent work shows that models that learn to reward hack on RL environments can become broadly misaligned, and that reframing reward hacking as acceptable behavior during training (inoculation prompting, or IP) blocks this generalization. We ask whether synthetic document finetuning (SDF) can inoculate a model against future training we don't intervene on. We add synthetic documents framing reward hacking as acceptable behavior to a model's midtraining corpus, and then train these models with RL on exploitable environments, teaching them to reward hack. Behaviorally, midtraining succeeds: model

Read original source ↗ Open in workspace

recordType

paper

region

Global