SOURCE-LINKED INTELLIGENCE

Optimizing Sparse Outcomes Through Dense Behavioral Signals via Value-Guided Preference Distillation

arXiv · AI, language, vision and robotics · article · Sep 13, 2026 · UTC

Aligning multi-turn dialogue agents is usually framed as matching turn-level human preferences, yet direct optimization of long-term outcomes is often ineffective and prone to reward hacking. We formulate long-horizon dialogue optimization as a multi-objective reinforcement learning problem and train a multi-head value model that predicts a vector of observed user behaviors across multiple look-ahead horizons. Our findings demonstrate that a scalarized composite of dense auxiliary behavioral signals enables effective credit assignment and optimization of sparse outcomes. However, optimizing un

Read original source ↗ Open in workspace

recordType

paper

region

Global