SOURCE-LINKED INTELLIGENCE

BRACE: Anchored Bellman-Residual Correction for Stale Critics in Asynchronous RL

arXiv · AI, language, vision and robotics · article · Sep 9, 2026 · UTC

Asynchronous reinforcement learning has become the standard way to scale training for large language models (LLM), but the resulting policy lag biases the critic toward the stale behavior policy. Existing work on asynchronous LLM training corrects the actor and leaves this bias unaddressed, while the off-policy value correction of classical RL does not carry over to long-horizon agentic tasks, since a short correction horizon leaves the regression target free of the reward and a long one lets the product of importance ratios drift exponentially with the trajectory length. We propose BRACE, an

Read original source ↗ Open in workspace

recordType

paper

region

Global