SOURCE-LINKED INTELLIGENCE

The Filter Metric is Safety-Critical: Phantom Advantages in Group-Relative RL under Shaped Rewards

arXiv · AI, language, vision and robotics · article · Sep 12, 2026 · UTC

Group-relative policy optimization (GRPO and descendants) can discard no-contrast rollout groups through dynamic sampling, while practical implementations expose a configurable filter metric. We identify and quantify a metric-predicate mismatch under composite shaped rewards. When filtering follows the shaped training score rather than the task outcome, all-fail groups retain nonzero within-group spread and pass the predicate; standard-deviation normalization then promotes shaping differences among failures to full-size phantom advantages. In a controlled GSM8K comparison (Qwen2.5-1.5B, LoRA),

Read original source ↗ Open in workspace

recordType

paper

region

Global