SOURCE-LINKED INTELLIGENCE

Policy Loopholes in Agent Evaluation: When Policy Ambiguity Masquerades as Agent Error

arXiv · AI, language, vision and robotics · article · Sep 13, 2026 · UTC

Agent benchmarks evaluate policy compliance but assume each policy determines a unique correct action. Natural-language policies can violate this assumption through silence, ambiguity, or contradiction, admitting multiple defensible readings that a single gold trajectory cannot capture. Auditing two $τ^2$-bench domains, we develop a taxonomy of such policy loopholes and show that affected tasks produce unreliable scores: they lower scores across different models in different ways and make every model less consistent across repeated trials. A cross-domain comparison reveals that exploitability

Read original source ↗ Open in workspace

recordType

paper

region

Global