SOURCE-LINKED INTELLIGENCE

SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents

arXiv · AI, language, vision and robotics · article · Sep 8, 2026 · UTC

SWE-Bench Pro has emerged as a standard benchmark for evaluating software engineering agents on challenging repository-level tasks. However, our analysis work show that its evaluation is undermined by two sources of unreliability: reward hacking, enabled by leakage of gold solutions or hidden evaluation information, and task quality issues, including misleading problem statements and improperly scoped tests. These issues can inflate benchmark performance and obscure agents' true coding ability. We present SWE-Bench Pro Verified, a verified version of SWE-Bench Pro that addresses both problems.

Read original source ↗ Open in workspace

recordType

paper

region

Global