SOURCE-LINKED INTELLIGENCE

RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests

arXiv · AI, language, vision and robotics · article · Aug 28, 2026 · UTC

Coding agents are now commonly evaluated on the SWE-bench family of benchmarks, whose tasks are built from curated GitHub issues: long, structured, and information-rich. Real user requests, however, are typically far shorter and less structured. To characterize this gap, we define a six-category information taxonomy and four dimensions of linguistic style, and apply them to real user prompts from SWE-chat and problem statements from SWE-bench Verified and Pro. We find that requests carrying only a problem statement, alone or with limited additional context, account for 88% of real prompts but

Read original source ↗ Open in workspace

recordType

paper

region

Global