SOURCE-LINKED INTELLIGENCE

SemVerBench: Benchmarking LLM Comprehension of Version-Constraint Resolution Semantics

arXiv · AI, language, vision and robotics · article · Sep 10, 2026 · UTC

Large language model (LLM) coding agents constantly decide whether a version satisfies a constraint such as ^1.2.3 or >=2.0, 1.2 means >=1.3.0) traps every model on Cargo (near 60%), and although standard PEP 440 prefix matching is universal, on zero-pad/post-release corner cases GPT-5.1 collapses (0/26) while Claude stays at 97-100% (verified on a 67-item oracle-validated set). Opus significantly outperforms all other models, and Sonnet outperforms the OpenAI models (McNemar). The failures look more like an activation/application gap than a knowledge gap: injecting the rule or a light correct

Read original source ↗ Open in workspace

recordType

paper

region

Global