SOURCE-LINKED INTELLIGENCE

Are We Grading Properly? Understanding Failure Modes in Medical Benchmarks

arXiv · AI, language, vision and robotics · article · Sep 10, 2026 · UTC

Medical evaluation is shifting from static option-based questioning to realistic clinical scenarios with open-ended output modes. Grading these at scale naively, however, is expensive, and rubric-based evaluation has become the dominant scalable alternative. We ask what happens when the rubrics themselves are not airtight, and whether such flaws can be detected and corrected. We apply RIFT, a global rubric failure taxonomy, to two clinical benchmarks (HealthBench Professional and LiveMedBench), and find failure modes are meaningful: on HealthBench Professional an LLM judge flags 29.6% of crite

Read original source ↗ Open in workspace

recordType

paper

region

Global