SOURCE-LINKED INTELLIGENCE

Auditing MCQA Benchmarks through Probability Landscapes

arXiv · AI, language, vision and robotics · article · Aug 31, 2026 · UTC

As Large Language Models rapidly advance, performance on standard multiple-choice question answering (MCQA) benchmarks is reaching saturation. While the community has responded by developing increasingly difficult datasets, validating question quality and filtering flawed items remains a labor-intensive process. To provide a scalable diagnostic approach, we propose a two-component probabilistic framework for auditing MCQA benchmarks using model output distributions. First, for benchmark-level analysis, we characterize the probability landscape using the top prediction probability ($P_{top1}$)

Read original source ↗ Open in workspace

recordType

paper

region

Global