SOURCE-LINKED INTELLIGENCE

Large Language Models Systematically Favor Popular Options: Evidence and Mitigation Across MCQs

arXiv · AI, language, vision and robotics · article · Aug 29, 2026 · UTC

Multiple-choice questions (MCQs) are a standard format for evaluating large language models (LLMs), yet the popularity of answer options can confound evaluation. Modern LLMs systematically prefer popular but incorrect options over less popular correct ones, a vulnerability we call \textbf{popularity bias}. This pattern aligns with confidence miscalibration: model confidence remains high even as accuracy collapses for popular options. To systematically isolate this phenomenon, we introduce \textbf{PopMCQ}, a benchmark with six controlled strategies that vary option popularity while keeping the

Read original source ↗ Open in workspace

recordType

paper

region

Global