SOURCE-LINKED INTELLIGENCE

Who Judges Matters: Measuring Family-Conditioned Preference in LLM-as-Judge Panels

arXiv · AI, language, vision and robotics · article · Sep 15, 2026 · UTC

Who the judge is can affect an LLM-as-judge result, but measuring that effect without confusing it with candidate quality is difficult. We study four open-weight families (Llama 3.1, Qwen 2.5, Gemma 2, and Yi 1.5) in a fully crossed pairwise design with 9,312 judgments. A common per-family statistic is strongly confounded with candidate quality and correlates with Bradley-Terry ability at r = 0.95. We derive a corrected estimator that holds the candidate family fixed and compares judges. All four families then show a positive same-family lift (3.4-8.4 percentage points), with global FPS 0.067

Read original source ↗ Open in workspace

recordType

paper

region

Global