SOURCE-LINKED INTELLIGENCE

A Judge Should Know What Changed:Construct Validity for LLM-as-a-Judge Evaluation

arXiv · AI, language, vision and robotics · article · Aug 25, 2026 · UTC

LLM-as-a-judge evaluation is usually assessed by agreement and robustness to surface perturbations, but reliability does not establish construct validity. We formalize construct validity for an evaluator as a two-dimensional profile: invariance S, the probability that a verdict is unchanged under construct-preserving edits, and construct sensitivity R, the probability that it changes under minimal construct-changing edits. We show that S and R are independent and that no scalar summary preserves all relevant comparisons. We measure the profile across 7 judges and 4 domains using 7 construct-ch

Read original source ↗ Open in workspace

recordType

paper

region

Global