SOURCE-LINKED INTELLIGENCE

Can We Trust the Judges? Validation of Factuality Evaluation Methods via Answer Perturbation

arXiv · AI, language, vision and robotics · article · Sep 14, 2026 · UTC

Evaluating the factual correctness of large language models (LLMs) is vital for many applications. But are our evaluation tools themselves trustworthy? Despite the rise of factuality-based metrics, their sensitivity and reliability remain underexplored. This paper introduces a meta-evaluation framework that systematically tests these metrics using controlled corruptions of gold standard answers. Our method generates ranked outputs with known degrees of degradation to probe how metrics capture nuanced changes in truthfulness. Our experiments reveal that pipeline-based methods, such as the RAGAS

Read original source ↗ Open in workspace

recordType

paper

region

Global