SOURCE-LINKED INTELLIGENCE

Beyond Scores: Understanding LLM-as-a-Judge Mechanisms in Summarization Evaluation

arXiv · AI, language, vision and robotics · article · Sep 1, 2026 · UTC

LLM-based evaluators of natural language generation (NLG) quality are widely deployed as scoring tools and as automated training signals, yet the internal procedure by which they assign a rating remains poorly understood. We investigate this procedure mechanistically through an eight-attack perturbation taxonomy across the Readability and Adequacy dimensions of NLG quality, a generation pipeline that produces paired clean and corrupt summaries with controlled error intensity and explicit token-level modification maps, and a four-experiment battery of causal tracing, logit-lens vocabulary proje

Read original source ↗ Open in workspace

recordType

paper

region

Global