SOURCE-LINKED INTELLIGENCE

Beyond Accuracy: How Procedural Traces Shift the Decision Criterion of LLM Overseers

arXiv · AI, language, vision and robotics · article · Sep 16, 2026 · UTC

Organizations increasingly use oversight loops where one large language model (LLM) audits another's outputs alongside procedural traces of claimed steps. A common concern about such LLM-as-a-judge pipelines is that detailed traces make overseers gullible. Using signal detection theory, we audit five LLM overseers on 19 compliance tasks (4,551 analyzed judgments), varying only trace detail and evidence labeling. With disconfirming evidence always visible, error detection remains near ceiling. Instead, elaborate traces shift the decision criterion toward rejection, increasing false alarms in su

Read original source ↗ Open in workspace

recordType

paper

region

Global