SOURCE-LINKED INTELLIGENCE

Does Reasoning Improve Psychological Depth in Large Language Models? It Depends on Who's Judging

arXiv · AI, language, vision and robotics · article · Sep 12, 2026 · UTC

LLM-as-a-Judge evaluators are increasingly used to score open-ended generation, yet a judge's correlation with human ratings on its development set may not guarantee valid measurement when outputs are closely matched and human preferences are subjective. We study this failure mode through psychological depth in short stories. Seven human readers and an LLM-judge ensemble selected on the original scalar Psychological Depth Scale dataset ($ρ= 0.646$) evaluated 60 blinded, prompt-matched story pairs from GPT-5 vs.\ GPT-4o and DeepSeek-R1 vs.\ DeepSeek-V3. Human preferences showed no universal rea

Read original source ↗ Open in workspace

recordType

paper

region

Global