SOURCE-LINKED INTELLIGENCE

An Analysis of Training-Free Self-Reported Confidence in Language Models

arXiv · AI, language, vision and robotics · article · Sep 17, 2026 · UTC

Large language models can report a numerical confidence together with generated content, but it is unclear whether this report is more than calibrated rhetoric. We analyze three training-free signals: confidence verbalized with the answer, post-hoc $P(\mathrm{True})$, and agreement with three additional generations on the same 100 TriviaQA questions for two model families. Direct verbalization is a surprisingly strong baseline: after auditing benchmark errors, it reaches AUROC 0.956 and 0.937 for correctness prediction. Three-sample agreement is substantially weaker (0.765 and 0.790), and a fi

Read original source ↗ Open in workspace

recordType

paper

region

Global