SOURCE-LINKED INTELLIGENCE

Single-Query Black-Box Calibration Auditing via Logit Bias

arXiv · AI, language, vision and robotics · article · Sep 4, 2026 · UTC

Evaluating the calibration of Large Language Models (LLMs) is critical for their safe deployment as zero-shot classifiers. Yet, commercial API providers increasingly hide the continuous output probabilities required by standard calibration metrics. To bypass this opacity, we demonstrate that any LLM API exposing a logit\_bias parameter can be mathematically manipulated to evaluate exact probability thresholds using strictly one query per sample. Leveraging this mechanism, we introduce a novel and provably consistent estimator of the True Calibration Error for binary tasks. Our approach therefo

Read original source ↗ Open in workspace

recordType

paper

region

Global