SOURCE-LINKED INTELLIGENCE

A Calibrated Instrument for Measuring How Inference Optimizations Affect Output Quality

arXiv · AI, language, vision and robotics · article · Sep 16, 2026 · UTC

Large language model optimization is an active research area, spanning quantization of model weights, early-exit methods for skipping layers, and speculative decoding. Each track uses its own quality measures, typically an idiosyncratic benchmark score. Few approach the measurement precision required by other scientific disciplines. We propose a rigorous methodology for measuring output quality, suitable for cross-system and cross-technique comparison. We score outputs with an LLM as a judge, but calibrate the judge formally: we compare its scores on two ordinary runs of a model given the same

Read original source ↗ Open in workspace

recordType

paper

region

Global