SOURCE-LINKED INTELLIGENCE

One Size Does Not Fit All: Setting Inference Depth from the Questions a Deployment Actually Asks

arXiv · AI, language, vision and robotics · article · Sep 12, 2026 · UTC

A transformer language model is trained to respond to any prompt, but each deployment asks only a narrow range of questions: a support assistant sees delivery complaints, a coding tool sees Python. Every deployment nonetheless pays the same computation per token. This paper measures how much of that cost is avoidable when the range of prompts is known in advance. The mechanism examined is early exit: a small, trained component - called a readout - is attached to an intermediate layer and proposes a token, and a confidence test decides whether to emit it or to run the remaining layers. The mode

Read original source ↗ Open in workspace

recordType

paper

region

Global