SOURCE-LINKED INTELLIGENCE

OUTLETS: Output-Length Prediction from Speculative Decoding Backbones

arXiv · AI, language, vision and robotics · article · Sep 1, 2026 · UTC

The heavy-tailed distribution of output lengths in Large Language Model (LLM) serving poses major challenges for resource provisioning and cluster scheduling. Although output-length prediction can mitigate these issues, existing approaches have key drawbacks: external proxy models add substantial latency and often have limited fidelity, whereas internal state-based methods are efficient but rely on shallow probes of current model states. We identify a structural connection between speculative decoding (SD) and length prediction: latent representations produced by the draft decoder in advanced

Read original source ↗ Open in workspace

recordType

paper

region

Global