SOURCE-LINKED INTELLIGENCE

Osprey: Target-agnostic Pre-training Makes Stronger Drafters in Speculative Decoding

arXiv · AI, language, vision and robotics · article · Sep 8, 2026 · UTC

Speculative decoding is critical for accelerating LLM inference. However, the speedup is fragile: drafters are typically trained against a narrow distribution for a single target model, and their acceptance rate collapses under workload shifts. This is a striking inversion of modern LLM development, where target models are valued precisely for the broad generalization they acquire through large-scale pretraining. We argue that the natural remedy, pretraining, has been hard to apply to drafters because existing recipes are target-specific: the drafter consumes the target's hidden states and is

Read original source ↗ Open in workspace

recordType

paper

region

Global