SOURCE-LINKED INTELLIGENCE

Carryover Drafting: Recycling Rejected States for Speculative Decoding

arXiv · AI, language, vision and robotics · article · Sep 13, 2026 · UTC

Speculative decoding accelerates LLM inference by verifying multiple drafted tokens in parallel, allowing a single target forward pass to accept several tokens. By construction, verification computes representations for both accepted and rejected tokens. Yet, conventional drafters retain only the representations of accepted tokens, leaving the substantial verifier computation spent on rejected tokens effectively wasted. We find that these discarded hidden states generated during target forward retain useful information about future tokens that can improve subsequent drafts. However, realizing

Read original source ↗ Open in workspace

recordType

paper

region

Global