SOURCE-LINKED INTELLIGENCE

Strong Drafts Need Compact Memories: Long-Context Speculative Decoding with Compressed KV Cache

arXiv · AI, language, vision and robotics · article · Aug 31, 2026 · UTC

Long-context LLM applications such as document summarization and multi-turn agents require generation from prefixes spanning tens of thousands of tokens, making decoding latency a major bottleneck. Speculative decoding (SD) reduces latency without changing model outputs, but its speedup depends on both accepted draft tokens and draft-step latency: Lightweight drafts are fast but lack the capacity to capture long-range dependencies, whereas strong independent drafts recover acceptance but incur growing KV-access cost at long prefixes. We introduce memory-augmented drafting for long-context SD,

Read original source ↗ Open in workspace

recordType

paper

region

Global