SOURCE-LINKED INTELLIGENCE

Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning

arXiv · AI, language, vision and robotics · article · Sep 3, 2026 · UTC

Large language models achieve superior performance on tasks that require extended reasoning, but long chains of thought make the KV cache a severe memory bottleneck. Existing KV cache compression methods share one paradigm: score each cached token by some estimate of how much it will matter later, and keep the top-scoring ones. We show that the selection signal contributes almost nothing. Random Attention keeps the prompt and evicts uniformly at random within each attention head, computing no score at all; across four models and six reasoning tasks it matches the strongest prior evictor while

Read original source ↗ Open in workspace

recordType

paper

region

Global