SOURCE-LINKED INTELLIGENCE

A Probabilistic Interpretation of KV Cache Eviction

arXiv · AI, language, vision and robotics · article · Aug 28, 2026 · UTC

The premise and promise of KV (cache) eviction is simple: higher throughput can be achieved by evicting some entries from the KV cache, at a negligible cost to quality. This holds empirically for many existing methods, though most rely on creative heuristics for selecting which entries to drop. Despite recent advances, the problem of KV eviction has remained informal in the literature. This paper aims to properly formalize this problem through the lens of probabilistic reasoning and reveal what can be learned from this perspective. Concretely, we (1) formalize the problem of KV eviction and, u

Read original source ↗ Open in workspace

recordType

paper

region

Global