SOURCE-LINKED INTELLIGENCE

Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Caches

arXiv · AI, language, vision and robotics · article · Sep 15, 2026 · UTC

When agentic sessions run to a million tokens with many sessions resident at once, the KV cache and the index that ranks it live in host memory, and the scan that ranks all n keys for a top-k step becomes the traffic that bounds decoding. We present Fathom, a key scan in which each query decides how many bits of each key channel to read. The 4-bit K cache is stored channel-major as bit planes, so a prefix of t planes is exactly the channel's t-bit quantizer, and the query spends its bit budget by reverse water-filling over the variance-weighted importance of its channels. At one million tokens

Read original source ↗ Open in workspace

recordType

paper

region

Global