SOURCE-LINKED INTELLIGENCE

Physically Partitioned KVCache Format for CPU--GPU Load Balancing in MoE Inference

arXiv · AI, language, vision and robotics · article · Sep 13, 2026 · UTC

Single-GPU long-context inference with Mixture-of-Experts (MoE) models requires spilling the key-value cache (KVCache) to CPU memory. The spilled KV serves two complementary purposes---transferring to the GPU for attention computation, or computing in-place on the CPU---which demand opposing physical states. The optimal split between them varies with workload, yet existing KVCache abstractions offer only storage semantics over a monolithic object of a single physical state, and cannot express dynamic load balancing. We propose InplaceKVCache, the first KVCache abstraction whose format fixes ea

Read original source ↗ Open in workspace

recordType

paper

region

Global