SOURCE-LINKED INTELLIGENCE

MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference

arXiv · AI, language, vision and robotics · article · Sep 7, 2026 · UTC

Key--value (KV) cache compression is an effective way to reduce the memory overhead of large language model (LLM) inference, particularly for long-context workloads. However, existing compression methods make different trade-offs among accuracy, inference latency, and peak KV cache memory utilization, making a single fixed configuration unsuitable across different prompts and resource constraints. We introduce MetaKV, an adaptive framework that selects a KV cache compression configuration for each input prompt based on user-specified latency and peak memory budgets. MetaKV uses lightweight pre

Read original source ↗ Open in workspace

recordType

paper

region

Global