arXiv:2607.01520 · v2

The risk of KV cache compression

首次提交 2026-07-02 06:36

版本更新 2026-11-11 17:13

雷达观测 2026-11-13 11:11

摘要

Transformer inference on long sequences is expensive because softmax attention repeatedly reads from a large KV cache. The prevalent approach to this bottleneck is KV cache compression, which replaces the full cache with a compact summary. Despite its practical importance, the design of such summaries is largely driven by empirical experimentation. On the theoretical side, existing results show that KV cache compression can be impossible in the worst case, but offer little systematic guidance for designing algorithms in regimes where accurate compression is possible. We bridge this gap by characterizing the minimax risk of KV cache compression in terms of the intrinsic compressibility of a cache, revealing when and how accurate compression is possible. In particular, we show that, once attention can sharply retrieve individual tokens, compression can only uniformly outperform random sampling by exploiting information about future queries, regardless how redundant the cache is inherently. Additionally, our results yield novel design principles for KV cache compression under causal masking that map efficiently to prefill and autoregressive decoding while achieving minimax-optimal risk. We instantiate these principles in a practical algorithm and illustrate how our theoretical findings can be put to practical use. Overall, our results opens a principled avenue for KV cache compression with theoretical guarantees.

作者备注：Accepted at NeurIPS 2026