- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper asks how much KV-cache merging can actually improve over plain eviction in frozen long-context models, and largely answers "not much." It proves that value-only merging can exactly recover dense attention for every future query only if eviction was already exact, and derives a calibrated criterion for when approximate merging beats eviction. Measuring that criterion under oracle routing across four rotary models from three families, only a few percent of evicted tokens qualify even under verbatim repetition, because contextualization and rotary embeddings keep repeated tokens' keys distinct; a non-rotary model is far more mergeable, as the theory predicts. A matched-protocol re-evaluation of published operators finds no resolved gains and several regressions, mostly driven by key rewriting. The authors' own fixed-key construction gives the only resolved development gain, which largely disappears on held-out data and reverses on a second model.
Critical strengths
- The related-work treatment of merging is organized and current enough to serve as a compact mini-survey of the field.
- The central finding that input repetition does not generally produce mergeable keys, because contextualisation and rotary embeddings keep them distinct, is both theoretically clean and independently validated across families, including a falsifiable contrast with a non-rotary model rather than a post-hoc story.
- It critiques a literature whose gains are small and often measured against weak baselines, and does so under a matched protocol that reports negative and unresolved results honestly. Since KV-cache merging sees little real deployment, this kind of skeptical re-evaluation is a useful service.
Critical weaknesses
- The load-bearing Gaussian model for the logit gap assumes query samples are independent, but supported only by a weak check. Decode queries are a correlated, drifting sequence, and the paper's own drift numbers show the gap's mean shifts during decoding, yet the break-even threshold that drives every downstream claim rests on that untested independence premise.
- The key quantities (variance, oracle routing ratio, query covariance) are all measured from the uncompressed run. In deployment the compressed cache emits different tokens and therefore different queries, so the merge/no-merge decision is made from statistics that deployment will not reproduce.
- As the author stated, the methods most used in practice, like trained compression and per-query retrieval over the full cache (e.g. Quest), are explicitly excluded. Because merging is rarely deployed anyway, a negative result about its ceiling has limited practical stakes.
Questions for authors
No major concerns. Overall, I like this paper, though there are a lot of complex theories done by LLMs.