acceptodds
Under review as a conference paper at ICLR 2027

On the Limits of KV Cache Merging

Abstract

Merging is an appealing alternative to evicting tokens from the key–value (KV) cache of a long-context language model: rather than discard an entry, fold it into one that is kept. Existing KV-cache merging methods broadly follow two strategies: directly folding an evicted entry into a retained key–value slot, or augmenting the merge with a correction for the attention mass represented by that slot. We examine both and show that their headroom is limited, while a larger opportunity lies in deciding what to evict. We first consider merges that leave keys untouched and only edit values. We prove that, for keys in general position, such a merge can be exact for every future query only when evicting the token would already have lost nothing, and derive how close two keys must be for an approximate merge to beat eviction. On four rotary models from three families, even with each evicted token paired with its best possible kept entry, at most a few percent meet this condition, including in text repeated verbatim: contextualisation and the rotary embedding keep the keys of repeated tokens apart. We then consider compensated merges, which add a per-slot attention-mass correction and escape the theorem, yet under a matched protocol no published operator we port shows a resolved LongBench gain over its own eviction baseline; those that rewrite retained keys lose, and in the clearest case the loss disappears once the same merges leave the keys in place. Our own construction keeps every key fixed, fits the value mixture and the mass correction jointly against the attention output, and keeps only the merges that lower it; its gain is +0.70 on the 400-sample development split, where it resolves, shrinks to +0.18, unresolved, on the 800-sample held-out split, and reverses to −0.74 on a second model, Qwen2.5-7B-Instruct. Merging is thus bounded by key geometry. The more effective strategy is to improve which tokens are evicted rather than how they are merged: since eviction error is the evicted attention mass times a value gap, weighting attention by value norm gives a selector indistinguishable from the strongest eviction methods on LongBench under a shortened-generation protocol, where margins of a few tenths of a point are below what the benchmark resolves, and better on needle retrieval.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache compression, token merging, token eviction, long-context inference, efficient LLM inference, rotary position embedding, negative results

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_D6UwOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks how much KV-cache merging can actually improve over plain eviction in frozen long-context models, and largely answers "not much." It proves that value-only merging can exactly recover dense attention for every future query only if eviction was already exact, and derives a calibrated criterion for when approximate merging beats eviction. Measuring that criterion under oracle routing across four rotary models from three families, only a few percent of evicted tokens qualify even under verbatim repetition, because contextualization and rotary embeddings keep repeated tokens' keys distinct; a non-rotary model is far more mergeable, as the theory predicts. A matched-protocol re-evaluation of published operators finds no resolved gains and several regressions, mostly driven by key rewriting. The authors' own fixed-key construction gives the only resolved development gain, which largely disappears on held-out data and reverses on a second model.

Critical strengths
  1. The related-work treatment of merging is organized and current enough to serve as a compact mini-survey of the field.
  2. The central finding that input repetition does not generally produce mergeable keys, because contextualisation and rotary embeddings keep them distinct, is both theoretically clean and independently validated across families, including a falsifiable contrast with a non-rotary model rather than a post-hoc story.
  3. It critiques a literature whose gains are small and often measured against weak baselines, and does so under a matched protocol that reports negative and unresolved results honestly. Since KV-cache merging sees little real deployment, this kind of skeptical re-evaluation is a useful service.
Critical weaknesses
  1. The load-bearing Gaussian model for the logit gap assumes query samples are independent, but supported only by a weak check. Decode queries are a correlated, drifting sequence, and the paper's own drift numbers show the gap's mean shifts during decoding, yet the break-even threshold that drives every downstream claim rests on that untested independence premise.
  2. The key quantities (variance, oracle routing ratio, query covariance) are all measured from the uncompressed run. In deployment the compressed cache emits different tokens and therefore different queries, so the merge/no-merge decision is made from statistics that deployment will not reproduce.
  3. As the author stated, the methods most used in practice, like trained compression and per-query retrieval over the full cache (e.g. Quest), are explicitly excluded. Because merging is rarely deployed anyway, a negative result about its ceiling has limited practical stakes.
Questions for authors

No major concerns. Overall, I like this paper, though there are a lot of complex theories done by LLMs.

Reviewer_U8RROpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the limits of KV cache merging in language models. It establishes an exact-recovery limitation for value-only merging and identifies key geometry and calibration drift as practical obstacles. Controlled experiments find limited and inconsistent gains from the evaluated merging operators. The authors also propose fixed-key compensated merging and value-weighted selection.

Critical strengths

S1. The paper combines theoretical analysis with empirical diagnostics to explain how key geometry and calibration drift can limit KV cache merging. S2. The paper shows that lower attention-output error or perplexity does not necessarily improve downstream task performance, highlighting an important limitation of these metrics.

Critical weaknesses

W1. The introduction briefly covers the background and problem setting before moving into a lengthy list of contributions. It does not sufficiently explain the motivation and research gap or establish a coherent narrative connecting the theoretical analysis, merging experiments, and proposed method.

W2. Many prior work has already examined limitations of conventional KV merging and developed specialized merging mechanisms. Although this paper adds theoretical analysis, it does not sufficiently clarify what new, generalizable design insights emerge beyond these existing findings, particularly for operators outside the value-only theorem. Moreover, the proposed fixed-key method does not provide consistent cross-model gains. These results leave the practical guidance offered by the analysis insufficiently established.

Questions for authors

See weak points

Reviewer_M7yYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates the theoretical and empirical limitations of training-free KV cache merging in large language models. The authors first establish an exactness theorem characterizing the limitations of value-only merging and develop a Gaussian logit-gap model to analyze approximate mergeability. They further examine how contextualization and RoPE affect key similarity and the feasibility of merging redundant tokens. Building on these analyses, the paper evaluates several existing KV cache merging methods against corresponding eviction-only baselines and proposes two approaches: a fixed-key merging method combining value mixing with attention-mass compensation, and value-weighted selection (VWS), which incorporates value information into token selection. The experimental results suggest that conventional KV cache merging offers limited advantages over eviction under the evaluated settings.

Critical strengths

S1. Relevant research problem and a useful theoretical perspective. The paper addresses an important question in efficient LLM inference: whether merging discarded KV cache entries provides meaningful advantages over direct eviction. The exactness theorem offers a theoretical characterization of the limitations of value-only merging, while the analysis of contextualization and RoPE provides useful insights into why seemingly redundant tokens may not be readily mergeable.

S2. Systematic evaluation of existing KV cache merging methods. The paper examines multiple existing merging approaches and compares them against corresponding eviction-only baselines. This comparison is valuable because improvements attributed to merging may otherwise originate from differences in token selection or cache budget allocation. The analysis also distinguishes between value-only merging, attention-mass compensation, and key-rewriting methods.

Critical weaknesses

W1. Limited methodological novelty and inconsistent empirical improvements. The paper proposes two methods, but neither demonstrates consistent improvements over strong baselines. The fixed-key merging method (ll. 302–323) combines existing ideas of value mixing and attention-mass compensation, while its LongBench improvement decreases from +0.70 on the development split to a statistically insignificant +0.18 on held-out data and becomes −0.74 on Qwen2.5-7B. Similarly, VWS primarily integrates existing techniques from SnapKV, VATP, and adaptive budget allocation, without outperforming strong baselines on LongBench (25.71 vs. 26.00 for SnapKV).

W2. Non-standard LongBench evaluation protocol. The evaluation code (evaluation/tasks/longbench.py, lines 97–100) uses a generic prompt instead of the official task-specific LongBench templates. For instance, summarization tasks such as GovReport and MultiNews may receive the raw context without an explicit summarization instruction, while their outputs are still evaluated using ROUGE-L. Furthermore, the implementation limits generation to 64 tokens and truncates inputs exceeding 8,000 tokens, whereas the official LongBench protocol specifies task-dependent generation lengths, reaching 512 tokens for several summarization tasks. Minor Weaknesses

M1. Poor presentation and readability. Both the abstract and introduction are overly dense, containing excessive technical details, numerical results, and statistical information that obscure the paper's main message. The introduction presents five lengthy contributions without clearly establishing the central research question or distinguishing their relative significance. Several paper-specific concepts are also introduced without sufficient explanation, making the paper difficult to follow.

M2. Missing citations and inaccurate references. Several important references are missing, including the original RoFormer paper (Su et al., 2021) for RoPE, LongBench (Bai et al., 2024), and RULER (Hsieh et al., 2024) for the evaluation benchmarks, and the technical reports of the evaluated models: Llama 3 (Grattafiori et al., 2024), Qwen2.5 (Yang et al., 2024), Qwen3 (Yang et al., 2025), Mistral 7B (Jiang et al., 2023), and GPT-2 (Radford et al., 2019). Additionally, the CAKE reference (ll. 594–596) incorrectly lists the second author as Yuxin Cao instead of Yuchen Cao and omits Jianguo Li from the author list. These issues should be corrected to ensure citation accuracy and completeness.

Questions for authors

Q1. Non-standard LongBench evaluation protocol. Could the authors explain why the evaluation uses a generic prompt and a fixed 64-token generation limit instead of the official LongBench protocol? Do the main conclusions remain consistent when using the official task-specific prompts and generation lengths?

Q2. Scope of the theoretical conclusions To what extent do the theoretical limitations established for value-only merging generalize to more flexible merging strategies, such as attention-mass compensation and key rewriting? Could the authors clarify which conclusions are theoretically guaranteed and which are supported only by empirical observations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.