- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2(+1)
Read review
Paper summary
This paper presents a surrogate-distribution framework for correcting the attention approximation introduced by KV cache merging. The authors conclude that existing methods can be understood within a common framework, where different correction strategies use different surrogate distributions. To handle cases where keys are not aligned, the authors introduce an additional correction in logit space. Experiments demonstrate improved approximation fidelity over the one-atom baseline.
Critical strengths
-
The paper presents rigorous score and value corrections formulas and establishes exactness for two-token merges under keyalignment. The appendix further provides error bounds for larger clusters and show that the error does not grow linearly with the number of merged clusters. The logit-space variance decomposition into parallel and orthogonal components demonstrates careful considerations of the attention structure.
-
The surrogate-distribution perspective provides a unified framework for understanding KV cache merging correction methods. It shows that KeepKV’s count-based correction corresponds to a one-atom surrogate, while Taylor expansion corresponds to a Gaussian surrogate. While the two-atom surrogate achieves exact merging for two tokens, it is under the strong key-alignment assumption. The correction formulas proposed in this paper may extend existing methods based on the key-alignment assumption to scenarios with misaligned keys.
Critical weaknesses
-
The research problem is not clearly presented. No measurement data or illustrative figure is given to help justify the limitation of existing works. The paper organization is also strange: the introduction is too short, and there is not clear summarization of the key design challenges and insights. It is hard to grasp the novelty of this paper.
-
All experiments cover only two 7B–8B models on a single H100 GPU. Although the results show that the proposed corrections are effective on the evaluated benchmarks, it remains unclear how robust the approximation and its performance are across different model sizes, attention configurations, and context lengths. A more comprehensive sensitivity study would strengthen the method’s generality.
-
The two-atom surrogate’s exactness property relies on the simplification of key-alignment assumption. Although the correction in logit space introduced by the authors partially addresses misaligned keys, the paper neither provides systematic analysis of how frequently this assumption is violated in practice nor quantifies the degradation when this assumption fails. A clear characterization of the relationship between the degree of key misalignment and the approximation error would strengthen the paper
Questions for authors
See the above weakness points.
2 comments
Thank you for the detailed response and the revised manuscript. The restructured introduction now states the problem, the requirements a valid correction must meet, and the key insight clearly, and the pointers to Figs. 1–2 make the limitation of existing corrections concrete. Your reading of Fig. 2 also answers part of my question on key alignment: the merge threshold bounds the worst-case misalignment, and at m = 2 the residual error is attributable to angular spread alone. I raise my score accordingly from 1 to 2.
Two concerns remain. First, the evaluation is still limited to two 7–8B models, with RULER accuracy reported only at 16K. Second, the alignment analysis covers m = 2, whereas the distribution of within-cluster misalignment and of cluster sizes at practical thresholds is not reported. Together with the observation that the better-approximating two-atom variants are not more accurate per byte of cache, the practical benefit over existing merging and eviction methods remains limited.
Thank you for the review, and for recognising the rigour of the corrections and the unifying value of the surrogate-distribution view. We would like to address the soundness concern directly and worry that this has led to the Clear Reject recommendation, since the whole aim of the paper is to put KV-cache merging on a sound footing, and believe that our surrogate distribution formulation does that cleanly.
Problem statement and presentation. We accept that the introduction is too short and does not make the challenges and insights explicit. In the revised version we have restructured it around three points: (i) the problem, that an averaged merged slot has the probability mass of one token and a fixed value, whereas the true contribution is query dependent; (ii) the challenge, that the correction must hold for queries not seen at merge time, from a constant amount of storage per slot; and (iii) the insight, that the exact contribution is a cumulant generating function, so that every correction is derived analytically from the choice of surrogate distribution.
On justifying the limitation of existing work: Figure 1 shows on a controlled cluster that the one-atom surrogate (the KeepKV correction) cannot represent the curvature of the true score, and Figure 2 measures the same failure on real Llama-3.1-8B activations. The uncorrected and count-corrected variants have the largest score and value errors at every threshold. We have pointed to both figures from the revised introduction.
Model sizes, attention configurations and context lengths. Please see our replies to Reviewer oLmh on model size and Reviewer zSZg on context length.
How often key alignment is violated, and the cost when it is. This is the question Figure 2 is designed to answer. Every two-token cluster formed at threshold t has key cosine of at least t in every head, so the threshold sets the worst-case violation. At m=2 the two-atom corrections are exact for aligned keys (Props. 3.2 and 3.3), so the residual plotted at each threshold is the error due to misalignment alone, and it falls monotonically as alignment tightens. The logit-space variant, which drops the assumption, removes most of the score residual. We will state this reading of Figure 2 explicitly.