- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
Let S be the softmax normalization mass, N the unnormalized weighted-value sum, and attention output A = N/S. This paper gives several theoretical analyses of the effect of KV cache reduction on A. And they try to analyze under what condition this reduction has less effect on A.
Critical strengths
- The proposed problem is meaningful.
Critical weaknesses
- As a theoretical paper, it is very hard to follow. I have the feeling that the formalization is more than the real conceptual content.
- There is a gap between the theoretical analysis and the experimental analysis. I don't think the experiment is strong enough to show the discussion of the theory part.
Questions for authors
How does the proposed transition-aware criterion correlate with end-to-end downstream degradation, such as LongBench accuracy, long-context perplexity, or generation divergence, rather than local attention-state error alone?
3 comments
I appreciate the author's reply. It addressed some of my concern. My remaining concern is mainly the presentation. I have a lot of experience on this topic; however, it makes it hard to understand the content. I think this paper could be a good paper if it had much clearer writing. So I encourage the author to keep refining it. I'll keep my original score. Thanks.
Response to the Review
We thank the reviewer for the follow-up and appreciate your acknowledgment that our response addressed some of your concerns. We take the remaining presentation concern seriously.
To make the conceptual contribution clearer, the paper's argument can be summarized in three steps:
1. State-substitution obstruction. Matching the current attention output does not ensure reliable cache replacement: two states may agree at the current read yet diverge after the same append.
2. Sharp recovery. Theorem 1 identifies conditions under which current-output error does control the complete post-append discrepancy, with a sharp, attainable guarantee for unchanged-row selection.
3. Constructor-relative transfer. Theorem 2 extends this guarantee beyond pure selection by comparing constructed representations with a same-support selection reference and separately controlling response and normalization-mass defects.
The experiments investigate these mechanisms rather than claiming a new eviction policy or an unconditional long-horizon guarantee.
We recognize that the manuscript could communicate this progression more directly, with the central conceptual results distinguished more clearly from the technical derivations and experimental protocol. We appreciate your constructive feedback on this issue.
Response to the Review
We thank the reviewer for recognizing the importance of the problem. We agree that the conceptual contribution and its connection to model behavior should be easier to identify. The central question is whether a compressed cache is a reliable persistent replacement, rather than merely whether it matches one current attention output.
1. Why does state substitution require a different analysis?
Decoding continues to read and extend a compressed KV cache. Attention reveals a normalized output, but hides normalization mass that can change the influence of the next common append. Thus, two states with similar current outputs need not remain similar after extension. Proposition 1 formalizes this obstruction. Theorem 1 establishes a sharp, attainable recovery bound for unchanged-row selection away from cancellation; Theorem 2 transfers that bound to support-anchored constructed states under explicit response and normalization-mass defects. This is a quantitative transition-reliability result, not simply a reformulation of current attention error.
2. What does the constructor-level analysis contribute?
Pure selection preserves the original keys and values. Many practical compressors instead modify their retained representations, so a sharp selection bound alone does not certify them. Theorem 2 provides a concrete bridge: for a constructor with a declared support, we compare it to unchanged-row selection on that same support and separately bound response distortion (ξ) and log normalization-mass distortion (η). This gives a sufficient condition for transferring common-append reliability to support-anchored constructed states, without assuming that arbitrary synthesis inherits selection guarantees.
CaM and KeepKV test different parts of this theory. CaM modifies retained values but preserves normalization mass; its fresh all-head audit retains 91.19%/96.42% of support-only certificate coverage on Qwen/Mistral. KeepKV also changes keys and effective multiplicities: removing its mass-defect term produces a maximum 0.01397 bound violation on fresh Qwen, while the complete bound has no observed violation. The empirical contrast shows why the two defects must be distinguished.
3. How does the criterion relate to downstream model behavior?
We distinguish controlled transition discrimination from native generation. In fresh 256-token, Read-matched attention-state tests (64 prompts/model), the transition-aware score improves post-append endpoint concordance over current-output error by 0.3205 on Qwen (95% CI [0.2364, 0.4114]) and 0.2112 on Mistral ([0.0405, 0.3824]). This is a retrospective test using realized transition information.
To assess whether the local phenomenon remains relevant when the compressed state generates its own queries and writes, we also evaluate native continuation after one compression, using prefixes up to 8192 tokens and 32 greedy steps. In the independent HotpotQA/MuSiQue replication (32 prompts/model), common-write/native error rankings have Spearman correlations of 0.9377/0.9384, with both 95% confidence intervals strictly positive. This supports a controlled-to-native correspondence in relative error difficulty.
Direct answer: We have not established a correlation between the transition-aware score and end-to-end LongBench accuracy or long-context perplexity, nor validated the certificate as a generation-lifetime predictor. The native result correlates common-write error with native error; it should not be confused with direct score-to-task prediction. It nevertheless provides evidence that the local transition geometry captures behavior relevant beyond the controlled attention endpoint.
The contribution is therefore a connected account of state-substitution obstruction → sharp recovery → constructor-level transfer, supported by controlled and native behavioral evidence at their respective scopes.