- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes RankKV, a training-free KV cache post-processing method that combines attention and information-value signals without changing the cache budget, enabling controlled correction of the retained token set and Gini-based adaptive intervention. Experiments span three models, three long-document summarization tasks, and four compressors. Most configurations show improvements in average quality, with larger gains on severely degraded samples. However, some configurations and previously well-performing samples deteriorate, and practical efficiency remains to be validated.
Critical strengths
-
The problem is well identified: KV compression harms different samples unevenly. Some samples are severely degraded, but this is hidden by average scores. The paper specifically focuses on this “severe degradation masked by average metrics,” which is a valuable angle.
-
The method design is clear: It sets a minimum attention guarantee for auxiliary information and uses it to control how much the auxiliary information intervenes, making it more controllable than fixed weighting. When no intervention is needed, it can revert to the original retention strategy.
-
Good compatibility: It requires no training and does not change the final KV budget. It can be directly attached after existing compressors as post-processing, offering good modularity and combination potential.
Critical weaknesses
-
Insufficient innovation: It mainly linearly combines existing information agents and adds a coverage constraint. It is not fairly compared with tuned fixed weighting, constrained replacement, and related post-processing methods, making it hard to see what the new framework actually contributes.
-
Mechanism is not fully explained: Looking only at damage grouping cannot prove that the bimodal distribution or attention monopolization is the cause; grouping by host performance may also be affected by regression to the mean. Whether tail samples are actually improved needs separate verification.
-
Theoretical guarantees are weak: It only guarantees normalized attention coverage, not generation quality. The naming of NDCG, the definition of maximum feasible weight, and the claim of optimality under finite search all need clarification.
-
Statistical and ablation analyses are insufficient: Some gains are small, and some configurations even degrade, but no confidence intervals, stability, or parameter sensitivity analyses are provided.
Questions for authors
-
Does RankKV still beat well-tuned fixed weighting, constrained replacement, and existing post-processing? Is the gain from coverage constraints or the auxiliary signal itself?
-
Is there full evidence for the “bimodal split,” and can attention concentration and controls rule out regression to the mean as the reason for tail improvement?
-
Is attention coverage theoretically linked to quality, or only empirical?
4 comments
- Recovery beyond summarization — Table 20. On native SnapKV, RankKV achieves stronger unhealthy-group recovery than CriticalKV in all four settings of each benchmark, LongBench and RULER, with healthy-group costs reported alongside recovery.
W3/Q3. Meaning and limits of the guarantee.
“Is attention coverage theoretically linked to quality, or only empirical?”
Thank you for asking about this connection. We provide both a theoretical rationale and empirical evidence for controlling attention coverage.
Theoretically:
Section 5 uses CriticalKV's fixed-attention bound. Fix normalized attention and projected values . For retained mass , deleting entries outside and renormalizing the retained attention gives
-\frac{1-p_S}{p_S}\sum_{i\in S}A_i\mathcal V_i.$$ The first term is the removed contribution. The second is the change from renormalizing the retained contribution. Applying the triangle inequality yields $\mathcal L=\|o-o_S\|_1\le(C-T_S)+(1-p_S)T_S/p_S$, which rearranges to $$\mathcal{L}\le C-\left(2-\frac{1}{p_S}\right)T_S,$$ where $T_S=\sum_{i\in S}A_i\|\mathcal V_i\|_1$ and $C=\sum_i A_i\|\mathcal V_i\|_1$. **At fixed $p_S>1/2$, increasing $T_S$ lowers this bound**. At $p_S=1/2$, its coefficient vanishes. This explains the half-mass condition, not a threshold for good generation. Appendix K.1 (p. 34) defines $M=\max_i\|\mathcal V_i\|_1$ and derives $$\mathcal{L}\le M(1-p_S)+\frac{1-p_S}{p_S}Mp_S=2M(1-p_S).$$ This uniform bound decreases for all positive $p_S$ and discards value differences. If host scores are proportional to that same attention, relative coverage at least $\delta$ implies $p_S\ge\delta p_*$, where $p_*$ is the host's absolute retained mass. Thus $\delta p_*>1/2$ suffices for $2-1/p_S>0$, and $\mathcal{L}\le2M(1-\delta p_*)$. The floor $\delta$ limits coverage loss. $\mu$ is chosen within this constraint. For accumulated host scores, this connection is conditional. RankKV returns the largest tested $\mu$ satisfying the declared coverage floor. $\delta=1$ returns Base. The guarantee concerns retained attention, while answer quality is evaluated empirically. **Empirically:** Figure 10 (p. 22, discussion on p. 21) compares both methods on identical input groups. RankKV has higher mean per-prompt FRI in 10/12 groups, with five direct paired 95% intervals wholly positive. Several low-coverage Fixed $\mu$ groups fall below Base, motivating attention-loss control. These exploratory, pointwise intervals and cross-input groups support adaptive intervention. They are not a within-input coefficient sweep.W4. Uncertainty and sensitivity.
“no confidence intervals, stability, or parameter sensitivity analyses are provided.”
The following table reports FRI point wins out of twelve settings, including every nested endpoint (Table 1, p. 9):
| Comparator | 150 prompts | 200 prompts | 250 prompts | 300 prompts |
|---|---|---|---|---|
| Base | 9/12 | 9/12 | 9/12 | 9/12 |
| Fixed | 9/12 | 10/12 | 8/12 | 9/12 |
| Pure value () | 8/12 | 7/12 | 7/12 | 6/12 |
| CriticalKV | 9/12 | 8/12 | 9/12 | 9/12 |
| Random | 11/12 | 11/12 | 11/12 | 11/12 |
These shared-prefix stages assess sample-size sensitivity, not independent replications or statistical significance.
Cache-budget sensitivity (Figure 6, p. 9). The table reproduces the Llama-3.1-8B budget sweep. Entries are FRI differences between RankKV and the corresponding Base at retained-cache ratio . Positive values indicate improvement. QMSum, GovReport, and arXiv use 124, 194, and 400 prompts, respectively. The 50% results reuse the original summarization evaluation.
| Task | Host | |||
|---|---|---|---|---|
| QMSum | H2O | +0.006 | +0.027 | −0.002 |
| QMSum | PyramidKV | +0.039 | +0.047 | +0.038 |
| QMSum | SnapKV | +0.037 | +0.066 | +0.045 |
| QMSum | Ada-SnapKV | +0.041 | +0.032 | +0.040 |
| GovReport | H2O | +0.020 | +0.030 | +0.054 |
| GovReport | PyramidKV | +0.017 | +0.026 | +0.063 |
| GovReport | SnapKV | −0.012 | +0.019 | +0.045 |
| GovReport | Ada-SnapKV | +0.021 | +0.031 | +0.020 |
| arXiv | H2O | 0.000 | +0.034 | +0.011 |
| arXiv | PyramidKV | +0.013 | +0.013 | +0.019 |
| arXiv | SnapKV | −0.009 | +0.025 | +0.022 |
| arXiv | Ada-SnapKV | +0.050 | +0.017 | +0.025 |
Gains extend beyond 50% retention, although their magnitude and sign depend on the task and host. The arXiv/H2O value at 35% is +0.00044 before rounding. Tables 9 and 11 additionally test the value signal and Gini-to-floor direction.
Clarification of NDCG and finite-search optimality.
“The naming of NDCG, the definition of maximum feasible weight, and the claim of optimality under finite search all need clarification.”
Thank you for identifying these ambiguities. NDCG denotes a flat-discount set adaptation of normalized discounted cumulative gain, with Järvelin and Kekäläinen (2002) cited at introduction. Retained entries have no ranking-dependent discount. The ratio measures the final set's retained host-attention mass relative to the host keep set (p. 2, lines 085–088, p. 4, lines 201–206).
The ideal continuous feasible boundary is more generally a supremum, which can be unattained or unbounded. The implementation instead searches a finite candidate set , including the feasible Base option , and returns
Coverage monotonicity supports this search. Theorem 5.2 and Appendix K.1 specify that the returned candidate is feasible, without requiring attainment of the continuous boundary or equality with the floor (p. 6, lines 315–321, p. 34, lines 1832–1835). “Strongest” therefore means the largest tested feasible coefficient, not a globally optimal keep set or a guarantee of optimal answer quality.
Dear Reviewer Mv1V,
As the authors' response deadline is fast approaching, please read through their rebuttal and let them know whether your concerns have been addressed or further clarification is still required.
Thank you.
AC
Response to Reviewer Mv1V
Thank you for distinguishing empirical recovery from causal and theoretical claims. We agree that controls and uncertainty are essential. The revised evidence separates these questions.
W1/Q1. Controls and source of gains.
RankKV makes intervention strength input-adaptive through an explicit retained-attention floor. We assess the value signal and adaptation rule through ablations, then compare complete methods.
1. Why combine value information with adaptive coverage control?
2. How does RankKV compare with existing post-processing?
CriticalKV protects a high-attention subset before its second-stage selection. RankKV reranks the eligible pool while constraining final retained attention mass (see xEqw Claim 1). This distinguishes a protected-window restriction from a coverage constraint.
For further discussion, FC25 Claims 1–3 address tuned weighting, component contributions, and coverage control. xEqw Claim 1 addresses the CriticalKV comparison.
W2/Q2. Grouping and regression to the mean.
Thank you for distinguishing the observed distribution from its possible causes. Our evidence follows this chain: substantial compression damage → scope for recovery by changing the retained set → a recovery–preservation tradeoff → benefits from informative value signals and controlled intervention.
Figure 1 shows substantial unequal damage: with H2O at 50% retention, 44% of Llama-3.1-8B inputs with defined FRI lose less than 2% relative to Full KV, while 38% lose at least 10%. These shares describe damage groups rather than establish statistical bimodality. Because Full KV performs no eviction, this contrast motivates the hypothesis that attention-greedy selection can discard useful KV entries. Figure 11 shows recovery on initially damaged inputs even under Random, suggesting scope to improve the retained set.
Recovery must also account for inputs the host already handles well. Random incurs substantial healthy-group losses. RankKV improves unhealthy-group FRI over Random in 9/12 settings and preserves healthy-group FRI better in 11/12. We therefore assess the selection strategy through direct method contrasts and ablations, rather than conditional tail movement alone. These comparisons support method-dependent recovery.
Supporting evidence: