acceptodds
Under review as a conference paper at ICLR 2027

RankKV: NDCG-Bounded KV Reranking against Attention Foam

Abstract

Long-context serving must evict KV positions once the cache exceeds a budget, and the default keep rule ranks tokens by accumulated attention. We find that attention greed does not degrade every prompt uniformly under a tight cache. A few structurally prominent tokens monopolize the keep set, and quality versus Full KV—the uncompressed cache—is bipolar: some prompts stay as good as Full KV, or even exceed it, while others collapse far below. We call this allocation attention foam and trace the Matthew split to single-surrogate eviction: most compressors still rank by one score, so they retune monopoly rather than leave it; the keep set is a local optimum of that score. However, replacing attention by another single criterion is the other pole, not the remedy. An effective correction must therefore rewrite the host compressor's greedy keep set at controllable strength, without abandoning that compressor. In this paper, RankKV is a training-free post-process behind the host: it swaps a bounded number of high-value evicted tokens into the greedy keep set while preserving a floor on greedy attention coverage. Behind , PyramidKV, SnapKV and Ada-SnapKV, across three model families and three long-form tasks, the method is net-positive per model. Mean lift is redistribution: healthy prompts stay near the host; hurt prompts recover.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache, KV eviction, attention foam, NDCG, long-context LLMs, training-free, cache compression

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Mv1VOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RankKV, a training-free KV cache post-processing method that combines attention and information-value signals without changing the cache budget, enabling controlled correction of the retained token set and Gini-based adaptive intervention. Experiments span three models, three long-document summarization tasks, and four compressors. Most configurations show improvements in average quality, with larger gains on severely degraded samples. However, some configurations and previously well-performing samples deteriorate, and practical efficiency remains to be validated.

Critical strengths
  1. The problem is well identified: KV compression harms different samples unevenly. Some samples are severely degraded, but this is hidden by average scores. The paper specifically focuses on this “severe degradation masked by average metrics,” which is a valuable angle.

  2. The method design is clear: It sets a minimum attention guarantee for auxiliary information and uses it to control how much the auxiliary information intervenes, making it more controllable than fixed weighting. When no intervention is needed, it can revert to the original retention strategy.

  3. Good compatibility: It requires no training and does not change the final KV budget. It can be directly attached after existing compressors as post-processing, offering good modularity and combination potential.

Critical weaknesses
  1. Insufficient innovation: It mainly linearly combines existing information agents and adds a coverage constraint. It is not fairly compared with tuned fixed weighting, constrained replacement, and related post-processing methods, making it hard to see what the new framework actually contributes.

  2. Mechanism is not fully explained: Looking only at damage grouping cannot prove that the bimodal distribution or attention monopolization is the cause; grouping by host performance may also be affected by regression to the mean. Whether tail samples are actually improved needs separate verification.

  3. Theoretical guarantees are weak: It only guarantees normalized attention coverage, not generation quality. The naming of NDCG, the definition of maximum feasible weight, and the claim of optimality under finite search all need clarification.

  4. Statistical and ablation analyses are insufficient: Some gains are small, and some configurations even degrade, but no confidence intervals, stability, or parameter sensitivity analyses are provided.

Questions for authors
  1. Does RankKV still beat well-tuned fixed weighting, constrained replacement, and existing post-processing? Is the gain from coverage constraints or the auxiliary signal itself?

  2. Is there full evidence for the “bimodal split,” and can attention concentration and controls rule out regression to the mean as the reason for tail improvement?

  3. Is attention coverage theoretically linked to quality, or only empirical?

4 comments

Response to Reviewer Mv1V

Thank you for distinguishing empirical recovery from causal and theoretical claims. We agree that controls and uncertainty are essential. The revised evidence separates these questions.

W1/Q1. Controls and source of gains.

“Does RankKV still beat well-tuned fixed weighting, constrained replacement, and existing post-processing? Is the gain from coverage constraints or the auxiliary signal itself?”

RankKV makes intervention strength input-adaptive through an explicit retained-attention floor. We assess the value signal and adaptation rule through ablations, then compare complete methods.

1. Why combine value information with adaptive coverage control?

  • Signal and adaptation — Tables 9 and 11: Compare original versus shuffled value scores, and the original versus reversed Gini-to-floor mapping to assess each component’s contribution.
  • Overall quality — Tables 12–15: On 300 paired prompts per setting, RankKV exceeds Base and tuned Fixed μ\mu in 9/12 FRI comparisons each, and pure value in 6/12. The component scores explain these overall differences.
  • Damage redistribution — Figure 9: Read the paired unhealthy/healthy panels together to assess recovery depth and preservation cost.
  • Coverage rationale — Appendix K.1 and Figure 10: At fixed absolute attention mass pS>1/2p_S>1/2, increasing retained value tightens the perturbation bound. If host scores are proportional to that attention, δp∗>1/2\delta p_*>1/2 suffices, where p∗p_* is the host’s retained mass. Figure 10 complements this rationale: RankKV is higher in 10/12 matched groups, while several low-coverage fixed-weight groups fall below Base. Together, these motivate limiting attention loss while allowing useful value exchanges.

2. How does RankKV compare with existing post-processing?

CriticalKV protects a high-attention subset before its second-stage selection. RankKV reranks the eligible pool while constraining final retained attention mass (see xEqw Claim 1). This distinguishes a protected-window restriction from a coverage constraint.

  • Overall quality — Tables 16–19: On the same 300-prompt settings, RankKV exceeds CriticalKV in 9/12 FRI comparisons.
  • Damage redistribution — Figure 11: Compare both methods on identical unhealthy/healthy groups, with Base and Random as references. RankKV achieves stronger unhealthy-group recovery than CriticalKV in 8/12 settings.

For further discussion, FC25 Claims 1–3 address tuned weighting, component contributions, and coverage control. xEqw Claim 1 addresses the CriticalKV comparison.

W2/Q2. Grouping and regression to the mean.

“Is there full evidence for the “bimodal split,” and can attention concentration and controls rule out regression to the mean as the reason for tail improvement?”

Thank you for distinguishing the observed distribution from its possible causes. Our evidence follows this chain: substantial compression damage → scope for recovery by changing the retained set → a recovery–preservation tradeoff → benefits from informative value signals and controlled intervention.

Figure 1 shows substantial unequal damage: with H2O at 50% retention, 44% of Llama-3.1-8B inputs with defined FRI lose less than 2% relative to Full KV, while 38% lose at least 10%. These shares describe damage groups rather than establish statistical bimodality. Because Full KV performs no eviction, this contrast motivates the hypothesis that attention-greedy selection can discard useful KV entries. Figure 11 shows recovery on initially damaged inputs even under Random, suggesting scope to improve the retained set.

Recovery must also account for inputs the host already handles well. Random incurs substantial healthy-group losses. RankKV improves unhealthy-group FRI over Random in 9/12 settings and preserves healthy-group FRI better in 11/12. We therefore assess the selection strategy through direct method contrasts and ablations, rather than conditional tail movement alone. These comparisons support method-dependent recovery.

Supporting evidence:

  • Recovery on fixed Base-defined groups — Figures 5, 8, and 9. Figures 5 and 8 show recovery in task quality and factual recall. In the expanded 300-prompt comparison, Figure 9 shows higher unhealthy-group FRI under RankKV than Base in 12/12 settings.
  • Value signal and intervention allocation — Tables 9 and 11. Original value scores outperform shuffled scores in 8/9 H2O settings. On Llama/H2O, reversing the Gini-to-floor mapping reduces FRI gains by 0.030 on QMSum and 0.022 on GovReport, supporting both informative value scores and the chosen adaptation direction.
  • Direct method comparisons — Tables 16–19 and Figure 11. RankKV exceeds Random in overall FRI in 11/12 settings and CriticalKV in 9/12. Its unhealthy-group recovery exceeds CriticalKV in 8/12.
  • Recovery beyond summarization — Table 20. On native SnapKV, RankKV achieves stronger unhealthy-group recovery than CriticalKV in all four settings of each benchmark, LongBench and RULER, with healthy-group costs reported alongside recovery.

W3/Q3. Meaning and limits of the guarantee.

“Is attention coverage theoretically linked to quality, or only empirical?”

Thank you for asking about this connection. We provide both a theoretical rationale and empirical evidence for controlling attention coverage.

Theoretically:

Section 5 uses CriticalKV's fixed-attention bound. Fix normalized attention AiA_i and projected values Vi=ViWO\mathcal V_i=V_iW^O. For retained mass pS>0p_S>0, deleting entries outside SS and renormalizing the retained attention gives

-\frac{1-p_S}{p_S}\sum_{i\in S}A_i\mathcal V_i.$$ The first term is the removed contribution. The second is the change from renormalizing the retained contribution. Applying the triangle inequality yields $\mathcal L=\|o-o_S\|_1\le(C-T_S)+(1-p_S)T_S/p_S$, which rearranges to $$\mathcal{L}\le C-\left(2-\frac{1}{p_S}\right)T_S,$$ where $T_S=\sum_{i\in S}A_i\|\mathcal V_i\|_1$ and $C=\sum_i A_i\|\mathcal V_i\|_1$. **At fixed $p_S>1/2$, increasing $T_S$ lowers this bound**. At $p_S=1/2$, its coefficient vanishes. This explains the half-mass condition, not a threshold for good generation. Appendix K.1 (p. 34) defines $M=\max_i\|\mathcal V_i\|_1$ and derives $$\mathcal{L}\le M(1-p_S)+\frac{1-p_S}{p_S}Mp_S=2M(1-p_S).$$ This uniform bound decreases for all positive $p_S$ and discards value differences. If host scores are proportional to that same attention, relative coverage at least $\delta$ implies $p_S\ge\delta p_*$, where $p_*$ is the host's absolute retained mass. Thus $\delta p_*>1/2$ suffices for $2-1/p_S>0$, and $\mathcal{L}\le2M(1-\delta p_*)$. The floor $\delta$ limits coverage loss. $\mu$ is chosen within this constraint. For accumulated host scores, this connection is conditional. RankKV returns the largest tested $\mu$ satisfying the declared coverage floor. $\delta=1$ returns Base. The guarantee concerns retained attention, while answer quality is evaluated empirically. **Empirically:** Figure 10 (p. 22, discussion on p. 21) compares both methods on identical input groups. RankKV has higher mean per-prompt FRI in 10/12 groups, with five direct paired 95% intervals wholly positive. Several low-coverage Fixed $\mu$ groups fall below Base, motivating attention-loss control. These exploratory, pointwise intervals and cross-input groups support adaptive intervention. They are not a within-input coefficient sweep.

W4. Uncertainty and sensitivity.

“no confidence intervals, stability, or parameter sensitivity analyses are provided.”

The following table reports FRI point wins out of twelve settings, including every nested endpoint (Table 1, p. 9):

Comparator150 prompts200 prompts250 prompts300 prompts
Base9/129/129/129/12
Fixed μ\mu9/1210/128/129/12
Pure value (μ→+∞\mu\to+\infty)8/127/127/126/12
CriticalKV9/128/129/129/12
Random11/1211/1211/1211/12

These shared-prefix stages assess sample-size sensitivity, not independent replications or statistical significance.

Cache-budget sensitivity (Figure 6, p. 9). The table reproduces the Llama-3.1-8B budget sweep. Entries are FRI differences between RankKV and the corresponding Base at retained-cache ratio B/TB/T. Positive values indicate improvement. QMSum, GovReport, and arXiv use 124, 194, and 400 prompts, respectively. The 50% results reuse the original summarization evaluation.

TaskHostB/T=0.35B/T=0.35B/T=0.50B/T=0.50B/T=0.65B/T=0.65
QMSumH2O+0.006+0.027−0.002
QMSumPyramidKV+0.039+0.047+0.038
QMSumSnapKV+0.037+0.066+0.045
QMSumAda-SnapKV+0.041+0.032+0.040
GovReportH2O+0.020+0.030+0.054
GovReportPyramidKV+0.017+0.026+0.063
GovReportSnapKV−0.012+0.019+0.045
GovReportAda-SnapKV+0.021+0.031+0.020
arXivH2O0.000+0.034+0.011
arXivPyramidKV+0.013+0.013+0.019
arXivSnapKV−0.009+0.025+0.022
arXivAda-SnapKV+0.050+0.017+0.025

Gains extend beyond 50% retention, although their magnitude and sign depend on the task and host. The arXiv/H2O value at 35% is +0.00044 before rounding. Tables 9 and 11 additionally test the value signal and Gini-to-floor direction.

Clarification of NDCG and finite-search optimality.

“The naming of NDCG, the definition of maximum feasible weight, and the claim of optimality under finite search all need clarification.”

Thank you for identifying these ambiguities. NDCG denotes a flat-discount set adaptation of normalized discounted cumulative gain, with Järvelin and Kekäläinen (2002) cited at introduction. Retained entries have no ranking-dependent discount. The ratio measures the final set's retained host-attention mass relative to the host keep set (p. 2, lines 085–088, p. 4, lines 201–206).

The ideal continuous feasible boundary is more generally a supremum, which can be unattained or unbounded. The implementation instead searches a finite candidate set P\mathcal P, including the feasible Base option μ=0\mu=0, and returns

μ^=max⁡{μ∈P:NDCG⁡(μ)≥δ}.\hat\mu=\max\{\mu\in\mathcal P:\operatorname{NDCG}(\mu)\ge\delta\}.

Coverage monotonicity supports this search. Theorem 5.2 and Appendix K.1 specify that the returned candidate is feasible, without requiring attainment of the continuous boundary or equality with the floor (p. 6, lines 315–321, p. 34, lines 1832–1835). “Strongest” therefore means the largest tested feasible coefficient, not a globally optimal keep set or a guarantee of optimal answer quality.

Dear Reviewer Mv1V,

As the authors' response deadline is fast approaching, please read through their rebuttal and let them know whether your concerns have been addressed or further clarification is still required.

Thank you.

AC

Reviewer_xEqwOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper finds that existing KV compression methods typically rely on a single importance metric (such as cumulative attention) to perform greedy pruning. This leads to a polarized distribution of quality across samples: while most samples remain nearly lossless, a significant portion suffers severe degradation. This paper propose RankKV—a training-free post-processing re-ranking step applied after existing compressors—to repair these degraded samples.

Critical strengths

This paper studies a problem that is both novel and important, and I believe it makes for a valuable and interesting piece of work.

Critical weaknesses

The experimental evaluation is insufficient.

  1. The choice of baselines is not well justified. The paper builds on value norms as an observation. However, recent methods, most notably CriticalKV, already include this query-agnostic signal in their design. I encourage the authors to discuss whether their approach remains effective when combined with these methods, and how it relates to them.

  2. The benchmark coverage is too narrow. The authors should evaluate on more diverse benchmarks, such as LongBench and RULER.

Questions for authors

None

5 comments

Response to Reviewer xEqw

We appreciate your recognition of the problem's importance. We have clarified the relationship to CriticalKV, added direct comparisons, and evaluated overall performance and damage redistribution on LongBench and RULER.

W1. Relationship, direct comparison and combination.

“The choice of baselines is not well justified. The paper builds on value norms as an observation. However, recent methods, most notably CriticalKV, already include this query-agnostic signal in their design. I encourage the authors to discuss whether their approach remains effective when combined with these methods, and how it relates to them.”

Thank you for highlighting this closely related baseline. We agree that value information alone does not establish RankKV's contribution. CriticalKV combines projected-value information with attention safeguards through two-stage selection motivated by output perturbation. RankKV explicitly adapts intervention strength μ\mu in a+μva+\mu v under a host-attention coverage constraint, NDCG⁡(μ)≥δ\operatorname{NDCG}(\mu)\ge\delta. Related Work clarifies this distinction (p. 3, lines 128–137). Section 5 and Appendix K.1 connect CriticalKV's perturbation analysis to the coverage constraint (p. 7, lines 324–335).

To make the distinction concrete, consider an attention-based host's eligible pool, excluding any fixed host-protected positions. Sort these KV entries by decreasing attention:

[p1,…,pr⏟P∣c1,…,cs⏟C∣e1,…,et⏟E].\left[\underbrace{p_1,\ldots,p_r}_{P}\mid\underbrace{c_1,\ldots,c_s}_{C}\mid\underbrace{e_1,\ldots,e_t}_{E}\right].

Here, P∪CP\cup C is the host's attention-based keep set of size B=r+sB=r+s, and EE contains the remaining candidates. CriticalKV first reserves the highest-attention subset PP, sized by α\alpha, as its attention safeguard. It then fills the remaining ss slots from C∪EC\cup E using attention weighted by projected-value norms.

RankKV jointly selects BB entries from the entire eligible pool P∪C∪EP\cup C\cup E using a+μva+\mu v, subject to NDCG⁡(μ)≥δ\operatorname{NDCG}(\mu)\ge\delta. It does not lock PP in advance: entries in PP can also be exchanged whenever the final set respects the coverage floor. This can admit exchanges that a fixed protected subset forbids. Controllability comes from adjusting δ\delta, which bounds the permitted loss of host-attention coverage. Precision comes from adapting μ\mu to each input/layer, searching toward the constraint boundary and retaining the strongest tested feasible intervention.

CriticalKV grounds attention/value selection in output-perturbation theory. RankKV explores a+μva+\mu v under a host-relative coverage constraint, turning “how much should the auxiliary signal intervene?” into “how much host-attention coverage may be sacrificed?”

Claim 1: RankKV exceeds the matched CriticalKV adaptation in overall FRI in 9/12 summarization settings and in damaged-group recovery in 8/12.

We assess mean performance through FRI (Evidence 1.1) and uneven compression damage through unhealthy/healthy groups (Evidence 1.2).

Evidence 1.1 — Overall and component quality (Tables 16–19). On 300 paired prompts per setting, RankKV has higher FRI in 9/12 point comparisons. Tables 16–19 give NativeScore, FactRecall, and UnsupportedRate (pp. 23–24, discussion: p.9, lines446–453).

FRI difference: RankKV minus CriticalKV.

ModelH2OSnapKVPyramidKVAda-SnapKV
Llama-3.1-8B+0.0213+0.0467+0.0301-0.0007
Qwen2.5-14B+0.0092-0.0202+0.0014+0.0148
Qwen3-32B+0.0381+0.0052-0.0444+0.0029

Further component results are provided in Tables 16–19 of the paper.

Evidence 1.2 — Damage redistribution (Figure 11). Compare paired unhealthy/healthy panels on identical Base-defined groups, with Base and Random as references (p. 25). RankKV has stronger unhealthy-group recovery in 8/12 settings. Recovery and healthy preservation are both better in 4/12. Llama/SnapKV's unhealthy-group FRI advantage is +0.0997 (paired 95% CI [0.0460, 0.1513]). These panels reveal where stronger recovery entails preservation costs that the overall mean alone would not show.

Regarding combination, the current experiment compares the two postprocessors on common hosts and budgets. Applying RankKV on top of CriticalKV remains a direction for future work.

W2. Broader benchmarks.

“The benchmark coverage is too narrow. The authors should evaluate on more diverse benchmarks, such as LongBench and RULER.”

Thank you for this suggestion. We agree broader evaluation is needed. We added all 16 English/code LongBench tasks (3,750 inputs/model) and all 13 RULER tasks at 16k (100 inputs/task, 1,300/configuration). Llama-3.1-8B and Mistral-7B, three hosts (SnapKV/H2O/PyramidKV), and 40%/20% retention give twelve configurations per benchmark (p. 10, lines 488–492). Official task scores use a 0–100 scale, with tasks weighted equally.

Claim 2: On native SnapKV, recovery of severely damaged inputs translates into higher overall scores on both benchmarks, with particularly strong gains on RULER.

We first compare overall quality (Evidence 2.1), then examine recovery and preservation within the same groups (Evidence 2.2).

Evidence 2.1 — Overall benchmark quality (Tables 21 and 28). Using the author CriticalKV implementation on the same native SnapKV interface, RankKV improves Base in 4/4 settings of each benchmark: +2.81–4.45 points on LongBench and +11.63–30.05 on RULER. RULER also shows 4/4 gains over CriticalKV (+6.32–11.51), all with positive paired 95% intervals. On LongBench, only Mistral/20% has a positive direct interval (p.10, lines496–498 and508–515). R/C denote RankKV/CriticalKV.

Benchmark / model / keepBaseRankKVCriticalKVR−C [95% CI]
LongBench / Llama / 40%44.0346.8446.82+0.02 [−0.71, +0.75]
LongBench / Llama / 20%38.7943.2342.54+0.69 [−0.18, +1.55]
LongBench / Mistral / 40%41.6044.9945.06−0.07 [−0.82, +0.66]
LongBench / Mistral / 20%37.0341.4840.07+1.41 [+0.56, +2.25]
RULER / Llama / 40%45.2375.2867.48+7.80 [+6.12, +9.50]
RULER / Llama / 20%31.1658.2546.73+11.51 [+9.61, +13.35]
RULER / Mistral / 40%31.0556.2947.09+9.20 [+7.08, +11.35]
RULER / Mistral / 20%24.5136.1429.82+6.32 [+4.47, +8.07]

Intervals use paired within-task document resampling and are pointwise. Tables 21 and 28 report all twelve configurations per benchmark (pp. 28 and 31), including the mixed results on dynamic H2O/PyramidKV.

Evidence 2.2 — Damage redistribution (Table 20). Both methods share groups defined by Full and Base: unhealthy (U) requires Full≥50 and Full−Base≥20. Healthy (H) requires Full≥50 and Full−Base≤5 (p.10, lines494–496). All inputs remain in overall scores. Group shares show prevalence. LongBench overall scores weight tasks equally, so raw shares are not their overall weights.

RULER, native SnapKV. Shares use all 1,300 inputs. Positive H differences indicate better preservation.

Model / keepU: n (share)U: R−CH: n (share)H: R−C
Llama / 40%718 (55.2%)+13.46507 (39.0%)+1.21
Llama / 20%917 (70.5%)+17.14315 (24.2%)-2.46
Mistral / 40%834 (64.2%)+16.74340 (26.2%)-6.97
Mistral / 20%935 (71.9%)+9.24239 (18.4%)-2.58

RankKV has stronger U recovery in 4/4 RULER settings, with worse H preservation in 3/4. For Llama/40%, equal task sizes allow a direct decomposition: U contributes +32.06 overall points relative to Base, outweighing −1.78 from H and −0.23 from other inputs, for a net +30.05. Recovering a large damaged group thus improves the mean despite some preservation cost (Appendix I, p. 27, lines 1417–1423).

LongBench also shows stronger U recovery and greater H cost in all four native settings. For Mistral/20%, U gains over Base are +39.40 for RankKV versus +25.78 for CriticalKV. H changes are −6.92 versus −4.77 (Table 20, p. 26). The +1.41 overall advantage therefore combines stronger recovery with a preservation cost.

Dear Reviewer xEqw,

As the authors' response deadline is fast approaching, please read through their rebuttal and let them know whether your concerns have been addressed or further clarification is still required.

Thank you.

AC

Reviewer_FC25OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a training-free, plug-and-play system for augmenting the quality of existing KV cache eviction mechanisms. Authors suggest reranking the long tail of attention scores over keys, incorporating primarily value norm as an auxiliary importance signal, up to a reordering budget that allows for smooth interpolation between aggressive and minimal reranking. Results show that reranked eviction produces favorable performance tradeoffs, raising performance on damaged prompts with minimal drop in undamaged ones, across a range of models and settings.

Critical strengths

The proposed technique is broadly applicable and easy to use, being training-free, plug-and-play, and largely orthogonal to choice of underlying eviction mechanism. The idea itself is sensible at a high level and motivation is sound. Results are exhaustively populated across a deployment matrix covering models, eviction schemes, benchmarks, and score metrics.

Critical weaknesses

All of my issues with the paper stem from what I believe to be an overly complicated mechanism and delivery: I suspect that both the presentation and the technique itself could be greatly simplified.

  • My main concern is that it isn’t clear to me whether all the discrete formulations, and resultant moving parts, are necessary in the first place. Fundamentally, the idea is to interpolate between attention mass and value vector norm as an updated eviction signal. The choice to implement this via discrete reordering leads to a multifaceted implementation involving active budget management, constrained discrete reranking with a separate dedicated solver, and dynamically updated dosing floor. But figure 7 suggests that in practice, the updated eviction boundary amounts to a linear interpolation between attention and value signal, with budget encompassing points already within the boundary transition region. This suggests that we could simply replace the attention score a_i with the proposed sum a_i+mu*v_i for all keys, and just like that achieve the same result. Keys with large attention mass are free to ignore the updated value signal, while the long tail self-reorders to a degree naturally determined by the degree of attention mass concentration, no dynamic floor or budget management required. How does RankKV compare to this simple baseline? Without this comparison, it is difficult to justify the extra complexity of RankKV.

  • Similarly, is the FRI metric needed? What is accomplished by reporting FRI as the primary metric, as opposed to the normal benchmark score?

  • Remaining issues mostly deal with presentation. Text is not particularly clear or concise, and alternates between overly abstract (L203 “The polarity—higher monopoly → lower floor—is load-bearing” is particularly egregious – what is this effectively communicating?) and rote recitation via sentence fragments (Section 6.1). Please unify style to improve clarity.

  • Many technical jargon terms are introduced with no context or explanation, such as “Full-Referenced Index”, “unsupported-rate recovery”, “NDCG”, “wrap policy”, and “IDF of the surface token” from page 1 alone. Even worse, the provenance for the (in the pdf, titular) “Matthew split” is never provided - as best I can tell, this might be a biblical reference to Matthew 25:29… maybe? Please define or cite the sources for your terms before using them, rather than after, and avoid this kind of jargon-making when it isn’t necessary.

Questions for authors

None, see weaknesses

10 comments
AuthorsOpenReview ↗

# Response to Reviewer FC25

Response to Reviewer FC25

We appreciate your careful reading and concern about unnecessary complexity. We have added the tuned baseline, connected the results to our ablations and coverage analysis, and clarified the presentation.

W1. Fixed weighting and extra complexity.

“How does RankKV compare to this simple baseline? Without this comparison, it is difficult to justify the extra complexity of RankKV.”

Thank you for identifying this essential comparison. We agree that the added complexity must deliver a measurable benefit. RankKV exceeds tuned Fixed μ\mu in most tested settings. To explain this result, we connect the direct comparison to component ablations, then examine why retained attention is the quantity to constrain.

All four strategies belong to the score family a+μva+\mu v:

StrategyCoefficient μ\muHow intervention is controlled
Baseμ=0\mu=0Original host selection.
Fixed μ\muTuned, then frozenSame coefficient within each task.
Pure valueμ→+∞\mu\to+\inftyValue signal alone.
RankKVAdaptive per input/layerLargest tested μ\mu satisfying NDCG⁡(μ)≥δ\operatorname{NDCG}(\mu)\ge\delta. δ=1\delta=1 returns Base.

Claim 1: Adaptive intervention improves overall FRI in most comparisons with fixed weighting while redistributing compression damage.

We examine overall FRI and its components (Evidence 1.1–1.2), then recovery of unhealthy inputs together with preservation of healthy inputs (Evidence 1.3). Together, these show improvement in overall fidelity over Base and fixed weighting in most settings, and recovery of the damaged group throughout. Pure value remains competitive: it can recover unhealthy inputs more strongly, but sometimes at greater cost to healthy inputs. RankKV makes this recovery–preservation trade explicit through a controllable attention floor.

Evidence 1.1 — Overall FRI (Table 1). On 300 paired prompts per setting, RankKV exceeds tuned Fixed μ\mu in 9/12, Base in 9/12, and pure value in 6/12 FRI comparisons. The table shows how these comparisons change as the sample count increases.

Table 1. RankKV FRI wins across nested sample sizes.

Comparator150 prompts200 prompts250 prompts300 prompts
Base (μ=0\mu=0)9/129/129/129/12
Fixed μ\mu9/1210/128/129/12
Pure value (μ→+∞\mu\to+\infty)8/127/127/126/12
CriticalKV9/128/129/129/12
Random11/1211/1211/1211/12

The majority advantage over fixed weighting persists from 150 to 300 prompts. These are point wins across twelve model–host settings.

Evidence 1.2 — Component quality (Tables 12–15). Compare NativeScore, FactRecall, and UnsupportedRate to see how task quality, retained factual content, and source support contribute to FRI. Repetition and output length provide additional diagnostics.

Evidence 1.3 — Damage redistribution (Figure 9). Compare the paired unhealthy/healthy panels on identical Base-defined groups. RankKV improves unhealthy-group FRI over Base in 12/12 settings and preserves healthy-group FRI better than pure value in 7/12.

Fixed-weight selection: We selected from nine weights by the official task score on 71 Llama/H2O development prompts from 42 documents disjoint from evaluation. The task-specific weights—QMSum: 0.1, GovReport: 0.1, arXiv: 0.3—were frozen across target models and hosts. Pure value reuses historical per-model predictions on the same 300-prompt subset.

Claim 2: Informative value scores vv and the chosen direction of adaptation contribute to the benefit.

We test these components by shuffling the value signal and reversing the adaptation direction, respectively.

Evidence 2.1 — Value signal (Table 9). Original scores outperform shuffled scores in 8/9 H2O settings, supporting the information carried by vv.

Evidence 2.2 — Adaptation direction (Table 11). Reversing the Gini-to-floor mapping reduces Llama/H2O FRI gains by 0.030 on QMSum and 0.022 on GovReport, supporting the chosen direction of intervention.

Claim 3: Sufficient coverage makes the value term favorable in the perturbation bound. Low fixed-weight coverage can accompany quality loss.

Theory explains the half-mass condition (Evidence 3.1). Figure 10 complements it with adaptive gains on matched inputs and quality losses in several low-coverage groups (Evidence 3.2–3.3). Together, they motivate a floor NDCG⁡(μ)≥δ\operatorname{NDCG}(\mu)\ge\delta that limits attention loss while leaving room for useful value exchanges.

Evidence 3.1 — Theory (Appendix K.1). Following CriticalKV's fixed-attention analysis, let pS>0p_S>0 be retained absolute mass, CC and TST_S the total and retained attention-weighted projected-value ℓ1\ell_1 norms, and MM the maximum projected-value ℓ1\ell_1 norm. The local output perturbation satisfies

=C-\left(2-\frac{1}{p_S}\right)T_S,$$ $$\mathcal L\le2M(1-p_S).$$ At fixed $p_S>1/2$, increasing $T_S$ tightens the first bound. At $p_S=1/2$, its coefficient vanishes. Higher coverage tightens the second bound. If host scores are proportional to the same attention, the floor gives $p_S\ge\delta p_*$ and $\mathcal L\le2M(1-\delta p_*)$, where $p_*$ is the host's retained mass. Thus $\delta p_*>1/2$ suffices for the favorable sign under this assumption. Accumulated host scores need not satisfy it, so this is a local-output rationale rather than a universal quality threshold. **Evidence 3.2 — Adaptive advantage (Figure 10).** Compare the two curves within each identical input group: RankKV has higher mean per-prompt FRI in 10/12 groups, with five positive direct paired 95% intervals (exploratory, unadjusted). This supports the practical benefit of adaptive post-processing. **Evidence 3.3 — Over-intervention risk (Figure 10).** Follow the Fixed $\mu$ curve from higher to lower coverage: gains fluctuate but generally weaken, and several low-coverage groups fall below Base. This motivates limiting attention loss. **W2. FRI and official benchmark scores.** > “Similarly, is the FRI metric needed? What is accomplished by reporting FRI as the primary metric, as opposed to the normal benchmark score?” Thank you for asking what FRI adds. Our motivating intuition was that attention-greedy eviction may preserve an answer's broad pattern while losing details—like recalling a history lesson's narrative but forgetting names and dates. FRI therefore combines three aspects: - **NativeScore:** task quality, measured by ROUGE-L F1 on normalized text in our summarization experiments. - **FactRecall:** coverage of reference factual-unit proxies. - **Source support ($1-U$):** penalizes output factual-unit proxies absent from the source, complementing recall. FRI is the geometric mean of the three ratios of task-level mean scores to their Full-KV counterparts, summarizing joint retention under compression. Overall comparisons average task FRIs equally. Tables 12–15 also report absolute components. LongBench/RULER retain official task metrics. **W3. Presentation.** > “Remaining issues mostly deal with presentation. Text is not particularly clear or concise, and alternates between overly abstract (L203 ‘The polarity—higher monopoly → lower floor—is load-bearing’ is particularly egregious – what is this effectively communicating?) and rote recitation via sentence fragments (Section 6.1). Please unify style to improve clarity.” Thank you for these concrete examples. We agree they obscured the method. We replaced “load-bearing” with the operational rule: higher keep-set Gini selects a lower floor, permitting more attention mass to be exchanged for higher-value entries, followed by the reversed-mapping ablation (p. 5, lines 240–242). The Introduction now distinguishes intervention control from value-based selection (p. 2, lines 085–090). Section 6.1 is rewritten as three connected paragraphs covering implementation/tasks, hosts/controls, and settings/metrics, with configurations preserved (pp. 7–8, lines 370–385). **W4. Definitions before use.** > “Many technical jargon terms are introduced with no context or explanation” We appreciate this concern and now explain these terms when introduced: FRI combines Full-KV recovery ratios (p. 1, lines 038–041). Unsupported-rate recovery measures relative source support (p. 1, lines 044–046). Host/wrap denotes the original compressor (p. 1, lines 033–034). NDCG is expanded, cited, and identified as our flat-discount set adaptation (p. 2, lines 085–088). “IDF of the surface token” becomes token rarity within the input (p. 2, lines 088–090), with Section 4.3 defining the prompt-local formula and distinguishing it from document-frequency IDF (p. 6, lines 287–288). The abstract describes coverage directly (p. 1, lines 014–016). NDCG and IDF references are added (p. 2, lines 085–088, p. 5, line 269).

W5. Meaning of the Matthew terminology.

“the provenance for the (in the pdf, titular) “Matthew split” is never provided”

Thank you for highlighting this omission. A title term should have a clear meaning and source. The Introduction now describes the unequal damage plainly—some prompts preserve or exceed Full KV, while others degrade—and defines “Matthew split” through analogy with cumulative advantage, citing Merton (1968), The Matthew Effect in Science at first use there (p. 1, lines 035–039).

Comments on the paper

1 comment

We thank the reviewers and the Area Chair for their time and careful consideration. This overview outlines the problem, motivation, and approach behind RankKV, then connects each main claim to its supporting evidence in the paper. It serves as a guide to the reasoning and experiments, with detailed responses provided separately for each reviewer.

Overview: repairing uneven KV-compression damage under an attention-coverage constraint

Research problem: KV compression harms inputs unevenly: some remain well preserved, while others degrade severely.

Motivation: Use some attention-retention headroom to recover important information, improving damaged inputs while maintaining overall quality. The Way of Heaven is to take from those who have too much and give to those who have too little——Arthur Waley

Methodology: At a fixed cache budget, post-process with a+μva+\mu v and adapt intervention strength under NDCG⁡(μ)≥δ\operatorname{NDCG}(\mu)\ge\delta.

Innovation: Turn attention–value combination into controllable correction with an explicit coverage guarantee.

Contribution: RankKV, evaluated through overall quality, the distribution of compression damage, and transfer across tasks.

Conclusion and implications: Attention retention and information recovery can work together; KV compression should be evaluated on both average performance and severely damaged inputs.

Logic 1: a+μva+\mu v Nonzero μ\mu can improve post-processing and mitigate uneven damage, beyond random perturbation.

Fig. 4 shows task-level FRI improvements; Tables 5–8 summarize the 12 model–host settings. Figs. 5/8 show preservation costs and damaged-group recovery. With 300 inputs per setting, RankKV exceeds Base in 9/12 overall-FRI comparisons and Random in 11/12 (Table 1). In 9/12 settings, it both preserves healthy inputs better and recovers damaged inputs more strongly than Random (Fig. 11), supporting informed correction.

Logic 2: a+μva+\mu v. Our requirement is that attention coverage constrains the intervention strength μ\mu.

H2O ablations show that aligned vv beats shuffled vv in 8/9 cells, reversing adaptation reduces FRI gains by 0.030/0.022 in the two tested scenes (Tables 9–11). At 300 inputs, RankKV exceeds Base, fixed μ\mu and pure value in 9/12, 9/12 and 6/12 overall-FRI comparisons (Tables 12–15). Fig. 9 shows the corresponding changes in the damage distribution.

Theoretically, Appendix K.1 starts from CriticalKV's perturbation bound:

$

L≤C−(2−1p)T.\mathcal L\le C-\left(2-\frac1p\right)T.

$

Here pp is retained attention mass, TT the retained attention-weighted value-norm contribution, and CC its uncompressed total. At fixed p>1/2p>1/2, increasing TT lowers this bound. When host scores are proportional to the same attention, with host-retained mass p∗p_*, NDCG⁡≥δ\operatorname{NDCG}\ge\delta implies p≥δp∗p\ge\delta p_*, so δp∗>1/2\delta p_*>1/2 certifies this regime. This motivates preserving attention while leaving room for value-guided correction.

Empirically, Fig. 10 groups inputs by fixed-weight coverage and compares both curves on identical members. RankKV has higher FRI gains in 10/12 groups, with five positive paired 95% intervals, supporting adaptive correction under the coverage constraint.

Logic 3: Coverage-constrained a+μva+\mu v improves overall performance and damaged-input recovery relative to other baselines.

Against adapted CriticalKV at 300 inputs, RankKV wins 9/12 overall-FRI comparisons (Tables 16–19), achieves stronger damaged-group recovery in 8/12, and improves both recovery and preservation in 4/12 (Fig. 11).

Robustness: varying sample count and compression budget.

Across nested 150/200/250/300 inputs, wins over Base and Random remain 9/12 and 11/12, respectively (Table 1). Fig. 6 further examines 35%/50%/65% cache retention.

Logic 4: RankKV's recovery transfers to LongBench and RULER.

Across four native SnapKV model–budget settings, official-score gains over Base are 2.81–4.45 points on LongBench and 11.63–30.05 on RULER (Tables 2, 21, 28). On each benchmark, RankKV recovers the damaged group more strongly than CriticalKV in 4/4 settings, with healthy-group costs (Table 20). Dynamic-host gains depend on the configuration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.