- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes RESCUE, a method for improving an existing KV cache eviction policy. It trains a scorer to recover future-important entries that the base policy drops, then uses one decoding step on the uncompressed cache to decide whether to apply the correction. On 16 LongBench tasks and five base policies, the authors report an average gain of 1.28 points on Llama-3.1-8B. The reported gains are +1.08 on Mistral-7B and +0.43 on Qwen3-8B. Gating reduces the number of cells with a drop of at least one point from 18 to 3, compared with applying the correction unconditionally.
Critical strengths
- The paper identifies a key limitation of existing eviction policies: their scoring metrics may not capture the future importance of cached entries. It proposes a learned correction that adjusts scores based on predicted future utility, along with an inference-time gate that determines whether to apply the correction.
- The authors describe a reproducibility package containing code, trained scorers, per-document scores, and scripts for checking reported results.
Critical weaknesses
- Gains are concentrated on weaker base policies. H2O benefits substantially, while R-KV gains only 0.27 points on Llama. On Qwen3, the average gain excluding H2O is about 0.12. This supports the authors’ explanation that policies with more missed entries offer more room for correction, but it also limits the method’s value for stronger policies. The paper should state more clearly which base policies and deployment settings are expected to benefit.
- Missing discussion with methods that calibrate local compression ratios using future importance (utility). LU-KV [1] already uses offline future-importance information to characterize the long-term utility of individual attention heads and optimize their local KV cache budgets, which should be also discussed.
[1] LU-KV: Predicting Future Utility: Global Combinatorial Optimization for Task-Agnostic KV Cache Eviction, ICML 2026.
Questions for authors
- How much of RESCUE’s gain remains when the base policy’s eviction is delayed by the same number of steps as the probe?
- Does the KL-based gate require calibration for each model or task? How sensitive are the results to the abstention threshold?
2 comments
Thank you for the detailed response and the new experiments. The core disagreement remains Q1: the w=1 deferral experiment honestly shows that on SnapKV a zero-cost one-step deferral (31.9) closes more of the gap than full RESCUE (24.8), and the authors report neither the cost of the deferral nor a w-sweep, so it is hard to judge whether a slightly larger deferral would simply replace the method. Another reviewer's observation points the same way — at the same number of decoding steps, two simple baselines without the learned correction already outperform RESCUE on average, and a larger cache without RESCUE is both more accurate and faster on the evaluated tasks. I therefore maintain my score.
We thank the reviewer for the careful and constructive assessment. Both questions motivated new experiments, which we report below.
Q1. How much of RESCUE's gain remains when the base policy's eviction is delayed by the same number of steps as the probe?
We evaluated delayed eviction on Llama-3.1-8B-Instruct at B=128 across all 16 LongBench tasks, for SnapKV and H2O, deferring each policy's eviction by w=1 decoding step — RESCUE's probe length.
Each figure is the share of the base-to-dense gap closed, 100 × (score − base) / (dense − base):
The deferral is a real effect, and on SnapKV it closes more of the gap than RESCUE does alone. The two compose: applying RESCUE after the deferral closes a further 12.3 percentage points for SnapKV and 22.4 for H2O — about half of RESCUE's standalone gain on SnapKV and three fifths on H2O survive the head start — so a one-step head start does not make the correction redundant.
Q2. Does the KL gate need per-model or per-task calibration? How sensitive is τ?
Table 17 sweeps one model and one policy, which does not answer "for each model". We extended it to 3 models × 5 base policies × 7 values of τ, with no retraining (Table 18). Six of them below; each cell is the mean gain over the base policy across all 80 (base policy, task) cells for that backbone — five policies × 16 tasks — and the last column is how often the gate abstains:
Moving τ four orders of magnitude takes Llama from +1.28 to +0.96 and Mistral from +1.08 to +0.97 while the abstaining set grows from 56% to 73%. One shared τ serves all three: the best setting is 1e−5 on Llama and Mistral and 1e−6 on Qwen, and above 1e−5 the gain falls with τ in every model. No setting makes the gate harmful; it degrades towards the base policy. Per task the shared value is not always the best one: the largest gain a task-specific τ would add is 1.01 points, on the two code tasks, where abstaining more often helps.
W1. State which base policies and deployment settings are expected to benefit.
The paper now says this explicitly. RESCUE is an auxiliary correction layered over whichever base policy is already in use, so the more a base policy misses, the more it stands to recover. At B=128, each policy's gap to dense against what RESCUE returns:
The gain tracks the gap, and at this budget the correction costs nothing on any of the five. On Qwen3 it is carried by H2O (+1.63): excluding it the mean is +0.12, and we do not claim the correction pays on Qwen3's other four.
The deployment condition is a tight budget. Over Llama-3.1's 18,750 evaluated documents, the median fraction of the prompt a budget retains, against the gain at that budget:
A wide budget removes the need from both sides: at 11.57% retained the base policy is unlikely to have dropped an entry that mattered, and the reason to evict is itself thin. The left of this table is where eviction earns its place and where we expect the correction to pay. §5.4 carries this scope.
W2. LU-KV is not discussed.
Thank you for the pointer; it was missing, and we have added it to the related work. LU-KV casts head-level budget allocation as a global combinatorial problem, solved by convex-hull relaxation and a marginal-utility greedy over offline profiles, leaving an existing rule to fill each head's allocation. RESCUE changes that second decision and leaves the first alone, so the two are complementary: how much budget a head gets, against which entries fill it. We have now run it on their released profile, with their allocation and the within-head rule unchanged. Over 16 tasks RESCUE adds +0.94 at B=128 (44.62 to 45.56) and +0.58 at B=256 (47.41 to 47.99).