acceptodds
Under review as a conference paper at ICLR 2027

RESCUE: Gated Correction for KV Cache Eviction

Abstract

KV cache eviction is essential for efficient long-context LLM inference. Existing policies score KV entries from evidence already visible at the eviction point, and share a blind spot: at an aggressive budget each misses most of what the continuation will attend to, and policies reading the same observation window fail on overlapping entries rather than independent ones. We propose RESCUE, which treats future-aware eviction as correcting a deployed policy rather than replacing it and then verifying the correction against the dense model before it is applied. A policy-conditioned residual scorer is supervised only on the entries its base policy missed, so it never relearns what that policy already ranks correctly; a verification step then decodes a probe from the uncompressed cache immediately after prefill and adopts the corrected cache only if it keeps the output distribution closer to the dense model's. Verification is the part that carries the result: applied unconditionally the correction is harmful on almost a quarter of the evaluated cells. Across all 80 cells of 16 LongBench tasks × five base policies on Llama-3.1-8B, RESCUE improves the average score of every policy it augments, by +1.28 points (95% CI [+0.51, +2.16]), and cuts the cells it harms from 18 to 3; the average stays positive on Mistral-7B (+1.08) and Qwen3-8B (+0.43), with the gain concentrated on the weakest base policy and falling towards zero for the strongest as the budget widens. RESCUE targets evict-once-after-prefill inference and buys that safety with one dense decoding step.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
KV cache eviction, long-context inference, large language models, attention, efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6dWhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RESCUE, a method for improving an existing KV cache eviction policy. It trains a scorer to recover future-important entries that the base policy drops, then uses one decoding step on the uncompressed cache to decide whether to apply the correction. On 16 LongBench tasks and five base policies, the authors report an average gain of 1.28 points on Llama-3.1-8B. The reported gains are +1.08 on Mistral-7B and +0.43 on Qwen3-8B. Gating reduces the number of cells with a drop of at least one point from 18 to 3, compared with applying the correction unconditionally.

Critical strengths
  • The paper identifies a key limitation of existing eviction policies: their scoring metrics may not capture the future importance of cached entries. It proposes a learned correction that adjusts scores based on predicted future utility, along with an inference-time gate that determines whether to apply the correction.
  • The authors describe a reproducibility package containing code, trained scorers, per-document scores, and scripts for checking reported results.
Critical weaknesses
  • Gains are concentrated on weaker base policies. H2O benefits substantially, while R-KV gains only 0.27 points on Llama. On Qwen3, the average gain excluding H2O is about 0.12. This supports the authors’ explanation that policies with more missed entries offer more room for correction, but it also limits the method’s value for stronger policies. The paper should state more clearly which base policies and deployment settings are expected to benefit.
  • Missing discussion with methods that calibrate local compression ratios using future importance (utility). LU-KV [1] already uses offline future-importance information to characterize the long-term utility of individual attention heads and optimize their local KV cache budgets, which should be also discussed.

[1] LU-KV: Predicting Future Utility: Global Combinatorial Optimization for Task-Agnostic KV Cache Eviction, ICML 2026.

Questions for authors
  • How much of RESCUE’s gain remains when the base policy’s eviction is delayed by the same number of steps as the probe?
  • Does the KL-based gate require calibration for each model or task? How sensitive are the results to the abstention threshold?
2 comments

We thank the reviewer for the careful and constructive assessment. Both questions motivated new experiments, which we report below.

Q1. How much of RESCUE's gain remains when the base policy's eviction is delayed by the same number of steps as the probe?

We evaluated delayed eviction on Llama-3.1-8B-Instruct at B=128 across all 16 LongBench tasks, for SnapKV and H2O, deferring each policy's eviction by w=1 decoding step — RESCUE's probe length.

Each figure is the share of the base-to-dense gap closed, 100 × (score − base) / (dense − base):

base policy+RESCUEdelay w=1delay + RESCUE
SnapKV24.831.944.2
H2O36.425.547.9

The deferral is a real effect, and on SnapKV it closes more of the gap than RESCUE does alone. The two compose: applying RESCUE after the deferral closes a further 12.3 percentage points for SnapKV and 22.4 for H2O — about half of RESCUE's standalone gain on SnapKV and three fifths on H2O survive the head start — so a one-step head start does not make the correction redundant.

Q2. Does the KL gate need per-model or per-task calibration? How sensitive is τ?

Table 17 sweeps one model and one policy, which does not answer "for each model". We extended it to 3 models × 5 base policies × 7 values of τ, with no retraining (Table 18). Six of them below; each cell is the mean gain over the base policy across all 80 (base policy, task) cells for that backbone — five policies × 16 tasks — and the last column is how often the gate abstains:

τLlama-3.1Mistral-7BQwen3-8Babstain
1e−6 (shipped)+1.28+1.08+0.4355.9%
1e−5+1.28+1.16+0.3858.2%
1e−4+1.20+1.14+0.3361.3%
1e−2+0.96+0.97+0.2972.5%
1e−1+0.90+0.88+0.2881.8%
1+0.61+0.67+0.2594.0%

Moving τ four orders of magnitude takes Llama from +1.28 to +0.96 and Mistral from +1.08 to +0.97 while the abstaining set grows from 56% to 73%. One shared τ serves all three: the best setting is 1e−5 on Llama and Mistral and 1e−6 on Qwen, and above 1e−5 the gain falls with τ in every model. No setting makes the gate harmful; it degrades towards the base policy. Per task the shared value is not always the best one: the largest gain a task-specific τ would add is 1.01 points, on the two code tasks, where abstaining more often helps.

W1. State which base policies and deployment settings are expected to benefit.

The paper now says this explicitly. RESCUE is an auxiliary correction layered over whichever base policy is already in use, so the more a base policy misses, the more it stands to recover. At B=128, each policy's gap to dense against what RESCUE returns:

Llama-3.1, B=128H2OSnapKVR-KVLAVaLaProx
gap to dense6.916.924.926.076.07
returned+2.51+1.71+0.27+1.09+0.81

The gain tracks the gap, and at this budget the correction costs nothing on any of the five. On Qwen3 it is carried by H2O (+1.63): excluding it the mean is +0.12, and we do not claim the correction pays on Qwen3's other four.

The deployment condition is a tight budget. Over Llama-3.1's 18,750 evaluated documents, the median fraction of the prompt a budget retains, against the gain at that budget:

Llama-3.1, B per layer1281922565121024
prompt retained1.45%2.17%2.89%5.78%11.57%
SnapKV+1.71+1.40+0.98+0.12−0.02
H2O+2.51+2.22+2.19+2.21+1.48

A wide budget removes the need from both sides: at 11.57% retained the base policy is unlikely to have dropped an entry that mattered, and the reason to evict is itself thin. The left of this table is where eviction earns its place and where we expect the correction to pay. §5.4 carries this scope.

W2. LU-KV is not discussed.

Thank you for the pointer; it was missing, and we have added it to the related work. LU-KV casts head-level budget allocation as a global combinatorial problem, solved by convex-hull relaxation and a marginal-utility greedy over offline profiles, leaving an existing rule to fill each head's allocation. RESCUE changes that second decision and leaves the first alone, so the two are complementary: how much budget a head gets, against which entries fill it. We have now run it on their released profile, with their allocation and the within-head rule unchanged. Over 16 tasks RESCUE adds +0.94 at B=128 (44.62 to 45.56) and +0.58 at B=256 (47.41 to 47.99).

Thank you for the detailed response and the new experiments. The core disagreement remains Q1: the w=1 deferral experiment honestly shows that on SnapKV a zero-cost one-step deferral (31.9) closes more of the gap than full RESCUE (24.8), and the authors report neither the cost of the deferral nor a w-sweep, so it is hard to judge whether a slightly larger deferral would simply replace the method. Another reviewer's observation points the same way — at the same number of decoding steps, two simple baselines without the learned correction already outperform RESCUE on average, and a larger cache without RESCUE is both more accurate and faster on the evaluated tasks. I therefore maintain my score.

Reviewer_c9GiOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces RESCUE to improve KV-cache eviction policies in the prefill-compress-decode setting. It does so by learning to predict the future importance of KV entries that past attention does not already capture. For each existing policy, a small scorer is trained to approximate what entries the policy evicts that are instead useful. RESCUE always builds both candidate caches, and uses a "fidelity-guided correction selection" to choose what cache to use (policy or policy + learned correction), keeping the one whose output on a probe token is closer to the full model one. Evaluated on LongBench the method gives a modest average gain.

Critical strengths
  1. The core idea of combining past and approximate future importance is interesting.
  2. The one-token validation is simple. It can be used to choose between other eviction methods or approximation. However it is not clear if its cost is justified, e.g., running only the extra one token to decide what to evict could be better than the policy.
Critical weaknesses
  1. The paper is extremely hard to read, entire sections read like unpolished LLM output, some examples:
    • "the arm that excludes zero is the gated one" (line 374)
    • "cuts the cells it harms from 18 to 3" (line 025)
    • "verification is the part that carries the result" (line 022) The paper even says that "two controls remain unrun and are what we run next" (line 474), with no results or crefs to appendix, which looks like an unfinished draft. The formatting is not refined (see Figure 1), and Quest is cited as a method that predicts future importance (line 045), which it is not. Although the papers discloses the use of AI for "drafting portions of the manuscript, and editing and polishing the writing" it should still meet the high quality standard of ICLR. Addressing these problems would require a major rewriting.
  2. Adding the probe token attention to the SnapKV score, without scorer or selection, beats the full method on average (line 423)
  3. It is not clear how much the cache is actually compressed, there is no granular curve of performance vs compression ratio (or cache size)
  4. The method doubles the time to first token. How does it compare with a larger KV cache that has the same TTFT?
  5. If I understood correctly, the improvement to the base policy is small, and it is present only at extreme compression ratios (40x or 20x). It disappears on stronger policies at larger KV budgets.
  6. With a perfect future prediction, the past attention would not be useful. Their combination helps only if the future scorer is an approximation. Thus, it is important to show how good the scorer used in the paper is, reporting different alternatives (performance vs scorer quality)
Questions for authors
  • How does the method performs in terms of performance vs compression ratio (from 100% of kv cache to ~0%, with 5% steps for example) or cache size.
  • How does the method perform with long outputs?
  • Is the method only compensating for an bad future prediction? How do different scorer compare and modify the performance?
2 comments

We thank the reviewer for a close and detailed review.

W1 (presentation). Thank you for catching the citation: QUEST scores against the current query, not against a response not yet generated, so it does not belong in that sentence. It is gone; we now cite ForesightKV, Moschella et al. (2026) and LookaheadKV.

The four sentences you point out are rewritten, along with other passages that had the same fault, and the formatting around Figure 1 is fixed. The revised PDF is uploaded.

Both controls are run; §5.5 carries them:

B=128, Llama-3.1-8BSnapKVH2O
gap to dense closed by deferring eviction (w=1)31.9%25.5%
the same, with RESCUE on top44.2%47.9%
gain kept by the other policy's scorer96%90%

W2 (a probe-attention baseline beats RESCUE on the mean). §5.4 discloses that baseline — the probe token's attention over the prompt, added to the base policy's vote — and decomposes its +1.95: that mean is one task. On TREC that row alone gains +20.50 against RESCUE's +5.50; over the other fifteen, +0.71 against +1.46, behind on nine. TREC is structural: its answers are 50-way class labels of two words, so the probe token is nearly the scored output there. The baseline also carries the deeper tail: its worst task costs −0.86 against RESCUE's −0.22, and it improves eleven of the sixteen against RESCUE's thirteen.

W3 / W5 / Q1 (the curve; the gain is small and only at 40x–20x). The submission reports three budgets for two policies. Here are all five, at five budgets, with the compression ratio at the median document and the gap to dense at 17x:

Llama-3.1, B per layer1281922565121024gap at 17x
compression69x46x35x17x8.6x
H2O+2.51+2.22+2.19+2.21+1.483.57
SnapKV+1.71+1.40+0.98+0.12−0.022.11
LAVa+1.09+0.92+0.73+0.10−0.051.68
LaProx+0.81+0.40+0.32+0.13+0.061.88
R-KV+0.27+0.28+0.18−0.17−0.101.62

The 40x–20x band is B=192 to B=512. RESCUE only recovers what the base policy missed, so the less a policy misses the less there is to recover. A wide budget is exactly that — at 8.6x the median document still keeps 11.57% of its prompt — and eviction has little point there either. The curve falling to the right is what the design predicts. §5.4 scopes the claim.

W4 (TTFT). The prefill cost is real: +272 ms per document at a 5K prompt, and it does not buy an accuracy win — SnapKV at B=256 stays +0.96 ahead of RESCUE at B=128.

The two are paid differently. A larger cache is paid per decoded token and by every sequence in a batch, so at high batch it is paid in throughput: at batch 128, B=256 decodes 30% slower than B=128 and B=1024 four times slower, and the uncompressed cache no longer fits. RESCUE's 277 ms is paid once after prefill and barely moves with the budget (277.2, 275.1, 274.2 ms; it tracks the prompt — 376/584/942 ms at 16K/32K/64K, Table 13), repaid after about 2,000 generated tokens at any batch of four or more.

W6 / Q3 (how the scorers compare). The two learned scorers differ in their training target — full-future predicts the whole top-B under true future attention, residual only what this base got wrong — and gated means applied only where the probe accepts it:

scorergain over SnapKV, B=128
full-future target+0.86
residual target+1.11
full-future target, gated+1.40
residual target, gated (RESCUE)+1.71
true future attention (oracle)+6.04

The narrower target wins at both settings — predicting less is what makes it learnable. Two published future-attention predictors land at or below the policy they replace: LookaheadKV 43.38, ForesightKV 40.43, against a five-policy base of 43.77.

Of the entries the base drops, only 35% are visible to a future signal, and RESCUE recovers 25.7% of them — three quarters of what any future signal could.

Q2 (long outputs). The gain holds on some base policies and fails on others — at B=128, on the three longest-output tasks:

taskoutputgapSnapKVLaProxLAVaR-KVH2O
QMSum73 w2.47+0.07+0.11+0.04−0.12+0.15
GovReport328 w16.02+1.68+1.26+0.06−0.26−2.17
MultiNews400 w8.02+0.75+0.57+0.29−0.31−0.95

On GovReport RESCUE returns its 16-task average on SnapKV (+1.68 against +1.71) and loses on H2O (−2.17). Length is not what bounds it: within a task it does not predict the gain (ρ = −0.015 over 3,550 documents), and the gap column explains the rows — QMSum's is 2.47. An oracle reaches 63.5% of the gap here against 94.8% elsewhere.

Thank you for the rebuttal and the new experiments. Unfortunately, my main concern remains. RESCUE needs one dense decoding step, and on SnapKV two simple baselines that use the same step without the learned correction beat it on average: adding the probe attention to the SnapKV score, and delaying the eviction by one step. The rebuttal also confirms that on the evaluated tasks a larger cache without RESCUE is both more accurate and faster. As the budget grows, the gain mostly disappears.

The revision fixes the sentences I quoted, but most of the text is unchanged and the edits add new errors, for example the conclusion now ends with "gated on a single-token probe 3". AI use is allowed, but submitting a paper that reads like unedited AI output and fixing it during the rebuttal moves the editing work onto the reviewers. I do not think this is fair to the review process. Overall, I will keep my score.

Reviewer_pja3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a rescuing method to correct a deployed policy with a dense model to predict the residual but important missed tokens. The method improve several baseline methods under the strict 128 budget per layer.

Critical strengths
  • The simple MLP based prediction model with the 18-dim per KV features as input. Stable accuracy improvement under the 128 token budget than various baseline policies.
  • The method can be seamlessly integrated with other KV cache eviction methods in a plug-and-play manner.
Critical weaknesses
  • The 128 layer-wise KV cache budget even takes less memory than the per-layer, per-head, per-token KV cache features, as the token level KV features take O(L) memory, which grows linearly with the context length.
  • The accuracy improvement than the strong base policies under the layer-wise 1024 or more budget is in-significant.
  • Only the Longbench benchmark is evaluated.
Questions for authors

How does the trained model generalize to unseen datasets?

1 comment

We thank the reviewer for the careful review. The memory question led us to attribute the full prefill peak, the budget question to add two budgets and state the method's scope, and the generalisation question to run a transfer control the paper had named but not reported. On W3 we think there is a misreading we would like to correct. All numbers are on Llama-3.1-8B-Instruct.

W1. The token-level features take O(L) memory and exceed the 128-token budget.

The features do grow with context length, and at 64K they are larger than the retained cache. They are not, however, where RESCUE's extra memory goes.

  1. Size. The 18 feature columns are averaged over KV heads rather than kept per head: 4.5 MiB per layer at 64K, 144 MiB over the 32 layers.
  2. Lifetime. They are used once for scoring and then freed, while the retained cache is held through all of decoding and for every sequence in a batch. Through decoding RESCUE runs on exactly its base policy's cache: 16.4 MB at 64K, the same for both methods (Table 13).
  3. Where the extra memory goes. RESCUE does raise the prefill peak: 68.3 GiB against SnapKV's 43.9 at 64K. To attribute it we logged every live CUDA tensor at the peak on one 64K document. There RESCUE stands at 63.1 GiB against SnapKV's 42.6, and 16.4 of the 20.6 GiB between them is one object — the prompt's hidden states, one tensor of 65,020 positions × 4,096 channels per layer. The released code asks for them at every position of every layer, then reads only the last position of each, which is all the scorer needs to form the current query. The remaining 4.2 GiB is the two candidate caches and the uncompressed prompt cache the selector needs for its one-step check.

The dominant term is therefore how the features are gathered, not the features themselves and not the budget: it scales with the prompt rather than with what is kept, and a per-layer pass would remove most of it. Appendix L carries this attribution and that estimate, and §5.5 lists the 4.2 GiB as a limitation.

W2. The improvement over strong base policies at budgets of 1024 or more is insignificant.

This follows from the design: RESCUE is an auxiliary correction that recovers the entries a base policy drops but the continuation still needs, so as the budget grows the base keeps more of them itself and less is left to recover.

B per layer1281922565121024
prompt retained1.45%2.17%2.89%5.78%11.57%
SnapKV+1.71+1.40+0.98+0.12−0.02
H2O+2.51+2.22+2.19+2.21+1.48

At B=1024 SnapKV is within 1.11 points of the uncompressed model (48.84 against 49.95), so little is left for any correction to recover; H2O, which misses more, still gains +1.48. Tight budgets are also the ones that matter in serving: at batch 128 the uncompressed cache no longer fits and B=1024 decodes at a quarter of B=128's throughput (Table 8). §5.4 states this scope.

W3. Only the LongBench benchmark is evaluated.

RULER is evaluated as well: the summary is in §5.2, per-task results in Tables 6 and 11, the protocol in Appendix J. The main tables are all LongBench, so we understand it was easy to miss; §5.2 now points to them. We would add that the gain there is carried by H2O, whose base score falls furthest at long context; the other four policies gain +0.05, +0.45 and +0.50 at 16K, 32K and 64K, growing with length.

Q. How does the trained model generalise to unseen datasets?

The scorer is trained only on Natural Questions and arXiv (Appendix B.2) and applied without retraining to all 16 LongBench tasks and to RULER, whose synthetic distribution is unrelated to either. For leakage we checked the pair that shares a domain, arXiv and Qasper: the largest 8-gram Jaccard similarity between any training document and any of Qasper's 200 is 0.0008. That shows the documents are disjoint, not that every distribution shift is covered.

It generalises across base policies too, a control §5.5 named but had not run. A scorer trained on H2O residuals, applied unchanged on top of SnapKV, keeps 96% of the gain; the reverse direction keeps 90%. And where the scorer is wrong the selector limits the damage: without it 18 of 80 (policy, task) cells lose a point or more, with it 3 (Table 2).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.