acceptodds
Under review as a conference paper at ICLR 2027

KV-Rescue: Recovering Reasoning Language Model KV Eviction Loss via Stepwise Interleaving

Abstract

KV-cache eviction caps the memory cost of long reasoning traces but is inherently lossy because the model decodes from a partial view of its history. Under aggressive budgets, this not only lowers accuracy but can also cause runaway degeneration, where the model produces incoherent or repetitive tokens until reaching the length limit. We characterize much of this loss as an information gap caused by missing context, rather than a capability gap caused by limited model capacity. An evicted 7B model and a full-context 1.5B model make complementary errors, and an oracle choice between their answers recovers 78% of the accuracy gap to the full-KV 7B model. Based on this observation, we propose KV-Rescue, a training-free inference framework that bridges the information gap introduced by KV eviction using a lightweight full-context helper. KV-Rescue interleaves reasoning steps from the two models into a shared trajectory. An online detector uses entropy and compressibility to terminate incoherent or repetitive base candidates during generation. Across five math benchmarks with Qwen2.5-Math 7B and 72B, KV-Rescue recovers an average of 87% of the accuracy lost to eviction at budget (B=64). A decode-cost analysis further shows that preventing runaway degeneration cuts base-model token generation by 43% on average.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Long-context Reasoning, Test-Time Scaling, KV Cache Compression, KV Cache Eviction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mLEROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the performance degradation of large language models caused by KV-cache eviction during long-chain reasoning. The authors argue that much of this degradation is associated with missing contextual information rather than insufficient model capability.

The authors observe that an evicted large model and a smaller full-context model exhibit complementary errors. Based on this observation, they propose KV-Rescue, a training-free inference framework that combines reasoning steps from both models. A process reward model (PRM) selects the candidate step at each iteration. The framework also incorporates an online detector based on entropy and compressibility to terminate incoherent or repetitive generation.

Across the evaluated settings, KV-Rescue recovers an average of 87% of the accuracy lost to KV eviction at B=64, where the physical base-cache bound is B+128 tokens. In the 7B, N=8, B=64 decode-cost analysis, it reduces base-model token generation by 43% on average across the five benchmarks.

Critical strengths

The paper addresses an important practical problem: preserving reasoning quality under aggressive KV-cache eviction. The accuracy gains are substantial across five mathematical benchmarks and two base-model scales. In Table 2, interleaving alone improves MATH500 accuracy from 52.8% to 81.8% at B=64, compared with 84.0% for the full-KV reference.

The framework also addresses wasted computation from degenerate generation. Table 2 shows that adding early exit reduces base-model token generation without lowering accuracy, while Table 3 reports end-to-end throughput measurements that include helper generation and PRM scoring. These results strengthen the practical significance of the method.

Critical weaknesses

My main concern is that the paper does not sufficiently demonstrate the independent contribution of asymmetric context access, which is a key component of KV-Rescue. Although the ablation study shows significant improvements from stepwise interleaving, it does not clearly distinguish the benefits of multi-model collaboration from those of asymmetric context access. Since multi-model collaboration and PRM-guided step selection have been explored in prior research, this distinction is important for evaluating the paper's novelty. A controlled comparison between Evicted + Evicted and Evicted + Full settings, using the same models, candidate counts, and PRM, would provide stronger evidence for the proposed design.

Another concern is the framework's dependence on a high-quality PRM. Although Appendix A discusses PRM quality, the experiments do not sufficiently examine how step-level selection errors affect overall performance. Furthermore, the evaluation focuses on mathematical reasoning tasks,leaving the framework's generalizability to other domains unclear.

Questions for authors
  1. The current ablation results demonstrate the effectiveness of stepwise interleaving, but do not fully isolate the contribution of asymmetric context access from that of multi-model collaboration. Could the authors compare the current Evicted + Full setting with an Evicted + Evicted setting, while keeping the models, candidate counts, and PRM unchanged? Reporting the corresponding accuracy and resource costs would help clarify the independent contribution of asymmetric context access.

  2. Appendix A evaluates the PRM's ability to distinguish successful and unsuccessful trajectories, but this does not directly measure its reliability in ranking step-level candidates. How sensitive is KVRescue to incorrect PRM selections? An analysis of step-level selection quality or performance under different PRM quality levels would help clarify the framework's robustness and applicability.

3 comments
AuthorsOpenReview ↗

Response to Reviewer mLER (1/2)

We thank the reviewer for the careful reading and valuable feedback, including the well-specified ablation, which we ran as suggested.

[W1, Q1] Independent contribution of asymmetric context access. To isolate the contribution of asymmetric context access, we ran the suggested ablation, in which the helper is evicted with the same method and budget as the base while the models, N=8, the PRM, the detector and the sampling stay unchanged (§5.2, Table 2). Except for this new setting, all rows are taken from the main results (Fig. 5):

7B base, 1.5B helper, N=8MATH500 B=64MATH500 B=128AIME24 B=64AIME24 B=128
7B base alone, full KV84.084.019.3 ± 2.519.3 ± 2.5
1.5B helper alone, full KV73.673.612.0 ± 2.712.0 ± 2.7
Evicted base only52.869.21.3 ± 1.64.7 ± 1.6
Evicted base + Evicted helper66.779.83.3 ± 2.19.3 ± 3.3
Evicted base + Full-KV helper (KV-Rescue)82.283.212.0 ± 2.714.0 ± 4.9

At B=64, Evicted base + Evicted helper falls well below KV-Rescue on both benchmarks, indicating that asymmetric context access is a key source of KV-Rescue's gains. On MATH500, KV-Rescue also clearly exceeds the full-KV helper alone, so it does not simply fall back to the helper, and the evicted base still contributes useful steps.

Interleaving with an evicted helper also improves over eviction-only best-of-N, but by a smaller margin than KV-Rescue. Its gap to KV-Rescue is a conservative estimate of the contribution of full context, since the two evicted caches retain different parts of the context and can still complement each other.

Resource cost. Both settings generate a similar number of tokens per problem. The only additional cost of full context is therefore the helper's KV cache. It grows with the context instead of staying at B+128 tokens, but each helper token takes half the memory of a 7B base token (28 vs. 56 KiB).

AuthorsOpenReview ↗

Response to Reviewer mLER (2/2)

[W2, Q2] Sensitivity to incorrect step-level PRM selections. To measure how much KV-Rescue depends on the PRM's selection quality, we inject selection errors at a controlled rate. At every step, the PRM's choice is replaced by a uniformly random candidate with probability p (7B, N=8, B=64; §5.2, Appendix H.4, Fig. 14). For a controlled comparison, we re-ran all settings, including p=0 and eviction-only, under the same conditions, so the numbers differ slightly from Table 2:

p00.10.250.5eviction-only
MATH50081.276.671.064.251.6
AIME24 (5 seeds)16.710.710.04.00.7

Accuracy degrades gracefully, and KV-Rescue stays well above eviction-only even when half of the selections are random. AIME24 is more sensitive to selection errors, as its longer solutions involve more selection steps over which errors accumulate. We additionally evaluate KV-Rescue with two naturally weaker selectors. The long-CoT experiments (Appendix H.3) use the Qwen2.5-Math PRM as an out-of-distribution selector, since it was never trained on R1 traces. The code experiments (Appendix A) use weak third-party PRMs, which reach an AUROC of only 0.61 to 0.73, compared with 0.91 for the math PRM. In both settings, KV-Rescue still outperforms the evicted base in nearly all configurations, demonstrating its robustness to selector quality.

[W2] Generalization beyond mathematical reasoning. KV-Rescue is not tied to a specific model family or task. It can be applied whenever a smaller helper model for the same task and a step-level scorer are available. We focused the main evaluation on math because few model families currently provide a matched step-level PRM, Qwen2.5-Math being one of them. We nevertheless evaluated KV-Rescue on code with one of the weaker third-party PRMs discussed above (Appendix A, Table 6; Qwen2.5-Coder, Skywork-o1-PRM, N=1, 3 seeds):

B7B evicted1.5B fullKV-Rescue
HumanEval (7B full-KV 87.4)6432.963.867.9
12869.363.876.0
25678.063.881.9
MBPP (7B full-KV 72.3)6425.846.757.6
12853.146.762.9
25666.346.768.0

Even with this weaker PRM, KV-Rescue outperforms the evicted base at every budget, although it recovers less of the accuracy lost to eviction than in math. A policy-matched code PRM would likely narrow this difference.

We again thank the reviewer for the valuable feedback. We have revised the paper accordingly, with all changes highlighted in blue.

AuthorsOpenReview ↗

Follow-up on the revised manuscript

Thank you again for your careful review. We briefly summarize below how the revised manuscript addresses your concerns.

  • Asymmetric context access (W1). In the suggested ablation, Evicted base + Evicted helper falls well below KV-Rescue at B=64 (§5.2, Table 2).
  • PRM errors (W2). With half of the selections replaced at random, KV-Rescue still stays well above eviction-only (64.2 vs. 51.6 on MATH500; Appendix H.4).
  • Beyond math (W2). On code with a weak third-party PRM, KV-Rescue outperforms the evicted base at every budget (Appendix A).

We hope these additions address your concerns, and we would be glad to clarify any remaining questions.

Reviewer_KCF9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes KV-Rescue, a training-free framework that mitigates reasoning performance degradation caused by KV-cache eviction. It interleaves reasoning steps from a KV-compressed large model and a lightweight full-context helper model, using a process reward model (PRM) to select candidate steps. An online detector further identifies and terminates incoherent or repetitive generations.

Critical strengths

1.The paper introduces a training-free framework that interleaves reasoning steps from a KV-evicted large model and a full-context lightweight helper model.

2.The overall methodology is clearly presented and easy to follow.

Critical weaknesses

1.The paper claims to study “reasoning models”, which typically refer to System 2 models capable of performing explicit reasoning before generating a final answer, such as modern models that expose their reasoning traces using <think> tags (e.g., DeepSeek-R1). However, the experiments use Qwen2.5-Math-Instruct, which is a System 1 model. This appears inconsistent with the paper's stated focus on reasoning models.

2.The paper introduces an additional helper model to generate candidate steps and a PRM to select among them. However, it lacks a detailed analysis of how these two additional models affect the overall decoding efficiency, including throughput, KV-cache memory consumption, and computation time. The authors should provide a more comprehensive theoretical analysis based on model sizes and parameter counts, and identify the break-even points at which the proposed method becomes more efficient.

3.Several KV-cache compression methods have been specifically designed for reasoning models, yet the paper primarily evaluates R-KV. SnapKV is not specifically designed for reasoning models, so evaluating only R-KV does not sufficiently demonstrate the robustness of the proposed method. The authors should compare different KV-cache eviction methods in the key experiments.

4.The experiments report only Pass@1 results. For small benchmarks such as AIME and AMC, performance estimates can have high variance. The authors should conduct multiple runs using sampling parameters recommended by the model providers and report the mean results to ensure more reliable comparisons.

Questions for authors

1.See weakness

2.The proposed combination of a helper model and a PRM appears conceptually similar to speculative decoding, where a draft model generates candidates and a verification mechanism determines which candidates to accept. The authors should clearly explain the differences between KV-Rescue and speculative decoding. Furthermore, they should compare the two methods under comparable model configurations, evaluating both reasoning accuracy and inference efficiency.

4 comments
AuthorsOpenReview ↗

Response to Reviewer KCF9 (1/3)

We thank the reviewer for the detailed and valuable feedback, which led to several new experiments in the revision.

[W1] Use of a System-1 model for "reasoning models". KV-Rescue targets models that reason in explicit steps, so that steps from different models can be interleaved. Qwen2.5-Math is a reasoning model in this sense, as it is trained to solve problems through step-by-step chain-of-thought reasoning. Recent reasoning models such as DeepSeek-R1 and Qwen3 extend this step-by-step reasoning inside a <think> block to tens of thousands of tokens, where the KV cache grows fastest and eviction matters most. However, unlike Qwen2.5-Math, no model family of this type includes a policy-matched step-level PRM, so our main evaluation centers on Qwen2.5-Math. KV-Rescue itself makes no assumption about the reasoning format and applies to these models unchanged (§5.5). To show this, we evaluate R1-Distill-Llama-8B (evicted) with R1-Distill-Qwen-1.5B (full KV) as the helper, using the Qwen2.5-Math PRM as the selector (N=1, sampling recommended by DeepSeek; Appendix H.3):

B8B evicted1.5B fullKV-Rescue
MATH500 (8B full-KV 88.4)25665.481.084.2
51275.481.086.2
102482.881.087.0
204885.081.089.6
AIME24, 5 seeds (44.0)25610.0 ± 2.132.0 ± 3.430.7 ± 3.9
51223.3 ± 3.732.0 ± 3.433.3 ± 3.0
102428.0 ± 2.732.0 ± 3.438.3 ± 4.2
204834.0 ± 2.532.0 ± 3.440.7 ± 4.4

Even with this out-of-distribution selector setup, KV-Rescue outperforms the evicted base at every budget, most strongly at small budgets (at B=256, 65.4 to 84.2 on MATH500 and 10.0 to 30.7 on AIME24). The full evaluation on all five benchmarks is in Appendix H.3 (Fig. 13), where KV-Rescue outperforms the evicted base in 19 of the 20 benchmark-budget combinations.

AuthorsOpenReview ↗

Response to Reviewer KCF9 (2/3)

[W2] Impact of the helper and PRM on decoding efficiency. Table 4 (§5.4) reports end-to-end throughput on the 72B base, with helper generation and PRM scoring included. At batch 64, KV-Rescue reaches 3.3× the throughput of eviction-only, while full-KV runs out of memory. Appendix G derives the KV memory, the decode compute and the helper's share of the compute from model constants.

Following the reviewer's suggestion, the revision extends this analysis to break-even points that include the helper's weights and the PRM scoring of all 2N candidates (Appendix G.4, Table 9). The thresholds are computed from model constants. For the 72B base used in Table 4, they are:

KV-Rescue is cheaper than full-KV when72B base, B=64
KV per candidate: ℓ > B'/(1 − m_h/m_b)ℓ > 210 tokens
Total memory incl. helper weights: S > W_h / (N[(m_b − m_h)ℓ − m_b B'])S > 2.8 problems
Total FLOPs incl. helper and PRMℓ > 7.8k tokens

KV-Rescue thus requires less KV memory than full-KV once a candidate exceeds about 210 tokens, and the memory overhead of the helper's weights is amortized once about three problems are processed concurrently. Its FLOPs exceed those of full-KV below 7.8k tokens, but decoding at these batch sizes is memory-bandwidth bound, so throughput depends on how many sequences fit in memory rather than on FLOPs. This is why KV-Rescue sustains batch 64 in Table 4, where full-KV runs out of memory.

[W3] Robustness to reasoning-specific KV eviction methods. KV-Rescue is agnostic to the eviction method, since it leaves the eviction decision of the base model unchanged. To demonstrate this beyond R-KV and SnapKV (Appendix H.1), we additionally evaluate KV-Rescue with VaSE [Chang et al., 2026], a recent KV eviction method designed for reasoning models, under the same setup as the main 7B experiments with R-KV replaced by VaSE (§4.1, Appendix H.2, Fig. 12). As with R-KV, eviction-only accuracy drops sharply at small budgets, while KV-Rescue recovers most of the loss:

VaSE, 7B, N=8Beviction-onlyKV-Rescue
MATH5006459.483.0
MATH50012877.085.4
AIME24 (5 seeds)644.014.7

Results on all five benchmarks, for N=1 and N=8 and every budget, are in Appendix H.2 (Fig. 12).

AuthorsOpenReview ↗

Response to Reviewer KCF9 (3/3)

[W4] Variance on small benchmarks. We agree that single runs are noisy on such small benchmarks. For this reason, all results on AIME24 and AMC23 (30 and 40 problems) were already reported as pass@1 averaged over five seeds (avg@5) in the submitted version (§4.1). The revision states this reason explicitly and shows the standard deviations as shaded bands in Fig. 5. Although the recommended setting for Qwen2.5-Math-Instruct defaults to greedy decoding, best-of-N selection requires diverse candidates, so we use stochastic sampling for all methods (Appendix E). The added DeepSeek-R1-Distill model experiments use the sampling recommended by DeepSeek (temperature 0.6, top-p 0.95). All new experiments in this response also average AIME24 and AMC23 over five seeds.

[Q2] Relation to speculative decoding. While speculative decoding (SD) and KV-Rescue both use a small model to assist the inference of a large base, their objectives are fundamentally different. SD accelerates decoding by generating multiple tokens per base model forward while preserving the target's output distribution, whereas KV-Rescue recovers the accuracy that the target loses under KV eviction through reasoning step interleaving. Applied to an evicted target, SD therefore yields the same accuracy as our eviction-only baseline (Fig. 5), well below KV-Rescue. It also adds the drafter's weights and KV cache rather than reducing memory. The two methods are orthogonal and can be combined. For example, the helper in KV-Rescue could also serve as the SD draft model to accelerate the base's candidate generation, which we leave to future work (§6).

We again thank the reviewer for the valuable feedback. We have revised the paper accordingly, with all changes highlighted in blue.

AuthorsOpenReview ↗

Follow-up on the revised manuscript

Thank you again for your detailed review. We briefly summarize below how the revised manuscript addresses your concerns.

  • Reasoning models (W1). KV-Rescue with R1-Distill-Llama-8B and R1-Distill-Qwen-1.5B outperforms the evicted base in 19 of the 20 benchmark-budget combinations (§5.5, Appendix H.3).
  • Decoding efficiency (W2). Break-even points including the helper's weights and PRM scoring show that KV-Rescue needs less total memory than full-KV once about three problems are processed concurrently (Appendix G.4).
  • Reasoning-specific eviction (W3). With VaSE, KV-Rescue recovers most of the accuracy lost to eviction, as with R-KV (Appendix H.2).
  • Variance (W4). All AIME24 and AMC23 results are averaged over five seeds, with standard deviations now shown in Fig. 5.

We hope these additions address your concerns, and we would be glad to clarify any remaining questions.

Reviewer_psZvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

In this paper, the author proposes KV-Rescue to recover reasoning accuracy lost through KV-cache eviction. The framework combines a large model with a reduced cache and a small model retaining the full context, using a process reward model to select their reasoning steps. It also detects and stops incoherent or repetitive generations. Experiments across five math benchmarks show substantial accuracy recovery under aggressive eviction and reduced wasted generation.

Critical strengths

A small model retaining full context can complement a larger model whose reasoning suffers from KV eviction, offering a promising way to recover accuracy without restoring the large model’s full cache.

Experiments span five math benchmarks, two base-model sizes, multiple cache budgets, and two eviction methods, supporting the approach across the evaluated settings.

Critical weaknesses

Reducing base-model token generation does not necessarily reduce total inference cost, since KV-Rescue also generates helper candidates, scores candidates with a 7B reward model, and synchronizes the models’ contexts. End-to-end latency, throughput, and total GPU-memory comparisons at matched accuracy are needed to establish whether the accuracy recovery translates into practical efficiency gains.

All experiments use Qwen2.5-Math models and math benchmarks with a matched process reward model. It remains unclear whether the approach transfers to other model families or tasks where reliable step-level reward models are unavailable.

Questions for authors

How well does KV-Rescue generalize to other model families and tasks?

3 comments
AuthorsOpenReview ↗

Response to Reviewer psZv

We thank the reviewer for the positive assessment and the valuable feedback.

[W1] Total inference cost including the helper and the PRM. Table 4 (§5.4) already measures the end-to-end cost on the 72B base (N=8, B=64), including helper generation and PRM scoring. At batch 64, KV-Rescue reaches 3.3× the throughput of eviction-only, while full-KV runs out of memory. Both baselines are also stepwise best-of-N guided by the same PRM, so KV-Rescue adds only the 1.5B helper and the PRM scoring of the helper's candidates. PRM scoring takes under 1% of the generation time, and the helper generates its candidates in parallel with the base. At batch 64, generation takes 4.58 s per problem with KV-Rescue against 12.84 s with eviction-only.

At matched accuracy. With one candidate per model per step (N=1, B=64), KV-Rescue outperforms eviction-only best-of-8 at B=128 on all five benchmarks (72B, Fig. 5). We measure both in the setup of Table 4, each at its highest-throughput batch size.

72B, MATH500candidates per stepBbatchprob/minbase tokens per problemKV per problem (Appendix G)
Eviction-only best-of-N8128649.94,523640 MiB
KV-Rescue1 + 16412847.7612172 MiB (base + helper KV)

These results show that sampling more candidates from the evicted base does not restore the context it has lost, whereas the full-context helper can effectively provide it with fewer candidates. KV-Rescue therefore reaches higher accuracy with one base candidate per step instead of eight, and decodes far fewer base tokens. At matched peak KV memory, KV-Rescue also outperforms eviction-only on all five benchmarks (Fig. 7). Against full-KV, the revision adds break-even points that include the helper's weights and the PRM (§5.4, Appendix G.4), which show that KV-Rescue needs less total memory than full-KV once about three problems are processed concurrently.

[W2, Q1] Transfer to other model families and tasks. KV-Rescue is not tied to a specific model family or task. It can be applied whenever a smaller helper model for the same task and a step-level scorer are available. We focused the main evaluation on math because few model families currently provide a matched step-level PRM, Qwen2.5-Math being one of them. Third-party code PRMs, for example, are clearly weaker selectors (Appendix A, Table 5). To evaluate generalization across tasks and model families, the revision adds the following experiments:

  • Code (Appendix A, Table 6). With Qwen2.5-Coder 7B/1.5B and a third-party PRM, KV-Rescue outperforms the evicted base at every budget on HumanEval and MBPP (32.9 to 67.9 on HumanEval at B=64), although it recovers a smaller share of the accuracy lost to eviction than in math.
  • Long-CoT models (§5.5, Appendix H.3). With a Llama-based base (R1-Distill-Llama-8B) and a Qwen-based helper (R1-Distill-Qwen-1.5B), KV-Rescue outperforms the evicted base in 19 of the 20 benchmark-budget combinations, with the largest gains at small budgets (65.4 to 84.2 on MATH500 at B=256). The two models come from different families and use different tokenizers. KV-Rescue does not require a shared tokenizer, because the two models exchange only the text of committed steps.

We again thank the reviewer for the valuable feedback. We have revised the paper accordingly, with all changes highlighted in blue.

AuthorsOpenReview ↗

Follow-up on the revised manuscript

Thank you again for your thoughtful review. We briefly summarize below how the revised manuscript addresses your concerns.

  • Total inference cost (W1). At matched accuracy, KV-Rescue reaches 4.8× the throughput of eviction-only best-of-8 on the 72B base. The revision also adds break-even points that include the helper's weights and the PRM (§5.4, Appendix G.4).
  • Other model families and tasks (W2, Q1). KV-Rescue outperforms the evicted base on code (Appendix A) and on long-CoT models from different families (§5.5, Appendix H.3).

We hope these additions address your concerns, and we would be glad to clarify any remaining questions.

Thank you for your response and clarification! I will keep my positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.