- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the performance degradation of large language models caused by KV-cache eviction during long-chain reasoning. The authors argue that much of this degradation is associated with missing contextual information rather than insufficient model capability.
The authors observe that an evicted large model and a smaller full-context model exhibit complementary errors. Based on this observation, they propose KV-Rescue, a training-free inference framework that combines reasoning steps from both models. A process reward model (PRM) selects the candidate step at each iteration. The framework also incorporates an online detector based on entropy and compressibility to terminate incoherent or repetitive generation.
Across the evaluated settings, KV-Rescue recovers an average of 87% of the accuracy lost to KV eviction at B=64, where the physical base-cache bound is B+128 tokens. In the 7B, N=8, B=64 decode-cost analysis, it reduces base-model token generation by 43% on average across the five benchmarks.
Critical strengths
The paper addresses an important practical problem: preserving reasoning quality under aggressive KV-cache eviction. The accuracy gains are substantial across five mathematical benchmarks and two base-model scales. In Table 2, interleaving alone improves MATH500 accuracy from 52.8% to 81.8% at B=64, compared with 84.0% for the full-KV reference.
The framework also addresses wasted computation from degenerate generation. Table 2 shows that adding early exit reduces base-model token generation without lowering accuracy, while Table 3 reports end-to-end throughput measurements that include helper generation and PRM scoring. These results strengthen the practical significance of the method.
Critical weaknesses
My main concern is that the paper does not sufficiently demonstrate the independent contribution of asymmetric context access, which is a key component of KV-Rescue. Although the ablation study shows significant improvements from stepwise interleaving, it does not clearly distinguish the benefits of multi-model collaboration from those of asymmetric context access. Since multi-model collaboration and PRM-guided step selection have been explored in prior research, this distinction is important for evaluating the paper's novelty. A controlled comparison between Evicted + Evicted and Evicted + Full settings, using the same models, candidate counts, and PRM, would provide stronger evidence for the proposed design.
Another concern is the framework's dependence on a high-quality PRM. Although Appendix A discusses PRM quality, the experiments do not sufficiently examine how step-level selection errors affect overall performance. Furthermore, the evaluation focuses on mathematical reasoning tasks,leaving the framework's generalizability to other domains unclear.
Questions for authors
-
The current ablation results demonstrate the effectiveness of stepwise interleaving, but do not fully isolate the contribution of asymmetric context access from that of multi-model collaboration. Could the authors compare the current Evicted + Full setting with an Evicted + Evicted setting, while keeping the models, candidate counts, and PRM unchanged? Reporting the corresponding accuracy and resource costs would help clarify the independent contribution of asymmetric context access.
-
Appendix A evaluates the PRM's ability to distinguish successful and unsuccessful trajectories, but this does not directly measure its reliability in ranking step-level candidates. How sensitive is KVRescue to incorrect PRM selections? An analysis of step-level selection quality or performance under different PRM quality levels would help clarify the framework's robustness and applicability.
3 comments
Response to Reviewer mLER (2/2)
[W2, Q2] Sensitivity to incorrect step-level PRM selections. To measure how much KV-Rescue depends on the PRM's selection quality, we inject selection errors at a controlled rate. At every step, the PRM's choice is replaced by a uniformly random candidate with probability p (7B, N=8, B=64; §5.2, Appendix H.4, Fig. 14). For a controlled comparison, we re-ran all settings, including p=0 and eviction-only, under the same conditions, so the numbers differ slightly from Table 2:
| p | 0 | 0.1 | 0.25 | 0.5 | eviction-only |
|---|---|---|---|---|---|
| MATH500 | 81.2 | 76.6 | 71.0 | 64.2 | 51.6 |
| AIME24 (5 seeds) | 16.7 | 10.7 | 10.0 | 4.0 | 0.7 |
Accuracy degrades gracefully, and KV-Rescue stays well above eviction-only even when half of the selections are random. AIME24 is more sensitive to selection errors, as its longer solutions involve more selection steps over which errors accumulate. We additionally evaluate KV-Rescue with two naturally weaker selectors. The long-CoT experiments (Appendix H.3) use the Qwen2.5-Math PRM as an out-of-distribution selector, since it was never trained on R1 traces. The code experiments (Appendix A) use weak third-party PRMs, which reach an AUROC of only 0.61 to 0.73, compared with 0.91 for the math PRM. In both settings, KV-Rescue still outperforms the evicted base in nearly all configurations, demonstrating its robustness to selector quality.
[W2] Generalization beyond mathematical reasoning. KV-Rescue is not tied to a specific model family or task. It can be applied whenever a smaller helper model for the same task and a step-level scorer are available. We focused the main evaluation on math because few model families currently provide a matched step-level PRM, Qwen2.5-Math being one of them. We nevertheless evaluated KV-Rescue on code with one of the weaker third-party PRMs discussed above (Appendix A, Table 6; Qwen2.5-Coder, Skywork-o1-PRM, N=1, 3 seeds):
| B | 7B evicted | 1.5B full | KV-Rescue | |
|---|---|---|---|---|
| HumanEval (7B full-KV 87.4) | 64 | 32.9 | 63.8 | 67.9 |
| 128 | 69.3 | 63.8 | 76.0 | |
| 256 | 78.0 | 63.8 | 81.9 | |
| MBPP (7B full-KV 72.3) | 64 | 25.8 | 46.7 | 57.6 |
| 128 | 53.1 | 46.7 | 62.9 | |
| 256 | 66.3 | 46.7 | 68.0 |
Even with this weaker PRM, KV-Rescue outperforms the evicted base at every budget, although it recovers less of the accuracy lost to eviction than in math. A policy-matched code PRM would likely narrow this difference.
We again thank the reviewer for the valuable feedback. We have revised the paper accordingly, with all changes highlighted in blue.
Follow-up on the revised manuscript
Thank you again for your careful review. We briefly summarize below how the revised manuscript addresses your concerns.
- Asymmetric context access (W1). In the suggested ablation, Evicted base + Evicted helper falls well below KV-Rescue at B=64 (§5.2, Table 2).
- PRM errors (W2). With half of the selections replaced at random, KV-Rescue still stays well above eviction-only (64.2 vs. 51.6 on MATH500; Appendix H.4).
- Beyond math (W2). On code with a weak third-party PRM, KV-Rescue outperforms the evicted base at every budget (Appendix A).
We hope these additions address your concerns, and we would be glad to clarify any remaining questions.
Response to Reviewer mLER (1/2)
We thank the reviewer for the careful reading and valuable feedback, including the well-specified ablation, which we ran as suggested.
[W1, Q1] Independent contribution of asymmetric context access. To isolate the contribution of asymmetric context access, we ran the suggested ablation, in which the helper is evicted with the same method and budget as the base while the models, N=8, the PRM, the detector and the sampling stay unchanged (§5.2, Table 2). Except for this new setting, all rows are taken from the main results (Fig. 5):
At B=64, Evicted base + Evicted helper falls well below KV-Rescue on both benchmarks, indicating that asymmetric context access is a key source of KV-Rescue's gains. On MATH500, KV-Rescue also clearly exceeds the full-KV helper alone, so it does not simply fall back to the helper, and the evicted base still contributes useful steps.
Interleaving with an evicted helper also improves over eviction-only best-of-N, but by a smaller margin than KV-Rescue. Its gap to KV-Rescue is a conservative estimate of the contribution of full context, since the two evicted caches retain different parts of the context and can still complement each other.
Resource cost. Both settings generate a similar number of tokens per problem. The only additional cost of full context is therefore the helper's KV cache. It grows with the context instead of staying at B+128 tokens, but each helper token takes half the memory of a 7B base token (28 vs. 56 KiB).