- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies KV cache eviction for long-context LLM inference. The authors observe that existing eviction methods mainly rely on prompt-side signals, such as attention patterns or recent queries, while the query distribution during autoregressive decoding can differ significantly from that during prefill. To address this issue, the authors propose LORE-KV, which formulates KV eviction as a future-use prediction problem. LORE-KV performs short autoregressive rollouts with the frozen target model to sample plausible future response trajectories, and uses the resulting response-side queries as Monte Carlo estimates of future decoding behavior. It then evaluates token importance based on projected leave-one-out attention-output deletion costs and aggregates scores across sampled futures with reliability weighting. Experiments on multiple long-context benchmarks and model architectures demonstrate that LORE-KV consistently improves over existing eviction methods under aggressive compression settings, with ablation studies showing that better alignment with future decoding queries is the main contributor to the observed gains.
Critical strengths
1.The paper introduces an insightful perspective by incorporating future query distributions into KV eviction decisions. Reframing eviction as a future-use prediction problem is well motivated and better aligns with the actual autoregressive decoding process, where token utility is determined by future queries rather than only prompt-side observations. 2.Instead of explicitly modeling a complex future query distribution, the proposed approach directly leverages the target LLM to generate short continuations and uses the corresponding query states as samples of future decoding behavior. This design is training-free, model-agnostic, and provides a practical way to approximate future query information. 3.The paper provides extensive mechanism analyses to understand the source of performance improvements. Such analyses strengthen the understanding and credibility of the proposed approach.
Critical weaknesses
1.LORE-KV requires performing M*T additional autoregressive generation steps before the actual decoding process. Although the authors show that the steady-state decoding memory usage and per-token inference cost remain unchanged after compression, the additional lookahead overhead can still affect the end-to-end latency in practical serving scenarios. In particular, for short generation tasks or high-throughput batch inference settings, the extra computation introduced by lookahead sampling may offset the efficiency benefits gained from KV cache compression. 2.LORE-KV makes cache retention decisions at a single point before decoding and permanently removes selected KV entries. However, the importance of cached tokens may evolve as generation proceeds and the query distribution changes over time. Exploring dynamic or recoverable cache eviction strategies could make the approach more suitable for realistic long-context generation scenarios.
Questions for authors
1.Could the authors provide more analysis on the stability of Monte Carlo trajectory sampling? For example, how do different sampling strategies (e.g., temperature-based sampling or greedy decoding) affect eviction quality and token ranking stability? 2.Can LORE-KV support decoding-time or dynamic KV eviction? Additional evaluation on long-horizon reasoning tasks (e.g., MATH or AIME with reasoning models) would help assess its applicability in more challenging generation scenarios. 3.Could the authors clarify the availability of the implementation code? The code release would improve reproducibility and facilitate further research.
3 comments
Response to Reviewer dv8t (Part 2 of 2)
Q1. Stability of Monte Carlo sampling
The default rollouts use temperature 1 without truncation (Appendix E.1). We add three seeds and four temperatures.
Table H. Seeds and temperature (LongBench, Mistral-7B, LORE-KV/E, M=4, T=8, mean ± standard deviation over three seeds).
| Method | B=128 | B=512 |
|---|---|---|
| Temperature 0.5 | 40.24 ± 0.02 | 41.59 ± 0.07 |
| Temperature 0.7 | 40.25 ± 0.15 | 41.63 ± 0.01 |
| Temperature 1.0 | 40.03 ± 0.18 | 41.72 ± 0.06 |
The standard deviation across seeds is at most 0.18 points. At B=128, each seed is at least 0.9 points above AnDPro. Temperature changes the score by at most 0.22 points. At temperature 1.3 (one seed) and with a greedy rollout, LORE-KV also stays above AnDPro at all four budgets. With a longer lookahead (T=32), LORE-KV/E is 0.31 to 1.17 points above AnDPro at all four budgets.
Limit. A greedy rollout gives the same continuation for all M, so it reduces to M=1. On LongBench, it is close to the sampled rollouts, so the value of sampled diversity depends on the task (Tables B and D in our response to Reviewer ikcN). The retained tokens for M=1 and M=4 overlap by 0.97 on LongBench and by 0.92 on RULER (Figure 9, Appendix F.3). The revised paper will add a full analysis of token-ranking stability.
Q3. Code availability
Our code is available at this anonymous repository: https://anonymous.4open.science/r/train_free_kv-68CD.
References.
- [1] D. Hendrycks et al. Measuring Mathematical Problem Solving With the MATH Dataset. NeurIPS 2021 Datasets and Benchmarks. https://arxiv.org/abs/2103.03874
- [2] H. Veeraboina. AIME Problem Set 1983-2024. Kaggle, 2023. https://www.kaggle.com/datasets/hemishveeraboina/aime-problem-set-1983-2024
Thank you again for these questions. The new results on long outputs, long reasoning, and stability make the practical scope of LORE-KV clear.
I have reviewed the authors' responses and the additional experiments. The authors have adequately addressed my concerns regarding stability, latency overhead, and performance on longer outputs and reasoning tasks. I will maintain my rating of 3 (Weak acceptance). I thank the authors for their constructive engagement.
Response to Reviewer dv8t (Part 1 of 2)
Thank you for the positive assessment and the questions on deployment and stability.
W1. Lookahead overhead in practical serving
We agree that the one-time cost matters most for short outputs and high-throughput batches. Table E in our response to Reviewer ikcN gives the measured time of each stage. Three points are specific to your comment:
W2 and Q2. Dynamic eviction and long-horizon reasoning
Dynamic eviction. The submission evaluates one eviction after the prompt (Section 3.1), as in SnapKV and AnDPro. Section 5 lists irreversible eviction as a limitation and names dynamic eviction as a direction. The score needs no change for decode-time eviction. At a selected step, we treat the current cache as the prompt, sample M continuations, score the tokens, and evict. The revised paper will discuss this extension as future work, with its cost at each eviction.
Long outputs. We increase the output limit to 1024 tokens on the three summarization tasks of LongBench.
Table F. Long generation (Mistral-7B, mean ROUGE-L of GovReport, QMSum, and MultiNews, 200 samples each, full cache 28.14).
LORE-KV/E is above AnDPro at every budget and for both lookahead lengths. The gain is largest on GovReport, which has the longest outputs (+1.0 to +1.7 points at T=8). A longer lookahead (T=32) adds 0.34 to 0.75 points, at a higher one-time cost (3.54 s against 0.88 s of lookahead per sample).
Long reasoning. Raw MATH [1] and AIME [2] problems have about 50 tokens, so we add a 32-shot context before each problem. We evaluate Mistral-7B (up to 1024 new tokens) and the reasoning model DeepSeek-R1-Distill-Qwen-14B (up to 4096 new tokens).
Table G1. MATH-500 accuracy (%) with Mistral-7B (full cache 9.4).
With equal lookahead tokens (M=4, T=32), LORE-KV/E is 1.0 to 1.2 points above the attention score and 1.4 to 2.0 above AnDPro. At B=512, it exceeds the full-cache accuracy (9.6 against 9.4). Both LORE-KV settings stay above AnDPro at all four budgets, also at B=1024 (8.2 and 8.4 against 7.8). Mistral-7B is not a reasoning model, so its absolute accuracy is low, and Table G2 gives the reasoning-model results.
Table G2. MATH-500 accuracy (%) with a reasoning model (full cache 79.0).
LORE-KV/E is above AnDPro at every budget and reaches the full-cache accuracy from B=512. Limit. An unpaired two-proportion z-test did not detect a significant difference in Tables G1 and G2 (p ≥ 0.17). Thus, LORE-KV keeps its accuracy with long reasoning outputs, and we do not claim a significant gain. AIME has 30 problems for each year, so one problem changes the accuracy by 3.3 points, and AIME does not separate the methods. The revised paper will report all MATH and AIME results.