acceptodds
Under review as a conference paper at ICLR 2027

Monte Carlo Future-Query Estimation for KV Cache Eviction

27 views

Abstract

Most KV-cache eviction methods ask, in effect, which memory appeared important while reading the prompt? We instead ask, which memory will matter while answering? Since decoding queries are unavailable at eviction time, prior future-aware methods rely on pseudo-responses or synthetic future-query estimates. We cast fixed-budget future-aware eviction as distributional estimation over plausible model-conditional query trajectories and introduce LORE-KV (Lookahead Output-perturbation with Reliability-weighted Ensembles for Key-Value caches), a training-free method that samples short autoregressive continuations from the frozen target model and uses their response-side query states to estimate prompt-token utility. Tokens are scored by projected leave-one-out attention-output deletion cost and aggregated across sampled futures with optional trajectory weighting. The temporary continuations are discarded before final decoding, requiring no auxiliary model or training. Ablations isolate the mechanism: at B=128, a single response-side continuation recovers about 89% of the gain over the prompt-window control, while additional futures provide smaller improvements. At B=128, LORE-KV raises the LongBench average on Qwen2.5-14B from 45.49 to 48.24 (+2.75) and the 16K RULER average on Mistral-7B from 45.20 to 51.05 (+5.85). Gains diminish at larger cache budgets and coexist with task-level regressions. LORE-KV incurs 1.46–2.77× AnDPro's per-sample wall-clock time as a one-time compression overhead across six dense and hybrid-attention backbones.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV Cache Compression, KV Cache Eviction, Efficient LLM Inference, Long-Context LLMs, Future-Query Estimation, Monte-Carlo Estimation, Test-Time Computation, Memory-Efficient Transformers, Attention, Large Language Models.

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dv8tOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies KV cache eviction for long-context LLM inference. The authors observe that existing eviction methods mainly rely on prompt-side signals, such as attention patterns or recent queries, while the query distribution during autoregressive decoding can differ significantly from that during prefill. To address this issue, the authors propose LORE-KV, which formulates KV eviction as a future-use prediction problem. LORE-KV performs short autoregressive rollouts with the frozen target model to sample plausible future response trajectories, and uses the resulting response-side queries as Monte Carlo estimates of future decoding behavior. It then evaluates token importance based on projected leave-one-out attention-output deletion costs and aggregates scores across sampled futures with reliability weighting. Experiments on multiple long-context benchmarks and model architectures demonstrate that LORE-KV consistently improves over existing eviction methods under aggressive compression settings, with ablation studies showing that better alignment with future decoding queries is the main contributor to the observed gains.

Critical strengths

1.The paper introduces an insightful perspective by incorporating future query distributions into KV eviction decisions. Reframing eviction as a future-use prediction problem is well motivated and better aligns with the actual autoregressive decoding process, where token utility is determined by future queries rather than only prompt-side observations. 2.Instead of explicitly modeling a complex future query distribution, the proposed approach directly leverages the target LLM to generate short continuations and uses the corresponding query states as samples of future decoding behavior. This design is training-free, model-agnostic, and provides a practical way to approximate future query information. 3.The paper provides extensive mechanism analyses to understand the source of performance improvements. Such analyses strengthen the understanding and credibility of the proposed approach.

Critical weaknesses

1.LORE-KV requires performing M*T additional autoregressive generation steps before the actual decoding process. Although the authors show that the steady-state decoding memory usage and per-token inference cost remain unchanged after compression, the additional lookahead overhead can still affect the end-to-end latency in practical serving scenarios. In particular, for short generation tasks or high-throughput batch inference settings, the extra computation introduced by lookahead sampling may offset the efficiency benefits gained from KV cache compression. 2.LORE-KV makes cache retention decisions at a single point before decoding and permanently removes selected KV entries. However, the importance of cached tokens may evolve as generation proceeds and the query distribution changes over time. Exploring dynamic or recoverable cache eviction strategies could make the approach more suitable for realistic long-context generation scenarios.

Questions for authors

1.Could the authors provide more analysis on the stability of Monte Carlo trajectory sampling? For example, how do different sampling strategies (e.g., temperature-based sampling or greedy decoding) affect eviction quality and token ranking stability? 2.Can LORE-KV support decoding-time or dynamic KV eviction? Additional evaluation on long-horizon reasoning tasks (e.g., MATH or AIME with reasoning models) would help assess its applicability in more challenging generation scenarios. 3.Could the authors clarify the availability of the implementation code? The code release would improve reproducibility and facilitate further research.

3 comments
AuthorsOpenReview ↗

Response to Reviewer dv8t (Part 1 of 2)

Thank you for the positive assessment and the questions on deployment and stability.

W1. Lookahead overhead in practical serving

We agree that the one-time cost matters most for short outputs and high-throughput batches. Table E in our response to Reviewer ikcN gives the measured time of each stage. Three points are specific to your comment:

  • The M rollouts run in one batch, so they need only T=8 sequential steps.
  • LORE-KV copies the live cache once for each rollout (Appendix E.2), so peak memory during lookahead is higher than steady-state memory. The revised paper will state this clearly.
  • For long outputs, the one-time cost is shared over more tokens, and the accuracy gain stays (Table F). For short outputs, M=1 is the lighter setting.

W2 and Q2. Dynamic eviction and long-horizon reasoning

Dynamic eviction. The submission evaluates one eviction after the prompt (Section 3.1), as in SnapKV and AnDPro. Section 5 lists irreversible eviction as a limitation and names dynamic eviction as a direction. The score needs no change for decode-time eviction. At a selected step, we treat the current cache as the prompt, sample M continuations, score the tokens, and evict. The revised paper will discuss this extension as future work, with its cost at each eviction.

Long outputs. We increase the output limit to 1024 tokens on the three summarization tasks of LongBench.

Table F. Long generation (Mistral-7B, mean ROUGE-L of GovReport, QMSum, and MultiNews, 200 samples each, full cache 28.14).

MethodB=128B=512B=1024
AnDPro21.9223.9125.24
LORE-KV/E (M=4, T=8)22.3224.7125.87
LORE-KV/E (M=4, T=32)23.0725.2426.21

LORE-KV/E is above AnDPro at every budget and for both lookahead lengths. The gain is largest on GovReport, which has the longest outputs (+1.0 to +1.7 points at T=8). A longer lookahead (T=32) adds 0.34 to 0.75 points, at a higher one-time cost (3.54 s against 0.88 s of lookahead per sample).

Long reasoning. Raw MATH [1] and AIME [2] problems have about 50 tokens, so we add a 32-shot context before each problem. We evaluate Mistral-7B (up to 1024 new tokens) and the reasoning model DeepSeek-R1-Distill-Qwen-14B (up to 4096 new tokens).

Table G1. MATH-500 accuracy (%) with Mistral-7B (full cache 9.4).

MethodB=128B=256B=512
AnDPro4.66.88.2
Attention score, same rollouts (M=4, T=32)5.47.48.6
LORE-KV/E (M=4, T=8)6.08.09.0
LORE-KV/E (M=4, T=32)6.68.49.6

With equal lookahead tokens (M=4, T=32), LORE-KV/E is 1.0 to 1.2 points above the attention score and 1.4 to 2.0 above AnDPro. At B=512, it exceeds the full-cache accuracy (9.6 against 9.4). Both LORE-KV settings stay above AnDPro at all four budgets, also at B=1024 (8.2 and 8.4 against 7.8). Mistral-7B is not a reasoning model, so its absolute accuracy is low, and Table G2 gives the reasoning-model results.

Table G2. MATH-500 accuracy (%) with a reasoning model (full cache 79.0).

MethodB=128B=256B=512B=1024
AnDPro69.673.877.478.2
LORE-KV/E69.874.079.279.0

LORE-KV/E is above AnDPro at every budget and reaches the full-cache accuracy from B=512. Limit. An unpaired two-proportion z-test did not detect a significant difference in Tables G1 and G2 (p ≥ 0.17). Thus, LORE-KV keeps its accuracy with long reasoning outputs, and we do not claim a significant gain. AIME has 30 problems for each year, so one problem changes the accuracy by 3.3 points, and AIME does not separate the methods. The revised paper will report all MATH and AIME results.

AuthorsOpenReview ↗

Response to Reviewer dv8t (Part 2 of 2)

Q1. Stability of Monte Carlo sampling

The default rollouts use temperature 1 without truncation (Appendix E.1). We add three seeds and four temperatures.

Table H. Seeds and temperature (LongBench, Mistral-7B, LORE-KV/E, M=4, T=8, mean ± standard deviation over three seeds).

MethodB=128B=512
Temperature 0.540.24 ± 0.0241.59 ± 0.07
Temperature 0.740.25 ± 0.1541.63 ± 0.01
Temperature 1.040.03 ± 0.1841.72 ± 0.06

The standard deviation across seeds is at most 0.18 points. At B=128, each seed is at least 0.9 points above AnDPro. Temperature changes the score by at most 0.22 points. At temperature 1.3 (one seed) and with a greedy rollout, LORE-KV also stays above AnDPro at all four budgets. With a longer lookahead (T=32), LORE-KV/E is 0.31 to 1.17 points above AnDPro at all four budgets.

Limit. A greedy rollout gives the same continuation for all M, so it reduces to M=1. On LongBench, it is close to the sampled rollouts, so the value of sampled diversity depends on the task (Tables B and D in our response to Reviewer ikcN). The retained tokens for M=1 and M=4 overlap by 0.97 on LongBench and by 0.92 on RULER (Figure 9, Appendix F.3). The revised paper will add a full analysis of token-ranking stability.

Q3. Code availability

Our code is available at this anonymous repository: https://anonymous.4open.science/r/train_free_kv-68CD.

References.

Thank you again for these questions. The new results on long outputs, long reasoning, and stability make the practical scope of LORE-KV clear.

I have reviewed the authors' responses and the additional experiments. The authors have adequately addressed my concerns regarding stability, latency overhead, and performance on longer outputs and reasoning tasks. I will maintain my rating of 3 (Weak acceptance). I thank the authors for their constructive engagement.

Reviewer_ikcNOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes LORE-KV, a training-free KV-cache eviction method that estimates token importance using future decoding queries. It samples several short continuations from the target model and scores prompt tokens by the attention-output change caused by deleting them. These scores are combined across continuations with reliability weighting, and the temporary continuations are discarded before actual generation. Experiments show improved accuracy over existing eviction methods, particularly at small cache budgets, at the cost of additional inference latency.

Critical strengths
  1. The motivation is clear and reasonable: queries observed during prompt processing may not represent the information needed during answering. Using response-side queries directly addresses this mismatch without additional training or an auxiliary model.

  2. The paper provides extensive evaluations across multiple backbones, benchmarks, and cache budgets. The ablations help identify where the gains come from, showing that response-side queries account for most of the improvement, while multiple continuations and reliability weighting provide smaller refinements.

Critical weaknesses
  1. The added value over existing lookahead methods is unclear. Table 14 shows that a single response-side continuation recovers about 89% of the gain over the prompt-only control. With the same continuations, the proposed scoring also performs very similarly to AnDPro (40.28 vs. 40.24). This makes it unclear how much the method adds beyond LAQ/LAQ++, which already use response-side lookahead queries. A controlled comparison is needed to establish the contribution of multiple futures and reliability weighting.

  2. The inference overhead limits the practical benefit. Sampling four responses before actual generation introduces additional sequential decoding and scoring, and the reported inference time is 1.46–2.77× that of AnDPro. Since the gains become small at larger cache budgets, it is unclear when the accuracy improvement justifies this extra latency.

Questions for authors
  1. Since most of the gain comes from using response-side queries, how much improvement remains over LAQ/LAQ++ under the same evaluation protocol and lookahead-token budget? Could you isolate the benefit of multiple trajectories and reliability weighting while keeping the scoring rule fixed?

  2. How does LORE-KV compare with KVzip? KVzip uses queries from teacher-forced context reconstruction as a proxy for future information needs, rather than sampling responses autoregressively. A comparison at the same retained-cache budget, reporting both accuracy and compression overhead, would help clarify the benefit of response-specific lookahead.

  3. In which workloads does sampling four responses provide a worthwhile practical trade-off? Could you report time to first token and end-to-end latency at matched accuracy against single-response lookahead and KVzip, including all lookahead and scoring costs?

2 comments
AuthorsOpenReview ↗

Response to Reviewer ikcN (Part 1 of 2)

Thank you for the rigorous review. For each point, we give our position, the evidence, and the limits.

W1 and Q1. Added value over LAQ and LAQ++

Position and contribution. Response-side future queries, which Lookahead Q-Cache [1] introduced, give most of the gain, and Section 2.2 will credit this work clearly. With the same rollouts, four objectives stay within 0.09 points at B=128 (Table 14, Appendix D), so the scoring rule is not our claim. LORE-KV treats the future as a distribution, estimates it with a sampled set of rollouts, and uses one rollout as its lightest setting. As in our response to Reviewer SB1H, one rollout gives most of the average gain, and the set adds reliability where one lookahead fails (Tables C and D). The control below isolates the number of rollouts, with fixed scoring and uniform weights.

Table B. Number of rollouts with fixed scoring and uniform weights (LongBench, Mistral-7B, T=8).

MethodB=128B=256B=512B=1024
M=139.6141.1241.4342.03
M=239.9241.2341.6342.10
M=440.0941.1541.6342.12
M=840.2841.1841.7342.15

M=2, 4, and 8 are at or above M=1 at every budget. The gain is largest at B=128 (+0.67 for M=8) and small at larger budgets. On LongBench, the reliability weight changes these values by at most 0.3 points with no consistent direction. On RULER at B=256, entropy weights add 1.1 to 1.4 points over log-probability weights (Table D).

Table B2. Seed stability (LongBench, Mistral-7B, LORE-KV/E, M=4, T=8, temperature 1.0).

BudgetSeed 42Seed 43Seed 44Mean ± SD
B=12839.8739.9940.2340.03 ± 0.18
B=51241.7641.6541.7541.72 ± 0.06

The results are stable across seeds, with a standard deviation of at most 0.18 points. The revised paper will add paired-seed comparisons for all values of M.

Settings of the values. Tables B and B2 come from the runs for this response, and Tables C and D come from the submission. Table B2 uses the submission default (LORE-KV/E, M=4, T=8). The submission reports 40.28 for this setting (95% confidence interval [39.97, 40.59]), and Table C shows this value at (4,8). Table B2 gives 40.03 for the same setting, inside that interval. The 40.28 at M=8 in Table B is a different setting (uniform weights), and the equal value is a coincidence.

The full cache (42.51 to 42.72) and AnDPro (38.96 to 38.89) also changed between the two sets of runs, but they use no sampling. Thus, the difference comes from the evaluation setup, not from the seeds. We did not yet identify the exact cause, so we compare methods only inside one set of runs. Inside each set, LORE-KV/E leads AnDPro by a similar margin (1.32 and 1.14 points).

Table C. Equal lookahead-token budget (LongBench, Mistral-7B, B=128, Table 13a in Appendix D).

Total tokens M×TOne rolloutSplit into several rolloutsMany short rollouts
8(1,8): 39.81(2,4): 39.49(4,2): 38.96
16(1,16): 39.62(2,8): 40.17(8,2): 39.01
32(1,32): 39.89(4,8): 40.28(8,4): 39.72

At 16 and 32 tokens, the best split has more than one rollout, and at 8 tokens one rollout is best. The (4,8) split also needs only 8 sequential steps, while (1,32) needs 32.

Table D. Several rollouts on RULER (Mistral-7B, Table 15 in Appendix D).

MethodB=128, 4K8K16KB=256, 4K8K16K
AnDPro54.8550.0645.2071.0564.0857.91
M=156.9054.2952.1568.4562.7757.70
M=4, log-probability weight58.3955.7250.9870.0864.4558.85
M=4, entropy weight (LORE-KV/E)58.4055.7751.0571.5165.5259.94

At B=256, M=1 is below AnDPro in all three settings. M=4 with log-probability weights adds 1.2 to 1.7 points over M=1. Table D has no uniform-weight M=4 row, so this is a combined change of the rollouts and the weights. A change from log-probability to entropy weights adds 1.1 to 1.4 more (2.2 to 3.1 in total). At B=128 and 16K, M=1 is better (52.15 against 51.05).

When extra rollouts are worth it. Extra rollouts help most at small budgets on LongBench and at B=256 on RULER. In other settings, one rollout is enough. This result motivates an adaptive M that stops when the top-B token set becomes stable. Section 5 already lists adaptive lookahead as future work, and we do not claim a result.

LAQ and LAQ++. On Mistral-7B (Table 1), published LAQ++ reaches 92.6%, 96.3%, and 97.6% of its full-cache score at B=128 to 512. LORE-KV/E reaches 94.8%, 97.3%, and 98.5%. These are published values, and we did not complete a matched comparison during the response period. LORE-KV adds only a one-time cost before decoding, and scoring takes most of this cost (Table E).

AuthorsOpenReview ↗

Response to Reviewer ikcN (Part 2 of 2)

Q2. Comparison with KVzip

The submission cites KVzip [2] in Appendix A.2, but we did not run a matched KVzip test for the submission or this response. KVzip is query-agnostic. It compresses a context once and reuses the cache for many queries, and its reconstruction pass approximately doubles the prefill cost. LORE-KV is prompt-conditional and targets one-shot requests at very small budgets, for example B=128 at 16K (less than 1% of the context). The KVzip paper evaluates ratios from 10% to 100%. We expect KVzip to be the better choice when many queries share one context.

W2 and Q3. Inference overhead and practical trade-off

Position. The overhead is a one-time cost before decoding. Section 4.4 and Appendix C.4 report its total time, and Table E now gives each stage. Time to first token includes prefill, lookahead, and scoring.

Table E. Seconds per sample (Mistral-7B, LongBench, ~6.2K-token prompts, 50 samples, B=128).

MethodPrefillLookaheadScoringTime to first tokenTotalDecode (ms/token)
AnDPro0.34-0.430.771.2923.0
LORE-KV, M=1, T=80.350.231.191.772.3522.2
LORE-KV/E, M=4, T=80.350.881.973.203.7621.6

The decode speed is the same for all methods (about 22 ms per token, as Section 4.4 states), so the relative cost decreases as the output becomes longer. Scoring is the largest stage, and an optimized implementation can reduce it. Single-response lookahead (M=1) decreases the extra time to first token from 2.43 s to 1.00 s.

LORE-KV is a good choice under these conditions:

  • Memory is the limit. To match LORE-KV/E at B=128, AnDPro needs about 208 tokens (62.5% more), so LORE-KV/E uses 38.5% fewer tokens (Section 4.1 and Table 16).
  • Long context, small budget. On RULER at B=128, the gain over AnDPro is 3.55 points at 4K and 5.85 at 16K (Section 4.3).
  • Poor match between prompt and future queries. The gain increases with this mismatch (Figure 6, Section 4.5).
  • Long outputs. With outputs of up to 1024 tokens, LORE-KV/E stays above AnDPro at every budget (Table F in our response to Reviewer dv8t).

Limit. Section 5 lists this one-time overhead as a limitation. For short outputs, strict latency targets, and high-throughput batches, M=1 or a method without lookahead is the better choice. The revised paper will give this guidance and the full latency results.

References.

Thank you again for these precise questions. The narrower claim and the new controls make the contribution of the paper exact.

Reviewer_SB1HOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper is proposing a learning-free KV eviction method. It is based on a simple idea of rolling the prefill out with monte-carlo continuation and measuring the importance of each token within KV cache within these rollouts. It measures the impact of each token in the KV-cache in a direct way of computing the impact of the token all the way to the output layer using the attention it receives. It also proposes a way to measure typicality of the continuation to weight the losses.

Critical strengths

The methodology is solid, specificially:

  • The model of token impact is direct as it is not a surrogate or heuristic.
  • Monte-carlo formulation is technically sound, though how useful is not clear from the experiments.
  • The importance is following actual rollouts so instead of past importance, it is modelling the future importance.
  • Empirical study is complete covering multiple datasets, benchmarks and models.
Critical weaknesses
  • I don't find it critical but there are parts of the method which turns out to be not that critical in the ablations. For example, the monte carlo setting might not be necessary as most of the gain comes with a single rollout anyway (eg. Table 14) If this is true, I suggest writing in a bit more direct way by emphasizing the rolling out part more than monte carlo part as currently it is discussed way less.
  • I find the exposition of the method a bit cumbersome with excessive notation and formulation. I believe the paper can be simplified as most of the notation is not entirely necessary to understand the paper.
  • One interesting comparison which is missing is performance against learning-based methods. Obviously, it is not a direct baseline as the target is learning-free but I am curious how does the method fare against learning based methods. Even if not a complete study but at least a simple mention of i) it is far from it and we need to close the gap or ii) learning-free and learning-based are somewhat close to each other would be nice to know for the reader. This is especially relevant since there is an additional wall-clock time for the method. Hence, what is more preferable loosing inference time or spending this time before any inference as a training cost.
Questions for authors

Overall I find the paper interesting and I would be curious to hear what authors think for the weaknesses part.

1 comment
AuthorsOpenReview ↗

Response to Reviewer SB1H

Thank you for the positive assessment and the constructive suggestions.

W1. The need for several rollouts

We agree that the rollout gives most of the average gain, and the revised abstract, introduction, and Section 3 will present the rollout first. On RULER at B=256, one rollout falls below AnDPro, and the sampled set recovers this loss. Four rollouts with log-probability weights add 1.2 to 1.7 points. This change combines the rollouts and the weights, and entropy weights add 1.1 to 1.4 more. With equal lookahead tokens at B=128, four rollouts of 8 tokens are better than one rollout of 32 tokens (40.28 against 39.89). In other settings, one rollout is enough, and the revised paper will state this (Tables B to D in our response to Reviewer ikcN).

W2. Excessive notation

We agree. Section 3 will explain the method in four steps:

  1. Run the model on the prompt.
  2. Sample M short continuations, and keep their query states.
  3. For each cached token, calculate the change in attention output if we remove the token.
  4. Keep the tokens with the largest average change.

The reliability derivations (Eq. 2 to Eq. 4 in Section 3.3) will move to Appendix C.3, which already gives their details. The main text will keep only the deletion score and the final score.

W3. Comparison with learning-based methods

Position. Cross-paper results suggest case (ii). On Llama-3.1-8B-Instruct, published LORE-KV scores are close to or above the published scores of IndexMem [1] and LKV [2]. The protocols are different, so these results do not show that LORE-KV is better than learned methods.

Table A1. RULER: LORE-KV/E against IndexMem [1].

ContextLORE-KV retention (tokens)LORE-KV/EIndexMem retention (tokens)IndexMemDifference
4K6.25% (256)72.2210% (~410)55.20+17.02
4K12.50% (512)83.1225% (1024)78.30+4.82
4K25.00% (1024)93.6025% (1024)78.30+15.30
16K3.125% (512)71.6010% (~1638)56.00+15.60
16K6.25% (1024)79.9810% (~1638)56.00+23.98
16K6.25% (1024)79.9825% (4096)74.30+5.68

Table A2. LongBench: LORE-KV/E against LKV [2] and IndexMem [1].

LORE-KV retention (tokens)LORE-KV/ELKV retention (tokens)LKVDifferenceIndexMem retention (tokens)IndexMemDifference
~1.7% (128)46.1110% (~743)43.93+2.1810% (~743)36.51+9.60
~3.4% (256)47.5010% (~743)43.93+3.5710% (~743)36.51+10.99
~6.9% (512)48.0510% (~743)43.93+4.1210% (~743)36.51+11.54
~13.8% (1024)48.7415% (~1114)46.73+2.0125% (~1857)42.18+6.56

Note. These are cross-paper comparisons with different protocols. They provide context, but they do not establish relative performance under matched conditions.

The papers do not report the same budgets. LORE-KV uses fixed token budgets, and IndexMem and LKV use retention ratios. Thus, we give the retention and the token number for each published result. We pair each LORE-KV budget with a published retention that is the same or higher. LORE-KV/E reaches a higher score with the same or less retention in all rows. LongBench token numbers use the mean input length of 7,426 tokens, and the IndexMem LongBench values come from Table 6 of [1].

Table A2 uses question-agnostic LKV. With the question, LKV reports 47.45 at 15% retention, and LORE-KV/E reaches 48.74 at ~13.8%. Also, nominal retention does not count the latent memory and the indexer key cache of IndexMem. LORE-KV adds no auxiliary memory, so its budget B is its full cache cost.

Training cost or request cost. LKV needs under 2 hours of training on 8 A100 GPUs, and IndexMem also needs training. LORE-KV needs no training and pays a one-time cost for each request (Table E in our response to Reviewer ikcN). For one fixed model with many requests, training is likely the better choice. For a new model, LORE-KV works immediately, as the submission shows on six backbones (Section 4.1 and Appendices E.2 and E.3). The revised paper will include these comparisons.

References.

Thank you again for these suggestions. They make the paper more direct, and they put LORE-KV in context with learned methods.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Rebuttal

Dear Reviewers and ACs,

We thank the reviewers for their careful reviews. Our responses add five experiments (Tables B, B2/H, E, F, and G1/G2) and a cross-paper comparison with two learned methods (Tables A1 and A2).

Strengths noted by the reviewers.

  • Reviewer SB1H found the methodology "solid", the token-impact model "direct as it is not a surrogate or heuristic", and the empirical study "complete covering multiple datasets, benchmarks and models".
  • Reviewer ikcN found the motivation "clear and reasonable" and noted that the ablations "help identify where the gains come from".
  • Reviewer dv8t called the future-use view of eviction "an insightful perspective" and found that the mechanism analyses "strengthen the understanding and credibility" of the approach.

Our responses.

  • Contribution (SB1H, ikcN). We agree that response-side lookahead queries, which LAQ introduced, give most of the average gain. We do not claim the scoring rule (Table 14). LORE-KV estimates the future with a sampled set of rollouts, and one rollout is its lightest setting. The set adds reliability where one rollout fails.
  • Multiple rollouts (ikcN, SB1H). On LongBench, M=8 adds 0.67 points at B=128 and at most 0.30 at larger budgets (Table B). On RULER at B=256, one rollout falls below AnDPro. Four rollouts with log-probability weights add 1.2 to 1.7 points, a combined change of rollouts and weights. Entropy weights add 1.1 to 1.4 more (Table D, from Table 15). In other settings, one rollout is enough.
  • Stability (dv8t, ikcN). The standard deviation across three seeds is at most 0.18 points. Temperature changes the mean by at most 0.22 points (Tables B2 and H).
  • Overhead (ikcN, dv8t). The one-time overhead raises time to first token from 0.77 s (AnDPro) to 1.77 s (M=1) and 3.20 s (M=4, Table E). Thus, we recommend LORE-KV when memory is the limit or outputs are long. For short outputs, strict latency targets, and high-throughput batches, M=1 or a method without lookahead is the better choice.
  • Long outputs and reasoning (dv8t). LORE-KV/E stays above AnDPro with outputs of up to 1024 tokens (Table F). On MATH-500, LORE-KV/E is at or above AnDPro, but an unpaired z-test did not detect a significant difference (p ≥ 0.17, Tables G1 and G2).
  • Learned methods (SB1H). Published LORE-KV/E scores are close to or above published IndexMem and LKV results at equal or lower retention (Tables A1 and A2). The protocols differ, so this is context, not evidence that LORE-KV is better than learned methods.
  • LAQ and KVzip (ikcN). We did not complete matched tests with the official LAQ, LAQ++, or KVzip code during the response period. M=1 (Tables B and D) is the closest control for single-rollout lookahead in our pipeline, but it is not the official LAQ code.
  • Other points (dv8t, SB1H). The score extends to decode-time eviction unchanged (future work), and the revision will simplify Section 3.

Code: https://anonymous.4open.science/r/train_free_kv-68CD

Sincerely, Authors

AuthorsOpenReview ↗

Thank You and Final Discussion

Dear Reviewers,

Thank you again for your careful reviews. We have thoughtfully acknowledged the strengths you have noted, addressed the concerns you expressed, and answered the questions you asked. The result, we believe, is a stronger paper thanks to your feedback. With just over one day remaining in the author-reviewer discussion period, we hope that you will take note of our comments and respond accordingly, including positive adjustments to your ratings if you deem them appropriate.

Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.