acceptodds
Under review as a conference paper at ICLR 2027

When to Evict, Not What to Keep: Draft-Guided Eviction for Training-Free KV-Cache Compression

Abstract

Training-free KV-cache compression methods such as SnapKV, H2O, and PyramidKV evict tokens at the end of prefill, aiming to preserve the attention mass that future queries are expected to use - optimizing what to keep. We show that this objective fails in two distinct ways. (1) Compensation: restoring the evicted attention mass can recover the attention-level target without recovering task quality. (2) Selection: covering more of the true decode-query mass can hurt quality when the recovered mass is fragmented rather than concentrated in coherent spans. These failures share a common cause: eviction occurs before the queries that determine the answer trajectory exist. We propose Draft-Guided Eviction (DGE), which defers eviction until after drafting the first k=2 answer tokens using the full cache - just one decode step beyond prefill. Because the draft is generated from the answer's own prefix, no cache entries are discarded before this trajectory signal becomes available. The per-head cache budget remains unchanged, and DGE can be applied directly to SnapKV, PyramidKV, H2O, and StreamingLLM without modifying their eviction scores. Unlike extra-pass methods, DGE changes when eviction occurs rather than what cache entries are selected. Extensive experiments demonstrate that DGE outperforms prior methods at every evaluated budget on five of six instruct-tuned backbones, achieving 44.2 on LongBench, nearly matching FullKV at 44.3. The timing-only control DGE-W achieves the same score, demonstrating that the gain comes from when eviction occurs rather than what is selected - an effect we term trajectory anchoring.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV-Cache Compression, Draft-Guided Eviction

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_JaZXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes DGE, a training-free KV-cache compression method that delays irreversible eviction until after generating the first k answer tokens with the full cache. DGE can be applied to existing eviction strategies such as SnapKV, PyramidKV, H2O, and StreamingLLM. Experiments across multiple backbones, cache budgets, and eviction strategies show consistent improvements over prior methods and performance close to FullKV.

Critical strengths

Overall, the method is simple, the empirical gains are substantial, and the timing-only control provides useful evidence that eviction timing matters. Details are as follows:

  1. Novel and simple method. Existing methods mainly optimize which tokens to retain. DGE introduces eviction timing as an independent design dimension and obtains clear gains with a minimal algorithmic change.

  2. Strong empirical results. DGE consistently outperforms existing eviction methods across multiple models and budgets, with some results approaching FullKV.

  3. Effective timing-only control. DGE-W keeps SnapKV’s scorer and selection rule unchanged while delaying eviction, yet achieves performance close to DGE. This shows that eviction timing is an important factor.

Critical weaknesses

Overall, the main weaknesses concern the motivation for studying eviction timing, the explanation of why the method works, and the choice of the key hyperparameter k. Details are as follows:

  1. The motivation for studying eviction timing is underdeveloped. The paper demonstrates limitations of compensation and selection methods, but does not clearly establish that premature eviction is their common cause. The transition from “improving what to keep is insufficient” to “when to evict is the main problem” lacks a direct and exclusive experiment.

  2. The mechanism is not sufficiently validated. DGE and DGE-W show that delayed eviction is effective, but the trajectory-anchoring explanation remains largely intuitive. DGE-W also lets the model generate an initial prefix with the full cache, so the gain may come from the quality of this prefix in addition to eviction timing. The paper does not directly measure whether the draft prefix anchors the subsequent generation trajectory.

  3. The choice of draft length kk lacks a principled basis. The main experiments use k=2k=2, but different backbones require different draft lengths. The paper does not provide a clear rule for selecting kk or sufficiently discuss the accuracy–latency trade-off.

Questions for authors
  1. About the motivation of eviction timing: Section 4 shows that attention-level objectives do not always translate into better task performance. However, the connection between this observation and the claim that premature eviction is the root cause is still not fully clear to me. Could the authors explain why these results point specifically to eviction timing, rather than simply showing that attention-based objectives are imperfect? This is my main question about the paper’s motivation.

  2. How to make choice of draft length k: The main experiments use k=2, but different backbones appear to have different sensitivities to the draft length. Could the authors clarify the rationale for choosing k=2, and whether the results support using it as a general default across models?

  3. Role of the FullKV prefix: It would be interesting to analyze how much of the gain comes from the initial FullKV prefix itself. For example, is prefix quality correlated with the final performance recovery? Recent work on prefix utility, such as From Correctness to Utility: Gain-Based Prefix Evaluation for LLM Reasoning(https://arxiv.org/abs/2606.07190), also suggests that early generated prefixes can influence the subsequent reasoning trajectory. A similar analysis here could help clarify the respective roles of the FullKV prefix and delayed eviction.

8 comments
AuthorsOpenReview ↗

Response to Reviewer JaZX

We sincerely thank the reviewer for the thoughtful questions and constructive feedback. The comments helped us clarify the motivation behind deferred eviction, better understand the role of the initial prefix, and provide a more systematic analysis of the choice of (k). Our responses are summarized below.

1. Why eviction timing matters.

Our results show that changing when eviction occurs can provide greater benefits than the tested adjustments to prefill-time selection objectives. In particular, DGE-W preserves SnapKV's scoring, selection rule, and budget, while delaying eviction by 16 tokens. It improves over SnapKV by 2.64 points and outperforms matched LAQ by 0.66 points. We will clarify this comparison and update the corresponding results using the corrected implementation.

2. Role of the initial FullKV prefix.

We added prefix-level probes to disentangle the effects of initial-token quality and full-cache KV preservation. The results indicate that the quality of the initial tokens accounts for most of the performance gain: using the FullKV-generated prefix retains approximately 95% of DGE-W's improvement, even when its KV is computed on the compressed cache. We will clarify this finding and discuss the related work (arXiv:2606.07190).

3. Choice of (k) and the accuracy–latency trade-off.

We added a systematic comparison across (k=8,12,16,) and (24), using matched LAQ look-ahead lengths. DGE consistently outperforms matched LAQ from (k=12) onward. Although increasing (k) to 24 provides a small additional gain, we select (k=16) as the default to balance accuracy and latency. At this setting, the additional full-cache decoding takes approximately 0.17 seconds per request. We will incorporate these results and the selection rationale into the revised paper.

We appreciate the reviewer's helpful feedback, which has enabled us to strengthen the empirical analysis and clarify the design choices underlying DGE.

AuthorsOpenReview ↗

Q1 — Why do the §4 results point at timing?

Q1 — Why do the §4 results point at timing?

Could the authors explain why these results point specifically to eviction timing, rather than simply showing that attention-based objectives are imperfect?

The results suggest that changing the eviction time has a larger effect than the tested prefill-time adjustments.

  • Without pooling, SnapKV covers 11.5% more held-out attention and loses 1.73 (Table 1(b)). LAQ's pseudo-queries add 1.2% coverage and 1.98 points.

  • DGE-W$_{16}$ keeps SnapKV's scores, selection rule and budget and only cuts 16 tokens later. In Table R1\color{red}{\text{Table R1}} it gains 2.64 points, more than LAQ's 1.98, and beats LAQ (m=16m=16) by 0.66 ([+0.32, +1.01]).

We will update the decode-query result in Table 1(b) using the corrected implementation and clarify this comparison at the end of §4.

AuthorsOpenReview ↗

Q3 — Mechanism not validated; role of the FullKV prefix

Q3 — Mechanism not validated; role of the FullKV prefix.

DGE-W also lets the model generate an initial prefix with the full cache, so the gain may come from the quality of this prefix in addition to eviction timing. The paper does not directly measure whether the draft prefix anchors the subsequent generation trajectory. … Is prefix quality correlated with the final performance recovery?

Table R9. Prefix probes, k=16k=16.

RunFirst 16 tokens fromTheir KV computed onLongBench (vs DGE-W$_{16}$)
SnapKVcompressed cachecompressed cache42.22
Draft cachecompressed cachefull cache42.42 (−2.44 [−2.95, −1.96])
Teacher forcingfull cachecompressed cache44.71 (−0.15 [−0.26, −0.04])
DGE-W$_{16}$full cachefull cache44.86

The additional probes in Table R9\color{red}{\text{Table R9}} indicate that the initial prefix accounts for most of the gain: FullKV's first 16 tokens retain about 95% of DGE-W's improvement even when their KV is computed on the compressed cache. By comparison, retaining full-cache KV for a prefix generated by SnapKV preserves less than 10% of the gain. We will clarify this interpretation and cite arXiv:2606.07190 in the related-work section.

Reviewer_bqBSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Draft-Guided Eviction (DGE), a KV-cache management technique that delays KV-cache eviction until the initial stage of decoding. DGE effectively avoids the need to recover prematurely evicted KV content through mechanisms such as merging or approximate compensation, and mitigates the coverage–contiguity trade-off in token selection. This technique can enhance various training-free KV-cache compression methods by improving the attention module's ability to retrieve information needed for decoding, simply by delaying the timing of eviction. The underlying intuition is straightforward: using the full KV cache during the initial decoding stage provides a higher-quality signal for guiding subsequent generation, and the reported gains are substantial.

Critical strengths

This paper opens up a new direction by introducing a new dimension for KV-cache compression. Previous work has devoted considerable effort to determining which KV-cache entries should be retained. This paper demonstrates that task performance can be improved simply by delaying KV-cache eviction until decoding has begun (that is, allowing real decode queries to anchor the answer trajectory before eviction). The ablation studies show that most of the gains come from the delay itself. This is an interesting new perspective on KV-cache management systems (i.e., when to evict).

Critical weaknesses

DGE-W, which changes only the timing of eviction, also achieves a high score on LongBench. The authors interpret this as evidence that the performance gains stem from when cache eviction occurs rather than from the specific content selected for retention. However, the paper invokes the irrecoverability of discarded content to motivate the information-compensation problem faced by prefill-time KV-cache compression, suggesting that access to the full information in the KV cache is important for high-quality decoding. The attribution ablation shows that DGE achieves a score very close to that of DGE-W, which retains SnapKV's window-based scoring mechanism (Figure 3(b), where DGE improves the score by only 0.7 points). This supports the authors' central claim that the timing of eviction itself is the primary driver of the gains. However, it raises a critical question: do the gains from delaying eviction actually stem from the information preserved by generating the initial tokens with the full KV cache, or does the quality of these initial tokens influence subsequent tokens through the generation trajectory? The contributions of these two intertwined mechanisms within the eviction-timing intervention have not been fully disentangled.

Questions for authors
  1. Although DGE reduces time to first token (TTFT), its total latency in the 32-token short-generation setting is still 1.45× that of SnapKV. This suggests that the additional overhead introduced by DGE is harder to amortize for short outputs. Most LongBench tasks have relatively short outputs, so it is necessary to further analyze whether the quality gains provided by DGE are sufficient to offset its additional latency on these tasks. The paper reports quality and latency separately but does not provide a sufficiently comprehensive joint cost–benefit analysis across different tasks and output lengths.
  2. DGE’s core mechanism performs eviction only once and does not re-select the original prompt KV entries as query demands change during subsequent generation. In long-running agentic generation, historical tokens deemed unimportant early on may become important later, but their original KV entries are no longer directly accessible. In contrast, H2O can perform online eviction based on accumulated attention scores, while StreamingLLM continuously updates its retained set by preserving attention sinks and maintaining a sliding window of recent tokens. Could DGE’s lack of adaptivity significantly limit its effectiveness in terms of cache hit rate, accessibility of historical information, and generation quality on long-generation workloads?
5 comments
AuthorsOpenReview ↗

Response to Reviewer bqBS

We sincerely thank the reviewer for the thoughtful comments and constructive suggestions. The feedback helped us better understand the mechanisms behind DGE and clarify its benefits and limitations. Our responses are summarized below.

1. Disentangling the effects of deferred eviction.

We added prefix-level probes to distinguish the effects of initial-token quality from those of full-cache KV preservation. The results show that the quality of the initial generated tokens explains most of DGE's gain, while retaining full-cache KV provides a smaller additional benefit. We will clarify this interpretation in the revised paper.

2. Quality–latency trade-off on short-output tasks.

We added a category-wise comparison with SnapKV. At batch size 1, both methods have the same total latency of 2.08 seconds for 32-token generation, including prefill. DGE improves performance across most categories, although we observe no gain on Synthetic tasks. We will clarify that the benefits of deferral may depend on the task category.

3. One-shot eviction versus online adaptivity.

We evaluated deferred eviction with four base eviction policies, including H2O and StreamingLLM. Deferral improves all four methods, with gains ranging from 2.87 to 7.88 points. However, we have not compared DGE with online eviction policies. We therefore agree that the potential benefits of online adaptivity in long-running agentic workloads remain an open question, and will explicitly acknowledge this limitation.

We appreciate the reviewer’s feedback, which has helped us strengthen the analysis and present a more balanced account of DGE's contributions and limitations.

AuthorsOpenReview ↗

Two intertwined mechanisms in the timing intervention.

Two intertwined mechanisms in the timing intervention.

Do the gains from delaying eviction actually stem from the information preserved by generating the initial tokens with the full KV cache, or does the quality of these initial tokens influence subsequent tokens through the generation trajectory? The contributions of these two intertwined mechanisms within the eviction-timing intervention have not been fully disentangled.

We conducted two additional probes using the corrected implementation (k=16k=16, SnapKV window scoring as in DGE-W, Llama-3.1-8B, B=128B=128). The results are reported in Table R9\color{red}{\text{Table R9}}.

Table R9. Prefix probes, k=16k=16.

RunFirst 16 tokens fromTheir KV computed onLongBench (vs DGE-W$_{16}$)
SnapKVcompressed cachecompressed cache42.22
Draft cachecompressed cachefull cache42.42 (−2.44 [−2.95, −1.96])
Teacher forcingfull cachecompressed cache44.71 (−0.15 [−0.26, −0.04])
DGE-W$_{16}$full cachefull cache44.86

Table R9\color{red}{\text{Table R9}} indicates that the quality of the initial generated tokens explains most of DGE-W's gain. Using FullKV's first 16 tokens while computing their KV on the compressed cache retains about 95% of the gain over SnapKV (44.71 vs. 44.86). In contrast, retaining full-cache KV for a prefix generated by SnapKV preserves less than 10% of the gain (42.42 vs. 42.22). The full-cache KV contributes a smaller additional improvement of 0.15 points. Draft-query scoring adds a further 0.23 points (Table R1\color{red}{\text{Table R1}}). We will clarify that deferral primarily benefits generation by allowing the answer's initial tokens to be produced using the full cache.

AuthorsOpenReview ↗

Q1 — Joint cost–benefit on short-output tasks.

Q1 — Joint cost–benefit on short-output tasks.

Its total latency in the 32-token short-generation setting is still 1.45× that of SnapKV … Most LongBench tasks have relatively short outputs, so it is necessary to further analyze whether the quality gains provided by DGE are sufficient to offset its additional latency on these tasks.

We added a category-wise comparison of DGE$_{16}$ and SnapKV. Table R10\color{red}{\text{Table R10}} reports the results for the corrected implementation on Llama-3.1-8B with B=128B=128.

Table R10. Gain per LongBench category.

CategorySnapKVDGE$_{16}$Gain
Single-Doc QA35.5140.97+5.45
Multi-Doc QA38.9740.89+1.92
Summarization22.1623.37+1.21
Few-shot64.8868.48+3.60
Synthetic40.4639.73−0.73
Code54.9860.47+5.48

For a 32-token output at batch size 1, DGE and SnapKV both take 2.08 s including prefill (Table R7\color{red}{\text{Table R7}}). The category-wise results show no gain on Synthetic, suggesting that deferral may be less suitable for this category (Table R10\color{red}{\text{Table R10}}).

Table R7. Throughput (requests/s).

32 tok, b=1b=132 tok, b=8b=832 tok, b=16b=16512 tok, b=1b=1512 tok, b=8b=8512 tok, b=16b=16Peak memory (b=16b=16)
FullKV0.5931.1181.1610.0660.2450.29943.0 GB
SnapKV0.4811.1661.2340.0520.3300.51529.3 GB
DGE$_{12}$0.5241.0261.0790.0500.3150.50443.0 GB
DGE$_{16}$0.4801.0111.0620.0490.3330.49643.0 GB
LAQ (m=12m=12)*0.4470.5000.4910.0660.0680.06221.3 GB
LAQ (m=16m=16)*0.4130.4380.4490.0570.0630.06421.3 GB
Reviewer_D87VOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Draft-Guided Eviction (DGE) delays KV-cache compression until an initial answer prefix has been generated using the full cache. It can preserve an existing eviction rule or use attention from the draft to select retained entries. Experiments suggest that delaying eviction accounts for most of the observed recovery, motivating the hypothesis that full-cache opening tokens help stabilize subsequent generation.

Critical strengths
  1. The paper identifies eviction timing as a meaningful design choice distinct from token selection. This is a simple, potentially useful contribution that can complement existing compression methods without training.
  2. The timing-only control is particularly informative. Keeping the base selection rule unchanged while delaying eviction provides stronger evidence for the scheduling effect than comparisons that simultaneously modify selection and timing.
  3. The evaluation includes multiple backbones, eviction policies, draft lengths, and paired statistical comparisons. Analyses of early termination and longer outputs also help assess whether improvements merely reflect answers completed before compression.
Critical weaknesses
  1. The conclusions about the limitations of selection and compensation appear broader than the evidence supports. Failure of the tested approximations to convert attention-level improvements into task gains does not establish a general ceiling for these method families or show that timing resolves their underlying limitations.
  2. Generalization beyond greedy, single-response generation remains uncertain. Longer summaries provide useful evidence, but they do not establish robustness when generation branches, revises earlier decisions, or repeatedly retrieves different contextual information.
  3. Improved time to first token does not necessarily imply improved request-level efficiency. Moving compression after generation begins can introduce an early inter-token stall, while cache editing can outweigh the benefit for short responses. The current measurements leave the serving-level trade-off incompletely characterized.
Questions for authors

Do the conclusions hold against faithfully implemented stronger selection and compensation methods, and under settings where substantial information must be retrieved after the initial answer prefix?

Does deferred eviction remain effective under sampling, extended reasoning, and multi-turn or agentic workloads where information requirements change during generation?

How does DGE compare with competing methods at matched output quality in total latency, inter-token latency, and throughput under concurrent requests?

9 comments
AuthorsOpenReview ↗

Response to Reviewer D87V

We sincerely thank the reviewer for the thoughtful and constructive feedback. The comments helped us clarify the scope of our claims and strengthen the experimental evaluation. Our responses and revisions are summarized below.

1. Scope of the conclusions.
We agree that our original conclusions were broader than the evidence warranted. We have narrowed our claims to the specific selection and compensation methods evaluated, without suggesting that deferred eviction resolves the general limitations of these approaches.

2. Generalization to more diverse generation settings.
To examine robustness beyond greedy single-response generation, we added multi-key, multi-query, and multi-value RULER evaluations, as well as multi-turn cache reuse and sampling experiments. We also clarify that branching generation, extended reasoning, and agentic workloads remain important directions for future evaluation.

3. Serving efficiency and latency.
We added per-token latency and throughput measurements under different output lengths and batch sizes. These results provide a more comprehensive view of the serving trade-offs, including the initial decoding overhead and the throughput behavior under concurrent requests.

4. Stronger baselines and retrieval-intensive settings.
We extended the comparison to include AdaKV and HeadKV and added retrieval-intensive RULER evaluations. We also revised the discussion to ensure that our conclusions remain within the scope of the evaluated baselines.

We appreciate the reviewer’s suggestions, which have helped us make the paper more precise and the experimental evaluation more comprehensive.

AuthorsOpenReview ↗

Conclusions about selection/compensation broader than the evidence

Conclusions about selection/compensation broader than the evidence.

The conclusions about the limitations of selection and compensation appear broader than the evidence supports. Failure of the tested approximations to convert attention-level improvements into task gains does not establish a general ceiling for these method families or show that timing resolves their underlying limitations.

Thank you for pointing this out. We agree that the original wording was too broad. We will limit the claim in §4 and the abstract to the four counterparts evaluated, and remove the suggestion that timing resolves the broader limitations of selection and compensation. The corrected results in Table R1\color{red}{\text{Table R1}} support this more focused conclusion.

AuthorsOpenReview ↗

Generalization beyond greedy, single-response generation

Generalization beyond greedy, single-response generation.

Longer summaries provide useful evidence, but they do not establish robustness when generation branches, revises earlier decisions, or repeatedly retrieves different contextual information.

To examine these settings, we added multi-key, multi-query, and multi-value RULER evaluations, as well as multi-turn and sampling experiments. Table R3\color{red}{\text{Table R3}} reports the RULER results at B=32B=32 (five tasks and 100 samples per task). FullKV, SnapKV, and LAQ are unaffected by the implementation correction.

Table R3. RULER multi-key/query/value, B=32B=32.

ContextFullKVSnapKVLAQ$_{12}$DGE-W$_{12}$DGE$_{12}$LAQ$_{16}$DGE-W$_{16}$DGE$_{16}$
4K98.9511.7047.5556.3559.8550.0563.6567.20
8K98.2510.5546.4057.4061.6048.4066.3070.80

LAQ$_m$ is LAQ with mm look-ahead tokens. Against LAQ with the same look-ahead length, DGE improves by 12–15 points at k=12k=12 and 17–22 points at k=16k=16; the gains are largest on multi-query and multi-value tasks. Multi-key 3 remains challenging for all compressed methods, and DGE is approximately 30 points below FullKV. Table R4\color{red}{\text{Table R4}} evaluates multi-turn cache reuse using 167 question pairs from shared LongBench contexts (NarrativeQA, Qasper, and MultiFieldQA-en). For the second turn, Q2Q_2 is appended to the first-turn cache without re-prefill (B=128B=128; QA F1; paired-bootstrap 95% confidence intervals).

Table R4. Two-turn cache reuse.

Turn-1 policyEvictsTurn 1Turn 2Turn 2 vs LAQ (m=km=k)
FullKV0/16737.0237.12
SnapKV167/16732.5518.56
LAQ (m=12m=12)167/16735.3818.70—
DGE$_{12}$52/16735.8328.81+10.11 [+4.65, +15.46]
DGE$_{12}$, always evicts*163/16735.8317.06−1.64 [−4.79, +1.37]
LAQ (m=16m=16)167/16735.0518.01—
DGE$_{16}$36/16736.0229.75+11.74 [+5.91, +17.35]
DGE$_{16}$, always evicts*163/16736.0217.15−0.86 [−3.95, +2.14]

Comments on the paper

2 comments
AuthorsOpenReview ↗

Dear Reviewers and Area Chair

Thank you for your thoughtful reviews. During the throughput evaluation for D87V, We identified a minor issue in part of the post-eviction decoding implementation that slightly affected the reported DGE results. We have corrected the implementation and rerun the experiments. The main finding remains: when to evict matters more than what to keep. Under a matched look-ahead setting, DGE$_{16}$ outperforms LAQ by 0.89 points (95% CI: [+0.56, +1.24]), is 2.87 points above SnapKV, and is 1.15 points below FullKV (Table R1\color{red}{\text{Table R1}}). We will update all affected results in the revised paper.

Fixed evaluation protocol. We corrected the post-eviction continuation so that decoding proceeds one token at a time using the B+kB+k cache. All results below use this implementation. For a matched comparison, each draft length kk is compared with LAQ using the same look-ahead length (m=km=k), starting at k=8k=8 (LAQ's default); we use k=16k=16 as our default. Table R1\color{red}{\text{Table R1}} reports the main setting from Table 2 (Llama-3.1-8B, B=128B=128, and the 16-set LongBench average), together with paired-bootstrap 95% confidence intervals.

Table R1. Fixed LongBench results.

MethodLongBenchvs SnapKV+LAQ (m=km=k)
FullKV46.24
SnapKV42.22
SnapKV+LAQ (m=8m=8)44.13
SnapKV+LAQ (m=12m=12)44.25
SnapKV+LAQ (m=16m=16)44.20
SnapKV+LAQ (m=24m=24)44.45
DGE-W$_8$ (timing only)44.02−0.11 [−0.44, +0.23]
DGE$_8$44.27+0.14 [−0.19, +0.49]
DGE-W$_{12}$ (timing only)44.61+0.36 [+0.04, +0.70]
DGE$_{12}$44.82+0.57 [+0.24, +0.93]
DGE-W$_{16}$ (timing only)44.86+0.66 [+0.32, +1.01]
DGE$_{16}$45.09+0.89 [+0.56, +1.24]
DGE$_{24}$45.31+0.85 [+0.53, +1.19]

Table R1\color{red}{\text{Table R1}} shows that DGE improves as the draft length increases, closing 51%, 65%, 71%, and 77% of SnapKV's 4.02-point gap at k=8,12,16,k=8,12,16, and 2424, respectively. DGE ties LAQ at k=8k=8 and outperforms it from k=12k=12 onward, with gains of 0.57, 0.89, and 0.85 points. At k=16k=16, DGE is ahead on 11 of the 16 LongBench sets. DGE evicts only when the answer exceeds the draft length; this occurs in 68.3% and 66.4% of samples at k=12k=12 and k=16k=16, respectively. On these samples, DGE exceeds LAQ by 0.85 and 1.31 points, respectively. These results support the benefit of deferring eviction under the evaluated settings.

DGE also has a lower estimated memory-read overhead than LAQ. LAQ performs mm additional forward passes, whereas DGE generates kk output tokens while attending to the full cache. For Llama-3.1-8B with a 7.5K-token prompt and k=16k=16, the estimated additional memory reads are approximately 15 GB for DGE and 257 GB for LAQ. The two estimates become comparable only at a prompt length of roughly 120K tokens.

Table R2\color{red}{\text{Table R2}} re-runs Table 2 of the paper with the fixed code: five backbones, B=32B=32, 64 and 128, LongBench average over 16 sets, against LAQ of the same look-ahead length (m=16m=16).

Table R2. Fixed LongBench results across backbones and budgets.

BackboneBBFullKVSnapKVLAQ (m=16m=16)DGE-W$_{16}$DGE$_{16}$DGE$_{16}$ vs LAQ
Llama-3.2-3B3242.4531.5735.7940.2940.59+4.80
6435.6439.1340.5740.97+1.84
12838.0740.9040.8041.27+0.37
Mistral-7B3241.3526.3731.8638.5039.11+7.25
6431.3536.0139.1839.52+3.51
12834.5338.1639.7139.76+1.60
Qwen2.5-7B3245.5431.5637.8143.2643.73+5.92
6436.7741.5943.6544.08+2.49
12839.9843.4744.0644.38+0.91
Llama-3.1-8B3246.2435.8540.6544.1444.55+3.90
6439.3843.2244.6144.87+1.65
12842.2244.2044.8645.09+0.89
Qwen2.5-14B3246.0529.9437.1043.8344.41+7.31
6436.4242.6344.2144.52+1.89
12840.5144.4644.6144.81+0.35
Average44.3335.3439.8042.4242.78+2.98

In Table R2\color{red}{\text{Table R2}}, DGE$_{16}$ outperforms LAQ in all 15 settings, with margins of 0.35–7.31 points, and exceeds DGE-W$_{16}$ by 0.05–0.61 points. The average margin over LAQ increases as the cache budget decreases: 0.82 points at B=128B=128, 2.28 at B=64B=64, and 5.84 at B=32B=32.

Revision plan. We will update all DGE results using the corrected implementation and revise the main claim to: deferring eviction by 16 decoded tokens outperforms prefill-time look-ahead of the same length. We will also update the mechanism and latency analyses, revise the affected discussion, and release the implementation and evaluation scripts with the final version.

Revision checklist.

  1. Re-run all DGE experiments with the corrected continuation implementation.
  2. Revise the central claim.
  3. Update the mechanism analyses and add the sampling, multi-turn, and base-evictor comparisons.
  4. Correct draft-cache storage and update latency and throughput measurements.
  5. Narrow claims to the evaluated methods and identify untested settings, including online eviction and extended agentic generation.
  6. Release the source code and evaluation scripts with the final version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.