- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes DGE, a training-free KV-cache compression method that delays irreversible eviction until after generating the first k answer tokens with the full cache. DGE can be applied to existing eviction strategies such as SnapKV, PyramidKV, H2O, and StreamingLLM. Experiments across multiple backbones, cache budgets, and eviction strategies show consistent improvements over prior methods and performance close to FullKV.
Critical strengths
Overall, the method is simple, the empirical gains are substantial, and the timing-only control provides useful evidence that eviction timing matters. Details are as follows:
-
Novel and simple method. Existing methods mainly optimize which tokens to retain. DGE introduces eviction timing as an independent design dimension and obtains clear gains with a minimal algorithmic change.
-
Strong empirical results. DGE consistently outperforms existing eviction methods across multiple models and budgets, with some results approaching FullKV.
-
Effective timing-only control. DGE-W keeps SnapKV’s scorer and selection rule unchanged while delaying eviction, yet achieves performance close to DGE. This shows that eviction timing is an important factor.
Critical weaknesses
Overall, the main weaknesses concern the motivation for studying eviction timing, the explanation of why the method works, and the choice of the key hyperparameter k. Details are as follows:
-
The motivation for studying eviction timing is underdeveloped. The paper demonstrates limitations of compensation and selection methods, but does not clearly establish that premature eviction is their common cause. The transition from “improving what to keep is insufficient” to “when to evict is the main problem” lacks a direct and exclusive experiment.
-
The mechanism is not sufficiently validated. DGE and DGE-W show that delayed eviction is effective, but the trajectory-anchoring explanation remains largely intuitive. DGE-W also lets the model generate an initial prefix with the full cache, so the gain may come from the quality of this prefix in addition to eviction timing. The paper does not directly measure whether the draft prefix anchors the subsequent generation trajectory.
-
The choice of draft length lacks a principled basis. The main experiments use , but different backbones require different draft lengths. The paper does not provide a clear rule for selecting or sufficiently discuss the accuracy–latency trade-off.
Questions for authors
-
About the motivation of eviction timing: Section 4 shows that attention-level objectives do not always translate into better task performance. However, the connection between this observation and the claim that premature eviction is the root cause is still not fully clear to me. Could the authors explain why these results point specifically to eviction timing, rather than simply showing that attention-based objectives are imperfect? This is my main question about the paper’s motivation.
-
How to make choice of draft length k: The main experiments use k=2, but different backbones appear to have different sensitivities to the draft length. Could the authors clarify the rationale for choosing k=2, and whether the results support using it as a general default across models?
-
Role of the FullKV prefix: It would be interesting to analyze how much of the gain comes from the initial FullKV prefix itself. For example, is prefix quality correlated with the final performance recovery? Recent work on prefix utility, such as From Correctness to Utility: Gain-Based Prefix Evaluation for LLM Reasoning(https://arxiv.org/abs/2606.07190), also suggests that early generated prefixes can influence the subsequent reasoning trajectory. A similar analysis here could help clarify the respective roles of the FullKV prefix and delayed eviction.
8 comments
Q1 — Why do the §4 results point at timing?
Q1 — Why do the §4 results point at timing?
Could the authors explain why these results point specifically to eviction timing, rather than simply showing that attention-based objectives are imperfect?
The results suggest that changing the eviction time has a larger effect than the tested prefill-time adjustments.
-
Without pooling, SnapKV covers 11.5% more held-out attention and loses 1.73 (Table 1(b)). LAQ's pseudo-queries add 1.2% coverage and 1.98 points.
-
DGE-W$_{16}$ keeps SnapKV's scores, selection rule and budget and only cuts 16 tokens later. In it gains 2.64 points, more than LAQ's 1.98, and beats LAQ () by 0.66 ([+0.32, +1.01]).
We will update the decode-query result in Table 1(b) using the corrected implementation and clarify this comparison at the end of §4.
Q3 — Mechanism not validated; role of the FullKV prefix
Q3 — Mechanism not validated; role of the FullKV prefix.
DGE-W also lets the model generate an initial prefix with the full cache, so the gain may come from the quality of this prefix in addition to eviction timing. The paper does not directly measure whether the draft prefix anchors the subsequent generation trajectory. … Is prefix quality correlated with the final performance recovery?
Table R9. Prefix probes, .
| Run | First 16 tokens from | Their KV computed on | LongBench (vs DGE-W$_{16}$) |
|---|---|---|---|
| SnapKV | compressed cache | compressed cache | 42.22 |
| Draft cache | compressed cache | full cache | 42.42 (−2.44 [−2.95, −1.96]) |
| Teacher forcing | full cache | compressed cache | 44.71 (−0.15 [−0.26, −0.04]) |
| DGE-W$_{16}$ | full cache | full cache | 44.86 |
The additional probes in indicate that the initial prefix accounts for most of the gain: FullKV's first 16 tokens retain about 95% of DGE-W's improvement even when their KV is computed on the compressed cache. By comparison, retaining full-cache KV for a prefix generated by SnapKV preserves less than 10% of the gain. We will clarify this interpretation and cite arXiv:2606.07190 in the related-work section.
Response to Reviewer JaZX
We sincerely thank the reviewer for the thoughtful questions and constructive feedback. The comments helped us clarify the motivation behind deferred eviction, better understand the role of the initial prefix, and provide a more systematic analysis of the choice of (k). Our responses are summarized below.
1. Why eviction timing matters.
Our results show that changing when eviction occurs can provide greater benefits than the tested adjustments to prefill-time selection objectives. In particular, DGE-W preserves SnapKV's scoring, selection rule, and budget, while delaying eviction by 16 tokens. It improves over SnapKV by 2.64 points and outperforms matched LAQ by 0.66 points. We will clarify this comparison and update the corresponding results using the corrected implementation.
2. Role of the initial FullKV prefix.
We added prefix-level probes to disentangle the effects of initial-token quality and full-cache KV preservation. The results indicate that the quality of the initial tokens accounts for most of the performance gain: using the FullKV-generated prefix retains approximately 95% of DGE-W's improvement, even when its KV is computed on the compressed cache. We will clarify this finding and discuss the related work (arXiv:2606.07190).
3. Choice of (k) and the accuracy–latency trade-off.
We added a systematic comparison across (k=8,12,16,) and (24), using matched LAQ look-ahead lengths. DGE consistently outperforms matched LAQ from (k=12) onward. Although increasing (k) to 24 provides a small additional gain, we select (k=16) as the default to balance accuracy and latency. At this setting, the additional full-cache decoding takes approximately 0.17 seconds per request. We will incorporate these results and the selection rationale into the revised paper.
We appreciate the reviewer's helpful feedback, which has enabled us to strengthen the empirical analysis and clarify the design choices underlying DGE.