- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
EpiKV is a KV cache eviction method for reasoning models that uses hidden-state signals and periodically recomputed query–key relevance without requiring full attention matrices or modified attention kernels. Its experiments suggest that much of the accuracy loss under tight cache budgets comes from repetitive, non-terminating generations, which retaining tokens relevant to recent queries helps mitigate. Although the method achieves competitive accuracy and integrates with vLLM, the contribution of its “epiphany” score, the fairness of adapted baselines, and its end-to-end serving benefits require further validation.
Critical strengths
-
The paper addresses a practically important problem. Controlling KV cache growth during long reasoning traces is relevant to memory-efficient inference, and the method explicitly considers compatibility with existing serving infrastructure.
-
The failure-mode analysis is informative. Separating errors into length-cap hits and terminated incorrect answers reveals that much of the degradation under tight cache budgets is associated with non-termination, providing a useful diagnostic beyond aggregate accuracy.
-
The method achieves competitive accuracy at small cache budgets. On MATH-500, EpiKV performs comparably to RaaS on Llama-8B and significantly outperforms it on Qwen-7B at several budgets. Component ablations also demonstrate the benefit of combining recent-query relevance with budget filling.
-
The implementation demonstrates practical feasibility. EpiKV runs inside vLLM without modifying attention kernels, with accuracy consistent with the evaluation harness on the tested subset. The paper also reports implementation overheads and deployment limitations, making its practical claims easier to assess.
Critical weaknesses
- The role of the epiphany score in the full method is unclear. According to §2.3, segment allocation uses key variance, while both within-segment ranking and budget filling use query relevance, leaving no explicit role for the epiphany score in token selection. This disconnect weakens the paper’s central motivation. The authors should clarify where the score affects eviction decisions and establish its contribution through an ablation of the full method.
- Baseline modifications and unequal cache occupancy complicate fair comparisons. R-KV and LongFlow are adapted to use a shared keep-mask instead of their original per-head or per-layer selection, potentially weakening these baselines. Meanwhile, EpiKV can retain up to tokens—nearly 25% above the nominal budget at —whereas some variants leave part of the budget unused. Comparisons should preserve the baselines’ original selection granularity and match actual cache occupancy rather than nominal budgets alone.
- Prompt protection is not adequately controlled, and a prompt-preserving random baseline is missing. EpiKV always preserves the full prompt, while several baselines do not receive equivalent protection. The paper lacks controlled comparisons between protecting the prompt and allowing it to compete with generated tokens for retention. Random Attention shows that preserving the prompt while randomly evicting generated tokens can provide a strong baseline. Without matched prompt protection and an independent per-head random eviction baseline, the incremental value of EpiKV’s selection mechanism remains insufficiently established.
- The claims about FlashAttention incompatibility are too categorical. Requiring attention scores does not necessarily require eager attention throughout the forward pass: the main computation can use FlashAttention while the scores needed for eviction are recomputed separately for a small subset of queries, as EpiKV itself does. The paper should distinguish limitations of existing implementations from inherent algorithmic incompatibility and assess whether baselines such as R-KV could adopt a similar approach, including its computational cost and accuracy implications.
- The model coverage is too limited to establish generality. Experiments cover only DeepSeek-R1-Distill-Llama-8B and DeepSeek-R1-Distill-Qwen-7B, both from the same distillation family, and exclusively use greedy decoding. Evaluation on newer model families, such as Qwen3/Qwen3.5, larger models, and sampling-based decoding would help establish whether the hidden-state signals, non-termination patterns, and eviction benefits generalize beyond the tested setting.
[1] Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning
Questions for authors
Seeing waeknesses
3 comments
Thank you for your response! I will raise my score.
Thank you for the constructive review and for reconsidering your score. Your suggestions made the paper stronger.
Reply to Reviewer G9ic
Thank you for the review. Each of your five points asked for an experiment, and we have run all five.
The score is the ranking in EpiKV^-Q, and it fixes where EpiKV reads. The relation and the ablation are in the general response, Section 3. The score alone already beats LagKV at every budget and the key norm at the tight budgets.
At their original granularity, EpiKV beats LongFlow outright, beats R-KV on Qwen-7B, and equals R-KV on Llama-8B at a twelfth of R-KV's compaction time. We ran the released R-KV unmodified (per head and per layer) and LongFlow on its own kernel. Both rows are in the head-to-head table (general response, Section 1), and the revised Table 2 reports the released R-KV. On occupancy, the released R-KV has the same peak as EpiKV, K+127, since both compact every 128 steps. With a strict trigger that keeps the cache at or below K, EpiKV at K=512 beats RaaS on Qwen3-8B (60.4 against 53.9) and Qwen3-14B (60.2 against 56.0) and is within 1.5 points of it on the two paper models. Strict EpiKV is within paired noise of the shipped EpiKV in every cell (revised appendix, controls table). Occupancy averaged over a generation is in the general response, Section 4.
We ran it, and prompt protection does not explain the margin. A policy that keeps the full prompt and the recent tokens, and evicts the rest at random, scores 12 points below EpiKV at K=512 on both models. It also loops on nearly half the problems. Keeping the tokens that the model's current queries point to prevents those loops. On matched protection, RaaS, LagKV and the key norm keep the full prompt exactly as EpiKV does, so those comparisons are already matched.
We ran exactly this. On EpiKV's query read, RaaS's rule matches EpiKV at every budget, and R-KV's rule falls well below it at tight budgets (general response, Section 2). RaaS can leave the eager kernel by adopting EpiKV's query read, and no released baseline does. The serving table in that section gives the cost of not adopting it: the released R-KV runs on FlashAttention only by patching every attention module, and to compact an 8k-token cache it takes 12× the time and over 100× the memory that EpiKV needs. The revised methods table marks R-KV's kernel as patched, and its caption states the claim: only our policies run on an unmodified model and stock FlashAttention.
Three more models with nothing tuned, and sampling on three seeds (general response, Section 4). Qwen3-14B is the larger model you ask for. We placed the bands at Llama-8B's relative depth, and the gains carried over.
We hope these clarifications and new measurements address your concerns, and we are happy to discuss further.