acceptodds
Under review as a conference paper at ICLR 2027

EpiKV: Epiphany-Aware KV Cache Eviction Without the Attention Matrix

Abstract

Reasoning models can generate chains of thought tens of thousands of tokens long, making the key–value (KV) cache that holds them a major bottleneck for inference throughput. Existing eviction policies for long reasoning traces typically rank cached tokens using attention weights, requiring access to the attention matrix and making them incompatible with fast inference kernels. In this work we study the limits of such policies under tight cache budgets. Surprisingly, we find that under the strongest of them the generations that finish are wrong about as often as without eviction; most of the accuracy loss comes from generations that enter loops and run until the length limit, and retaining more tokens according to a fixed importance score exacerbates this behavior. What stops the looping is keeping the tokens the model’s recent queries point to, and the forward pass the model already runs reveals them without the attention matrix. Motivated by this observation, we introduce epiphany-aware KV cache eviction (EPIKV), which combines hidden-state shifts with the model’s recent query–key relevance to rank cached tokens without materializing the attention matrix. On multiple benchmarks,EPIKV matches or outperforms the strongest attention-based eviction baselines while running directly in vLLM with unmodified attention kernels.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache eviction, efficient LLM inference, reasoning models, long chain-of-thought, hidden states, FlashAttention, vLLM, non-termination

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_G9icOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EpiKV is a KV cache eviction method for reasoning models that uses hidden-state signals and periodically recomputed query–key relevance without requiring full attention matrices or modified attention kernels. Its experiments suggest that much of the accuracy loss under tight cache budgets comes from repetitive, non-terminating generations, which retaining tokens relevant to recent queries helps mitigate. Although the method achieves competitive accuracy and integrates with vLLM, the contribution of its “epiphany” score, the fairness of adapted baselines, and its end-to-end serving benefits require further validation.

Critical strengths
  1. The paper addresses a practically important problem. Controlling KV cache growth during long reasoning traces is relevant to memory-efficient inference, and the method explicitly considers compatibility with existing serving infrastructure.

  2. The failure-mode analysis is informative. Separating errors into length-cap hits and terminated incorrect answers reveals that much of the degradation under tight cache budgets is associated with non-termination, providing a useful diagnostic beyond aggregate accuracy.

  3. The method achieves competitive accuracy at small cache budgets. On MATH-500, EpiKV performs comparably to RaaS on Llama-8B and significantly outperforms it on Qwen-7B at several budgets. Component ablations also demonstrate the benefit of combining recent-query relevance with budget filling.

  4. The implementation demonstrates practical feasibility. EpiKV runs inside vLLM without modifying attention kernels, with accuracy consistent with the evaluation harness on the tested subset. The paper also reports implementation overheads and deployment limitations, making its practical claims easier to assess.

Critical weaknesses
  1. The role of the epiphany score in the full method is unclear. According to §2.3, segment allocation uses key variance, while both within-segment ranking and budget filling use query relevance, leaving no explicit role for the epiphany score in token selection. This disconnect weakens the paper’s central motivation. The authors should clarify where the score affects eviction decisions and establish its contribution through an ablation of the full method.
  2. Baseline modifications and unequal cache occupancy complicate fair comparisons. R-KV and LongFlow are adapted to use a shared keep-mask instead of their original per-head or per-layer selection, potentially weakening these baselines. Meanwhile, EpiKV can retain up to K+127K+127 tokens—nearly 25% above the nominal budget at K=512K=512—whereas some variants leave part of the budget unused. Comparisons should preserve the baselines’ original selection granularity and match actual cache occupancy rather than nominal budgets alone.
  3. Prompt protection is not adequately controlled, and a prompt-preserving random baseline is missing. EpiKV always preserves the full prompt, while several baselines do not receive equivalent protection. The paper lacks controlled comparisons between protecting the prompt and allowing it to compete with generated tokens for retention. Random Attention shows that preserving the prompt while randomly evicting generated tokens can provide a strong baseline. Without matched prompt protection and an independent per-head random eviction baseline, the incremental value of EpiKV’s selection mechanism remains insufficiently established.
  4. The claims about FlashAttention incompatibility are too categorical. Requiring attention scores does not necessarily require eager attention throughout the forward pass: the main computation can use FlashAttention while the scores needed for eviction are recomputed separately for a small subset of queries, as EpiKV itself does. The paper should distinguish limitations of existing implementations from inherent algorithmic incompatibility and assess whether baselines such as R-KV could adopt a similar approach, including its computational cost and accuracy implications.
  5. The model coverage is too limited to establish generality. Experiments cover only DeepSeek-R1-Distill-Llama-8B and DeepSeek-R1-Distill-Qwen-7B, both from the same distillation family, and exclusively use greedy decoding. Evaluation on newer model families, such as Qwen3/Qwen3.5, larger models, and sampling-based decoding would help establish whether the hidden-state signals, non-termination patterns, and eviction benefits generalize beyond the tested setting.

[1] Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning

Questions for authors

Seeing waeknesses

3 comments
AuthorsOpenReview ↗

Reply to Reviewer G9ic

Thank you for the review. Each of your five points asked for an experiment, and we have run all five.

W1. "Clarify where the score affects eviction decisions and establish its contribution through an ablation of the full method."

The score is the ranking in EpiKV^-Q, and it fixes where EpiKV reads. The relation and the ablation are in the general response, Section 3. The score alone already beats LagKV at every budget and the key norm at the tight budgets.

W2. "Comparisons should preserve the baselines' original selection granularity and match actual cache occupancy."

At their original granularity, EpiKV beats LongFlow outright, beats R-KV on Qwen-7B, and equals R-KV on Llama-8B at a twelfth of R-KV's compaction time. We ran the released R-KV unmodified (per head and per layer) and LongFlow on its own kernel. Both rows are in the head-to-head table (general response, Section 1), and the revised Table 2 reports the released R-KV. On occupancy, the released R-KV has the same peak as EpiKV, K+127, since both compact every 128 steps. With a strict trigger that keeps the cache at or below K, EpiKV at K=512 beats RaaS on Qwen3-8B (60.4 against 53.9) and Qwen3-14B (60.2 against 56.0) and is within 1.5 points of it on the two paper models. Strict EpiKV is within paired noise of the shipped EpiKV in every cell (revised appendix, controls table). Occupancy averaged over a generation is in the general response, Section 4.

W3. "A prompt-preserving random baseline is missing."

We ran it, and prompt protection does not explain the margin. A policy that keeps the full prompt and the recent tokens, and evicts the rest at random, scores 12 points below EpiKV at K=512 on both models. It also loops on nearly half the problems. Keeping the tokens that the model's current queries point to prevents those loops. On matched protection, RaaS, LagKV and the key norm keep the full prompt exactly as EpiKV does, so those comparisons are already matched.

W4. "Assess whether baselines such as R-KV could adopt a similar approach, including its computational cost and accuracy implications."

We ran exactly this. On EpiKV's query read, RaaS's rule matches EpiKV at every budget, and R-KV's rule falls well below it at tight budgets (general response, Section 2). RaaS can leave the eager kernel by adopting EpiKV's query read, and no released baseline does. The serving table in that section gives the cost of not adopting it: the released R-KV runs on FlashAttention only by patching every attention module, and to compact an 8k-token cache it takes 12× the time and over 100× the memory that EpiKV needs. The revised methods table marks R-KV's kernel as patched, and its caption states the claim: only our policies run on an unmodified model and stock FlashAttention.

W5. "Evaluation on newer model families ... larger models, and sampling-based decoding."

Three more models with nothing tuned, and sampling on three seeds (general response, Section 4). Qwen3-14B is the larger model you ask for. We placed the bands at Llama-8B's relative depth, and the gains carried over.

We hope these clarifications and new measurements address your concerns, and we are happy to discuss further.

Thank you for your response! I will raise my score.

Thank you for the constructive review and for reconsidering your score. Your suggestions made the paper stronger.

Reviewer_Vjb9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes EpiKV, a training-free KV cache eviction method for long COT reasoning. The method allocates segment-wise retention budgets using key variance and ranks tokens via recomputed window-query attention, avoiding full attention matrix materialization. Results on DeepSeek-R1 models suggest EpiKV reduces length-cap looping failures while remaining competitive with attention-based baselines.

Critical strengths
  1. Diagnostic of reasoning eviction failures: Decomposing eviction errors into length-cap loops versus early wrong terminations is a simple and valuable analysis. Showing that competitive policies like RaaS and EpiKV largely fail due to repetitive looping rather than emitting incorrect final answers provides useful intuition for why eviction degrades reasoning traces.

  2. Practicality of training-free approach: EpiKV achieves good accuracy on MATH-500 without requiring model fine-tuning or full attention weight materialization. In particular, it matches the strongest baselines on Llama-8B and consistently outperforms them across tight budgets on Qwen-7B. Furthermore the authors provide a functional vLLM integration.

Critical weaknesses
  1. Overstated novelty of attention-free scoring and missing key baselines: The paper presents compatibility with FlashAttention (not needing to materialize the attention matrix) as a primary novel contribution. However, avoiding attention materialization by recomputing small query windows or scoring externally is already standard practice in modern eviction methods (e.g., SnapKV [1] via kvpress [2], R-KV [3]). Furthermore, the paper omits relevant learned eviction baselines that also operate without reading attention weights, such as DMS [4] or KVpop [5].

  2. Evaluation protocol and unmatched memory budgets weaken empirical claims: The empirical results rely on greedy decoding with relatively short length caps (8k on MATH-500), under which even the dense attention baseline frequently hits the length cap. Appendix E.2 tests a doubled cap on a subset, but evaluating sampled decoding with longer caps or reporting effective compression ratios would provide a much clearer picture. In addition, to my understanding, the comparison is not strictly memory-matched as EpiKV evicts every τ=128\tau=128 steps and retains all tokens from a new chunk between compactions. Thus its peak cache capacity can be higher than some of the baselines.

Nitpicks:

  • Many ablations in the appendix use different settings than the main text (such as rewound positions), making it difficult to compare numbers directly to Table 2.

[1] https://arxiv.org/abs/2404.14469
[2] https://github.com/NVIDIA/kvpress
[3] https://arxiv.org/abs/2505.24133
[4] https://arxiv.org/abs/2506.05345
[5] https://arxiv.org/abs/2607.05061

Questions for authors
  1. Role of the epiphany score in the final method: I am not sure if I missed this in the paper, but does the final EpiKV policy actually use the epiphany score s(t)s(t)? As I read Section 2.3 and Figure 2, EpiKV tiers segments by key variance and ranks tokens using r(j)r(j). s(t)s(t) appears to be used in the score-based ablations. Could the authors clarify whether and where s(t)s(t) enters the final method? Furthermore, how expensive is the layer-correlation check across new model families, and does the two-band anatomy hold for models not distilled from DeepSeek-R1?
  2. Operational meaning of "still reading" (lines 193–195): What does it mean that the model is "still reading" tokens within a segment? That part was not really understandable for me.
  3. Comparison with DMS and FlashAttention-adapted baselines: Can you provide a comparison against at least DMS [1] at matched compression on Qwen-7B? Additionally, can RaaS [2] and R-KV [3] be evaluated using the same recomputed query window as EpiKV to separate the scoring heuristic from kernel-level implementation choices?
  4. Sampled decoding and harder benchmarks: Do the loop findings and EpiKV's performance advantages hold under standard sampling (e.g., temperature 0.6, top-pp 0.95) with longer length caps? Why are the AIME results buried in the appendix? I think this is an important benchmark here. Could you discuss these results in the main text and also report rollout distributions?
  5. Memory-matched evaluation: Could you report peak and average cache occupancy for all methods, or evaluate EpiKV with a strict budget trigger that eliminates the +127+127 token overshoot at K=512K=512 and K=1024K=1024?

[1] https://arxiv.org/abs/2506.05345
[2] https://arxiv.org/abs/2502.11147
[3] https://arxiv.org/abs/2505.24133

4 comments
AuthorsOpenReview ↗

Reply to Reviewer Vjb9

Thank you for the review.

Q1. "Does the final EpiKV policy actually use the epiphany score? ... how expensive is the layer-correlation check across new model families?"

EpiKV^-Q ranks by the score; EpiKV reads the recent queries at the layer the score identified (general response, Section 3). The layer check cost nothing on other families, because we did not run it: for Qwen3-8B, Qwen3-14B and Nemotron-Nano-8B we placed the two bands at Llama-8B's relative depth, and EpiKV beats RaaS at K=512 on both Qwen3 models.

Q2. "What does it mean that the model is 'still reading' tokens within a segment?"

It means the last 8 queries still place attention on tokens of that segment. The revised Section 2 says so: the second decision is which tokens of a segment the recent queries read.

W1/Q3. "Recomputing small query windows ... is already standard practice (SnapKV via kvpress, R-KV)." "Can RaaS and R-KV be evaluated using the same recomputed query window as EpiKV?"

The methods you cite recompute attention inside every layer. EpiKV shows that one layer is enough, and RaaS on EpiKV's query read equals EpiKV. We ran both experiments (general response, Section 2). RaaS's rule on EpiKV's query read matches EpiKV at every budget, which separates the rule from the kernel, as you asked. R-KV's rule falls 11 to 17 points below at K=512: 90% of its weight is on key redundancy, which one read cannot rescue. kvpress and the released R-KV both lose the head-to-head (Section 1), and R-KV pays 12× EpiKV's compaction time for reading every layer.

Q3. "Can you provide a comparison against at least DMS at matched compression on Qwen-7B?"

No DMS checkpoint exists for Qwen-7B, so we ran the one that does, on its own model. DMS is retrofitted by training, and the single public checkpoint is Qwen3-8B at 8× compression. The comparison, on Qwen3-8B (DMS was retrofitted on mathematical reasoning, OpenR1-Math per its model card; GPQA-Diamond is outside that domain):

DMS (trained)EpiKV (no training)No eviction
MATH-50084.284.2 at K=409685.1
GPQA-Diamond46.053.5 at K=409654.0

Training-free EpiKV reaches DMS on MATH-500, and on GPQA-Diamond the trained policy falls 8 points below its own base model. DMS was trained on math, and its learned policy does not carry over to science questions. It is one operating point per training run, tied to one model and one data distribution. EpiKV needs no training, and its budget is a hard cap set at runtime, where DMS's cache grows with the trace. The revised paper includes DMS as a trained reference and cites KVpop.

W2/Q4. "Do the loop findings and EpiKV's performance advantages hold under standard sampling with longer length caps? Why are the AIME results buried in the appendix?"

Yes, on three seeds and with the cap raised to 16k (general response, Section 4). EpiKV stays above RaaS on Qwen-7B and equals it on Llama-8B. The loop findings hold as well: under sampling on Qwen-7B at K=512, most of the errors that eviction adds are loops (38 of 59 for EpiKV, 40 of 67 for RaaS). The revised main text cites the AIME result in Section 3.1, and the AIME tables are in Appendix D.

On effective compression: the budget only binds on generations longer than K, so we report how often and by how much (MATH-500, EpiKV):

K=512K=1024K=2048K=4096
Qwen-7B: generations longer than K100%93%55%26%
Qwen-7B: mean compression on those7.1×3.4×2.3×1.7×
Llama-8B: generations longer than K98%88%53%31%
Llama-8B: mean compression on those7.0×3.7×2.6×1.7×

The paper's claims sit at K≤1024, where the policy acts on nearly every problem at 3 to 7× compression.

Q5. "Evaluate EpiKV with a strict budget trigger that eliminates the token overshoot."

With a strict trigger the cache never exceeds K, and the result stands. Strict EpiKV beats RaaS on both Qwen3 models and is within 1.5 points of it on the two paper models (K=512):

Qwen3-8BQwen3-14BQwen-7BLlama-8B
EpiKV, strict trigger60.460.263.863.6
RaaS53.956.065.364.8

The overshoot is at most 127 tokens, and EpiKV's accuracy does not come from it. Strict EpiKV is within paired noise of the shipped EpiKV in every cell (revised appendix, controls table). We discuss occupancy over whole generations in the general response, Section 4.

We hope these clarifications address your concerns, and we are happy to discuss further.

Thank you for the detailed response and new experiments, which address Q2, Q5 and partly Q3. However, several main concerns remain:

  1. The way I understand the reply, EpiKV does not actually use the epiphany score s(t) for ranking or eviction, only to select which layer to recompute queries from (which on Qwen3 and Nemotron was set by relative depth without analysis). In the revised version Table 25 was added which applies the RaaS rule to the same recomputed queries. In this setting the RaaS rule matches EpiKV. Therefore the main contribution seems to be the single-layer query read rather than s(t), even though the title, abstract and introduction still frame it around hidden-state shifts.
  2. The DMS comparison in Table 32 does not convince me. Rollouts on MATH-500 and GPQA-D are considerably shorter than on AIME, and on AIME, the only benchmark in this table where long rollouts are common, EpiKV is clearly behind DMS at K=4096 (51.1 vs. 67.8). Also, the DMS numbers in this table could be formatted better as they should be placed in the column corresponding to the 8× compression of the DMS checkpoint (e.g., K=4096 for AIME) instead of being centered under each benchmark. The DMS result on AIME is also omitted from the rebuttal text and I had to search for this table myself.
  3. More generally, the evaluation remains limited in my opinion, as most MATH-500 traces are relatively short. Therefore the budget is only conistently a factor at K≤1024 where accuracy drops substantially on most models. Results on longer-trace benchmarks are either uninformative on the main models (Tables 35–36) or unconvincing (Table 32).

Finally, the revised manuscript does not mark its changes, rewrites substantial parts of the main text, and adds five pages, making it hard to assess during the discussion period. I will therefore keep my score.

AuthorsOpenReview ↗

Response to Reviewer Vjb9's follow-up

Thank you for reading the revision closely and for following up. We are glad Q2, Q5 and part of Q3 are resolved. On the remaining points we respectfully disagree, and answer each below.

"the main contribution seems to be the single-layer query read rather than s(t), even though the title, abstract and introduction still frame it around hidden-state shifts"

The contribution is the whole policy, and the experiment you cite shows it: on our query read, R-KV's rule falls 11 and 17 points below EpiKV at K=512. The paper frames it this way: the title names eviction without the attention matrix, the abstract says EpiKV "ranks cached tokens by hidden-state shifts and recent query–key relevance", and the introduction lists the query read as part of the policy.

The hidden-state shift earns its place before any query is read. At the band of layers our analysis locates, it tracks counterfactual token importance more closely than cumulative attention, the signal H2O ranks by. Ranked by that shift alone, EpiKV^-Q reads no attention at all and is the strongest attention-free policy we tested, with 1.8× LagKV's accuracy on Qwen-7B at K=512.

The read alone does not carry the result. As we have shown, each part of EpiKV costs accuracy when removed:

  • Remove the query read: EpiKV^-Q falls significantly below EpiKV at the tight budgets, from 66.5 to 57.8 on Llama-8B and from 64.2 to 58.8 on Qwen3-8B at K=512, and its lost accuracy is generations that never stop.
  • Remove the segment tiers as well (EpiKV^flat): accuracy drops again, by 7 to 11 points at K=512 and 1024.
  • Swap the rule: R-KV's rule on our read falls behind

That is the paper's line: the hidden-state shift finds what to keep, splitting its errors shows the loss is loops, and the fix reads the queries from the same hidden states at the same Band-A layer. RaaS matching EpiKV on that read means RaaS's own rule no longer needs the attention matrix, which is the claim in our title.

The placement carrying over is a strength: the analysis is done once and transfers. We placed the bands on Qwen3-8B/14B and Nemotron-Nano-8B by relative depth, with no analysis or tuning on those models, as the anatomy predicts: Band A sits in the layers interpretability work ties to factual recall and feature composition (Meng et al., 2022; 2023; Lad et al., 2025). With nothing retuned, EpiKV is the best eviction row at every budget on both Qwen3 models and significantly above every attention-based row at K=512.

"on AIME ... EpiKV is clearly behind DMS at K=4096"

DMS has to be trained for each model and breaks when the data shifts. EpiKV needs no training, and it matches DMS on MATH-500, beats it on GPQA, and is above it on AIME at K=8192. EpiKV ran unchanged on all 5 models; DMS exists for one, a Qwen3-8B ckpt trained on math reasoning. On GPQA that checkpoint falls 8 points below its own base model (46.0 against 54.0), and EpiKV at K=4096 was significantly above it (53.5). On AIME (Qwen3-8B, n=90, cap 32,768):

K=4096K=8192
EpiKV (no training)51.168.9
RaaS51.166.7
DMS (trained, no budget)67.867.8
No eviction65.665.6

We understand why that cell stands out: its the one place DMS is clearly ahead. At that budget EpiKV is level with RaaS, the strongest attention-based policy, and one step up, at K=8192, it is above both DMS and the uncapped model. We are sorry the reply did not point you to it. The revision reports it with the MATH-500 and GPQA results.

"Results on longer-trace benchmarks are either uninformative on the main models (Tables 35–36) or unconvincing (Table 32)"

Respectfully, the same benchmark cannot be uninformative for EpiKV and decisive against it. On the two paper models, AIME and GPQA separate no policy because the uncapped model itself fails to stop on ~62% of AIME and ~47% of GPQA. So we added models that do stop, and on their long traces EpiKV holds:

  • GPQA: the uncapped traces have a median of 7,800 to 9,000 tokens, so K=1024 keeps ~1/8. There, and at all budgets, EpiKV is within paired noise of RaaS on all models.
  • AIME, Qwen3-8B: the median trace is 13,700 tokens. At K=8192 EpiKV is above the uncapped model, above RaaS and above DMS.
  • MATH-500 is not short either on the added models: on Qwen3-8B the median trace is 3,600 tokens and 45% exceed 4,096. At K=512 EpiKV is significantly above every attention-based row on both Qwen3 models.

In the final version we will state this beside the paper models' AIME and GPQA tables, so this ambiguity doesn't occur again.

AuthorsOpenReview ↗

Response to Reviewer Vjb9's follow-up (cont'd)

"the revised manuscript does not mark its changes, rewrites substantial parts of the main text, and adds five pages"

Three of the additions answer your own questions: the query-read rules (W1), DMS (Q3) and the strict budget trigger (Q5). Every other added table answers a question from the reviews or this discussion. They sit mostly in the appendix. The main text uses the extra page usually allowed for the camera-ready, and the PDF as a whole is 32 pages, one fewer than at the initial submission.

Main text:

  • Abstract: the second sentence.
  • Section 1: the order of the contributions.
  • Section 2.1 and Table 1: the definitions of FlashAttention-compatible and attention-based policies; kvpress added.
  • Section 3.1: Table 2 covers 5 models, with R-KV and LongFlow from their released code and a DMS row.
  • Section 3.3: the serving table (requests per GPU).

Appendix:

  • Baselines run from their released code and the query-read rules.
  • The 5-model GPQA-Diamond table.
  • The strict budget trigger.
  • Sampled decoding over three seeds
  • The tables and text these results update, merged or replaced where a new result supersedes an older one (eg., adapted baselines replaced by their released code)

We hope this makes the revision easier to review, and thank you again for the time you have given it.

References

Reviewer_dspkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper studies KV cache eviction for efficient long chain-of-thought reasoning, focusing on the limitations of existing approaches that rely on attention weights and may introduce overhead when used with FlashAttention. The authors propose EpiKV, which avoids constructing attention matrices and estimates token importance from hidden-state changes, retaining selected KV entries under a fixed memory budget. Experiments on mathematical reasoning benchmarks compare EpiKV with prior eviction methods in accuracy, inference efficiency, and memory usage.

Critical strengths
  1. An interesting perspective on token importance through hidden-state dynamics.
    The paper explores the relationship between hidden-state changes and token importance. Using hidden-state changes as an alternative to attention scores may provide complementary evidence about token importance without materializing attention matrices. This is a potentially useful direction for lightweight KV eviction design.
Critical weaknesses
  1. Limited novelty and weak differentiation from prior attention-free KV eviction work.
    Avoiding attention-matrix construction is not a new direction; several KV cache methods already estimate importance without attention. The evaluation does not convincingly show that the Epiphany Score yields consistently better eviction decisions than simpler or previously studied importance signals under comparable KV budgets. A more systematic comparison with closely related attention-free methods is needed to establish the incremental contribution.
  2. Efficiency evidence is incomplete, and runtime overhead is not clearly characterized.
    Although efficiency and FlashAttention compatibility are central motivations, the paper provides limited support for practical performance gains. It does not break down the costs of hidden-state scoring, token selection, and KV cache management. Since reported end-to-end speedups may largely stem from FlashAttention, these added costs could be masked. Throughput and latency across sequence lengths and KV budgets, with explicit overhead accounting, are needed to assess the eviction mechanism itself.
Questions for authors
  1. Please compare EpiKV systematically with prior attention-free KV eviction methods under matched KV budgets and evaluation protocols, and clarify the advantages of EpiKV compared with existing importance signals.
  2. Please report throughput and latency across a wide range of sequence lengths and KV cache budgets, and separate the runtime costs of hidden-state scoring, token selection, and KV cache management, so that the gain from FlashAttention compatibility can be distinguished from that of the proposed eviction mechanism.
5 comments
AuthorsOpenReview ↗

Reply to Reviewer dspk

Thank you for the review.

W1/Q1. "Compare EpiKV systematically with prior attention-free KV eviction methods under matched KV budgets ... and clarify the advantages of EpiKV compared with existing importance signals."

At matched budgets, the epiphany score makes better eviction decisions than every simpler signal. The table in the general response, Section 3, is that comparison, at the same budgets on the same 495 problems. Ranking by the epiphany score alone (EpiKV^-Q) scores 1.8× LagKV and 2.4× the key norm of Devoto et al. on Qwen-7B at K=512.

The simpler signals fail in one way. They are computed from the keys and values once, when a token is cached, so they keep evicting tokens the model later needs, and the generation loops. The key norm loops on 400 of 495 problems. The epiphany score records what a token did to the model's state, and EpiKV's query read adds what the model is asking for now.

Note that kvpress and R-KV avoid the attention matrix by recomputing queries inside every layer. EpiKV wins both head-to-heads while reading just a single layer (general response, Section 1).

W2/Q2. "Separate the runtime costs of hidden-state scoring, token selection, and KV cache management, so that the gain from FlashAttention compatibility can be distinguished from that of the proposed eviction mechanism."

The whole policy costs 1% of a decode step.

Cost per generated token, Llama-8B, batch 1, FlashAttention-2, one A100:

K=512K=4096
Model forward pass33.8 ms33.9 ms
Hidden-state read, every step0.28 ms0.28 ms
Query recompute, token selection and KV gather, once per 128 steps0.06 ms0.04 ms
Whole policy0.34 ms (1.0%)0.32 ms (0.9%)

And its gain is separate from FlashAttention's. We measured each alone, with every configuration at the largest batch that fits on the card:

ConfigurationTokens/sSequences per GPU
FlashAttention, no eviction (8k context / 32k context)169 / 4723 / 5
Eviction without FlashAttention: RaaS, H2O (K=4096)8223
EpiKV on FlashAttention (K=4096 / K=2048)278 / 47845 / 80

Sequences that fit on the card at each budget:

K=512K=1024K=2048K=4096
EpiKV2961288045
kvpress1281066334
RaaS and H2O188944723

FlashAttention alone slows as the trace grows, because the cache grows with it. Eviction alone is slow, because reading attention weights forces the eager kernel. EpiKV gets both gains: its cache is capped, and it never leaves FlashAttention, so its prefill costs exactly what FlashAttention's does. At batch 1 it adds 1 ms to the 33 ms decode step at every budget, where RaaS adds 3 to 7 ms. Energy per generated token is 2 to 4× lower for EpiKV than for R-KV, RaaS and H2O.

Thank you for your response. The additional experiments and results have addressed some of my concerns about efficiency. However, the manuscript overstate the novelty of the attention-score-free approach. Addressing this issue would require substantial revisions to the manuscript. Therefore, I will maintain my original score.

AuthorsOpenReview ↗

Reply to Reviewer dspk (follow-up)

Thank you for the follow-up, and for noting that the added results address concerns about efficiency. On novelty we respectfully disagree. Your follow-up does not name a sentence that overstates or a prior method the comparison leaves out, so we answer against the text of the review and the text of the paper.

"A more systematic comparison with closely related attention-free methods is needed to establish the incremental contribution." (review, W1)

The submitted Table 2 already made that comparison against LagKV, and the revision adds the key norm. Same budget, same 495 problems (MATH-500, Qwen-7B, K=512):

ScoreAccuracy
Key norm (added in the revision)25.9
LagKV (submitted Table 2)34.5
Epiphany score alone, EpiKV^-Q (submitted Table 2)63.0
EpiKV (submitted Table 2)66.1

The two prior attention-free scores are fixed when a token is cached. They evict tokens that the model's current queries still point to, and the generation loops.

Reviewers Vjb9 and G9ic named the methods that avoid forming the attention matrix, and we are grateful for it: SnapKV through kvpress, R-KV, and the trained DMS. We ran each from its released code (general response, Sections 1 and 2; Tables 1 and 5 of the revised PDF). Two results:

  1. EpiKV matches or beats each of them. On Qwen-7B at K=1024 it beats kvpress and the released R-KV (both significantly on paired problems), and on Llama-8B it ties R-KV. Without training, it equals DMS on MATH-500 and is above it on GPQA-Diamond.
  2. Each costs more, and the training-free ones still install attention code in every layer. kvpress hooks every attention module and R-KV replaces the attention forward. EpiKV computes no attention inside the model: it reads one layer's hidden states, which the standard output_hidden_states flag returns. DMS needs a training run per model. To compact an 8k-token cache R-KV takes 792 ms and 9.6 GB against 65 ms and 0.08 GB for EpiKV, and at K=512 one GPU holds 296 requests under EpiKV against 128 under kvpress.

"The manuscript overstates the novelty of the attention-score-free approach." (follow-up)

The originally submitted manuscript states that attention-free scores exist, and compares against one. Three places, all present at submission:

  • Section 4: "Attention-free scores exist: the ℓ2 norm of the cached key, block-level statistics inside vLLM's paged cache, and lag-relative KV statistics."
  • Table 2 has a LagKV row.
  • That paragraph was closed by our claim (which is still true): no prior decode-time policy for reasoning traces scores tokens from the residual stream or recomputes the query from it.

The follow-up does not dispute that sentence or the comparison above.

"Addressing this issue would require substantial revisions to the manuscript." (follow-up)

The revised PDF makes the scope explicit in two sentences and one table. The abstract's second sentence and the definition in Section 2.1 now say that attention-based policies take their weights from the eager kernel or from attention code of their own installed in every layer; Table 1 now also groups the baselines by which of the two they need. No result changes.

The review raised two weaknesses. The follow-up reports the second, efficiency, as addressed in part, and the first asked for the comparison above. If a specific sentence is still at issue, or an attention-free method we have not compared against, naming it would let us address it within the discussion period.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response (to all reviewers)

We thank the reviewers. If there is one thing we want you to take away, it is that EpiKV asks nothing of the serving stack beyond one layer's hidden states and the cached keys. On Qwen-7B it beats all five training-free methods the reviews named (RaaS, H2O, the released R-KV, kvpress, LongFlow) on the same problems. None of them beats EpiKV at any budget from K=512 to K=4096 on Qwen-7B or Llama-8B. It serves 2 to 3.4× the tokens per second of R-KV, RaaS and H2O.

Two terms used below. The query read: every 128 steps, EpiKV recomputes the model's last 8 queries at one layer, from hidden states the forward pass already returns, and keeps the tokens those queries attend to. EpiKV^-Q: the attention-free variant that ranks by the epiphany score alone.

We appreciate that the reviewers agree on what the paper found: Splitting eviction errors into length-cap loops and wrong answers is "a simple and valuable analysis" (Vjb9) and "informative" (G9ic). Reading token importance from hidden-state dynamics is "an interesting perspective" (dspk). The vLLM integration is "functional" (Vjb9) and "demonstrates practical feasibility" (G9ic). The reviews, however, question the evidence around that finding. For the rebuttals, we ran every experiment they ask for. The results are below and in the revised PDF.

1. The baselines, from their authors' code: EpiKV wins every head-to-head

"R-KV and LongFlow are adapted ... potentially weakening these baselines" (G9ic W2). "Missing key baselines ... SnapKV via kvpress, R-KV" (Vjb9 W1). "Compare EpiKV systematically ... under matched KV budgets" (dspk Q1).

We ran each method as its authors released it. R-KV is the official code, unmodified, selecting per head and per layer. kvpress is NVIDIA's library with its decoding press and SnapKV scoring. LongFlow runs on its own kernel. We paired each method with EpiKV on the same 495 MATH-500 problems (Qwen-7B, K=1024).

MethodAccuracyProblems only EpiKV solves : only the method solvesLoops (generations that hit the length cap)EpiKV's lead at K=512 (points)
EpiKV, reference for the three rows below (length cap 8,192)75.275
RaaS69.950 : 24101+0.8
H2O65.367 : 18130+13.6
R-KV (released code)71.932 : 16100+3.5
EpiKV, reference for the two rows below (length cap 16,384)75.867
kvpress (SnapKV, decoding)72.335 : 1870+9.9
LongFlow (released kernel)65.169 : 16129+26.7

Every row is a significant win for EpiKV (exact McNemar test). RaaS and the released R-KV are the two strongest of these methods. On Llama-8B they tie EpiKV on accuracy, and EpiKV serves 3.4× and 2.1× their throughput. The revised Table 2 reports the released R-KV in place of our adaptation. EpiKV is at or above the released R-KV in every cell, and is far cheaper (next section): it reads one layer where R-KV reads all of them.

The baselines lose by looping, as the paper's failure analysis predicts. Once a method evicts a token that the model's current queries still point to, the generation runs on to the length cap (the Loops column). At K=512, of the 138 problems that EpiKV solves and LongFlow does not, LongFlow looped on 117. Random eviction with the prompt protected (the control G9ic (W3) proposed) loops on nearly half of all problems, so prompt protection does not explain EpiKV's margin. The released R-KV, kvpress and LongFlow's own kernel were not in our initial submission, and yet, our loop diagnostic holds for each of them.

2. What each method needs from the serving stack, and what that costs

"The main computation can use FlashAttention while the scores needed for eviction are recomputed separately" (G9ic W4). "Can RaaS and R-KV be evaluated using the same recomputed query window as EpiKV?" (Vjb9 Q3). "Throughput and latency ... with explicit overhead accounting" (dspk W2).

We ran the experiment G9ic and Vjb9 propose. We gave RaaS's rule and R-KV's rule EpiKV's query read and changed nothing else (MATH-500, n=495, length cap 16,384):

Eviction rule, on EpiKV's query readLlama-8B, K=512K=1024Qwen-7B, K=512K=1024
EpiKV70.576.867.775.8
RaaS's rule71.177.066.176.2
R-KV's rule59.672.150.564.8

On EpiKV's query read, RaaS's rule equals EpiKV at every budget on both models. RaaS as published reads the attention weights of every layer at every step, and no released RaaS runs on a query read. One read at one layer therefore supplies everything RaaS takes from those weights. The query read is EpiKV's contribution. R-KV's rule on the same query read falls well below EpiKV, and it loops more (217 loops against 102 on Qwen-7B at K=512).

AuthorsOpenReview ↗

General response (to all reviewers), continued

2, continued. What each method needs from the serving stack

The methods fall into three tiers by what they need from the serving stack, and only the third works out of the box (one A100, Llama-8B, K=4096, each at its largest batch):

TierThe method needsMethodTokens/sSequences per GPUWhat that costs
1The kernel's attention weights, at every stepRaaS, H2O8223Cannot run on FlashAttention; on eager attention, prefill runs out of memory at a 4,096-token prompt
2Its own code inside every layerR-KV (patched attention modules)1336792 ms and +9.6 GB to compact an 8k-token cache; pinned to the transformers versions its patches target
2kvpress (hooks on every attention module)2673410 points below EpiKV at K=512; Hugging Face only, no serving-engine path; decoding press marked experimental
3Only one layer's hidden states and the cached keysEpiKV2784565 ms and +0.08 GB for the same compaction; the policy is 1% of a decode step
A training run per model and per data distributionDMSn/an/aOne public checkpoint, one fixed ratio; 8 points below its own base model on GPQA
No eviction, 8k to 32k context169 to 4723 to 5The cache grows with the trace

Tier 1 needs attention weights, which the fast kernel never computes. Tier 2 avoids this by putting its own code inside the model, so it runs only where that code can be installed. EpiKV is the only method here that changes neither the model nor the kernel. It reads only output_hidden_states and the cached keys, so it runs in vLLM with unmodified attention kernels.

3. Where the epiphany score enters

"Does the final EpiKV policy actually use the epiphany score?" (Vjb9 Q1). "Establish its contribution through an ablation of the full method" (G9ic W1).

Section 2 of the paper finds that the shift a token causes in the hidden state, at a middle band of layers, marks the tokens that later reasoning needs. Ranking by that score alone gives EpiKV^-Q, which reads no attention at all, and is the strongest training-free policy that reads none. A score fixed when a token is cached cannot see which tokens the model's current queries point to, and our failure analysis shows that evicting them starts a loop. EpiKV therefore adds the query read, at the layer the score identified. The last two rows are the ablation G9ic asks for (MATH-500, n=495, the paper's length cap of 8,192):

Llama-8B, K=512K=1024Qwen-7B, K=512K=1024
Key norm53.562.625.936.0
LagKV52.760.434.553.5
EpiKV^-Q: the score alone57.863.263.070.7
EpiKV: the score's layer, plus the query read66.570.366.175.2

On Qwen-7B the score alone scores 1.8× the best prior attention-free policy. The query read adds the rest at the budgets where loops dominate.

4. Beyond two models and greedy decoding

"Evaluation on newer model families ... and sampling-based decoding" (G9ic W5). "Does the two-band anatomy hold for models not distilled from DeepSeek-R1?" (Vjb9 Q1). "Sampled decoding with longer caps" (Vjb9 W2).

We changed nothing for the added models. Qwen3-8B, Qwen3-14B and Nemotron-Nano-8B use Llama-8B's two layer bands at the same relative depth, with no layer check. The Qwen3 models are not distilled from DeepSeek-R1, so the two bands hold beyond that family. On all three, EpiKV at K=4096 is statistically indistinguishable from the full cache, and under sampling it stays above RaaS on Qwen-7B across three seeds and two length caps. Rows are MATH-500 unless they name another benchmark:

SettingEpiKVReference
Qwen3-8B, K=51264.253.9 RaaS
Qwen3-14B, K=51263.856.0 RaaS
Nemotron-Nano-8B, K=51275.474.3 RaaS, 66.9 H2O
Qwen3-8B, K=409684.285.1 with the full cache, 83.0 RaaS
Sampling at temperature 0.6, Qwen-7B, K=102477.273.9 RaaS
AIME 2024-26, Qwen3-8B, K=819268.965.6 with no eviction
GPQA-Diamond, Qwen3-8B, K=409653.546.0 DMS (trained)

Occupancy (G9ic W2, Vjb9 Q5). EpiKV compacts every 128 steps, so its cache grows from K to at most K+127 and is cut back to K, as the released R-KV's does. Averaged over a generation it holds about as many tokens as RaaS, and fewer at the larger budgets:

Tokens in the cache, averaged over a generation (Llama-8B)K=512K=1024K=2048K=4096
EpiKV5048031,1481,457
RaaS4607521,1711,636

We hope these clarifications address all your concerns, and we are happy to discuss further on any of them.

AuthorsOpenReview ↗

Closing note: thanks and plans for the final version

We thank all reviewers for their time and for questions that made the paper stronger. Their requests shaped most of the revision:

  • Baselines run from their authors' code: Tables 2 and 22 (G9ic, Vjb9).
  • Comparisons with attention-free methods: Tables 2 and 23 (dspk).
  • Serving measurements: Table 4 (dspk).
  • Query-read experiments: Table 22 (G9ic, Vjb9).
  • Added model families: Tables 2 and 24 (G9ic, Vjb9).
  • Sampled decoding: Table 25 (G9ic, Vjb9).
  • The trained-policy comparison: DMS rows in Tables 2, 24 and 26 (Vjb9).

After rereading the revision, dspk finds that the hidden-state approach "provides a new perspective on KV cache eviction", which is the core of the paper. The central result held under every test proposed: on MATH-500, no eviction policy scores significantly above EpiKV on any of five models, including the released-code baselines and RaaS's own rule on our query read.

Toward camera-ready. We are completing every baseline on every model and benchmark, so that no table carries an in-progress mark: kvpress, LongFlow and R-KV (released) on every model their code supports, H2O and ThinKV on GPQA-Diamond for the two paper models, and the remaining AIME columns. In the cells already complete, neither kvpress (Llama-8B, Qwen-7B) nor LongFlow (Qwen3-8B, K=512 and 1024) scores significantly above EpiKV, and both fall significantly below it at K=512. Once every cell is in, the five-model GPQA-Diamond table moves into the main text.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.