acceptodds
Under review as a conference paper at ICLR 2027

ER-KV: Eviction Risk-Aware KV Cache Compression for Efficient Long Reasoning

Abstract

Reasoning language models generate long chain-of-thought sequences, causing KV-cache memory to grow substantially during decoding. KV-cache eviction mitigates this cost, but deciding which tokens to retain requires accounting for both future queries and the attention-output error caused by token removal. Existing methods address either future attention allocation or eviction-induced attention-output error, but do not jointly account for both when making eviction decisions. We propose Eviction Risk-Aware KV Cache Compression (ER-KV), which ranks cached tokens by their expected eviction-induced attention-output error over future queries. ER-KV models the future-query distribution with a Gaussian fitted to observed pre-RoPE query statistics. To estimate eviction risk, which depends nonlinearly on each query, it samples virtual queries to represent this distribution, applies RoPE at multiple future positions, and aggregates the resulting eviction errors. Cholesky factorization reduces sampling overhead while yielding the same Gaussian sampling distribution as eigendecomposition. Under repeated pruning, ER-KV reduces attention-output MSE by 24.4% and output-distribution KL divergence by 48.6% relative to TriAttention. Across AIME24, AIME25, and MATH-500 with three reasoning models, ER-KV matches or outperforms TriAttention in eight of nine model-benchmark settings, improving Qwen3-8B accuracy on AIME24 from 42.1% to 49.2%.

Primary area:
generative models
Keywords:
KV cache eviction, reasoning language models, efficient LLM inference, pre-RoPE query geometry

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pHNeOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ER-KV picks which KV entries to keep based on how much removing each one would change future attention outputs. It fits a Gaussian to past pre-RoPE queries, samples virtual queries from it, and applies them at several future positions. Tokens with a high estimated removal cost are kept. Experiments use three reasoning models on AIME24, AIME25 and MATH-500. The paper also reports attention-output error, output KL divergence, memory and throughput.

Critical strengths

1.The scoring idea fixes a real problem with attention-only scores. A token can get high attention weight, but removing it may barely change the output (e.g. if its value vector is close to the others). ER-KV takes the value vector and the softmax renormalization into account, and averages this cost over plausible future queries. The motivation for combining future-query estimation with an output-error criterion is clear.

2.The paper does not rely only on task accuracy. ER-KV matches or beats TriAttention in 8 of 9 model–task settings, including 42.1% → 49.2% on Qwen3-8B/AIME24 (Tables 1–2). The repeated-pruning study also shows lower attention-output MSE and KL. The mean-only and identity-covariance ablations are helpful, since they show the full query distribution actually matters.

Critical weaknesses

The efficiency advantage over other eviction methods is not shown. Table 3 only compares against Full Attention, at different max batch sizes. That shows ER-KV saves memory, which every eviction method does. It does not tell us how ER-KV compares with TriAttention or Expected Attention at the same batch size and budget. This matters because the scoring is expensive: 512 ms per pruning event by default (Section 4.4). Figure 4 trades off fidelity and throughput, but on only 11 replayed trajectories, and it is not a matched serving comparison. Whether the accuracy gain is worth the extra cost is still open.

Questions for authors
  1. At a fixed batch size and KV budget, how do ER-KV, TriAttention and Expected Attention compare in decode latency and peak memory? Please include the memory for query statistics and temporary scoring buffers.

  2. EVD and Cholesky should give the same Gaussian, but Section 4.4 says EVD is usually better at the same sample count. Does this hold across seeds? If so, is there a difference in regularization or how samples are drawn?

3 comments
AuthorsOpenReview ↗

1. Efficiency

We thank the reviewer for recognizing the motivation behind ER-KV and for providing constructive feedback. Below, we address both questions with additional experiments and clarifications.

[W1, Q1] Decode Latency and Memory Overhead

We compared ER-KV, TriAttention, and Expected Attention using the official TriAttention implementation under matched settings: Qwen3-8B, A100 40GB GPU, batch size 8, K=2,048, 16K generated tokens per request, identical prompts, and pruning every 128 tokens. Decoding latency includes scoring and pruning, while peak GPU memory includes query statistics and temporary scoring buffers.

Matched Decoding Latency and GPU Memory (Qwen3-8B, A100 40GB, Batch 8, K=2,048)

MethodDecode latency (ms/token) ↓Peak GPU (MiB)Query stats (MiB/seq)Scoring buffer (MiB)
TriAttention76.9925,1580*54.3
Expected Attention53.7125,44536.0192.3
ER-KV (Ours)68.6925,45336.8107.5

*TriAttention uses 0.84 MiB of shared calibration statistics instead of per-sequence query statistics. Query statistics and scoring buffers are included in peak GPU memory.

Decode Latency. ER-KV achieves 10.8% lower decoding latency than TriAttention (68.69 vs. 76.99 ms/token), while remaining 27.9% slower than Expected Attention.

Memory Overhead. ER-KV uses 295 MiB (1.2%) more peak GPU memory than TriAttention and only 8 MiB more than Expected Attention. Its query statistics require 36.8 MiB per sequence, while its temporary scoring buffer is smaller than Expected Attention's (107.5 vs. 192.3 MiB).

Accuracy–Efficiency Trade-off. In the separate Qwen3-8B/AIME24 evaluation at K=2,048, ER-KV achieves 49.2% accuracy, compared with 43.3% for TriAttention and 23.75% for Expected Attention. ER-KV thus achieves higher accuracy and lower batch-8 decoding latency than TriAttention, with only a modest increase in peak GPU memory, while trading speed for accuracy relative to Expected Attention.

Our budget sweep as shown in figure 3, further shows that ER-KV matches or outperforms TriAttention across all evaluated KV budgets (512–3K) on DS-Qwen/AIME24 and AIME25. Notably, ER-KV achieves higher accuracy than Expected Attention with half the KV-cache budget.

DS-Qwen Accuracy at Different KV Budgets (Fig. 3 and Table 1)

MethodKV BudgetAIME24 (%)AIME25 (%)
TriAttention1,02431.723.8
Expected Attention2,04835.426.2
ER-KV1,02437.526.7

These results demonstrate that ER-KV achieves a favorable accuracy–efficiency trade-off, improving both reasoning accuracy and decoding latency over TriAttention under matched settings while requiring only a modest increase in peak GPU memory.

Action Plan. We will incorporate these matched latency and memory results and clarify the accuracy–efficiency trade-off in the revised manuscript.

AuthorsOpenReview ↗

2. EVD vs. Cholesky and Summary

[Q2] EVD vs. Cholesky

EVD and Cholesky theoretically sample from the same Gaussian distribution. While EVD achieved lower KL divergence in Figure 4, this advantage does not consistently hold across random seeds. To verify this, we repeated the evaluation on 11 trajectories using 5 different seeds (20260905–20260909). The mean KL divergence was nearly identical for Cholesky (0.05278) and EVD (0.05290), with EVD achieving lower KL in only 25 of 55 trajectory–seed combinations.

The statement in Section 4.4 reflected the observation in Figure 4 and was not intended to imply a general advantage for EVD.

KL divergence across random seeds (lower is better)

SeedCholesky KL ↓EVD KL ↓EVD wins
202609050.05380.05317/11
202609060.05330.05304/11
202609070.05090.05262/11
202609080.05260.05265/11
202609090.05330.05327/11

Both methods use the same initial random samples, without jitter or shrinkage. EVD clips small negative eigenvalues caused by numerical round-off, while Cholesky uses pivoted factorization when the covariance matrix is not positive definite. Despite representing the same Gaussian distribution, different factorizations can produce different Monte Carlo estimates with a finite number of samples.

Action Plan. We will revise Section 4.4 to clarify that the advantage of EVD observed in Figure 4 does not consistently hold across random seeds.

Summary

To summarize, we have addressed both concerns through additional experiments and clarifications:

  • Efficiency (Q1): ER-KV achieves 10.8% lower decoding latency than TriAttention with only 1.2% higher peak GPU memory, while delivering substantially higher accuracy than the faster Expected Attention.
  • EVD vs. Cholesky (Q2): Multi-seed evaluations show no consistent advantage for EVD over Cholesky (25/55 cases).

Beyond the experiments addressing the reviewer's specific concerns, we conducted further evaluations during the rebuttal period. We integrated ER-KV into SGLang and vLLM, achieving 4.9× and 2.3× the end-to-end throughput of Full Attention, respectively, under memory-constrained serving settings. We also extended our AIME25 evaluations to AceReason-Nemotron-14B and GPT-OSS-120B, where ER-KV outperformed TriAttention by 5.8 and 10.4 percentage points, respectively.

Importantly, these findings demonstrate that ER-KV's accuracy improvements do not necessarily come at the cost of higher decoding latency: under the evaluated batch-8 setting, ER-KV achieves both higher accuracy and lower decoding latency than TriAttention, while trading decoding speed for substantially higher accuracy relative to Expected Attention. The multi-seed results also clarify that EVD has no consistent advantage over Cholesky. We believe these additional findings strengthen the empirical case for ER-KV and support a favorable reassessment of the paper. Thank you again for your careful review.

Reviewer_pHNeOpenReview ↗

response to author's rebuttal

Thanks for your detailed rebuttal, most of my concerns are addressed. So, i raise my rating to 3.

Reviewer_vJ9vOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ER-KV, a method for reducing KV-cache memory consumption during long-chain-of-thought reasoning. Unlike existing approaches that primarily estimate token importance from historical or predicted future attention, ER-KV introduces an eviction-risk criterion that measures the expected attention-output distortion caused by removing a cached token under future queries.

Critical strengths

A well-motivated and practically meaningful eviction-risk formulation. The paper introduces a principled criterion that jointly considers future attention patterns and eviction-induced attention-output distortion, addressing limitations of methods that rely primarily on attention scores. By explicitly accounting for attention renormalization and value-vector differences, the approach provides a more faithful estimate of token importance. This is a meaningful methodological contribution to KV-cache compression for long reasoning.

Critical weaknesses

Limited methodological novelty beyond existing approaches. The core method primarily combines established techniques: eviction-induced attention-output error from CAOTE, Gaussian-based future-query modeling from Expected Attention, and position-aware importance estimation from TriAttention. Cholesky factorization is also a standard computational technique. Although their integration is reasonable and effective, the paper provides limited theoretical insights or fundamentally new algorithmic mechanisms beyond this combination, weakening its methodological contribution.

Insufficient experimental rigor and generalizability. Several baseline results are taken from prior work rather than reproduced under a fully controlled experimental setup, making comparison fairness difficult to assess. Moreover, the evaluation is restricted to mathematical reasoning, while the key attention-output fidelity analysis uses only 11 trajectories from a single model. The absence of statistical significance analysis further limits confidence in the reported improvements, particularly on small AIME benchmarks. These limitations weaken the evidence that ER-KV consistently generalizes across tasks and deployment settings.

Questions for authors
  1. Since several baseline results are taken from prior work, can the authors clarify whether the decoding configurations, KV-cache budgets, and evaluation protocols are fully consistent? Can they also provide confidence intervals or statistical significance tests for the improvements on AIME24 and AIME25?
  2. Given that ER-KV combines established components from CAOTE, Expected Attention, and TriAttention, can the authors clarify which specific methodological innovation accounts for the observed improvements? In particular, how much of the gain comes from eviction-risk estimation itself versus improved future-query modeling under otherwise identical conditions?
  3. The method relies on covariance alignment between past and future pre-RoPE queries. How accurately does this Gaussian approximation capture actual future eviction risk, particularly when reasoning trajectories undergo substantial distribution shifts? Can the authors provide quantitative evidence connecting query-distribution alignment to eviction-risk estimation accuracy?
8 comments
AuthorsOpenReview ↗

1. Evaluation Protocol and Statistical Significance

[W2, Q1] Evaluation Protocol and Statistical Significance

We thank the reviewer for raising these concerns. In our study, we directly used TriAttention's official reproduction code and its Hugging Face generate() pipeline. We also verified that our reproduced Full Attention and TriAttention results closely match those reported in the original work. We further conducted additional SGLang evaluations on AIME24/25 to assess statistical significance, along with complementary evaluations on MATH-500 and larger models.

(1) Protocol Consistency and Baseline Reproduction.

We used TriAttention's official Hugging Face generate() pipeline with batch size 1, matching prompts (plain text, no chat template), decoding settings (T=0.6, top-p=0.95, maximum total sequence length of 32,768), KV-cache budgets, seed assignments, and answer extraction procedures, with 8 generations per AIME problem.

To verify baseline comparability, we re-ran Full Attention and TriAttention on Qwen3-8B/AIME24 (K=2,048), obtaining 57.5% and 43.3%, closely matching the reported 57.1% and 42.1%. ER-KV achieved 49.2% under the same settings.

MethodReportedReproduced
Full Attention57.1%57.5%
TriAttention42.1%43.3%
ER-KV—49.2%

These results confirm that our baseline reproductions and ER-KV evaluations used consistent evaluation settings.

(2) Additional AIME Evaluation and Statistical Significance.

Repeating additional AIME baseline evaluations with TriAttention's official Hugging Face pipeline at batch size 1 would require approximately 400 A100 GPU-hours. We therefore conducted additional evaluations using our SGLang implementations developed during the rebuttal period.

Following Qwen3's official thinking-mode evaluation settings, we compared both methods using the same model, KV-cache budget, and evaluation problems.

BenchmarkTriAttentionER-KVImprovementPaired 95% CI
AIME2461.7%67.1%+5.4%p[−1.7, +12.9]
AIME2535.4%48.8%+13.3%p[+5.8, +21.7]

Paired 95% confidence intervals are computed at the problem level.

On AIME25, the paired 95% CI excludes zero, demonstrating a statistically significant improvement over TriAttention. On AIME24, ER-KV also improves accuracy by +5.4 percentage points, although its confidence interval includes zero.

(3) Statistical Evidence on MATH-500.

We extended MATH-500 evaluation from 1 to 8 generations per problem while preserving TriAttention's original Hugging Face evaluation protocol, using matched KV-cache budgets and paired random seeds.

ModelTriAttentionER-KVImprovementPaired 95% CI
Qwen3-8B53.2%55.3%+2.1%p[−0.7, +4.9]
DS-Llama-8B77.5%80.2%+2.7%p[+1.5, +3.9]
DS-Qwen-7B80.2%81.8%+1.6%p[+1.0, +2.2]

ER-KV achieves higher mean accuracy across all three models, with statistically significant improvements on DS-Llama-8B and DS-Qwen-7B, where the paired 95% CIs exclude zero. These results provide additional evidence of ER-KV's accuracy advantage under the original Hugging Face evaluation protocol.

(4) Larger Models — Additional Generalization Evidence.

We further evaluated ER-KV on larger models. On AIME25, we compared ER-KV and TriAttention using the same KV-cache budget (K=2,048) and matched evaluation settings within each model.

ModelTriAttentionER-KVImprovementPaired 95% CI
AceReason-Nemotron-14B50.0%55.8%+5.8%p[−1.27, +12.94]
GPT-OSS-120B67.9%78.3%+10.4%p[+6.1, +14.7]

ER-KV achieves higher accuracy on both larger models. On GPT-OSS-120B, the paired 95% CI excludes zero, confirming a statistically significant improvement over TriAttention. AceReason-Nemotron-14B also shows a +5.8 percentage-point improvement, although its confidence interval includes zero.

Overall, our matched baseline reproductions verify protocol consistency, while the additional paired evaluations provide statistical evidence of ER-KV's accuracy improvements. We observe statistically significant gains on AIME25, two MATH-500 models, and GPT-OSS-120B. These results strengthen the empirical support for ER-KV across evaluation settings and model scales.

AuthorsOpenReview ↗

2. Contributions of Future-Query Modeling and Eviction-Damage Scoring

[W1, Q2] Contributions of Future-Query Modeling and Eviction-Damage Scoring

We thank the reviewer for this question. ER-KV's key contribution is estimating expected future eviction-induced attention-output damage through three complementary components, including the two highlighted by the reviewer and sampling-based risk estimation. We clarify their roles and evaluate their effects through controlled ablations below.

(1) Trajectory-Conditioned Future-Query Modeling. Figure 1 shows strong past–future covariance alignment among pre-RoPE queries from the same reasoning trajectory. This motivates estimating future-query distributions using cumulative trajectory-specific statistics, rather than the recent-window statistics used in Expected Attention.

(2) Future-Aware Eviction-Damage Scoring. Rather than relying on estimated future attention as the eviction criterion, ER-KV scores expected eviction-induced attention-output damage, accounting for value-vector differences and softmax renormalization.

(3) Sampling-Based Risk Estimation. While the Gaussian moment-generating function (MGF) used in Expected Attention provides closed-form expectations of exponentiated attention logits, its formulation does not directly extend to expected eviction damage, which involves softmax normalization and a query-dependent attention output. We therefore sample virtual future queries from the trajectory-conditioned Gaussian and estimate expected eviction damage across future RoPE offsets. We use Cholesky factorization for efficient Gaussian sampling.

Controlled Ablation. To assess the contribution of each component, we evaluate matched variants on Qwen3-8B with K=2,048, measuring output-distribution KL divergence against Full Attention across 11 reasoning trajectories. We additionally report AIME24/AIME25 accuracy using SGLang following our evaluation protocol.

MethodScoringQuery ModelingQuery Stats.KL ↓AIME24 ↑AIME25 ↑Avg. ↑
CAOTEDamage——0.101827.514.621.0
Expected AttentionAttnMGFRecent0.173217.914.616.2
+Cumul query statsAttnMGFCumul.0.092440.027.533.8
+Virtual queryAttnVirtual queryCumul.0.063150.037.944.0
+Damage (Ours)DamageVirtual queryCumul.0.053851.241.246.2

Component-Wise Contributions. We isolate the effect of each component through matched comparisons, changing one component at a time:

  • Damage-based scoring: Replacing attention-based scoring with damage-based scoring reduces KL by 14.7% (0.0631 → 0.0538), with virtual-query sampling and cumulative statistics fixed.
  • Cumulative query statistics: Replacing recent-window statistics with cumulative statistics reduces KL by 46.7% (0.1732 → 0.0924), with attention-based scoring and MGF estimation fixed.
  • Virtual-query sampling: Replacing MGF-based estimation with virtual-query sampling reduces KL by 31.7% (0.0924 → 0.0631), with attention-based scoring and cumulative statistics fixed.

Overall, ER-KV reduces KL divergence by 47.2% and 68.9% relative to CAOTE and Expected Attention, respectively, while achieving the highest AIME24/AIME25 accuracy (51.2%/41.2%).

These results demonstrate the complementary contributions of trajectory-conditioned query modeling, eviction-damage scoring, and sampling-based risk estimation to improving output fidelity.

AuthorsOpenReview ↗

3. Future Eviction-Risk Prediction Accuracy

[W2, Q3 — Part 1] Future Eviction-Risk Prediction Accuracy

To assess the reliability of ER-KV under query-distribution shifts, we conduct two complementary analyses: (1) prediction accuracy against actual future eviction risk and (2) the relationship between past–future covariance alignment and eviction-risk ranking agreement. We present the first analysis below and the second in a separate comment.

Oracle Eviction-Risk Evaluation. We define Oracle Eviction Risk as the average attention-output error caused by removing each KV token over actual future queries, providing a retrospective reference unavailable during inference. For each of the three models, we evaluate all 30 AIME24 problems and measure the Spearman correlation between each method's predicted token-importance ranking and the oracle ranking across decoding stages.

Please refer to the newly added Figure 5. ER-KV achieves the highest Spearman correlation with the oracle eviction-risk rankings at most decoding stages across all three models. Expected Attention relies on recent-query statistics and exhibits a progressive decline in correlation as decoding proceeds, suggesting sensitivity to evolving query distributions. In contrast, ER-KV leverages cumulative, trajectory-specific query statistics and maintains relatively stable correlations throughout generation.

These results suggest that ER-KV's Gaussian-based estimator more accurately predicts actual future eviction-risk rankings while maintaining robust prediction accuracy across decoding stages.

Reviewer_D454OpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes ER-KV, a token eviction method for long reasoning. It ranks cached tokens by the expected attention-output error caused by removing them, where the expectation is taken over a distribution of future queries. Following CAOTE, the error accounts for renormalization of the remaining attention weights. Future queries are modeled as a Gaussian fitted online to pre-RoPE query statistics. M virtual queries are sampled via Cholesky factorization and rotated to several future RoPE offsets, and the resulting errors are averaged. ER-KV matches or exceeds TriAttention in 8 of 9 settings and reduces attention-output MSE and KL relative to TriAttention.

Critical strengths
  • The score is well motivated. It extends an output-error criterion from the current query to a distribution of future queries and positions.
  • The overhead is reported transparently, with a per-stage latency breakdown of each pruning event.
Critical weaknesses
  1. Full Attention baseline.

The Qwen3-8B Full Attention numbers on MATH-500 / AIME24 / AIME25 are 69.6 / 57.1 / 40.8. They are marked "reported from prior work" and are identical to the TriAttention paper (Tables 1 and 2). The Qwen3 report gives 97.4 / 76.0 / 67.3 in thinking mode. Why is the gap so large? Can you re-run Full Attention and TriAttention in the same pipeline?

  1. Significance.

Against TriAttention, ER-KV has 7 wins, 1 tie and 1 loss. MATH-500 uses a single sample at T = 0.6 (p7), and the differences are −1.0 / +0.2. Can these differences be separated from sampling noise? Please report the mean and confidence interval over multiple samples.

  1. Scoring latency.

Pruning runs every 128 tokens, and each event takes 512 ms, of which scoring is 424 ms (p8, Fig. 4c). The throughput in Table 3 is measured at each setting's own maximum batch size, with no accuracy column. Does Table 3 include this cost? At what batch size and cache length was the 512 ms measured? Please report throughput at the same batch size, including scoring.

  1. Inference framework.

The paper does not mention SGLang, vLLM, CUDA graphs, paged KV or prefix caching, and does not state the inference framework. The implementation builds on the TriAttention codebase (p10, Reproducibility Statement). Which framework is used? Is the KV reorganization after pruning compatible with paged KV, prefix caching and CUDA graphs, and how much speedup remains with them?

  1. Related works.

Head-level and query-agnostic methods. ER-KV keeps the same budget B in every KV head (p5), and its related work covers only token-level eviction (p3). It does not discuss head-level methods (DuoAttention, RazorAttention, PruLong, RLKV) or query-agnostic eviction (KVzip, Fast KVzip). How does ER-KV relate to these methods, and would a head-level budget change the results?.

Questions for authors

Please address the weaknesses above.

25 comments
AuthorsOpenReview ↗

1. Full Attention Baseline and Evaluation Protocol

We thank the reviewer for carefully reading our paper and providing constructive feedback. Below, we address your questions with additional experiments and clarifications.

[W1] Full Attention Baseline and Evaluation Protocol

The performance gap between the official Qwen3 report and TriAttention's reported Full Attention results stems from differences in evaluation protocols; we follow TriAttention's (ICML 2026) public reproduction code. We also re-ran Full Attention and major baselines under matched settings, obtaining results consistent with TriAttention's reported values.

Evaluation Protocol. We use the same Hugging Face Transformers generate() pipeline, decoding settings, per-problem seeds, sequence-length limits, and answer extraction and grading procedures as TriAttention. The main differences from the official Qwen3 report are:

  • Prompt: Plain-text math prompts without a chat template, versus a chat template with thinking mode.
  • Length: 32,768 total tokens (including the prompt) for AIME, versus 38,912 maximum output tokens.
  • Sampling: 8 generations per problem (T=0.6, top-p=0.95), versus 64 generations.

Thus, absolute accuracies are not directly comparable across these protocols.

Baseline Reproduction. We re-ran Full Attention and major baselines on Qwen3-8B/AIME24 using the same evaluation pipeline.

Qwen3-8B/AIME24 (K=2,048 for eviction methods; accuracy %).

MethodOur Reproduction (%)Tri Reported (%)
Full Attention57.557.1
SnapKV36.334.6
R-KV24.225.4
Expected Attention23.75—
TriAttention43.342.1
ER-KV (Ours)49.2—

Our reproduced Full Attention and TriAttention results closely match the reported values (within 0.4 and 1.2 percentage points, respectively). ER-KV outperforms the reproduced TriAttention by 5.9 percentage points.

Given the limited computational budget during the rebuttal period, we prioritized baseline reproduction on Qwen3-8B/AIME24. A single AIME evaluation requires approximately 33–40 A100 GPU-hours, making it impractical to reproduce all baselines across every model–benchmark setting. We plan to extend these reproductions to the remaining settings in the camera-ready version.

AuthorsOpenReview ↗

2. Statistical Significance and Large-Model Evaluation

[W2] Statistical Significance

We thank the reviewer for this comment. To assess sampling variability, we conducted an additional 8-draw evaluation on MATH-500. ER-KV achieved higher mean accuracy than TriAttention across all three models, with statistically significant improvements in two of them.

Multi-Draw Evaluation. Using TriAttention's decoding settings, we extended the MATH-500 evaluation from 1 to 8 draws per problem, with paired random seeds for both methods. We report mean accuracy with 95% CIs, and 95% paired CIs for method differences computed from draw-wise accuracy differences on the fixed MATH-500 problem set.

ModelER-KV (95% CI)TriAttention (95% CI)Difference (95% paired CI)
Qwen3-8B55.3 [54.2, 56.5]53.2 [51.0, 55.4]+2.1 [−0.7, +4.9]
DS-Llama-8B80.2 [79.5, 80.9]77.5 [76.6, 78.4]+2.7 [+1.5, +3.9]
DS-Qwen-7B81.8 [80.9, 82.7]80.2 [79.4, 81.0]+1.6 [+1.0, +2.2]

ER-KV outperformed TriAttention in all 8 draws on DS-Llama and DS-Qwen, with paired 95% CIs excluding zero. Qwen3-8B also showed a positive mean improvement of +2.1 percentage points, although its paired CI includes zero.

While additional draws could further refine the confidence intervals, we believe the existing evidence already provides strong support for ER-KV's overall performance advantage. Beyond the multi-draw results, ER-KV matches or outperforms TriAttention in 8 of 9 original model–benchmark settings, with accuracy gains of up to 7.9 percentage points in the KV-budget analysis (Figure 3).

To examine whether this consistent trend extends to larger reasoning models, we additionally evaluated AceReason-Nemotron-14B and GPT-OSS-120B on AIME25 using GH200 GPUs (K=2,048; T=0.6, top-p=0.95; 8 generations per problem). AceReason followed the same evaluation pipeline as Qwen3-8B. GPT-OSS used the MXFP4 checkpoint with the Harmony chat template (medium reasoning), final-channel answer evaluation, and eviction restricted to its 18 full-attention layers using the unmodified TriAttention statistics.

AIME25 accuracy (%).

MethodAceReason-Nemotron-14B (%)GPT-OSS-120B (%)
Full Attention62.178.8
Expected Attention41.771.3
TriAttention50.067.9
ER-KV (Ours)55.878.3

ER-KV outperforms TriAttention by 5.8 and 10.4 percentage points, respectively, reaching within 0.5 percentage points of Full Attention on GPT-OSS-120B.

Taken together, these results provide strong evidence for ER-KV's accuracy advantage over TriAttention, with statistically significant improvements on two MATH-500 models and substantial gains on larger models in AIME25.

AuthorsOpenReview ↗

3. Scoring Overhead and Matched Throughput

[W3] Scoring Overhead and Matched Throughput

Yes. Table 3 includes both scoring and pruning costs. Throughput was measured from end-to-end wall-clock time for generating 16,384 tokens, with pruning every 128 tokens. For Table 3, we measured ER-KV throughput at the maximum supported batch sizes of 46, 72, and 104 for K=2,048, 1,024, and 512, respectively.

Pruning Latency. The 512 ms latency in Fig. 4(c) (which includes 424 ms for scoring) was measured on Qwen3-8B using an A100 40GB GPU at batch size 1, pruning from 2,176 to 2,048 cached tokens (K=2,048). The latency was averaged over pruning events across 11 AIME24 trajectories, excluding initial events. With pruning every 128 tokens, this corresponds to approximately 4.0 ms/token of amortized overhead.

Matched Throughput. We measured aggregate end-to-end decoding throughput (tokens/s) for Qwen3-8B on an A100 40GB GPU at K=2,048 and matched batch sizes of 1, 8, and 16. All results include scoring and pruning costs. At batch size 1, each request generated 16,384 tokens. At batch sizes 8 and 16, each request generated 16K tokens, and results were averaged over two runs.

Aggregate end-to-end throughput (tokens/s) ↑

BatchTriAttentionExpected AttentionER-KV
120.3821.6220.73
8103.91148.94116.46
16142.64241.25163.32

At K=2,048 and batch size 1, none of the three KV-compression methods outperformed Full Attention (23.75 tokens/s). At batch sizes 8 and 16, ER-KV achieved 12.1% and 14.5% higher throughput than the official TriAttention implementation, respectively, while remaining slower than Expected Attention. Thus, the throughput gains in Table 3 primarily reflect larger batch sizes enabled by KV-cache memory savings, rather than faster single-request decoding.

Action Plan. We will update Figure 4 to reflect the matched end-to-end throughput measurements for ER-KV, TriAttention, and Expected Attention.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Summary for Area Chair and Reviewers

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair and all reviewers for their time, careful evaluation, and constructive feedback. Your comments have helped us further clarify our methodological contributions and strengthen the empirical evidence supporting our work.

Key Contribution

Our core contribution is a future-aware KV eviction criterion that estimates expected attention-output damage caused by token removal, rather than relying solely on attention-based importance. By jointly accounting for future-query distributions, value-vector differences, and softmax renormalization, ER-KV provides a principled approach to identifying which cached tokens should be retained during long reasoning.

Additional Evidence and Clarifications

Throughout the rebuttal and discussion period, we have substantially strengthened our paper through extensive additional experiments.

These include:

  • Evaluation Protocol (D454, vJ9v): Independently evaluated Qwen3 under its official thinking-mode settings using SGLang, rather than relying solely on previously reported results.

  • Statistical Validation (D454, vJ9v): Conducted multi-draw evaluations and reported paired 95% confidence intervals across MATH-500, AIME24/25, and larger models.

  • Model Generalization (D454, vJ9v): Extended our evaluation to 14B and 120B reasoning models.

  • Serving Efficiency: Integrated ER-KV into SGLang and vLLM, validated continuous batching.

  • Head-Level Compatibility (D454): Integrated ER-KV with DuoAttention, RLKV, and Ada-KV, demonstrating additional accuracy and efficiency improvements.

  • Same-Run Evaluation (D454): Conducted additional A100 experiments using SGLang continuous batching on actual AIME24 workloads, measuring accuracy and throughput in the same serving runs.

  • Component-wise Ablation (vJ9v): Conducted controlled ablations to isolate and quantify the individual contributions of trajectory-conditioned query modeling, virtual-query sampling, and eviction-damage scoring.

  • Oracle-Based Validation (vJ9v): Added Figure 5 and quantitative Spearman correlation analyses comparing predicted eviction-risk rankings against oracle rankings computed from actual future queries, providing direct empirical evidence of ER-KV's predictive accuracy across decoding stages.

  • EVD vs. Cholesky Clarification (pHNe): Conducted additional multi-seed experiments and clarified that the apparent advantage of EVD in the original figure does not consistently hold across random seeds, with both methods yielding nearly identical average KL divergence.

Taken together, these additional experiments and analyses reinforce our original findings, provide stronger empirical support for our methodological contributions, and address the key concerns raised during the review process.

We sincerely appreciate the reviewers' thoughtful feedback and the opportunity to strengthen our work. We respectfully ask that these substantial additions be taken into consideration in the final assessment of our paper.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.