- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ER-KV picks which KV entries to keep based on how much removing each one would change future attention outputs. It fits a Gaussian to past pre-RoPE queries, samples virtual queries from it, and applies them at several future positions. Tokens with a high estimated removal cost are kept. Experiments use three reasoning models on AIME24, AIME25 and MATH-500. The paper also reports attention-output error, output KL divergence, memory and throughput.
Critical strengths
1.The scoring idea fixes a real problem with attention-only scores. A token can get high attention weight, but removing it may barely change the output (e.g. if its value vector is close to the others). ER-KV takes the value vector and the softmax renormalization into account, and averages this cost over plausible future queries. The motivation for combining future-query estimation with an output-error criterion is clear.
2.The paper does not rely only on task accuracy. ER-KV matches or beats TriAttention in 8 of 9 model–task settings, including 42.1% → 49.2% on Qwen3-8B/AIME24 (Tables 1–2). The repeated-pruning study also shows lower attention-output MSE and KL. The mean-only and identity-covariance ablations are helpful, since they show the full query distribution actually matters.
Critical weaknesses
The efficiency advantage over other eviction methods is not shown. Table 3 only compares against Full Attention, at different max batch sizes. That shows ER-KV saves memory, which every eviction method does. It does not tell us how ER-KV compares with TriAttention or Expected Attention at the same batch size and budget. This matters because the scoring is expensive: 512 ms per pruning event by default (Section 4.4). Figure 4 trades off fidelity and throughput, but on only 11 replayed trajectories, and it is not a matched serving comparison. Whether the accuracy gain is worth the extra cost is still open.
Questions for authors
-
At a fixed batch size and KV budget, how do ER-KV, TriAttention and Expected Attention compare in decode latency and peak memory? Please include the memory for query statistics and temporary scoring buffers.
-
EVD and Cholesky should give the same Gaussian, but Section 4.4 says EVD is usually better at the same sample count. Does this hold across seeds? If so, is there a difference in regularization or how samples are drawn?
3 comments
2. EVD vs. Cholesky and Summary
[Q2] EVD vs. Cholesky
EVD and Cholesky theoretically sample from the same Gaussian distribution. While EVD achieved lower KL divergence in Figure 4, this advantage does not consistently hold across random seeds. To verify this, we repeated the evaluation on 11 trajectories using 5 different seeds (20260905–20260909). The mean KL divergence was nearly identical for Cholesky (0.05278) and EVD (0.05290), with EVD achieving lower KL in only 25 of 55 trajectory–seed combinations.
The statement in Section 4.4 reflected the observation in Figure 4 and was not intended to imply a general advantage for EVD.
KL divergence across random seeds (lower is better)
| Seed | Cholesky KL ↓ | EVD KL ↓ | EVD wins |
|---|---|---|---|
| 20260905 | 0.0538 | 0.0531 | 7/11 |
| 20260906 | 0.0533 | 0.0530 | 4/11 |
| 20260907 | 0.0509 | 0.0526 | 2/11 |
| 20260908 | 0.0526 | 0.0526 | 5/11 |
| 20260909 | 0.0533 | 0.0532 | 7/11 |
Both methods use the same initial random samples, without jitter or shrinkage. EVD clips small negative eigenvalues caused by numerical round-off, while Cholesky uses pivoted factorization when the covariance matrix is not positive definite. Despite representing the same Gaussian distribution, different factorizations can produce different Monte Carlo estimates with a finite number of samples.
Action Plan. We will revise Section 4.4 to clarify that the advantage of EVD observed in Figure 4 does not consistently hold across random seeds.
Summary
To summarize, we have addressed both concerns through additional experiments and clarifications:
- Efficiency (Q1): ER-KV achieves 10.8% lower decoding latency than TriAttention with only 1.2% higher peak GPU memory, while delivering substantially higher accuracy than the faster Expected Attention.
- EVD vs. Cholesky (Q2): Multi-seed evaluations show no consistent advantage for EVD over Cholesky (25/55 cases).
Beyond the experiments addressing the reviewer's specific concerns, we conducted further evaluations during the rebuttal period. We integrated ER-KV into SGLang and vLLM, achieving 4.9× and 2.3× the end-to-end throughput of Full Attention, respectively, under memory-constrained serving settings. We also extended our AIME25 evaluations to AceReason-Nemotron-14B and GPT-OSS-120B, where ER-KV outperformed TriAttention by 5.8 and 10.4 percentage points, respectively.
Importantly, these findings demonstrate that ER-KV's accuracy improvements do not necessarily come at the cost of higher decoding latency: under the evaluated batch-8 setting, ER-KV achieves both higher accuracy and lower decoding latency than TriAttention, while trading decoding speed for substantially higher accuracy relative to Expected Attention. The multi-seed results also clarify that EVD has no consistent advantage over Cholesky. We believe these additional findings strengthen the empirical case for ER-KV and support a favorable reassessment of the paper. Thank you again for your careful review.
response to author's rebuttal
Thanks for your detailed rebuttal, most of my concerns are addressed. So, i raise my rating to 3.
1. Efficiency
We thank the reviewer for recognizing the motivation behind ER-KV and for providing constructive feedback. Below, we address both questions with additional experiments and clarifications.
[W1, Q1] Decode Latency and Memory Overhead
We compared ER-KV, TriAttention, and Expected Attention using the official TriAttention implementation under matched settings: Qwen3-8B, A100 40GB GPU, batch size 8, K=2,048, 16K generated tokens per request, identical prompts, and pruning every 128 tokens. Decoding latency includes scoring and pruning, while peak GPU memory includes query statistics and temporary scoring buffers.
Matched Decoding Latency and GPU Memory (Qwen3-8B, A100 40GB, Batch 8, K=2,048)
*TriAttention uses 0.84 MiB of shared calibration statistics instead of per-sequence query statistics. Query statistics and scoring buffers are included in peak GPU memory.
Decode Latency. ER-KV achieves 10.8% lower decoding latency than TriAttention (68.69 vs. 76.99 ms/token), while remaining 27.9% slower than Expected Attention.
Memory Overhead. ER-KV uses 295 MiB (1.2%) more peak GPU memory than TriAttention and only 8 MiB more than Expected Attention. Its query statistics require 36.8 MiB per sequence, while its temporary scoring buffer is smaller than Expected Attention's (107.5 vs. 192.3 MiB).
Accuracy–Efficiency Trade-off. In the separate Qwen3-8B/AIME24 evaluation at K=2,048, ER-KV achieves 49.2% accuracy, compared with 43.3% for TriAttention and 23.75% for Expected Attention. ER-KV thus achieves higher accuracy and lower batch-8 decoding latency than TriAttention, with only a modest increase in peak GPU memory, while trading speed for accuracy relative to Expected Attention.
Our budget sweep as shown in figure 3, further shows that ER-KV matches or outperforms TriAttention across all evaluated KV budgets (512–3K) on DS-Qwen/AIME24 and AIME25. Notably, ER-KV achieves higher accuracy than Expected Attention with half the KV-cache budget.
DS-Qwen Accuracy at Different KV Budgets (Fig. 3 and Table 1)
These results demonstrate that ER-KV achieves a favorable accuracy–efficiency trade-off, improving both reasoning accuracy and decoding latency over TriAttention under matched settings while requiring only a modest increase in peak GPU memory.
Action Plan. We will incorporate these matched latency and memory results and clarify the accuracy–efficiency trade-off in the revised manuscript.