- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The submisson looks at the error that comes from evicting entries out of the KV cache. It chooses which entries to keep by sampling with replacement, and then reweights each kept entry in later attention using it's sampling probability and how many times it was drawn. Evicted entries are thrown away for good, and their combined contribution is estimated from the entries that remain.
I agree with the starting point. Dropping tokens affects both the weighted sum over values and the softmax normalizer, and simply keeping the tokens that look important leaves both of those shifts unaccounted for. The proposal is to keep correction weights around after eviction, and the paper checks whether doing so actually helps.
I find Fig 2 to be the most informative. It puts corrected sampling, uncorrected sampling and deterministic selection side by side on the same underlying scores, with expected compression matched seperately per attention head, so the correction is tested directly. The authors also show that under very aggressive compression there are too few samples and variance gets high. The setup is fairly narrow, it only measures attention-output error on the first generated token for Llama3.2-3B, but it does back up the proposed mechanism.
One concern is with how novelty is framed. MagicPIG (ICLR 25) already uses self-normalized importance sampling for attention. This paper evicts permanently, which is a difference and the submission's contribution which I find interesting. But it also means that head-to-head timing comparisons need to be done carefully.
Downstream, the evaluation uses Llama3.2-3B and Qwen3-4B, with 130 RULER examples plus HotpotQA, QASPER and TriviaQA from LongBench restricted to prompts under 3,000 tokens. The method is competitive and comes out ahead in aggregate rankings. That is a good sign, but with this scope it can't show that the method holds up on long contexts or on bigger models.
I'm also not sure how much of the task gain is due to the correction itself. In Fig 3 expected compression is matched over the whole cache, yet sampling shifts how much budget each attention head ends up with, so part of the gain may come from that reallocation.
I lean towards accept. The paper is relevant and the contribution is meaningful but I dont think the evidence so far shows robustness in general. The authors should state that scope more plainly and discuss or show the effect of the downstream results and how the are related to correction and allocation.
Additional related work:
MagicPIG, ICLR 2025 ManifoldKV, ICML 2026
Critical strengths
See above
Critical weaknesses
See above
Questions for authors
See above
We thank Reviewer 8PrH for the useful feedback and suggestions. Please see the revised PDF for the plots and tables referred in the rebuttal below.
We remain available to answer any further questions.