- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper is addressing the problem of KC cache eviction using a learned scorer. The main algorithmic methodology is using a light per-head scorer to guide eviction decisions. Similar to existing approaches, it keeps three kind of tokens at every step as sink tokens, recent tokens, and top-k scored tokens. The paper proposes a few interesting and elegant algorithmic ideas:
- Using boundary aware supervision signal. The paper directly models the impact of not keeping a token in the cache when exactly it got dropped after it leaves the recent cache boundary.
- Transposed attention idea is a good addition to control computation and improve efficiency.
The evaluation follows Qwen3 family with 4B and 8B models on two tasks of AIME and HMMT.
Critical strengths
-
The two proposed ideas are technically sound and they are meaningful.
-
I strongly appreciate statistical significance test.
-
The results seem quite strong.
Critical weaknesses
-
The most important missing evaluation to me is evaluation of the model after fine-tuning without eviction. You are fine-tuning and modifying the model entirely, is it still a good model on large set of benchmarks? Perhaps, is it a better model? Is the teacher comparison a fine-tuned one or the original model. If you improved the model by injecting a high quality Nemotron-Math v2 and then later lost some performance with eviction, you might still end-up at the level of the original model. Did the model specialize to math and forgot everything else? Overall, a significantly extended analysis on impact of fine-tuning the model is important.
-
Coverage of the literature and empirical study can improve significantly. I at least expect citing KVPolicy (Learning to Evict from Key-Value Cache) from ICML and ideally comparison expanding with comparison to ForesightKV and KVPolicy.
-
An empirical evaluation of the impact of transposed attention can be interesting to include. It is discussed as a novelty but not really experimented.
-
I understand the concurrency of Neural Garbage Collection, but RL methods like KVPolicy is a prior art. The actual eviction is a counter-factual decision. So, intuitively it is a better match for RL like methods. However, KVPop works quite well. I assume this is because the surrogate loss is effective. However, some discussion on why use supervised learning instead of RL and why is it effective would improve the paper.
-
(Minor) The empirical setup is rather unintuitive to me. The model family is very old as multiple new generations of Qwen family exist and it is primarily evaluated on two arbitrarily chosen benchmarks of AIME and HMMT. It does not cover any of the common benchmarks like LongBench or Ruler suite.
Questions for authors
Overall, I am rather positive with the paper assuming some response to my concerns. I at least minimally expect discussion/experiments on the impact of fine-tuning. Also, some discussion about comparison to RL approaches like KVPolicy and NGC, even some experiments but experiment might very well be delayed to camera-ready.
4 comments
Response to Reviewer YqoE (2/2)
Supervised learning versus RL. We agree that eviction is a counterfactual decision. We use supervised learning because a token's future attention mass can be computed exactly and cheaply on the training continuations with the transposed pass. This gives a dense, token-level signal for every keep/drop decision, so no credit has to be assigned across many eviction decisions from a sparse sequence-level reward (NGC [6]), or learned from a separate ranking reward (KVPolicy). The boundary loss keeps supervision close to the counterfactual by directly training the inference decision (i.e. whether an entering token ranks above the top-k cutoff) instead of regressing scores for all tokens. Empirically this proxy is sufficient, as KVpop shows no significant loss relative to its own no-eviction model in all four settings (Table R1). The two approaches are also complementary. Since KVpop's target and loss can be computed on any continuation, they can be applied to the model's own rollouts, for example in on-policy distillation [3], which combines dense token-level supervision with on-policy data. We will add this discussion.
Model family and benchmarks. We use Qwen3-4B-Instruct-2507 and Qwen3-8B because they are full-attention models in which every layer holds a growing KV cache. Newer Qwen releases such as Qwen3-Next and Qwen3.5 place three Gated DeltaNet linear-attention layers before every full-attention layer, so only a quarter of their layers hold a KV cache, which makes them a less clean test bed for KV cache eviction. Qwen3 models are also widely used in recent KV cache compression work [2, 4, 5], and the paper additionally reports SmolLM3 3B. AIME and HMMT are standard benchmarks for reasoning with long rollouts in this line of work [2, 5], and they produce traces of typically 12k to 15k tokens on 8B, where KV cache growth dominates memory and latency.
For long prompts, the paper already reports RULER single-needle retrieval, where KVpop stays close to the teacher at 75% and the errors at 88% come mostly from the needle task with random UUID values. The general benchmarks in Table R2 and the paper's out-of-domain results (Table 3) cover broader capabilities.
Summary. We hope these clarifications and new results resolve your concerns and strengthen your assessment of the paper. If any questions remain or come up during the rebuttal process, we would be happy to engage in further discussion.
References.
[1] Learning to Evict from Key-Value Cache (KV Policy) - https://arxiv.org/abs/2602.10238
[2] ForesightKV - https://arxiv.org/abs/2602.03203
[3] On-Policy Distillation of Language Models - https://arxiv.org/abs/2306.13649
[4] KVzap - https://arxiv.org/abs/2601.07891
[5] Value-Aware Stochastic KV Cache Eviction for Reasoning Models - https://arxiv.org/abs/2606.03928
[6] Neural Garbage Collection - https://arxiv.org/abs/2604.18002
All my concerns are addressed
All my concerns are addressed I am happy to increase my score.
Thank you for your constructive feedback and for raising your score. Your review has made the paper stronger.
Response to Reviewer YqoE (1/2)
Thank you for the thoughtful review and for your positive assessment of the boundary-aware supervision, the transposed-attention target and the statistical testing. We address your points below and will include all new results in the revision. Tables abbreviate Qwen3 4B/8B as 4B/8B, and KVpop refers to the mLSTM variant. For math results we report the average pass@1 numbers over AIME 2024/2025 and HMMT Feb/Nov 2025 (16 rollouts). Significance refers to paired bootstrap tests over the 120 problems.
Fine-tuning without eviction. The teacher in all our tables is the original, unmodified Qwen3 model. We agree that isolating the effect of fine-tuning from eviction and checking for general-capability forgetting are important controls. To isolate the fine-tuning effect, we evaluated every KVpop checkpoint with eviction disabled, i.e. with the same weights and a cache budget larger than any sequence (Table R1). Fine-tuning does not inflate the results. Without eviction, the fine-tuned models stay within 1.6 points of the teacher, with no significant differences. Comparing the last two columns of Table R1 then measures the effect of eviction alone. KVpop matches the no-eviction baseline within 0.3 points in three of the four settings, and differs by a non-significant 1.2 points on 4B at 88%.
Table R1. Math accuracy of the teacher, KVpop without eviction and KVpop.
To test for specialisation and forgetting, we evaluated general benchmarks (Table R2). All differences are at most 2 points, so the math-focused retrofitting does not cause noticeable forgetting. The out-of-domain results in Table 3 of the paper point the same way as KVpop stays within 3 points of the teacher on GPQA-Diamond and LiveCodeBench although it is trained only on math.
Table R2. General benchmarks (MMLU 5-shot, GSM8K 5-shot CoT, IFEval prompt-level strict).
KVPolicy and ForesightKV. Thank you for the pointer. We will cite KVPolicy [1] and discuss it in Related Work. KVPolicy trains lightweight per-head RL agents on pre-computed generation traces to rank cached tokens from their keys and values, with a reward over all cache budgets, and leaves the LLM unchanged. KVpop instead supervises the keep/drop decision at the deployed budget directly with future attention mass and fine-tunes the model together with the scorer.
We reimplemented ForesightKV [2] (supervised stage) and trained it under cache budgets and training conditions matched to KVpop. Table R3 shows the results. On 8B, KVpop is significantly better at both ratios and maintains teacher-level accuracy, while ForesightKV drops 7.5 points below it at 88%. On 4B, both methods perform similarly at 75% and ForesightKV is significantly ahead at 88%. We will add ForesightKV as a baseline to Table 1.
Table R3. ForesightKV and KVpop under matched training budget.
Empirical evaluation of transposed attention. Standard fused attention kernels only expose per-query statistics, not the per-key future attention mass our target requires. We benchmarked the transposed pass per layer on Qwen3 4B (one H100) against computing the target explicitly in blocks of 1,024 queries, as materialising the full attention matrix runs out of memory at 16k and above (Table R4). Both agree within 2e-6 (maximum absolute difference), and at the 16k training length the transposed pass is about 30x faster and uses about 24x less memory, which makes computing the exact target practical during training.
Table R4. Time and extra memory per layer for computing the target.
(Continued in the next comment - 1/2)