acceptodds
Under review as a conference paper at ICLR 2027

KVpop - Key-Value Cache Compression with Predictive Online Pruning

24 views

Abstract

Key-value (KV) cache growth is a major bottleneck in autoregressive decoding, as memory and bandwidth scale linearly with context length. Existing KV eviction methods often rely on static heuristics or proxy scores, which poorly track future token utility and cause brittle eviction as relevance shifts. To address this, we introduce KVpop, which learns a fixed-budget KV eviction policy by directly supervising the keep-or-drop decision. We evaluate both stateless and delayed memory-based scorers, showing that boundary supervision succeeds with lightweight MLPs while recurrent scoring leverages near-future context before eviction. On AIME and HMMT mathematical reasoning, KVpop retains 98% of full-attention performance on Qwen3-4B at 75% KV cache compression and 97% at 88% compression, consistently outperforming established eviction baselines. Qwen3-8B shows even stronger results, reaching near-full teacher performance. These results show that supervising eviction with future-attention signals cuts memory costs while maintaining quality.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV Cache Compression, Efficient Inference, Long Context Modeling, Model Efficiency

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_YqoEOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper is addressing the problem of KC cache eviction using a learned scorer. The main algorithmic methodology is using a light per-head scorer to guide eviction decisions. Similar to existing approaches, it keeps three kind of tokens at every step as sink tokens, recent tokens, and top-k scored tokens. The paper proposes a few interesting and elegant algorithmic ideas:

  • Using boundary aware supervision signal. The paper directly models the impact of not keeping a token in the cache when exactly it got dropped after it leaves the recent cache boundary.
  • Transposed attention idea is a good addition to control computation and improve efficiency.

The evaluation follows Qwen3 family with 4B and 8B models on two tasks of AIME and HMMT.

Critical strengths
  • The two proposed ideas are technically sound and they are meaningful.

  • I strongly appreciate statistical significance test.

  • The results seem quite strong.

Critical weaknesses
  • The most important missing evaluation to me is evaluation of the model after fine-tuning without eviction. You are fine-tuning and modifying the model entirely, is it still a good model on large set of benchmarks? Perhaps, is it a better model? Is the teacher comparison a fine-tuned one or the original model. If you improved the model by injecting a high quality Nemotron-Math v2 and then later lost some performance with eviction, you might still end-up at the level of the original model. Did the model specialize to math and forgot everything else? Overall, a significantly extended analysis on impact of fine-tuning the model is important.

  • Coverage of the literature and empirical study can improve significantly. I at least expect citing KVPolicy (Learning to Evict from Key-Value Cache) from ICML and ideally comparison expanding with comparison to ForesightKV and KVPolicy.

  • An empirical evaluation of the impact of transposed attention can be interesting to include. It is discussed as a novelty but not really experimented.

  • I understand the concurrency of Neural Garbage Collection, but RL methods like KVPolicy is a prior art. The actual eviction is a counter-factual decision. So, intuitively it is a better match for RL like methods. However, KVPop works quite well. I assume this is because the surrogate loss is effective. However, some discussion on why use supervised learning instead of RL and why is it effective would improve the paper.

  • (Minor) The empirical setup is rather unintuitive to me. The model family is very old as multiple new generations of Qwen family exist and it is primarily evaluated on two arbitrarily chosen benchmarks of AIME and HMMT. It does not cover any of the common benchmarks like LongBench or Ruler suite.

Questions for authors

Overall, I am rather positive with the paper assuming some response to my concerns. I at least minimally expect discussion/experiments on the impact of fine-tuning. Also, some discussion about comparison to RL approaches like KVPolicy and NGC, even some experiments but experiment might very well be delayed to camera-ready.

4 comments
AuthorsOpenReview ↗

Response to Reviewer YqoE (1/2)

Thank you for the thoughtful review and for your positive assessment of the boundary-aware supervision, the transposed-attention target and the statistical testing. We address your points below and will include all new results in the revision. Tables abbreviate Qwen3 4B/8B as 4B/8B, and KVpop refers to the mLSTM variant. For math results we report the average pass@1 numbers over AIME 2024/2025 and HMMT Feb/Nov 2025 (16 rollouts). Significance refers to paired bootstrap tests over the 120 problems.

Fine-tuning without eviction. The teacher in all our tables is the original, unmodified Qwen3 model. We agree that isolating the effect of fine-tuning from eviction and checking for general-capability forgetting are important controls. To isolate the fine-tuning effect, we evaluated every KVpop checkpoint with eviction disabled, i.e. with the same weights and a cache budget larger than any sequence (Table R1). Fine-tuning does not inflate the results. Without eviction, the fine-tuned models stay within 1.6 points of the teacher, with no significant differences. Comparing the last two columns of Table R1 then measures the effect of eviction alone. KVpop matches the no-eviction baseline within 0.3 points in three of the four settings, and differs by a non-significant 1.2 points on 4B at 88%.

Table R1. Math accuracy of the teacher, KVpop without eviction and KVpop.

SettingTeacherKVpop, no evictionKVpop
4B, 75%0.4510.4480.451
4B, 88%0.4510.4370.425
8B, 75%0.4290.4450.445
8B, 88%0.4290.4330.435

To test for specialisation and forgetting, we evaluated general benchmarks (Table R2). All differences are at most 2 points, so the math-focused retrofitting does not cause noticeable forgetting. The out-of-domain results in Table 3 of the paper point the same way as KVpop stays within 3 points of the teacher on GPQA-Diamond and LiveCodeBench although it is trained only on math.

Table R2. General benchmarks (MMLU 5-shot, GSM8K 5-shot CoT, IFEval prompt-level strict).

ModelMMLUGSM8KIFEval
4B Teacher0.7220.9350.830
4B KVpop 75%0.7210.9310.823
4B KVpop 88%0.7210.9330.810
8B Teacher0.7190.9260.832
8B KVpop 75%0.7150.9230.830
8B KVpop 88%0.7030.9170.832

KVPolicy and ForesightKV. Thank you for the pointer. We will cite KVPolicy [1] and discuss it in Related Work. KVPolicy trains lightweight per-head RL agents on pre-computed generation traces to rank cached tokens from their keys and values, with a reward over all cache budgets, and leaves the LLM unchanged. KVpop instead supervises the keep/drop decision at the deployed budget directly with future attention mass and fine-tunes the model together with the scorer.

We reimplemented ForesightKV [2] (supervised stage) and trained it under cache budgets and training conditions matched to KVpop. Table R3 shows the results. On 8B, KVpop is significantly better at both ratios and maintains teacher-level accuracy, while ForesightKV drops 7.5 points below it at 88%. On 4B, both methods perform similarly at 75% and ForesightKV is significantly ahead at 88%. We will add ForesightKV as a baseline to Table 1.

Table R3. ForesightKV and KVpop under matched training budget.

Method8B, 75%8B, 88%4B, 75%4B, 88%
Teacher0.4290.4290.4510.451
ForesightKV0.4190.3540.4630.476
KVpop0.4450.4350.4510.425

Empirical evaluation of transposed attention. Standard fused attention kernels only expose per-query statistics, not the per-key future attention mass our target requires. We benchmarked the transposed pass per layer on Qwen3 4B (one H100) against computing the target explicitly in blocks of 1,024 queries, as materialising the full attention matrix runs out of memory at 16k and above (Table R4). Both agree within 2e-6 (maximum absolute difference), and at the 16k training length the transposed pass is about 30x faster and uses about 24x less memory, which makes computing the exact target practical during training.

Table R4. Time and extra memory per layer for computing the target.

ContextTransposed attention (ours)Explicit, chunked
8k2.3 ms / 132 MiB58 ms / 3.2 GiB
16k7.8 ms / 264 MiB232 ms / 6.4 GiB
32k30 ms / 527 MiB929 ms / 12.5 GiB

(Continued in the next comment - 1/2)

AuthorsOpenReview ↗

Response to Reviewer YqoE (2/2)

Supervised learning versus RL. We agree that eviction is a counterfactual decision. We use supervised learning because a token's future attention mass can be computed exactly and cheaply on the training continuations with the transposed pass. This gives a dense, token-level signal for every keep/drop decision, so no credit has to be assigned across many eviction decisions from a sparse sequence-level reward (NGC [6]), or learned from a separate ranking reward (KVPolicy). The boundary loss keeps supervision close to the counterfactual by directly training the inference decision (i.e. whether an entering token ranks above the top-k cutoff) instead of regressing scores for all tokens. Empirically this proxy is sufficient, as KVpop shows no significant loss relative to its own no-eviction model in all four settings (Table R1). The two approaches are also complementary. Since KVpop's target and loss can be computed on any continuation, they can be applied to the model's own rollouts, for example in on-policy distillation [3], which combines dense token-level supervision with on-policy data. We will add this discussion.

Model family and benchmarks. We use Qwen3-4B-Instruct-2507 and Qwen3-8B because they are full-attention models in which every layer holds a growing KV cache. Newer Qwen releases such as Qwen3-Next and Qwen3.5 place three Gated DeltaNet linear-attention layers before every full-attention layer, so only a quarter of their layers hold a KV cache, which makes them a less clean test bed for KV cache eviction. Qwen3 models are also widely used in recent KV cache compression work [2, 4, 5], and the paper additionally reports SmolLM3 3B. AIME and HMMT are standard benchmarks for reasoning with long rollouts in this line of work [2, 5], and they produce traces of typically 12k to 15k tokens on 8B, where KV cache growth dominates memory and latency.

For long prompts, the paper already reports RULER single-needle retrieval, where KVpop stays close to the teacher at 75% and the errors at 88% come mostly from the needle task with random UUID values. The general benchmarks in Table R2 and the paper's out-of-domain results (Table 3) cover broader capabilities.

Summary. We hope these clarifications and new results resolve your concerns and strengthen your assessment of the paper. If any questions remain or come up during the rebuttal process, we would be happy to engage in further discussion.

References.
[1] Learning to Evict from Key-Value Cache (KV Policy) - https://arxiv.org/abs/2602.10238
[2] ForesightKV - https://arxiv.org/abs/2602.03203
[3] On-Policy Distillation of Language Models - https://arxiv.org/abs/2306.13649
[4] KVzap - https://arxiv.org/abs/2601.07891
[5] Value-Aware Stochastic KV Cache Eviction for Reasoning Models - https://arxiv.org/abs/2606.03928
[6] Neural Garbage Collection - https://arxiv.org/abs/2604.18002

Reviewer_YqoEOpenReview ↗

All my concerns are addressed

All my concerns are addressed I am happy to increase my score.

Thank you for your constructive feedback and for raising your score. Your review has made the paper stronger.

Reviewer_SU7tOpenReview ↗
Rating
4(+2)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes KVPOP, a learned policy for keeping the KV cache within a fixed cache capacity during generation. The idea is to train a lightweight scorer to predict which tokens will matter later, using future attention as supervision. Once a token leaves a protected recent window, its score determines whether it enters the long-range cache. The authors also explore a recurrent scorer that waits before making this decision, allowing it to use later context. On the tested Qwen models, the method preserves nearly all of the dense model’s math accuracy under substantial cache compression.

Critical strengths

I paper tackles a concrete practical problem and makes memory usage predictable through a fixed per-head budget. The numerical results at high compression are particularly encouraging, with clear improvements over DMS. The lightweight scorer already performs well, which makes the approach appealing even without the recurrent component. The efficient target computation and the inclusion of a trained StreamingLLM baseline also strengthen the work.

Critical weaknesses

My main concern is that the paper does not compare against ForesightKV (cited in this work), despite its close overlap in future-attention supervision and pairwise ranking. This makes it difficult to judge how much the proposed objective adds over existing approaches. The boundary loss also compares candidates against the oracle’s cutoff rather than the student’s actual weakest entry, so getting the supervised comparison right does not necessarily produce the right eviction decision.

I am also not fully convinced by the case for recurrent delayed scoring: its downstream advantage over the MLP is not statistically significant. The broader long-context claims feel stronger than the evidence supports, since the efficiency experiments increase the cache budget with sequence length and do not establish accuracy at 128k. Retrieval is another clear limitation, with accuracy dropping substantially in the hardest reported setting. Overall, I see a useful practical method with promising results, but the paper needs stronger comparisons and ablations to establish which ingredients deserve the credit.

Questions for authors
  • Given the overlap with ForesightKV in future-attention supervision and pairwise ranking, could the authors provide a direct comparison under matched cache budgets and training conditions?

  • Since the recurrent scorer does not significantly outperform the MLP on downstream tasks, what evidence supports the benefit of recurrent memory and delayed scoring? Could an ablation separate these two components?

5 comments
AuthorsOpenReview ↗

Response to Reviewer SU7t (1/2)

Thank you for the constructive review and for noting the predictable memory, the high-compression gains over DMS and the efficient target computation. We address your points below and will include all new results in the revision. Tables abbreviate Qwen3 4B/8B as 4B/8B, and KVpop refers to the mLSTM variant. For math results we report the average pass@1 numbers over AIME 2024/2025 and HMMT Feb/Nov 2025 (16 rollouts). Significance refers to paired bootstrap tests over the 120 problems.

Q1 / Comparison with ForesightKV. We reimplemented ForesightKV (supervised stage) and trained it under cache budgets and training conditions matched to KVpop. Table R1 shows the results. On 8B, KVpop is significantly better at both ratios and maintains teacher-level accuracy, while ForesightKV drops 7.5 points below it at 88%. On 4B, both methods perform similarly at 75% and ForesightKV is significantly ahead at 88%.

Both methods use future attention and pairwise ranking. ForesightKV ranks cached entries from key, value and attention features against a schedule derived from block-pooled future attention, and prunes the cache in chunks. KVpop instead computes the exact per-token future attention mass with the transposed attention pass, without materialising attention maps, and supervises exactly the per-token admission decision at the top-k boundary of the deployed budget. We will add ForesightKV as a baseline to Table 1 and sharpen its positioning in Related Work.

Table R1. ForesightKV and KVpop under matched training budget.

Method8B, 75%8B, 88%4B, 75%4B, 88%
Teacher0.4290.4290.4510.451
ForesightKV0.4190.3540.4630.476
KVpop0.4450.4350.4510.425

Q2 / Recurrent memory versus delayed scoring. As requested, we separate recurrence from delay using an mLSTM that scores tokens immediately on arrival and report downstream math results (Table R2). Recurrence and delay add +0.9 and +0.6 points respectively (+1.5 together). We agree with the reviewer that gains are modest and not statistically significant and will temper the claim. The delayed mLSTM is an optional instantiation, and the main contribution, the future-attention target with the boundary-aware objective, is scorer-agnostic. We still see value in the stateful variant because it shows that a recurrent, delayed scorer can be trained in parallel and deployed with a compact constant state in the same framework, opening the door to richer stateful scoring policies.

Table R2. Separating recurrence from delay (4B, 75%).

ScorerRecurrentDelayedMath
MLPnono0.436
mLSTM, immediateyesno0.445
mLSTM, delayed (default)yesyes0.451

Boundary loss and the oracle cutoff. We compare the entering token with the oracle's cutoff rather than the student's weakest entry because the oracle cutoff is a fixed target that does not depend on the student. It provides informative comparisons exactly at the decision margin from the first step, whereas early in training the student's own cutoff is essentially arbitrary, and the student's errors would otherwise shape its own training pairs. The two coincide when the policy is accurate, since a student that ranks like the oracle holds the oracle's cache, so the oracle's boundary token is its weakest entry. Figure 7 of the paper shows that this largely holds in practice, as the cache kept by the learned scorer contains on average 81% of the oracle's top-k tokens (4B at 75%). The remaining differences cost little accuracy. In all four settings, KVpop is within 1.2 points of the same fine-tuned model evaluated without eviction, and none of these differences is significant. Replacing the boundary loss with direct regression of the target lowers accuracy by 1.1 points (Table R4).

Efficiency at a fixed budget. The paper scales KVpop's budget with sequence length only to match DMS, whose cache grows because it is trained to keep a fixed fraction of tokens. With a fixed budget, KVpop's advantage is larger. Table R3 reports total decoding throughput of Qwen3 8B at batch size 8 on one H100, with KVpop at a fixed 2,048 entries per head and DMS trained for 8x compression. For DMS we use NVIDIA's released implementation. KVpop keeps constant throughput and memory up to 128k and is 2.6x faster than DMS at 128k with half the memory. The lower memory also allows much larger batches. At 128k, each sequence requires about 2.8 GB for DMS, compared to just 0.4 GB for KVpop. An 80 GB GPU can therefore serve roughly 23 sequences with DMS versus 150 with KVpop.

Table R3. Total decoding throughput (tokens/s) and peak memory.

Method16k32k64k128kPeak memory
Dense11361OOMOOM37 GB to OOM
DMS17017817417220 to 38 GB
KVpop44544945145119 GB

(Continued in the next comment - 1/2)

AuthorsOpenReview ↗

Response to Reviewer SU7t (2/2)

Accuracy at 128k. We agree that the paper establishes speed, not accuracy, at 128k. We will scope the accuracy claims to the evaluated lengths and state length generalisation of the learned policy as future work. All policies were trained at 16k due to the compute cost of 128k training, while profiling decoding throughput at 128k is inexpensive.

Retrieval in the hardest setting. On RULER single-needle retrieval at 16k, KVpop stays close to the teacher at 75%. At 88% the standard single-needle task stays at 1.00 and the errors come mostly from the task with random UUID values. DMS, which retrieves these strings at about the same overall cache size, distributes its cache very unevenly. On the 16k prompts at 88%, 13 to 17 of its 36 layers keep at least one head at full attention. KVpop instead gives every head the same budget, which keeps memory and decoding cost predictable but makes retrieval of arbitrary strings harder at high compression. We will state this trade-off as a limitation in the revision.

Which ingredients deserve the credit. The clearest isolation of the learned policy is the trained StreamingLLM baseline (StreamingLLM+, sinks and a sliding window). It is trained with the same data, steps and budget and differs only in the learned selection, and KVpop outperforms it by 4 to 15 points in all four settings (all significant). Table R4 lists further ablations within KVpop as changes relative to the full model, and Table R2 covers the scorer. In the paper's ablations, removing margin weighting or keep/drop balancing in the boundary loss costs 1 to 2.5 points. The individual effects are small at 75% because KVpop is already close to the teacher there, while the large gains over the baselines come from the learned selection itself.

Table R4. Ablations within KVpop (4B, 75%), change in math accuracy relative to the full model.

VariantChange
Without temporal decay-1.9 points
MSE regression instead of the boundary loss-1.1 points

Summary. We hope these clarifications and new results resolve your concerns, and if so, we would be grateful if you would consider raising your score. If any questions remain or come up during the rebuttal process, we would be happy to engage in further discussion.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer SU7t,

Thank you again for your constructive review. We understand this is a busy period, and as the discussion window nears its close, we would be grateful to know whether our response addresses your concerns, in particular the ForesightKV comparison, the fixed-budget throughput up to 128k, and the ablations on recurrent scorers, delayed scoring and the boundary loss. If any concerns remain, we would be glad to address them.

Best regards,
The authors

Reviewer_bPd6OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes KVpop, a retrofit that keeps sinks, a protected recent window, and a learned top-k cache under a fixed per-head budget. Small per-head scorers, an MLP or a delayed mLSTM, are trained with a pairwise loss against the oracle top-k cutoff, where the oracle ranks tokens by future attention mass computed through a transposed attention pass. On AIME and HMMT, KVpop retains 97% to 100% of teacher accuracy on Qwen3-4B and Qwen3-8B at 75% and 88% compression and beats DMS, KVzap, and TOVA.

Critical strengths
  1. Supervising the exact admission decision is a clean idea, and the transposed attention target keeps training cheap without dense attention maps.

  2. The baselines are well controlled, including budget-matched DMS and a fine-tuned StreamingLLM, and Appendix F.2 reports paired significance tests. The uniform per-head budget also gives constant memory and decoding speed close to StreamingLLM.

Critical weaknesses
  1. ForesightKV already trains decoding-time eviction with future-attention pairwise ranking, but it is not compared, so the novelty over it remains unclear to me.

  2. The mLSTM scorer brings no significant gain over the MLP in Appendix F.2, and delayed scoring improves token accuracy by only 0.2 points, so contribution (iii) is weakly supported.

  3. Compression ratios are counted against a fixed 16,384 reference rather than each trace's real length, and on RULER at 16k and 88% KVpop falls to 0.80 while DMS stays at 1.00.

Questions for authors
  1. What realized compression ratio does each method reach on AIME and HMMT when measured on actual trace lengths?

  2. Can you compare with ForesightKV under the setting of Table 1?

  3. What justifies the extra state of the mLSTM scorer given its small gain?

  4. How does throughput compare with dense attention and DMS at larger batch sizes?

5 comments
AuthorsOpenReview ↗

Response to Reviewer bPd6 (1/2)

Thank you for the detailed review and for recognising our supervision target and efficient transposed-attention computation. We address your questions below and will include all new results in the revision. Tables abbreviate Qwen3 4B/8B as 4B/8B, and KVpop refers to the mLSTM variant. For math results we report the average pass@1 numbers over AIME 2024/2025 and HMMT Feb/Nov 2025 (16 rollouts). Significance refers to paired bootstrap tests over the 120 problems.

W1, Q2 / Comparison with ForesightKV and novelty. We reimplemented ForesightKV (supervised stage) and trained it under cache budgets and training conditions matched to KVpop. Table R1 shows the results. On 8B, KVpop is significantly better at both ratios and maintains teacher-level accuracy, while ForesightKV drops 7.5 points below it at 88%. On 4B, both methods perform similarly at 75% and ForesightKV is significantly ahead at 88%.

Both methods use future attention and pairwise ranking. ForesightKV ranks cached entries from key, value and attention features against a schedule derived from block-pooled future attention, and prunes the cache in chunks. KVpop instead computes the exact per-token future attention mass with the transposed attention pass, without materialising attention maps, and supervises exactly the per-token admission decision at the top-k boundary of the deployed budget. We will add ForesightKV as a baseline to Table 1 and sharpen its positioning in Related Work.

Table R1. ForesightKV and KVpop under matched training budget.

Method8B, 75%8B, 88%4B, 75%4B, 88%
Teacher0.4290.4290.4510.451
ForesightKV0.4190.3540.4630.476
KVpop0.4450.4350.4510.425

W2, Q3 / mLSTM versus MLP. We agree that the reported token accuracy gain of the mLSTM scorer is small, and we will temper contribution (iii) accordingly. To clarify where the gain comes from, we separate recurrence from delay using an mLSTM that scores tokens immediately on arrival and report downstream math results (Table R2). Recurrence and delay add +0.9 and +0.6 points respectively (+1.5 together), which remains modest and is not statistically significant.

We would also like to highlight that the KVpop mLSTM variant reserves 64 fewer KV entries than the MLP variant to offset its extra state, which is already small relative to the cache (64 KV entries contain the same number of scalars as a 128x128 memory and the memory accounts for only 1.6% / 3.1% of the cache at 75% / 88% compression). Additionally, we see value in the stateful variant because it shows that a recurrent, delayed scorer can be trained in parallel and deployed with a compact constant state in the same framework, opening the door to richer stateful scoring policies.

Table R2. Separating recurrence from delay (4B, 75%).

ScorerRecurrentDelayedMath
MLPnono0.436
mLSTM, immediateyesno0.445
mLSTM, delayed (default)yesyes0.451

W3, Q1 / Realized compression on actual trace lengths. The 16,384 reference is the training context length, which also bounds the evaluated sequences, and 75% / 88% denote a hard per-head cap of 4,096 / 2,048 entries. Table R3 reports the realized compression for Qwen3 8B at 88%, the setting with the longest traces (median about 15k tokens). KVpop compresses slightly less because it fills its budget before evicting anything and never compresses traces shorter than the cap. Its peak memory is still about the same as that of DMS, whose cache at the end of long traces holds ~1,900 entries per head on average compared with KVpop's cap of 2,048, and at this peak memory KVpop matches or exceeds DMS's accuracy in all four settings. The cache sizes of KVpop and DMS show the same pattern in the other settings.

Table R3. Realized compression on the AIME/HMMT traces (8B, 88%). Compression refers to the reduction of KV entries at the end of generation relative to dense attention and is averaged over traces.

MethodCompression
KVpop81%
DMS87%
StreamingLLM+81%
KVzap91%
Expected Attention85%

(Continued in the next comment - 1/2)

AuthorsOpenReview ↗

Response to Reviewer bPd6 (2/2)

Q4 / Throughput at larger batch size. Table R4 reports total decoding throughput of Qwen3 8B at batch size 8 on one H100, with KVpop at a fixed 2,048 entries per head and DMS trained for 8x compression. For DMS we use NVIDIA's released implementation. KVpop keeps constant throughput and memory up to 128k and is 2.6x faster than DMS at 128k with half the memory. The lower memory also allows much larger batches. At 128k, each sequence requires about 2.8 GB for DMS, compared to just 0.4 GB for KVpop. An 80 GB GPU can therefore serve roughly 23 sequences with DMS versus 150 with KVpop.

Table R4. Total decoding throughput (tokens/s) and peak memory.

Method16k32k64k128kPeak memory
Dense11361OOMOOM37 GB to OOM
DMS17017817417220 to 38 GB
KVpop44544945145119 GB

W3 / RULER at 16k and 88%. At 75% KVpop stays close to the teacher on the single-needle tasks. At 88% the standard single-needle task stays at 1.00 and the errors come mostly from the task with random UUID values. At about the same overall cache size, DMS distributes its cache very unevenly. On the 16k prompts at 88%, 13 to 17 of its 36 layers keep at least one head at full attention. KVpop instead gives every head the same budget, which keeps memory and decoding cost predictable but makes retrieval of arbitrary strings harder at high compression. We will state this trade-off as a limitation in the revision.

Summary. We hope these clarifications and new results resolve your concerns and strengthen your assessment of the paper. If any questions remain or come up during the rebuttal process, we would be happy to engage in further discussion.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer bPd6,

Thank you again for your thoughtful review and helpful questions. As the discussion period nears its end, we wanted to check whether our response addresses your concerns, in particular the realized compression on actual trace lengths, the ForesightKV comparison, the mLSTM scorer's extra state, and throughput at batch size 8. If anything remains unclear, we would be happy to clarify further.

Best regards,
The authors

Comments on the paper

2 comments

Dear Reviewers,

The authors have posted their rebuttals and updated their manuscript/responses. Please carefully read the authors' responses and engage in the discussion to clarify any remaining concerns.

In light of the rebuttal, please let the authors know whether your concerns have been adequately addressed, and update your review comments and scores if appropriate. Active participation is essential for us to reach a well-calibrated decision.

Please note that the author-reviewer discussion phase ends on November 18 at 11:59 PM AoE, after which no further responses can be posted. Thank you for your continued time and effort.

Best regards,

Area Chair

AuthorsOpenReview ↗

Summary of new results and planned revisions

We thank the Area Chair and all reviewers for their detailed and constructive feedback. Below we summarize the new experiments and planned revisions added across our individual responses.

Summary of new results

  • ForesightKV comparison. Under matched cache budgets and training conditions (supervised stage), KVpop significantly outperforms ForesightKV on Qwen3-8B at both compression ratios. On Qwen3-4B, both perform similarly at 75% and ForesightKV is significantly ahead at 88%.
  • No-eviction and forgetting controls. With eviction disabled, the fine-tuned KVpop checkpoints stay within 1.6 points of the original, unmodified teacher on math, with no significant differences. With eviction, KVpop matches its no-eviction model within 0.3 points in three of four settings (In the fourth setting it is behind by a non-significant 1.2 points). Scores on MMLU, GSM8K and IFEval stay within 2 points of the teacher.
  • Component ablations. We ablate individual scorer and objective components for Qwen3-4B at 75%.
    • Recurrence. Adding an mLSTM state with immediate scoring yields a modest, non-significant gain over the stateless MLP scorer.
    • Delayed scoring. Scoring tokens after later context is available, rather than on arrival, adds another modest, non-significant gain.
    • Temporal decay. Removing temporal decay lowers accuracy by 1.9 points relative to the full model.
    • Boundary loss. Replacing the boundary loss with direct MSE regression of the target lowers accuracy by 1.1 points.
  • Fixed-budget throughput at batch size 8. For Qwen3-8B with a fixed 2,048 token budget, KVpop maintains constant throughput (~450 tokens/s) and 19 GB peak memory from 16k to 128k. At 128k, KVpop achieves ~2.6x the throughput of DMS with half the peak memory.
  • Realized compression. Measured on actual trace lengths for Qwen3-8B at 88%, KVpop averages 81% KV reduction at the end of generation (vs. 87% for DMS), since traces shorter than the cap are not compressed. Its peak cache size on long traces is about the same as that of DMS.
  • Transposed attention. We benchmark our transposed attention formulation for computing the KVpop targets vs. a chunked explicit computation. Transposed attention runs ~30x faster with ~24x less extra memory.

Planned revisions

  • Include all new tables, ablations, and controls in the revision.
  • Add ForesightKV to Table 1 and sharpen its positioning in Related Work.
  • Cite and discuss KVPolicy alongside the supervised learning vs. RL trade-off (and note compatibility with on-policy distillation) and clarify the full-attention Qwen3 model choice.
  • Temper contribution (iii) on the mLSTM scorer and emphasize the scorer-agnostic objective.
  • Scope 128k claims to efficiency, noting length generalization of the learned policy as future work.
  • State the uniform per-head budget retrieval trade-off on random-UUID needles at 88% as a limitation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.