acceptodds
Under review as a conference paper at ICLR 2027

Merge Then Drop, Adaptive KV Cache Compaction for Long-Context Reasoning

Abstract

As Large Language Models (LLMs) generate increasingly long reasoning traces, the Key-Value (KV) cache grows linearly and becomes a memory bottleneck. Learned KV Cache compactors typically assign the same budget to every layer and head, even though heads differ in how much context they need. We propose an online KV cache compactor that jointly learns the compressed representation and its per-head budget. The compactor produces an overcomplete set of merged candidate latents, from which differentiable Gumbel-Sigmoid gates decide which to retain under a global memory budget. Although gate allocation is fixed at inference, the compactor learns to route the relevant reasoning context into the surviving latents. Because backpropagating through the full sequential objective incurs an intractable memory cost for long contexts, we introduce an unbiased dual-sampling gradient estimator that retains only one compactor graph per trajectory. We evaluate on Qwen3-1.7B and 8B at compression ratios up to 16. On 8B, our method lies on the accuracy-memory and accuracy-latency Pareto frontier for math reasoning, improves AIME Avg@4 by up to 18.3 points over a uniform-budget compactor and matches a state-of-the-art learned eviction baseline at about 350 MiB of peak KV cache.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Online KV Cache Compaction, Reasoning, Discrete Gating

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XQ6jOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an online KV cache compression method for long chain-of-thought reasoning that combines token merging with learned token dropping. Building on STILL, candidate latents are extracted per chunk and pruned per head using learned gates under a target budget. Experiments on Qwen3 models across math and code benchmarks show substantial memory savings over uniform-budget baselines at high compression ratios.

Critical strengths
  1. Controlled baseline comparison: Comparing directly against vanilla STILL under identical training data, objective, and backend cleanly isolates the benefit of overcomplete latents and learned per-head gating. The resulting gains at higher compression ratios (12× and 16× on AIME) are clear, and the learned head-wise budget allocations are interpretable.

  2. Practical systems-level implementation: Rather than relying purely on theoretical or simulated KV cache reductions, the authors implement paged storage and a custom Triton kernel to realize physical memory savings for variable-length per-head caches. Reporting actual peak KV memory, latency, and throughput makes the empirical evaluation much more practically relevant.

Critical weaknesses
  1. Limited and unfair baseline comparison against DMS: The claim of matching SOTA eviction rests on an incomplete comparison with DMS [1]. DMS is evaluated only at 8×, where the proposed method actually is worse on AIME and uses more peak KV memory. More importantly, the comparison is confounded by mismatched token-protection policies as the proposed method preserves the prompt and final answer in full, whereas DMS can evict prompt tokens (protecting only a trailing window). Evaluating DMS under a matched retention policy is needed to isolate the compression mechanism itself. Although Appendix F specifies an adapter-only DMS recipe with a frozen base LLM, its adequacy as a baseline and its training budget should be clarified.

  2. Evaluation noise and incomplete length-cap analysis: With only four samples per problem and a single training seed on small benchmarks, the evaluation carries substantial variance, making small reported gains on the math benchmarks hard to distinguish from noise. In addition, the method suffers from a clear practical failure mode as rollouts hit the maximum length cap without emitting EOS far more frequently than DMS (12.5% vs. 3.3% on AIME). Although Appendix I analyzes this behavior for Qwen3-8B at 8×, corresponding completion length and unfinished rate statistics are missing for Qwen3-1.7B and for the 8B model at higher compression ratios (12× and 16×).

Nitpick:

  • The straight-through Gumbel-Sigmoid trick at line 210 would benefit from a citiation (Maddison et al., The Concrete Distribution, ICLR 2017; Jang et al., 2017) [2].

[1] https://arxiv.org/abs/2506.05345
[2] https://arxiv.org/abs/1611.00712

Questions for authors
  1. DMS baseline setup: How many training steps and tokens were used to train the DMS baseline? Could you evaluate DMS at higher compression ratios (12× and 16×), or clarify where the DMS paper reports training collapse beyond 8×?
  2. Token-protection policies: Could you evaluate DMS under a matched retention policy (preserving the prompt and final answer uncompressed, rather than protecting only a trailing window) to isolate the compression method's gains?
  3. Statistical variance on math benchmarks: Given the variance with 4 rollouts per problem and a single seed, could you provide paired significance tests against DMS on AIME and HMMT (similar to the paired intervals reported on LiveCodeBench) or evaluate over more rollouts/seeds?
  4. Length and termination statistics: Could you report completion lengths and unfinished rates for Qwen3-1.7B, as well as for the 8B model at higher compression ratios (12× and 16×)?
  5. Budget loss formulation: What is the motivation for using a symmetric Huber loss rather than a one-sided penalty (e.g., ℓ1\ell_1) for the budget constraint? Given that realized compression diverges from the targets compression ratio, how can a practitioner reliably hit a fixed memory budget?
  6. Model diversity: Since all experiments are conducted on Qwen3, could you evaluate on a model from another family (e.g., llama or smollm) to show whether the gains generalize?
6 comments
AuthorsOpenReview ↗

Response to Reviewer XQ6j (1/3)

We thank the reviewer for the time and effort in assessing our paper. We appreciate the recognition that our method shows substantive gains over the baseline method, demonstrates practical efficiency gains in both training and inference, and reflects careful effort in its practical system implementation. We appreciate the valuable feedback and have integrated it into our responses below. We have also added the citation you requested. We hope this response addresses your questions and clarifies the strengths of our work.

Weakness 1. We originally evaluated DMS only at 8× because both the DMS paper and NVIDIA Model-Optimizer published configurations up to 8× compression. To address this concern, we clarified DMS’s adapter-only training recipe and trained Qwen3-8B DMS at 12× and 16× ourselves using the exact official Model-Optimizer recipe (detailed in Question 1). As shown in Question 1, Ours and DMS perform at parity when evaluated on AIME and HMMT across all tested CRs.

We also disagree that differing token-protection policies create a confound between methods. DMS is trained end-to-end to evict across the full sequence while retaining a trailing window, whereas Ours is trained to compact only reasoning chunks while preserving prompts. Evaluating DMS (and each method in general) under its native training policy is crucial to extract the best performance and avoid a distribution shift during evaluation; This is why we argue that token-protection policy is a part of the compression mechanism itself. Furthermore, reserving the full prompt imposes higher KV memory overhead on Ours, which we acknowledge as a limitation of our current design. To address this directly, we evaluated DMS with prompt protection forced at inference and show that DMS accuracy does not improve while incurring higher KV overhead (detailed in Question 2).

Weakness 2. Regarding evaluation noise, our four-run averaging (Avg@4) follows the MathArena evaluation protocol (https://matharena.ai). We use this metric to estimate single-run response correctness (similar to expected pass@1) and measure whether model reliability is preserved. While this reduces sampling noise, we acknowledge it does not eliminate uncertainty entirely. To address this, we conducted a paired uncertainty analysis by bootstrapping matched problem differences with 10,000 resamples (95% CIs), with p-values from a paired sign-flip test. In our matched-memory setting (Ours 12× vs. DMS 8×), the difference is −0.83% [−9.17, +6.67] on AIME and +0.76% [−1.52, +3.79] on HMMT, confirming that our performance is on par with DMS. The full significance table is reported in Question 3, and the paired CIs have been added to the revised paper (Appendix L, Table 14).

On the delayed EOS behavior, we agree that this is indeed a limitation of our method and soft KV compaction in general; we observe this behavior across all soft KV compaction methods that we tested. However, we argue that this is not a clear practical failure mode, as despite reaching the generation cap more frequently, our overall accuracy matches DMS and significantly outperforms other baselines—the model still generates the correct answer within the text even if it fails to halt, preserving reasoning capability. Furthermore, we have expanded our generation length analysis to include both the Qwen3-1.7B model and the 8B model at higher compression ratios (12× and 16×). These extra analyses are detailed in Question 4, and the full plots and statistics have been added to Appendix I (Figures 7–8 and Table 8).

Question 1. DMS trains a small adapter on a frozen model, like our method. We follow NVIDIA Model-Optimizer’s Qwen3-8B DMS recipe: 544 steps at effective batch size 16, 8,192-token context, a 512-token window, AdamW at 3×10−53\times 10^{-5}, and a compression ramp over the first 510 steps, on 4,000 OpenR1-Math-220k samples totaling about 71M training tokens. We trained Qwen3-8B 12× and 16× DMS using this same recipe and evaluated DMS on both AIME 26 and HMMT-Feb 26, as detailed below:

CRMethodAIME 26 Avg@4 ↑HMMT-Feb 26 Avg@4 ↑
8×DMS62.534.1
Ours58.336.4
12×DMS55.034.1
Ours61.734.8
16×DMS59.233.3
Ours56.730.3

Across all six CR–benchmark pairs, Ours and DMS are at parity, confirming that our soft compactor are competitive with state-of-the-art hard eviction. Notably, at 12×, Ours beats DMS on both AIME (61.7 vs. 55.0) and HMMT (34.8 vs. 34.1), and also leads on 8× HMMT (36.4 vs. 34.1), while DMS leads at 16× (59.2/33.3 vs. 56.7/30.3). We thank the reviewer’s comment and thoroughness on exploring the baselines; We retract our earlier Section 4.1 phrasing: the DMS paper reported collapse for their prior learned compression (DMC) work rather than DMS itself (which was evaluated up to 8×), and we have corrected that sentence in the revision.

AuthorsOpenReview ↗

Response to Reviewer XQ6j (2/3)

Question 2. We emphasize that protecting the prompt is not how DMS was trained or evaluated in its original paper. Using this retention policy during evaluation introduces a distribution shift from training and, in theory, worsens its efficiency by increasing both peak KV memory and total KV reads as unimportant tokens are not evicted. To concretely address your question, we evaluated Qwen3-8B DMS with this protected policy. Protecting the prompt does not change DMS accuracy (−0.83 points, paired 95% CI [−7.50, +6.67]) while increasing peak KV by 7.6% and KV reads by 22.2%. Because prompt protection degrades DMS's efficiency without changing accuracy, we retain DMS’s native evaluation policy in the main results tables.

Question 3. To address the variance issue with 4 rollouts, we conducted a paired uncertainty analysis as requested by bootstrapping matched problem differences. We average the four rollouts within each problem, resample 10,000 times for the 95% CIs, and compute p-values with a paired sign-flip test. As shown in the table below (with the paired CIs also included in Appendix L, Table 14), our method's performance is statistically indistinguishable from DMS across both benchmarks:

OursBenchmarkDifferencePaired 95% CIp
8×AIME−4.2[−10.8, +2.5]0.341
8×HMMT+2.3[−3.8, +9.1]0.656
12×AIME−0.8[−9.2, +6.7]1.000
12×HMMT+0.8[−1.5, +3.8]1.000

This confirms that the small performance differences are well within the margin of error, demonstrating that our method is competitive with state-of-the-art hard eviction.

Question 4. We have expanded our generation length analysis to include the Qwen3-8B model at higher compression ratios (12× and 16×), as well as the Qwen3-1.7B model. The full statistics and plots have been added to the paper (Appendix I, Figures 7 and 8). Curiously, the 1.7B length and EOS statistics are very similar across all models.

As shown in the graphs, the trend of delayed termination and longer responses persists at higher compression ratios for all methods tested. Although the average generation length and the number of unfinished generations increase, Ours remains strictly below Vanilla STILL while having roughly the same statistics as AM. As discussed previously, this behavior is observed across all soft KV compaction methods we tested; however, this cannot be equated with overall performance, as TOVA's generation length and termination statistics are consistently much lower than the other methods while having much lower accuracy, and all 1.7B methods show similar generation length statistics despite large differences in performance. Importantly, we argue that this delayed EOS does not equate to a reasoning failure or a practical failure mode, as our overall accuracy remains competitive with DMS and the model successfully generates the correct answer even if it subsequently struggles to halt.

Question 5. We use a symmetric Huber loss rather than a one-sided penalty because a one-sided penalty stops providing gradients once retention drops below the target. During the early part of training, the mean number of kept tokens dropped rapidly, and the symmetric loss restored gradients to prevent premature eviction and overfitting. This is especially important in our hard-mask design, where gradients only flow through selected tokens. We chose Huber loss because it provides smooth gradients near the target while keeping gradients bounded for large deltas (same as L1 loss), and we use a linear curriculum to control how quickly the target decreases.

Our training uses a surrogate objective over expected soft tokens kept to stabilize optimization and prevent unnecessary constraints on the expressiveness of the model. Furthermore, training adds Gumbel noise to mitigate overfitting, while inference uses the raw gate logits, meaning realized token counts will naturally differ. While it does not guarantee an exact match with the nominal compression ratio, it serves as an effective proxy during training. Previous work (e.g., DMS) also uses this design choice, where the final CR does not match the nominal target CR.

In practice, we do not need an exact nominal ratio to deploy the model: because our gates are static, each trained checkpoint’s retained token count is fixed and fully inspectable before deployment. This also makes it possible to adjust the gate threshold at inference to deterministically control the nominal CR, though in our experiments the performance difference is negligible.

AuthorsOpenReview ↗

Response to Reviewer XQ6j (3/3)

Question 6. To test generalization beyond Qwen3, we trained both Ours and Vanilla STILL on Ministral3-3B [1], compacting every global-attention KV cache (512-token chunks to targets of 64 and 43 tokens for 8× and 12× CR) using the exact same Open-R1 recipe, data, and seed as Qwen3. While we are still resolving a system issue (which hopefully can be solved before the end of rebuttal) and have not yet completed downstream evaluation, training has completed and the table below reports mean training KL divergence over the final 300 steps:

CompressionOurs (LKL\mathcal{L}_{\text{KL}})Vanilla STILL (LKL\mathcal{L}_{\text{KL}})
8×0.02930.0564
12×0.03890.0714

Across both ratios, Ours' final distillation KL is substantially lower than Vanilla STILL's for both configurations, and Ours at 12× achieves lower KL loss than Vanilla STILL at 8×. This closely mirrors the optimization behavior for Vanilla STILL vs. Ours observed on Qwen3 (Figure 4)—where lower post-anneal loss directly translates into large downstream AIME gains—and provides direct evidence that our overcomplete merging and per-head allocation mechanism transfers to another model family.

[1] Liu, A. H., et al. "Ministral 3." arXiv preprint arXiv:2601.08584 (2026).

Reviewer_Ek7YOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper combines overcomplete STILL-based KV latents with static per-head Gumbel-Sigmoid gates, trained through chunk-level self-distillation and a sampled-gradient procedure. A paged cache and Triton attention kernel realize memory savings.

Critical strengths
  1. Matched uniform-budget controls show substantial benefits from heterogeneous head budgets at aggressive compression.
  2. Physical cache measurements and an implemented attention backend support practical efficiency claims.
Critical weaknesses
  1. It is hard to tell what learned allocation adds beyond extra latent capacity. Vanilla STILL has neither the overcomplete latents nor the gates. The 8B results show that the combination works, but do not tell us which part drives the gains. The smaller-model ablations help, though it is unclear whether the same explanation holds for 8B.
  2. The comparison with DMS is less conclusive than the headline suggests. The mean accuracies are close, but there is no paired uncertainty analysis for this particular math comparison. The method also generates longer responses, so faster decoding per token may not mean a faster final answer.
Questions for authors
  1. Why is a fixed set of head budgets sufficient across different reasoning tasks? Could the learned allocation be specific to the Open-R1 training data, even though the compactor itself processes each chunk dynamically?
  2. What does the dual-sampling approach add to the cited workshop version? Is the sampling procedure different, or is the main addition the explicit weighting, proof, and variance analysis?
  3. Do the 8B gains mainly come from starting with more latents, or from learning different budgets across heads? Do the 1.7B ablations provide evidence that the same mechanism explains the 8B results?
  4. How much does single-chunk sampling affect convergence? Figure 4 shows that larger batches help, but does not separate the effect of batch size from the noise introduced by sampling.
5 comments
AuthorsOpenReview ↗

Response to Reviewer Ek7Y (1/3)

We thank the reviewer for the time and effort in assessing our paper. We appreciate the recognition that our method shows substantive gains over the baseline method and practical efficiency gains, although we acknowledge the lack of clearer separation of our proposed contributions. We appreciate the valuable feedback and have integrated it into our responses below. We hope this response addresses your questions and clarifies the strengths of our work.

Weakness 1. We emphasize that the overcomplete latents and the learned gates are joint and complementary contributions of our work. Overcomplete latents enable us to obtain high-fidelity representations during compression, while the learned per-head gates act as the adaptive routing mechanism to determine which of those latents to retain. Together, they allow the model to jointly learn both the routing and the positions (which implicitly control the per-head budget) of the compressed tokens across different layers and heads, which emerge as highly asymmetrical as we show in Figure 5.

Furthermore, regarding your concerns on the 8B behavior: during the development of the paper, we relied on the 1.7B model for our structural ablations. As our method performed well and looked promising, we scaled up our experiments to the 8B model. We initially refrained from running ablations at the 8B scale in the main paper due to heavy compute costs, while noting that it exhibits a similar allocation profile and performs substantially better than Vanilla STILL. To directly address both of your concerns, we have now conducted the requested controlled ablations on Qwen3-8B, detailed in our response to Question 3.

Weakness 2. We acknowledge this trade-off and do not claim overall superiority in terms of performance and end-to-end efficiency. This fundamentally stems from a methodological difference between the two approaches: DMS performs hard eviction of the original tokens to sparsify the cache, whereas our method performs cache compaction by synthesizing soft latents. As you correctly observed, our method generates longer responses and thus takes longer to reach the answer for some questions, which is a limitation of our method and soft KV compaction methods in general. We view mitigating this delayed EOS behavior in soft compaction as a highly promising direction for future work.

Rather than strictly beating DMS on all efficiency metrics, our core contribution and novelty lie in exploring a different paradigm, which is strictly more expressive than pure token dropping. Our architecture successfully combines the two main approaches in the field: optimizing high-fidelity token merging using overcomplete latents, while jointly learning to drop unused tokens and enable adaptive per-head budgets. We believe demonstrating that this hybrid approach can match the accuracy of state-of-the-art hard eviction baselines is a valuable contribution to the field.

Regarding the statistical significance of the math results, we have now included a paired uncertainty analysis for AIME 26 and HMMT-Feb 26 in the table below (paired CIs across all baselines are also in Appendix L, Table 14), which confirms our competitive accuracy with DMS:

OursBenchmarkDifferencePaired 95% CIp
8×AIME−4.2[−10.8, +2.5]0.341
8×HMMT+2.3[−3.8, +9.1]0.656
12×AIME−0.8[−9.2, +6.7]1.000
12×HMMT+0.8[−1.5, +3.8]1.000

Question 1. We distinguish capacity allocation (the gates) from content compression (the Perceiver). The static gate for each head only determines how many latent slots that head receives, and as you note, the representations are computed dynamically from every input chunk. The cross-attention in the Perceiver still decides what information each retained latent summarizes and is still trained after the budget is fixed, meaning static gates do not result in retaining the same tokens across different chunks.

Our static gates are positioned as a simple yet effective design choice for our main contribution of a hybrid merging-and-dropping method. Our rationale is that these budgets capture persistent differences in compression sensitivity across the frozen model’s layers and heads for a given task (e.g., reasoning concentrates capacity in middle KV heads). While overspecificity to Open-R1 remains possible, strong performance on unseen math benchmarks (AIME 26, HMMT-Feb 26) and on OOD coding (LCB v6) supports our hypothesis of transferability across reasoning tasks. We have added this clarification to Appendix H. A well-designed input-dependent architecture could in theory generalize better to multiple tasks and thus remains a promising future work direction.

AuthorsOpenReview ↗

Response to Reviewer Ek7Y (2/3)

Question 2. We appreciate the question and the thoroughness in exploring our work. Our earlier workshop paper started out with the same problem statement and core idea, although we used naive training on a very small and short dataset for the initial workshop submission. After submitting, we continued to expand on our method and arrived at the training strategy and data we have now, which we then used to obtain the results presented in the camera-ready version of the workshop paper.

With regard to our dual-sampling training strategy, our workshop version was sparse and provided little analysis; in this version, we substantially expanded our theoretical and empirical analysis: proofs of both the attention-level composability of chunks (as our motivation) and the unbiasedness of the gradient, as well as the variance analysis used to guide our training approach. Overall, this current version also adds scaled-up experiments (up to 16× compression and the 8B model), clearer ablations, more baselines, and better-tuned models and baselines.

Question 3. As emphasized in our answer to Weakness 1, our core contribution is the joint mechanism of overcomplete latents (for high-fidelity merging) with learned per-head gates (for adaptive budget allocation). Note that the Qwen3-8B compactor and gates are trained independently from scratch, as we only tuned design hyperparameters on 1.7B before scaling the full pipeline to 8B. As shown in Figure 5, the same highly asymmetric head-budget profile (concentrating tokens in middle KV heads while aggressively pruning early and late heads) emerges independently at both 1.7B and 8B, demonstrating that the learned allocation mechanism remains active and scales across model sizes.

To directly address the concerns regarding learned budget effects and the mechanism on the 8B model, we have conducted the requested controlled ablations on Qwen3-8B at 12× CR to isolate the contributions of overcomplete latents, learned heterogeneous per-head allocation, and the budget annealing curriculum. Specifically, we compare our full method against:

  • (A) Uniform Top-K (isolating per-head learned budget): Uses the exact same overcomplete architecture and training curriculum as Ours, except that we enforce a strictly uniform per-head budget for training and inference while forgoing the explicit budget loss.
  • (B) No curriculum (isolating the annealing schedule): Uses the exact same architecture and learned per-head gating as Ours, but removes the linear budget-annealing curriculum and trains directly at the final target budget BB from step 0.
ModelVariant12× AIME 26 ↑12× Peak KV (MiB) ↓
Qwen3-8BVanilla STILL43.30395.0
Uniform Top-K45.83407.6
Ours (no curr.)52.50231.1
Ours (full)61.67322.4

As shown in the table, at 12× compression, both learned per-head allocation and the annealing curriculum are critical. Introducing overcomplete candidates via Uniform Top-K improves over Vanilla STILL by +2.53 points (45.83 vs. 43.30 on AIME 26). Replacing uniform retention with learned per-head allocation adds a further +15.83 points (paired 95% CI [+5.83, +26.67]) with a smaller cache (322.4 vs. 407.6 MiB Peak KV; 34.4 vs. 43 realized tokens/head in Table 7). Finally, removing the annealing curriculum costs 9.17 points (paired 95% CI [+3.33, +15.00]) as gates prematurely over-prune the cache (231.1 MiB Peak KV), confirming that gradual dropping is essential for stable joint training. These results have been added to Section 4.3, cleanly separating the mechanisms and proving that the same mechanism drives the 8B results.

AuthorsOpenReview ↗

Response to Reviewer Ek7Y (3/3)

Question 4. We appreciate the question, and although we agree that a comprehensive evaluation of the convergence behavior of our sampling strategy is a useful future contribution, providing a full theoretical convergence analysis to isolate the noise dynamics is out of scope for this paper. As we empirically show in Figure 4 and our main results, the strategy still converges stably w.r.t. the loss and produces strong downstream performance on reasoning tasks, proving that it provides a highly effective learning signal.

A direct empirical comparison to the full objective is practically infeasible, and without our dual-sampling formulation, training the full model is near impossible due to the memory requirements of unrolling the full sequential objective. Therefore, our sampling strategy is strictly necessary to enable training with practical batch sizes at our desired sequence length. While sampling a single prefix chunk inherently increases gradient variance (which we analyze and provide variance-reduction strategies for in Appendix C.2), the ability to fit larger batch sizes effectively mitigates this issue.

To emphasize this point, we trained two compactors of the same Qwen3-1.7B model with the maximum batch size that fits on a single 80GB H100 GPU for each method, plotting the loss vs. steps and recording statistics across 2,500 training steps under the same budget curriculum with a max sequence length of 16k:

MetricDual samplingBoundary-only sampling
Largest tested batch that fits41
Peak allocated memory, full run (GiB)37.175.7
Memory at batch 1, probe (GiB)12.776.3
Samples processed10,0002,500
Total token count (teacher + student)39,540,71310,009,961
Samples per second1.830.80
Mean training KL (last 300 steps)0.06660.0634

As these results demonstrate, boundary-only sampling consumes 76.3 GiB at batch size 1 for a 1.7B model, which is impossible to run on the 48GB GPUs used in our initial iteration and barely fits on an 80GB H100. These results again emphasize that training without any sampling is completely intractable, and even boundary-only sampling is not feasible for models larger than 2B. In contrast, dual sampling reduces single-batch memory footprint by ~6×, increases sample throughput by ~2.3×, and processes far more tokens over the same 2,500 steps, which is crucial for generalization.

Moreover, dual sampling exhibits smoother convergence with fewer loss spikes than boundary-only sampling due to the larger batch size. Despite using less than half the peak memory of boundary-only sampling, dual sampling achieves a comparable final training KL. We have added a short summary of these results to the "Training stability of learned allocation" part of Section 4.3, and included the full analysis with the table and training graphs in Appendix J. While we do not deny that full un-sampled backpropagation or boundary-only sampling may converge better per step, both are intractable at scale, and dual sampling provides a training method that is far more memory-efficient, faster per sample, converges reliably on modest hardware, and performs well on downstream tasks as seen in our results section.

Reviewer_QsZPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an online KV-cache compactor for long reasoning trajectories by generating an overcomplete set of latent KV entries and jointly learns input-independent gates that allocate different retained budgets across layers and heads. The backbone remains frozen, and the compactor is trained through distillation with a budget penalty. A dual-sampling procedure selects one target boundary and one gradient-bearing prefix chunk to reduce training memory. The main results show improvements over a uniform-budget compactor at aggressive compression and competitive accuracy with DMS at similar occupied KV-cache memory.

Critical strengths

The method shows gains over STILL under a reasonably controlled setup. The implementation translates head-specific budgets into actual cache savings. Reporting measured cache occupancy, accumulated KV reads, and throughput is valuable. The approximately matched-memory comparison with DMS also shows competitive accuracy on both math benchmarks.

Critical weaknesses
  • The experiments do not clearly isolate the benefit of learned per-head budgeting. A controllable ablation experiment to isolate the effect of learnable budget allocation, candidate capacity, pruning curriculum would be important to show the effectiveness of the proposed method.
  • The practical efficiency advantage over DMS is unclear. At approximately matched memory, accuracy is similar to DMS and occupied KV memory is lower, but accumulated KV reads are higher on both benchmarks. This is a useful tradeoff, but does not establish overall efficiency superiority.
Questions for authors
  • How does the proposed method compare to other learnable KV eviction methods such as [1, 2] in terms of performance and efficiency?
  • How does the method compare with an overcomplete compactor using the same pruning curriculum but uniform per-head retention, matched by realized total cache entries?
  • Can you compare peak training memory and time to a comparable validation loss for dual sampling, boundary-only sampling, and a feasible checkpointed reference? Figure 4’s batch-size comparison does not directly establish the memory–convergence tradeoff.

[1] Huang, Yen-Chieh, et al. "Kv admission: Learning what to write for efficient long-context inference." arXiv preprint arXiv:2512.17452 (2025). [2] Bui, Ngoc, et al. "Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction." arXiv preprint arXiv:2605.09649 (2026).

4 comments
AuthorsOpenReview ↗

Response to Reviewer QsZP (1/3)

We thank the reviewer for the time and effort in assessing our paper. We appreciate the recognition that our method shows gains over the baseline method, practical efficiency gains, and competitive accuracy with the state-of-the-art method, while acknowledging the lack of clearer separation of our proposed contributions. We appreciate the valuable feedback and have integrated it into our responses below. We hope this response addresses your questions and clarifies the strengths of our work.

Weakness 1. We again emphasize that the overcomplete latents and the learned gates are joint and complementary contributions of our work. Overcomplete latents enable us to obtain high-fidelity representations during compression, while the learned per-head gates act as the adaptive routing mechanism to determine which of those latents to retain. Together with our chunk composability insight and dual-sampling method, they allow the model to jointly learn both the routing and the positions (which implicitly control the per-head budget) of the compressed tokens across different layers and heads, which emerge as highly asymmetrical as we show in Figure 5.

To directly address your concern and assess whether uniform allocation can perform comparably, we have conducted the requested controlled ablations (detailed in Question 2), which show the gain of our full method versus its individual components. Furthermore, regarding your concern on candidate capacity, we have already conducted an ablation on this in Section 4.3 and Table 11, which shows the advantage of using overcomplete latents up to 4× the target budget.

Weakness 2. We acknowledge this trade-off and do not claim overall efficiency superiority. This fundamentally stems from a methodological difference between the two approaches: DMS performs hard eviction of the original tokens to sparsify the cache, whereas our method performs cache compaction by synthesizing soft latents, leading to more tokens being generated and more tokens being read during the later part of inference as we buffer up to 512 + 128 uncompacted tokens before compacting each chunk.

Rather than strictly beating DMS on all efficiency metrics, our core contribution and novelty lie in exploring a different paradigm, which is strictly more expressive than pure token dropping. Our architecture successfully combines the two main approaches in the field, optimizing high-fidelity token merging using overcomplete latents while jointly learning routing/positions to drop unused tokens and enable adaptive per-head budgets. Along with that, we also introduce a dual-sampling training strategy motivated by our chunk composability insight that reduces training memory load while remaining unbiased. We believe demonstrating that this hybrid approach and training strategy can match the performance of state-of-the-art hard eviction baselines is a valuable contribution to the field.

Question 1. Our work shares the goal of optimizing cache utility with other learned KV eviction methods. WG-KV [1] learns which original token KVs enter a global cache while maintaining a local window. Similarly, DBTrimKV [2] learns globally comparable retention scores, allowing original KV entries to compete for capacity across layers and heads. Both methods, as with other hard KV eviction methods, are restricted to permanently dropping unselected original tokens. While both models report strong results, direct performance comparison is not possible due to differing models, data, and workloads.

Methodologically, our work differs in how we compact and retain the caches. Previous hard KV eviction methods learn which cache positions to evict while keeping a strict subset of the original KV cache, whereas ours learns both the positions and representations of the compacted cache. We synthesize a learned overcomplete set of latent representations of the cache from our network and apply differentiable Gumbel-Sigmoid gates to prune them over the course of training. By actively merging distributed reasoning features into these dense latents rather than performing hard KV eviction on original tokens, our method preserves higher representational utility and is able to sustain strong performance (on par with DMS at approximately matched memory).

AuthorsOpenReview ↗

Response to Reviewer QsZP (2/3)

Question 2. We have conducted the requested controlled ablations on Qwen3-8B at 12× CR to isolate the contributions of overcomplete latents, learned heterogeneous per-head allocation, and the budget annealing curriculum. Specifically, we compare our full method against:

  • (A) Uniform Top-K (isolating per-head learned budget): Uses the exact same overcomplete architecture and training curriculum as Ours, except that we enforce a strictly uniform per-head budget for training and inference while forgoing the explicit budget loss.
  • (B) No curriculum (isolating the annealing schedule): Uses the exact same architecture and learned per-head gating as Ours, but removes the linear budget-annealing curriculum and trains directly at the final target budget BB from step 0.
ModelVariant12× AIME 26 ↑12× Peak KV (MiB) ↓
Qwen3-8BVanilla STILL43.30395.0
Uniform Top-K45.83407.6
Ours (no curr.)52.50231.1
Ours (full)61.67322.4

As shown in the table, at 12× compression, both learned per-head allocation and the annealing curriculum are critical. Introducing overcomplete candidates via Uniform Top-K improves over Vanilla STILL by +2.53 points (45.83 vs. 43.30 on AIME 26). Replacing uniform retention with learned per-head allocation adds a further +15.83 points (paired 95% CI [+5.83, +26.67]) with a smaller cache (322.4 vs. 407.6 MiB Peak KV; 34.4 vs. 43 realized tokens/head in Table 7). Finally, removing the annealing curriculum costs 9.17 points (paired 95% CI [+3.33, +15.00]) as gates prematurely over-prune the cache (231.1 MiB Peak KV), confirming that gradual dropping is essential for stable joint training. These results have been added to Section 4.3.

To address your realized total cache entries comment: while we train both variants at the same nominal target (43 tokens/head at 12× CR) rather than matching realized cache entries directly (which is not known until training finishes), learned head budgets provide the flexibility that enables Ours to converge to fewer realized tokens per head (34.4 vs. 43, Table 7) while still outperforming Uniform Top-K.

AuthorsOpenReview ↗

Response to Reviewer QsZP (3/3)

Question 3. We appreciate the suggestion to compare these strategies, but a direct comparison with naive sequential training is infeasible due to the sheer memory requirement. Implementing gradient checkpointing w.r.t. the full objective to directly compare the trade-offs is practically infeasible—as it would require a complete rewrite of our sequential, chunk-wise compactor and model implementation while introducing huge time overheads due to massive forward-pass recomputations—and is out of scope for this rebuttal and the paper.

We have empirically shown that dual sampling exhibits stable convergence and yields a final model with strong downstream performance on reasoning tasks in the main results of our work (Figures 2 and 4). Our dual-sampling formulation is necessary to enable training with practical batch sizes and provides a highly effective learning signal at much lower memory cost than the alternatives. Although the full objective itself is unbiased, we acknowledge that sampling a single prefix chunk inherently increases gradient variance, which we explicitly analyze in Appendix C.2, but the ability to fit larger batch sizes effectively mitigates this issue.

To emphasize this point, we trained two compactors of the same Qwen3-1.7B model with the maximum batch size that fits on a single 80GB H100 GPU for each method, plotting the loss vs. steps and recording statistics across 2,500 training steps under the same budget curriculum with a max sequence length of 16k:

MetricDual samplingBoundary-only sampling
Largest tested batch that fits41
Peak allocated memory, full run (GiB)37.175.7
Memory at batch 1, probe (GiB)12.776.3
Samples processed10,0002,500
Total token count (teacher + student)39,540,71310,009,961
Samples per second1.830.80
Mean training KL (last 300 steps)0.06660.0634

As these results demonstrate, boundary-only sampling consumes 76.3 GiB at batch size 1 for a 1.7B model, which is impossible to run on the 48GB GPUs used in our initial iteration and barely fits on an 80GB H100. These results again emphasize that training without any sampling is completely intractable, and even boundary-only sampling is not feasible for models larger than 2B. In contrast, dual sampling reduces the single-batch memory footprint by ~6×, increases sample throughput by ~2.3×, and processes far more tokens over the same 2,500 steps, which is crucial for generalization.

Moreover, dual sampling exhibits smoother convergence with fewer loss spikes than boundary-only sampling due to the larger batch size. Despite using less than half the peak memory of boundary-only sampling, dual sampling reaches a comparable final training KL. We have added a short summary of these results to the "Training stability of learned allocation" part of Section 4.3, and included the full analysis with the table and training graphs in Appendix J.

While we do not deny that full un-sampled backpropagation or boundary-only sampling may converge better per step, both are intractable at scale, and dual sampling provides a training method that is far more memory-efficient, faster per sample, converges reliably on modest hardware, and performs well on downstream tasks as seen in our results section. Furthermore, because the budget linearly anneals and realized sparsity continues to change during training, intermediate validation losses are incomparable; we evaluate over a fixed 2,500-step curriculum, where both curves flatten at the same rate and dual sampling runs faster per sample at lower total memory. Thus, Table 9's higher total run time over 2,500 steps solely reflects our choice to scale the batch size 4× to improve generalization (while sample throughput is 2.3× faster) rather than slower convergence.

Dear Reviewer QsZP,

We extend our gratitude again for the time and feedback. We would like to remind you that the rebuttal period draws to a close, and note that we have extended our experiments to address your main concerns and questions. We did extra ablation to differentiate the contribution of learned budget allocation and pruning curriculum. Given that your concerns and questions are largely similar and coincide with Reviewer Ek7Y, which has been resolved, we encourage that you reevaluate your review of our work. We would be happy to offer up some explanation if you still have unresolved concerns.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Changes and Experiments

Thank you to Reviewers QsZP, Ek7Y, and XQ6j for the thorough and constructive feedback. Below is a concise summary of the additional experiments and manuscript revisions completed for the rebuttal to help reviewers and the AC easily track all updates, experiments, and results that was asked by other reviewers.

Additional Experiments

  • Learned allocation and curriculum ablations (Reviewers QsZP, Ek7Y): Evaluated Qwen3-8B at 12× compression on AIME 26 comparing (a) Uniform Top-K selection over the same 256 learned candidates with curriculum, and (b) Ours trained without the annealing curriculum directly at the target budget.
    • Result: Ours outperforms Uniform Top-K by +15.83 points at 21% lower Peak KV and beats the curriculum-free variant by +9.17 points, proving that learned per-head allocation and gradual pruning enables the 8B gains.
  • Dual vs. boundary-only sampling comparison (Reviewers QsZP, Ek7Y): Trained Qwen3-1.7B at 8× compression for 2,500 steps under the same curriculum at each method's maximum fitting batch size on a single 80 GB H100, tracking loss curves, peak memory, and total processed tokens.
    • Result: Dual sampling cuts single-batch memory by ~6× , fits 4× larger batches at less than half the peak memory , boosts sample throughput by 2.3×, and converges smoothly to a comparable final KL, enabling training where boundary sampling and full objective are intractable.
  • DMS higher compression ratios and prompt-protected DMS evaluation (Reviewer XQ6j): Trained Qwen3-8B DMS at 12× and 16× CR, and evaluated Qwen3-8B DMS at 8× compression with full initial-prompt protection to isolate the effect of token-protection policies.
    • Result: Ours achieves parity with DMS across all six CR–benchmark pairs (outperforming DMS at 12× on both AIME and HMMT), while forcing prompt protection on 8× DMS yields no accuracy gain (−0.83 pts) at +7.6% Peak KV and +22.2% KV reads.
  • Cross-architecture generalization (Reviewer XQ6j): Implemented Ministral-3-3B and trained both Ours and Vanilla STILL at 8× and 12× compression.
    • Result: Ours cuts final distillation KL nearly in half relative to Vanilla STILL across both ratios, demonstrating that our merging-and-allocation mechanism transfers beyond Qwen3.

Manuscript Revisions

  • Added the requested straight-through Gumbel-Sigmoid citation in Section 3.2 (Reviewer XQ6j).
  • Clarification on the DMS 8× evaluation in Section 4.1 (Reviewer XQ6j).
  • Added an ablation subsection in Section 4.3 (Table 4) on the effects of learned overcomplete latents, per-head allocation, and the budget curriculum (Reviewers QsZP, Ek7Y).
    • Takeaway: Together with the candidate-capacity sweep in Table 11, this cleanly isolates each design component and establishes that the combination of overcomplete latents, learned budget, and gradual pruning is essential.
  • Added paired uncertainty analysis on the math benchmarks in Appendix L (Table 14) (Reviewers Ek7Y, XQ6j**).**
    • Takeaway: Confirms statistical parity with DMS at matched memory alongside statistically significant gains over Vanilla STILL.
  • Added complete generation length and unfinished-rate plots and analysis across Qwen3-1.7B and Qwen3-8B (8×, 12×, 16×) in Appendix I (Figures 7–8) (Reviewer XQ6j).
    • Takeaway: Shows that delayed EOS is shared across soft compactors without indicating reasoning collapse, and that Ours remains strictly below Vanilla STILL in both length and truncation rate.
  • Added the dual vs. boundary-only sampling memory, throughput, and convergence analysis in Appendix J (Figure 9, Table 9), along with a summary in Section 4.3 (Reviewers QsZP, Ek7Y).
    • Takeaway: Quantifies the memory–throughput–convergence trade-off, proving dual sampling makes long-context compactor training tractable without sacrificing final loss quality.
  • Added analysis in Appendix H on how static per-head capacity allocation transfers across reasoning tasks (Reviewer Ek7Y).
    • Takeaway: Clarifies that static gates only set per-head slot counts while Perceiver cross-attention synthesizes chunk content dynamically, explaining strong transfer from Open-R1 to unseen math and OOD coding.
AuthorsOpenReview ↗

Rebuttal Summary and General Response

Dear Area Chair and Reviewers,

We thank you for the time spent providing valuable feedback and overseeing our work. As the discussion period draws to a close, we summarize our key contributions and how our rebuttal resolves every major weakness raised.

We introduce an end-to-end online KV cache compactor that jointly learns overcomplete latent token merging and discrete per-head budget allocation, trained with an unbiased dual-sampling estimator that enables long-context training on modest hardware, and deployed with a custom head-paged Triton decode kernel. All three reviewers acknowledged the practical systems implementation, real measured cache savings, substantial gains over uniform-budget STILL, and competitive accuracy against state-of-the-art DMS at matched peak KV memory.

During the rebuttal, we directly addressed each of the core weaknesses raised by each reviewer:

  • Reviewers QsZP and Ek7Y both asked us to isolate each component of our method and verify the memory–convergence trade-off of dual sampling. We addressed this through controlled ablations on Qwen3-8B (Section 4.3, Table 4), along with our candidate-capacity sweep in Table 11, shows that both learned per-head gating and the annealing curriculum, rather than just latent capacity, enables the full improvement at high compression. Furthermore a direct dual vs. boundary-only sampling benchmark (Appendix J, Table 9) showing ~6× lower single-batch memory and 2.3× higher sample throughput while converging to comparable training KL.
  • Reviewer XQ6j raised concerns regarding evaluating DMS only at 8×, mismatched prompt-protection policies, and evaluation variance. We resolved these by training and evaluating Qwen3-8B DMS at 12× and 16× (Reviewer XQ6j Reply) and adding paired bootstrap significance tests (Appendix L, Table 14) to verify parity of our method with DMS, and evaluating prompt-protected DMS at 8× confirming prompt protection brings no improvement and the need to evaluate each method with their intended evaluation.

Moreover, we addressed all remaining reviewer questions and strengthened the paper by clarifying why static per-head gates still enable dynamic content compression and transfer across tasks (Appendix H), expanding generation-length and EOS statistics across both 1.7B and 8B at all compression ratios (Appendix I), and experimenting on cross-architecture transfer on Ministral-3-3B. While we acknowledge that soft compaction is not strictly superior on every efficiency metric, our contribution establishes that a strictly more expressive hybrid merge-and-drop paradigm, training of which is tractable by our dual sampling, eliminates the collapse of uniform compactors and matches state-of-the-art hard eviction at matched physical memory. We hope the Area Chair and Reviewers agree on the value of this contribution and reflect these resolved weaknesses in the final evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.