- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes an online KV cache compression method for long chain-of-thought reasoning that combines token merging with learned token dropping. Building on STILL, candidate latents are extracted per chunk and pruned per head using learned gates under a target budget. Experiments on Qwen3 models across math and code benchmarks show substantial memory savings over uniform-budget baselines at high compression ratios.
Critical strengths
-
Controlled baseline comparison: Comparing directly against vanilla STILL under identical training data, objective, and backend cleanly isolates the benefit of overcomplete latents and learned per-head gating. The resulting gains at higher compression ratios (12× and 16× on AIME) are clear, and the learned head-wise budget allocations are interpretable.
-
Practical systems-level implementation: Rather than relying purely on theoretical or simulated KV cache reductions, the authors implement paged storage and a custom Triton kernel to realize physical memory savings for variable-length per-head caches. Reporting actual peak KV memory, latency, and throughput makes the empirical evaluation much more practically relevant.
Critical weaknesses
-
Limited and unfair baseline comparison against DMS: The claim of matching SOTA eviction rests on an incomplete comparison with DMS [1]. DMS is evaluated only at 8×, where the proposed method actually is worse on AIME and uses more peak KV memory. More importantly, the comparison is confounded by mismatched token-protection policies as the proposed method preserves the prompt and final answer in full, whereas DMS can evict prompt tokens (protecting only a trailing window). Evaluating DMS under a matched retention policy is needed to isolate the compression mechanism itself. Although Appendix F specifies an adapter-only DMS recipe with a frozen base LLM, its adequacy as a baseline and its training budget should be clarified.
-
Evaluation noise and incomplete length-cap analysis: With only four samples per problem and a single training seed on small benchmarks, the evaluation carries substantial variance, making small reported gains on the math benchmarks hard to distinguish from noise. In addition, the method suffers from a clear practical failure mode as rollouts hit the maximum length cap without emitting EOS far more frequently than DMS (12.5% vs. 3.3% on AIME). Although Appendix I analyzes this behavior for Qwen3-8B at 8×, corresponding completion length and unfinished rate statistics are missing for Qwen3-1.7B and for the 8B model at higher compression ratios (12× and 16×).
Nitpick:
- The straight-through Gumbel-Sigmoid trick at line 210 would benefit from a citiation (Maddison et al., The Concrete Distribution, ICLR 2017; Jang et al., 2017) [2].
[1] https://arxiv.org/abs/2506.05345
[2] https://arxiv.org/abs/1611.00712
Questions for authors
- DMS baseline setup: How many training steps and tokens were used to train the DMS baseline? Could you evaluate DMS at higher compression ratios (12× and 16×), or clarify where the DMS paper reports training collapse beyond 8×?
- Token-protection policies: Could you evaluate DMS under a matched retention policy (preserving the prompt and final answer uncompressed, rather than protecting only a trailing window) to isolate the compression method's gains?
- Statistical variance on math benchmarks: Given the variance with 4 rollouts per problem and a single seed, could you provide paired significance tests against DMS on AIME and HMMT (similar to the paired intervals reported on LiveCodeBench) or evaluate over more rollouts/seeds?
- Length and termination statistics: Could you report completion lengths and unfinished rates for Qwen3-1.7B, as well as for the 8B model at higher compression ratios (12× and 16×)?
- Budget loss formulation: What is the motivation for using a symmetric Huber loss rather than a one-sided penalty (e.g., ) for the budget constraint? Given that realized compression diverges from the targets compression ratio, how can a practitioner reliably hit a fixed memory budget?
- Model diversity: Since all experiments are conducted on Qwen3, could you evaluate on a model from another family (e.g., llama or smollm) to show whether the gains generalize?
6 comments
Response to Reviewer XQ6j (2/3)
Question 2. We emphasize that protecting the prompt is not how DMS was trained or evaluated in its original paper. Using this retention policy during evaluation introduces a distribution shift from training and, in theory, worsens its efficiency by increasing both peak KV memory and total KV reads as unimportant tokens are not evicted. To concretely address your question, we evaluated Qwen3-8B DMS with this protected policy. Protecting the prompt does not change DMS accuracy (−0.83 points, paired 95% CI [−7.50, +6.67]) while increasing peak KV by 7.6% and KV reads by 22.2%. Because prompt protection degrades DMS's efficiency without changing accuracy, we retain DMS’s native evaluation policy in the main results tables.
Question 3. To address the variance issue with 4 rollouts, we conducted a paired uncertainty analysis as requested by bootstrapping matched problem differences. We average the four rollouts within each problem, resample 10,000 times for the 95% CIs, and compute p-values with a paired sign-flip test. As shown in the table below (with the paired CIs also included in Appendix L, Table 14), our method's performance is statistically indistinguishable from DMS across both benchmarks:
| Ours | Benchmark | Difference | Paired 95% CI | p |
|---|---|---|---|---|
| 8× | AIME | −4.2 | [−10.8, +2.5] | 0.341 |
| 8× | HMMT | +2.3 | [−3.8, +9.1] | 0.656 |
| 12× | AIME | −0.8 | [−9.2, +6.7] | 1.000 |
| 12× | HMMT | +0.8 | [−1.5, +3.8] | 1.000 |
This confirms that the small performance differences are well within the margin of error, demonstrating that our method is competitive with state-of-the-art hard eviction.
Question 4. We have expanded our generation length analysis to include the Qwen3-8B model at higher compression ratios (12× and 16×), as well as the Qwen3-1.7B model. The full statistics and plots have been added to the paper (Appendix I, Figures 7 and 8). Curiously, the 1.7B length and EOS statistics are very similar across all models.
As shown in the graphs, the trend of delayed termination and longer responses persists at higher compression ratios for all methods tested. Although the average generation length and the number of unfinished generations increase, Ours remains strictly below Vanilla STILL while having roughly the same statistics as AM. As discussed previously, this behavior is observed across all soft KV compaction methods we tested; however, this cannot be equated with overall performance, as TOVA's generation length and termination statistics are consistently much lower than the other methods while having much lower accuracy, and all 1.7B methods show similar generation length statistics despite large differences in performance. Importantly, we argue that this delayed EOS does not equate to a reasoning failure or a practical failure mode, as our overall accuracy remains competitive with DMS and the model successfully generates the correct answer even if it subsequently struggles to halt.
Question 5. We use a symmetric Huber loss rather than a one-sided penalty because a one-sided penalty stops providing gradients once retention drops below the target. During the early part of training, the mean number of kept tokens dropped rapidly, and the symmetric loss restored gradients to prevent premature eviction and overfitting. This is especially important in our hard-mask design, where gradients only flow through selected tokens. We chose Huber loss because it provides smooth gradients near the target while keeping gradients bounded for large deltas (same as L1 loss), and we use a linear curriculum to control how quickly the target decreases.
Our training uses a surrogate objective over expected soft tokens kept to stabilize optimization and prevent unnecessary constraints on the expressiveness of the model. Furthermore, training adds Gumbel noise to mitigate overfitting, while inference uses the raw gate logits, meaning realized token counts will naturally differ. While it does not guarantee an exact match with the nominal compression ratio, it serves as an effective proxy during training. Previous work (e.g., DMS) also uses this design choice, where the final CR does not match the nominal target CR.
In practice, we do not need an exact nominal ratio to deploy the model: because our gates are static, each trained checkpoint’s retained token count is fixed and fully inspectable before deployment. This also makes it possible to adjust the gate threshold at inference to deterministically control the nominal CR, though in our experiments the performance difference is negligible.
Response to Reviewer XQ6j (3/3)
Question 6. To test generalization beyond Qwen3, we trained both Ours and Vanilla STILL on Ministral3-3B [1], compacting every global-attention KV cache (512-token chunks to targets of 64 and 43 tokens for 8× and 12× CR) using the exact same Open-R1 recipe, data, and seed as Qwen3. While we are still resolving a system issue (which hopefully can be solved before the end of rebuttal) and have not yet completed downstream evaluation, training has completed and the table below reports mean training KL divergence over the final 300 steps:
| Compression | Ours () | Vanilla STILL () |
|---|---|---|
| 8× | 0.0293 | 0.0564 |
| 12× | 0.0389 | 0.0714 |
Across both ratios, Ours' final distillation KL is substantially lower than Vanilla STILL's for both configurations, and Ours at 12× achieves lower KL loss than Vanilla STILL at 8×. This closely mirrors the optimization behavior for Vanilla STILL vs. Ours observed on Qwen3 (Figure 4)—where lower post-anneal loss directly translates into large downstream AIME gains—and provides direct evidence that our overcomplete merging and per-head allocation mechanism transfers to another model family.
[1] Liu, A. H., et al. "Ministral 3." arXiv preprint arXiv:2601.08584 (2026).
Response to Reviewer XQ6j (1/3)
We thank the reviewer for the time and effort in assessing our paper. We appreciate the recognition that our method shows substantive gains over the baseline method, demonstrates practical efficiency gains in both training and inference, and reflects careful effort in its practical system implementation. We appreciate the valuable feedback and have integrated it into our responses below. We have also added the citation you requested. We hope this response addresses your questions and clarifies the strengths of our work.
Weakness 1. We originally evaluated DMS only at 8× because both the DMS paper and NVIDIA Model-Optimizer published configurations up to 8× compression. To address this concern, we clarified DMS’s adapter-only training recipe and trained Qwen3-8B DMS at 12× and 16× ourselves using the exact official Model-Optimizer recipe (detailed in Question 1). As shown in Question 1, Ours and DMS perform at parity when evaluated on AIME and HMMT across all tested CRs.
We also disagree that differing token-protection policies create a confound between methods. DMS is trained end-to-end to evict across the full sequence while retaining a trailing window, whereas Ours is trained to compact only reasoning chunks while preserving prompts. Evaluating DMS (and each method in general) under its native training policy is crucial to extract the best performance and avoid a distribution shift during evaluation; This is why we argue that token-protection policy is a part of the compression mechanism itself. Furthermore, reserving the full prompt imposes higher KV memory overhead on Ours, which we acknowledge as a limitation of our current design. To address this directly, we evaluated DMS with prompt protection forced at inference and show that DMS accuracy does not improve while incurring higher KV overhead (detailed in Question 2).
Weakness 2. Regarding evaluation noise, our four-run averaging (Avg@4) follows the MathArena evaluation protocol (https://matharena.ai). We use this metric to estimate single-run response correctness (similar to expected pass@1) and measure whether model reliability is preserved. While this reduces sampling noise, we acknowledge it does not eliminate uncertainty entirely. To address this, we conducted a paired uncertainty analysis by bootstrapping matched problem differences with 10,000 resamples (95% CIs), with p-values from a paired sign-flip test. In our matched-memory setting (Ours 12× vs. DMS 8×), the difference is −0.83% [−9.17, +6.67] on AIME and +0.76% [−1.52, +3.79] on HMMT, confirming that our performance is on par with DMS. The full significance table is reported in Question 3, and the paired CIs have been added to the revised paper (Appendix L, Table 14).
On the delayed EOS behavior, we agree that this is indeed a limitation of our method and soft KV compaction in general; we observe this behavior across all soft KV compaction methods that we tested. However, we argue that this is not a clear practical failure mode, as despite reaching the generation cap more frequently, our overall accuracy matches DMS and significantly outperforms other baselines—the model still generates the correct answer within the text even if it fails to halt, preserving reasoning capability. Furthermore, we have expanded our generation length analysis to include both the Qwen3-1.7B model and the 8B model at higher compression ratios (12× and 16×). These extra analyses are detailed in Question 4, and the full plots and statistics have been added to Appendix I (Figures 7–8 and Table 8).
Question 1. DMS trains a small adapter on a frozen model, like our method. We follow NVIDIA Model-Optimizer’s Qwen3-8B DMS recipe: 544 steps at effective batch size 16, 8,192-token context, a 512-token window, AdamW at 3×10−5, and a compression ramp over the first 510 steps, on 4,000 OpenR1-Math-220k samples totaling about 71M training tokens. We trained Qwen3-8B 12× and 16× DMS using this same recipe and evaluated DMS on both AIME 26 and HMMT-Feb 26, as detailed below:
Across all six CR–benchmark pairs, Ours and DMS are at parity, confirming that our soft compactor are competitive with state-of-the-art hard eviction. Notably, at 12×, Ours beats DMS on both AIME (61.7 vs. 55.0) and HMMT (34.8 vs. 34.1), and also leads on 8× HMMT (36.4 vs. 34.1), while DMS leads at 16× (59.2/33.3 vs. 56.7/30.3). We thank the reviewer’s comment and thoroughness on exploring the baselines; We retract our earlier Section 4.1 phrasing: the DMS paper reported collapse for their prior learned compression (DMC) work rather than DMS itself (which was evaluated up to 8×), and we have corrected that sentence in the revision.