- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a KV Cache compression method with only lightweight training. The implementation shows that the proposed method achieves state-of-the-art results compared with other methods, with and without training, and also shows the applicability of the method under different compression ratios.
Critical strengths
- The proposed method only requires lightweight training, which is helpful for the huge training overhead of current large language models, and also increases the practicality of the method.
- This paper shows the effectiveness of the method, but more experiments are needed to further support it.
Critical weaknesses
- The paper does not report inference speed or memory-saving improvements, which are essential for evaluating the practical benefits of KV Cache compression.
- The comparison method is insufficient. In Section 2, the paper mainly discusses works from 2023 to 2024, without covering more advanced methods or conducting comparisons from the past year. This makes it difficult to highlight the innovation of the methods.
- Evaluating only on Qwen2.5-1.5B is insufficient. Further discussion is needed on performance across different architectures, larger parameter sizes, and the more advanced Qwen3 series to highlight the applicability of the method.
- The overall quality of the paper is not high. There is much room for improvement in both the writing and layout. Section 3.1 seems to belong in the experimental section.
- Further clarification is needed on the contribution of the proposed method, as well as the differences from previous methods.
- Authors need to recognize the difference between /cite and /citep in LaTeX writing.
Questions for authors
- Could the author be able to present the actual curve records of the training process in the next version? This would be beneficial for enhancing the reliability of the results.
- Can the proposed method be evaluated on more datasets or tasks?
- Based on the results presented and the overall quality of the writing, I remain skeptical about the reliability of the method. I hope the author can supplement the results with more experimental data; otherwise, I will still consider further lowering the score.
2 comments
Thanks to the author for the last-minute reply.
Although the author's reply can address some of my concerns, I think the overall quality of the paper is not up to the acceptance standard, and KV cache compression needs a more efficient comparison to highlight the effect. The results shown in Table 6 are not convincing to me, and the corresponding description is not readable.
We hope that in future submissions and Rebuttal processes, authors can mark the key parts of the revision in blue or other colors.
At the same time, the author can pay more attention to the writing quality in the future. The planning and layout of Figure 2 can be further modified to make the information density higher and the comparison more intuitive. Other typography and readability aspects have room for improvement.
In summary, we have added a new model and task, R-KV as a baseline, inference and training cost measurements, training curves, an explicit contributions list, and corrected the formatting.
We thank the reviewer for the constructive feedback. Below, we address each point in turn, including new experiments on additional models, tasks, and baselines, inference and training cost measurements, training curves, and revisions to the writing.
Models. We repeat the full pipeline on Phi-4-mini-instruct (3.8B), a model from a different family and at a larger scale (Section 4.3, Table 3). We train a new teacher and distill both BR and our method from the same teacher checkpoint and trajectories. Our method matches or exceeds BR at every compression ratio on both LinSys and StarGraph, with the clearest gains on LinSys at higher compression (0.297 vs. 0.172 at 8×).
Tasks. We add MATH (Hendrycks et al., 2021), an open-ended mathematical reasoning benchmark that is more diverse than our synthetic tasks (Section 4.3, Table 3; setup in Appendix A.1). Our method outperforms BR at four of five compression ratios, with the largest margin at 2× (0.426 vs. 0.340). Together with LinSys and StarGraph, our evaluation now covers structured algebraic deduction, graph search, and open-ended mathematical problem solving.
Comparisons. We have added R-KV (Cai et al., 2025), a recent KV-cache compression method designed specifically for reasoning models, as a training-free baseline (Table 2). R-KV is the strongest training-free baseline, matching our method on LinSys at 2× (0.926), but it collapses beyond 4× (0.086 at 8×) and remains well below the adapted policies on StarGraph (0.598 vs. 0.820 at 2×). We have also expanded Section 2 to cover recent reasoning-specific KV compression methods, including R-KV and ThinKV (Ramachandran et al., 2026). ThinKV has no publicly available implementation at the time of writing, so we could not include it as a baseline.
Training curves. We thank the reviewer for this suggestion. We have added the distillation loss curves for both BR and our method across all compression ratios on LinSys and StarGraph (Appendix A.3). Both methods converge at every ratio, with final loss increasing with the compression ratio, and our method reaches lower final loss, most markedly on LinSys. At higher ratios (c ≥ 8), our method shows transient loss spikes from which training recovers.
Inference speed and memory. We thank the reviewer for this comment. Our original submission reported KV-cache memory savings in Section 4.2 (KV-cache memory scaling): for example, at c = 4 on LinSys, our method reduces the generated cache by 3.99× and the total cache by 3.56×. We have now added measured decoding latency and peak memory for the uncompressed teacher, BR, and our method (Appendix A.4, Table 6). At each window boundary, both compressed methods incur one additional forward step, but the shorter cache makes every other decoding step cheaper. At high compression, this saving dominates: at c = 32, our method decodes at 70.4 tokens/s and BR at 72.3 tokens/s, compared with 66.4 tokens/s for the uncompressed teacher. At lower compression ratios, the boundary overhead outweighs this saving. At the model size and sequence lengths we evaluate, peak memory is dominated by model weights, since the KV cache occupies only about 29 MB per 1,000 tokens. The memory benefit of compression therefore grows with generation length and batch size, where the cache dominates memory use.
Contributions. We thank the reviewer for this comment. Prior learned approaches to reasoning-time KV compression, such as Breadcrumbs Reasoning, train a dedicated module to produce each compressed summary. We ask whether this learned aggregation is necessary. Our method replaces each window of generated KV entries with their mean and adapts the model with lightweight LoRA, adding no new tokens or auxiliary compression parameters. Under a controlled comparison, it matches or outperforms BR (Tables 1–2), and this generally holds across tasks and models (Table 3). Ablations show that both the aggregation rule and adaptation are necessary (Section 4.4).
Writing and formatting. We thank the reviewer for bringing this to our attention. We have corrected the citation formatting and reorganized the paper’s structure.