- Rating
- 1
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes the MILO compression framework, which features a block-wise compression approach that compresses many-shot examples' KV cache at block granularity. The framework dynamically allocates budgets based on entropy to avoid overcompression. Experiments on Qwen2.5 with ICL benchmarks demonstrate that the proposed MILO reduces up to half storage and improves 1.8x efficiency while the accuracy loss is negligible.
Critical strengths
- The motivation of this paper is well-defined. Two challenges have been identified: the compression granularity and the information density within a KV cache. The novelty appears to be appropriately directed toward addressing these issues (block-wise and dynamic rank).
- The presentation of this paper appears to be clear and logical, making it easy to follow and understand.
- The experimental part is clear in showing the performance of the proposed method compared to two baselines and full cache. Each illustration is direct and easy to understand.
Critical weaknesses
I have two critical concerns about this paper. If these can be properly addressed, I would like to recommend accepting the paper.
- The testing model is too small and out of date. For example, Qwen now has the newest Qwen3.8 models released. It is necessary to prove the approaches effective from the new up-to-date architectures. And it is necessary to prove the approach is scalable. And the computing resources are limited, I recommend scaling this test to 20B-level models.
- Only one model series is tested in this paper. This may hinder the demonstration of generalization ability for this approach. Although I saw many KV cache papers testing on one model family, I strongly believe this is not appropriate or rigorous due to the performance stability nature of LLM evaluation.
Questions for authors
I don't have any more questions for the authors. The presentation of the paper is very good. I sincerely hope the experiments can be substantially enhanced.
2 comments
Response to author's rebuttal.
The concerns are not addressed. The experiments are not comprehensive enough. The work is far below the acceptance threshold.
Dear Reviewer 8ccx,
Thank you for your valuable and constructive comments. To address your concerns about the experimental scope, we have added additional experiments on Ministral-3-8B and Ministral-3-14B and extended the number of shots to 1024, with the results summarized below. We have also updated Table 1 and Table 3 in the manuscript.