- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
We all know that KV-cache compression aims to preserve generation quality while reducing memory usage and inference overhead as much as possible. This paper questions the practice of using intermediate computations to represent final quality and provides empirical evidence of this issue. In addition, it makes some theoretical contributions.
Critical strengths
The paper has a relatively comprehensive experimental design and does a good job of supporting its core claims.
Critical weaknesses
My main concern is the paper's motivation. In fact, research on K -cache compression does not rely solely on the metric mainly discussed in this paper. On the contrast, it is only proxy metric, and the community has never widely used it or regarded the proxy as perfect. When evaluating a paper that primarily questions existing metrics or common research paradigms, I usually ask myself how the research practices of the entire field would change if everyone accepted the paper's conclusions. However, for this paper, I find that the main takeaway after reading it is that we should not trust local reconstruction errors alone, but should also examine final generation quality, memory usage, and speed. Yet these are things the field is already doing.
Questions for authors
There are some strange things in the appendix that look like commit hashes, as well as some references that directly point to filenames, even though the paper does not include any supplemental materials. In fact, even if supplemental materials were provided, writing things this way would still be somewhat strange. These details should appear in a README instead. If these contents were directly generated by AI (although I do not consider AI-generated content itself a reason for rejection, low-quality content is), I hope the authors can review all AI-assisted work, as they stated in their AI USE STATEMENT.