- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work finds that the reliance on head-specific token importance gives rise to two potential limitations: 1) individual attention heads capture part of the token importance; 2) close attention tokens lead to semantic omission while overlooking other crucial information. This work proposes CoCoKV, which is a novel KV cache compression framework. The experiment evaluates the method with NIAH benchmarks and 16 LongBench datasets.
Critical strengths
- This work proposes a cross-head consensus module that calibrates head-specific attention with collective evidence to improve importance reliability. This considers cross-head information.
- This work proposes a cross-token completeness module that selects informative tokens with broader semantic coverage. This could expand the semantic coverage.
- The method is easy to understand, and the paper is well-written.
- This work conducts experiments to support that the proposed method is useful.
Critical weaknesses
- The author could provide code to help reproduce the results.
- For Figure 6, the author should provide the full attention method to provide the baselines performance
- What is performance on math dataset, such as MATH datasets?
Questions for authors
N/A
3 comments
I will keep the positive score for this work. The author resonse addresses the major concerns.
We are glad that our response has addressed your questions. Thanks again for your valuable suggestions.
Thank you again for your constructive comments and positive assessment of our work.
As the discussion period is approaching its deadline, we would like to kindly follow up and ask whether our rebuttal has adequately addressed your concerns. We would be happy to provide any further clarifications or engage in additional discussion if needed.