- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies selecting the best KV cache among different compression methods instead of relying on a single compression policy. The authors propose using output-based features, including KL divergence and top-token overlap, to select a compressed cache. The experiments show better results compared to internal-state-based selection on QA tasks, while state-based methods perform better on synthetic retrieval. The evaluation includes multiple models, up to 70B parameters, and long-context inputs.
Critical strengths
-
The idea of selecting among different compressed KV caches based on their outputs is interesting and provides a useful direction for improving KV-cache compression.
-
The paper provides extensive experiments across multiple models and tasks. The comparison between output-based and internal-state-based features uses the same candidates and forward passes, which makes this particular comparison fair.
Critical weaknesses
Please refer to the questions below.
Questions for authors
-
The proposed method requires significantly more computation compared to existing KV-cache compression methods and still needs access to the full KV cache. While the comparison between output and state features is fair, the overall method is not compared under the same computation and peak memory budgets. Could the authors provide such a comparison? For example, would using a larger compressed cache with a single compression method achieve better results with the same computational resources?
-
The paper reports that retained attention mass performs worse than a coin flip, with a 54.38% ranking error. However, reversing the ranking would result in 45.62% error, which is better than random. This suggests that the metric still contains useful information. Have the authors considered reversing or learning the direction of this metric? How would this affect the conclusions?
-
Have the authors compared the proposed learned selector with simply selecting the cache that minimizes KL divergence from the full-cache output distribution? It would be interesting to understand how much improvement comes from learning the selector versus using the output-based metric directly.
Any other concerns or experiments
Dear Reviewers, Area Chairs,
If there are any other concerns or comments, we are happy to clarify and answer specific questions, including any additional concerns you may have after our rebuttal.
Best regards, Authors