- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes KVRecon, a merging-based KV cache compression method that addresses the memory bottleneck of the KV cache in long-context inference. The key idea is to formulate KV cache merging as a reconstruction problem of the attention output. Specifically, KVRecon groups KV entries into sets and merges each set into a single entry, where both the set assignment and the merged representations account for reconstruction errors between attention quantities computed with the original and merged caches at different stages of the attention computation: , , and . The merged key and its additive bias are derived using a first-order Taylor approximation, while the merged value is obtained by solving a least-squares problem. To prevent inaccurate merging, KVRecon further excludes outlier KV entries with large reconstruction errors from merging. The method is evaluated on LongBench and a shortened version of SCBench using Llama3.1-8B, Qwen2.5-7B, and Mistral-7B, and KVRecon achieves the highest average performance among the compared compression methods.
Critical strengths
1. Consistent gains in the multi-query setting. KVRecon substantially improves performance in multi-query scenarios, where existing eviction- and merging-based KV cache compression methods struggle. Although a gap to the full-cache performance remains, the improvements are meaningful in practical settings where a compressed cache is reused across multiple queries.
2. Simple and interpretable formulation. The additive bias, the merged key/value representations, and the set assignment are derived consistently from a single perspective of attention output reconstruction.
Critical weaknesses
1. Insufficient analysis of compression overhead and missing reproducibility details. I am concerned that the iterative clustering in Algorithm 1 may be computationally expensive, yet the paper does not analyze the time required for compression, leaving its practical applicability unsupported. In addition, although the Reproducibility Statement claims that the details required for reproduction are provided in the main text and appendix, the paper does not specify the number of merging rounds , nor how many queries () are sampled from the repeat-prompt sequence and how they are selected. The final values of and for each model are also not reported, and no code is provided to fill these gaps.
2. Missing comparisons with the most closely related work. KeepKV directly addresses the output perturbation caused by KV cache merging, ReST-KV uses output reconstruction to decide which entries to evict, and CentroidKV, like KVRecon, merges KV entries through set-level clustering. All three methods are discussed in Section 2 but are not included in the experimental comparisons. In particular, KeepKV shares the core objective of preserving the attention output during merging, so without this comparison it is difficult to assess the contribution of reconstruction-based merging.
Questions for authors
Q1. What is the computational complexity of compressing KV entries into sets using queries? What are the actual peak memory usage and compression latency (including the repeat-prompt pass)? In particular, how many decoding steps are required for the speedup from the compressed cache to offset KVRecon’s compression overhead (i.e., the break-even point)?
Q2. KeepKV, ReST-KV, and CentroidKV are discussed in Section 2 but not included as baselines. Why were they excluded? I understand that KeepKV is designed to preserve the output for recent queries, so it may not directly fit the query-agnostic setting. Could you still compare KVRecon with KeepKV under the same cache budget, e.g., by computing its merging weights from the same repeat-prompt queries, or by adding its merging operation to Table 3?
Q3. What fraction of the final cache consists of merged entries, and what is the average set size? In addition, what values of , , , and were used in the experiments, and how were the positions of the queries selected?