- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents KVCompose, a KV cache compression framework that combines head-specific token selection with adaptive layer-wise budgeting while preserving regular, dense cache layouts. It ranks cached KV pairs within each head using attention-based importance scores, then groups entries with the same rank across heads into composite tokens. The aggregated composite-token scores determine how a global cache budget is allocated across layers. Experiments on three models demonstrate improved accuracy–compression trade-offs over the evaluated structured baselines.
Critical strengths
The proposed method is simple and achieves promising accuracy–compression results while maintaining dense KV cache layouts. Its adaptive layer-wise budgeting provides a flexible alternative to fixed or predefined allocation schedules.
Critical weaknesses
A major concern is the unquantified overhead of attention-based token scoring. FlashAttention does not return the full attention-weight matrix by default. The authors should clarify how the required attention statistics are obtained and quantify the additional computation and memory costs, particularly for long contexts.
Questions for authors
-
Despite the emphasis on inference efficiency, the paper does not report end-to-end latency or throughput comparisons with the evaluated baselines. These measurements should include compression overhead and cover both single-query and cache-reuse scenarios.
-
The evaluation is limited to RULER-4K and RULER-16K, with the 16K experiments covering only one model and lacking comparisons against competing methods. Evaluation at longer context lengths and on more diverse benchmarks, such as LongBench [1] and LongBench v2 [2], would better support the claimed effectiveness on long-context tasks.
-
The ablation studies are insufficient to establish the contribution of the core design. Although the appendix examines several scoring choices, it does not directly isolate the benefit of composite-token construction. A controlled comparison between shared and head-specific token selection, using the same scoring method and per-layer budgets, would help clarify this contribution.
-
The presentation of the paper should be improved. The main text extends to page 10 and appears to exceed the initial submission page limit. Section 2.1 could be shortened, as it mainly presents standard attention formulations. In addition, Figure 3 is blurry and should be provided at higher resolution.
-
The task-aware setting should be clarified. The paper does not clearly explain how the scoring task set is constructed or whether evaluation queries are included. In practical serving, future queries may be unknown when the cache is compressed, so generalization to unseen queries and changes in the task distribution should be evaluated.
References:
[1] LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding. ACL 2024.
[2] LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks. ACL 2025.
Post-Rebuttal Assessment
I did not see a response from the authors, so I am keeping my original rating.