acceptodds
Withdrawn from ICLR 2027

KVCompose: Efficient Structured KV Cache Compression with Composite Tokens

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) rely on key-value (KV) caches for efficient autoregressive decoding; however, cache size grows linearly with context length and model depth, becoming a major bottleneck in long-context inference. Prior KV cache compression methods either enforce rigid heuristics, disrupt tensor layouts with per-attention-head variability, or require specialized compute kernels. We propose a simple, yet effective, KV cache compression framework based on attention-guided, layer-adaptive composite tokens. Our method aggregates attention scores to estimate token importance, selects head-specific tokens independently, and aligns them into composite tokens that respect the uniform cache structure required by existing inference engines. A global allocation mechanism further adapts retention budgets across layers, assigning more capacity to layers with informative tokens. This approach achieves significant memory reduction while preserving accuracy, consistently outperforming prior structured and semi-structured methods. Crucially, our approach remains fully compatible with standard inference pipelines, offering a practical and scalable solution for efficient long-context LLM deployment.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV-cache, compression, LLM

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_k8LgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents KVCompose, a KV cache compression framework that combines head-specific token selection with adaptive layer-wise budgeting while preserving regular, dense cache layouts. It ranks cached KV pairs within each head using attention-based importance scores, then groups entries with the same rank across heads into composite tokens. The aggregated composite-token scores determine how a global cache budget is allocated across layers. Experiments on three models demonstrate improved accuracy–compression trade-offs over the evaluated structured baselines.

Critical strengths

The proposed method is simple and achieves promising accuracy–compression results while maintaining dense KV cache layouts. Its adaptive layer-wise budgeting provides a flexible alternative to fixed or predefined allocation schedules.

Critical weaknesses

A major concern is the unquantified overhead of attention-based token scoring. FlashAttention does not return the full attention-weight matrix by default. The authors should clarify how the required attention statistics are obtained and quantify the additional computation and memory costs, particularly for long contexts.

Questions for authors
  1. Despite the emphasis on inference efficiency, the paper does not report end-to-end latency or throughput comparisons with the evaluated baselines. These measurements should include compression overhead and cover both single-query and cache-reuse scenarios.

  2. The evaluation is limited to RULER-4K and RULER-16K, with the 16K experiments covering only one model and lacking comparisons against competing methods. Evaluation at longer context lengths and on more diverse benchmarks, such as LongBench [1] and LongBench v2 [2], would better support the claimed effectiveness on long-context tasks.

  3. The ablation studies are insufficient to establish the contribution of the core design. Although the appendix examines several scoring choices, it does not directly isolate the benefit of composite-token construction. A controlled comparison between shared and head-specific token selection, using the same scoring method and per-layer budgets, would help clarify this contribution.

  4. The presentation of the paper should be improved. The main text extends to page 10 and appears to exceed the initial submission page limit. Section 2.1 could be shortened, as it mainly presents standard attention formulations. In addition, Figure 3 is blurry and should be provided at higher resolution.

  5. The task-aware setting should be clarified. The paper does not clearly explain how the scoring task set is constructed or whether evaluation queries are included. In practical serving, future queries may be unknown when the cache is compressed, so generalization to unseen queries and changes in the task distribution should be evaluated.

References:

[1] LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding. ACL 2024.

[2] LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks. ACL 2025.

1 comment
Reviewer_k8LgOpenReview ↗

Post-Rebuttal Assessment

I did not see a response from the authors, so I am keeping my original rating.

Reviewer_JHpeOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes KVCompose, a post-prefill structured KV cache compression framework designed to bridge the gap between head-wise token selection flexibility and dense tensor regularity. The method operates in three stages: (1) computing token importance scores via attention weight aggregation with Grouped Query Attention (GQA) pooling and cross-head mean augmentation (supporting both task-aware and task-agnostic settings); (2) independently sorting and realigning retained tokens per head by their importance rank to construct Composite Tokens, thereby preserving a uniform rectangular tensor shape (RHkv×Nℓ×dh\mathbb{R}^{H_{\text{kv}} \times N_\ell \times d_h}) within each layer; and (3) dynamically allocating layer-specific token budgets (NℓN_\ell) by selecting the global top-BtotalB_{\text{total}} composite tokens across all layers. Empirical evaluations on the Ruler-4096 benchmark across LLaMA-3.1-8B, Qwen2.5-7B, and Qwen3-14B demonstrate that KVCompose achieves a significantly better accuracy-compression trade-off than existing structured and semi-structured eviction baselines.

Critical strengths

(S1) Clever Structured Formulation via Composite Tokens

By leveraging the head-wise independence and permutation invariance of attention alongside pre-embedded RoPE in key vectors, the proposed "Composite Token" mechanism retains head-specific tokens and layer-adaptive budgets while strictly preserving a dense, rectangular tensor shape (RHkv×Nℓ×dh\mathbb{R}^{H_{\text{kv}} \times N_\ell \times d_h}) per layer without requiring custom sparse attention kernels.

(S2) Strong Empirical Accuracy Retention on Ruler

Across LLaMA-3.1-8B, Qwen2.5-7B, and Qwen3-14B, KVCompose consistently outperforms existing structured and semi-structured baselines—achieving a 79.8% max compression ratio at ϵ0=20%\epsilon_0=20\% (surpassing runner-up TOVA by 18.7%p) and the highest average AUC of 82.3, while its unstructured variant (88.4 AUC) approaches the heavyweight KVzip (89.5 AUC).

Critical weaknesses

(W1) Complete Absence of System-Level Efficiency Metrics

Despite claiming "practical and deployable efficiency" as a core contribution, the paper reports zero wall-clock latency, decoding throughput, or memory benchmarks. Because KVCompose introduces non-trivial prefill overhead—materializing attention weights, head-wise argsort, and tensor gather—quantifying the prefill latency penalty and the multi-query break-even point is essential to validate its practical utility.

(W2) Limited Context Length Evaluation & Missing Baselines at Scale

For a long-context KV cache compression paper, the primary evaluation is restricted to a relatively short 4K context (Ruler-4096). The only longer-context experiment (Ruler-16k in Table 4) evaluates solely Qwen2.5-7B in the task-agnostic setting without any baseline comparisons, and exhibits severe performance collapse at 90% compression (dropping to 23.5%–29.9%).

(W3) Ad-Hoc Scoring Formulation & Architecture-Dependent Sensitivity

The cross-head mean augmentation in Eq. 16 uses a fixed, unweighted 1:1 addition between head-specific scores and the cross-head mean without theoretical justification or sensitivity analysis on a balancing hyperparameter (e.g., (1−α)Shead+αSmean(1-\alpha) S_{\text{head}} + \alpha S_{\text{mean}}). Furthermore, the ablation studies in Appendix A.1.1 and A.1.3 reveal inconsistent behavior across model families—such as Value Norm (∥Vi∥\|V_i\|) weighting improving Qwen models while severely degrading LLaMA-3.1-8B, and optimal aggregation operators (Aggtask\text{Agg}_{\text{task}}) shifting between settings—raising questions about the generalizability of the default scoring recipe.

Questions for authors

(Q1) Given the non-trivial operations introduced by KVCompose (e.g., attention weight materialization, head-wise argsort, and tensor gather), what are the actual wall-clock prefill latency, decoding throughput, and peak GPU memory footprint compared to standard baselines.

(Q2) Computing explicit attention weights for Eq. 14–16 bypasses standard fused kernels like FlashAttention. How does KVCompose mitigate the potential O(N²) memory spikes during the prefill phase, particularly under long-context inputs?

(Q3) To demonstrate KVCompose's competitiveness at scale, can you provide comparisons against established long-context baselines on the Ruler-16k benchmark under identical compression ratios?

(Q4) What is the underlying architectural or representational reason for the performance inversion on Qwen2.5-7B, where the task-aware variant significantly underperforms the task-agnostic counterpart?

(Q5) To justify the unweighted 1:1 formulation in Eq. 16, can you provide an ablation study using a balancing hyperparameter, i.e., Sfinal=(1−α)Shead+αSmeanS_{final} = (1-α) S_{head} + αS_{mean}, to demonstrate how performance scales with different values of α\alpha?

Reviewer_PkzYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes KVCompose, a KV cache compression method combining attention-based importance scoring, head-specific token selection, and adaptive allocation of retention budgets across layers. Its “composite tokens” align selected KV entries from different original positions across heads while maintaining equal retained lengths within each layer. Experiments evaluate three models primarily on RULER-4096, with an additional RULER-16k evaluation for Llama-3.1-8B. The reported results show improved average accuracy–compression trade-offs over the evaluated structured baselines.

Critical strengths
  1. Promising accuracy–compression results across three models. Table 3 reports the highest AUC in five of six model/setting combinations and the strongest overall average, 82.3 versus 73.4 for TOVA. These results support the potential usefulness of the proposed scoring and allocation strategy.

  2. A concrete allocation procedure with a regular cache layout. Equations 17–24 specify how to select head-specific entries while enforcing equal retained lengths within each layer. This provides a plausible approach to balancing flexible selection with dense tensor storage.

Critical weaknesses
  1. Unclear novelty and incomplete ablations. Composite-token construction appears similar to SnapKV’s head-specific selection and gathering. The paper does not clearly establish the distinction or isolate adaptive layer allocation under matched scoring and cache budgets, leaving its central contribution insufficiently validated.

  2. Practical efficiency remains unverified. The paper claims memory and compute savings but reports no compression latency, peak memory, throughput, or end-to-end benchmarks. Attention-scoring costs and their amortization through cache reuse also remain unquantified.

Questions for authors
  1. What operation distinguishes composite-token construction from SnapKV’s head-specific selection and gather? Can a controlled comparison isolate the benefit of adaptive layer budgets while holding scoring and total retained cache size fixed?

  2. How is attention scoring implemented without excessive intermediate storage, and what are its measured overhead and end-to-end benefits on a specified inference engine? How many reused queries are needed to amortize compression?

  3. Which tokens are available when constructing task-aware scores, and do all baselines receive equivalent query information? Please clarify whether scoring uses the evaluated query, separate queries, or answer tokens.

  4. How should the superiority claim in §4.3 be interpreted given Table 2’s task-aware Qwen2.5 results? At 20% tolerance, KVCompose achieves 71.4% compression versus TOVA’s 80.8%; at 10%, it achieves 57.8% versus DuoAttention’s 74.9%.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.