acceptodds
Under review as a conference paper at ICLR 2027

BELT : BEhavior-Aligned Low-Rank KV Cache Compression with Learned Top-k Rank Allocation

17 views

Abstract

Long-context inference makes the key-value (KV) cache a major memory bottleneck for large language models. We propose BELT, a low-rank KV cache compression framework. Its key-side objective targets attention distributions rather than raw keys or pre-softmax logits, yielding a quadratic surrogate with a closed-form low-rank solution. A learned global rank allocator uses differentiable soft selection during training and hard selection at deployment to meet a user-specified component budget. Experiments demonstrate improved model quality under aggressive compression and increased decoding throughput at comparable accuracy. Evaluated across three LLMs and multiple benchmarks, BELT achieves up to 70% low-rank KV cache compression while outperforming the strongest baseline by up to 23.3 points on LongBench, and up to overall KV cache reduction when combined with quantization. With Triton-based GPU kernels and a quantized cache, BELT delivers up to speedup for the attention module and end-to-end decoding throughput.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
KV-Cache, Low-Rank Compression, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_uUhYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes BELT, a low-rank KV cache compression method to avoid KV cache bottleneck in long-context inference. It propose to compress the K side cache using attention-distribution KL rather than pre-softmax logit reconstruction. Specifically, the value projection W^V\widehat{W}_V is obtained by solving a LV(W^V)\mathcal{L}_V(\widehat{W}_V) in a closed form, and LK(W^K)\mathcal{L}_K(\widehat{W}_K) is approximated via a second-order Taylor expansion in logit space using the query-dependent softmax curvature, followed by a rank-1 Kronecker factorization and Cholesky decomposition to yield a closed-form solution. Also, to allocate a global rank budget, BELT assigns a learnable score per block and defines component-wise score ti,j=si−jt_{i,j} = s_i − j . At deployment, selection is fixed considering the largest scores ti,j and the budget.

Critical strengths

1. Direct Minimization of an Upper Bound on the Final Attention Output Error

Unlike prior approaches that reconstruct the raw weight or activation matrices (W_K, W_V), BELT directly targets the post-attention output perturbation ∥ox−o^x∥22\|o_x - \hat{o}_x\|_2^2 and cleanly decouples it into an additive theoretical upper bound consisting of the Key error and the Value reconstruction error.

2. Key Compression Accounting for the Shift-Invariance of Softmax

Although the softmax function is invariant to adding a constant shift across all logits, minimizing pre-softmax Euclidean logit error unnecessarily penalizes such harmless uniform shifts. By incorporating the softmax Jacobian as a weighting metric, BELT ignores constant logit shifts and focuses compression capacity strictly on directions where the resulting attention probabilities are truly sensitive.

Critical weaknesses

1. Questionable Tightness of the Upper Bound

While the paper motivates BELT by deriving an upper bound on the post-attention output error J(W^K,W^V)≤2LV(W^V)+4M2LK(W^K)\mathcal{J}(\widehat{W}_K, \widehat{W}_V) \le 2\mathcal{L}_V(\widehat{W}_V) + 4M^2\mathcal{L}_K(\widehat{W}_K) in Equation (2), the tightness of this bound is neither theoretically guaranteed nor empirically verified. LLMs are known to exhibit severe activation outliers, a single outlier token with near-zero attention weight can arbitrarily inflate M2M^2, rendering the 4M2LK4M^2\mathcal{L}_K term extremely loose. This may explain why the theoretical bound is only used to justify intra-block SVD directions, while cross-block K vs. V rank allocation must be learned from scratch via knowledge distillation.

2. Lack of Component-Wise Ablation Studies

Although BELT combines behavior-aligned K/V decomposition with learned global rank allocation, Section 4 only evaluates the full end-to-end pipeline without isolating each module's contribution. Without step-by-step ablations, such as testing the proposed decomposition bases under uniform ranks or applying the learned allocator to existing bases, it is unclear whether the performance gains stem primarily from the decomposition objective or the rank allocator.

Questions for authors
  1. Can the authors empirically evaluate the tightness of the upper bound in Equation (2) on calibration data? Additionally, how sensitive is the worst-case constant M2M^2 to activation outliers in practice, and could the bound be tightened enough to guide cross-block K/V rank allocation without relying entirely on knowledge distillation?

  2. To disentangle the contribution of the behavior-aligned decomposition objectives from the learned rank allocator, could the authors provide a step-by-step ablation in the main text? Specifically, how do LK≈\mathcal{L}_K^{\approx} and LV≈\mathcal{L}_V^{\approx} perform compared to prior baselines under uniform static rank allocation, and how much does the proposed global rank allocator improve existing bases when applied on top of them?

  3. Could the authors clean up and unify the mathematical notations? It seems like Z^X,t\widehat{Z}_{X,t} and index tt in Equation (2) are used without definition in the main text, and Equation (4) defines Ax:=Jxdiag(1/ax)JxA_x := J_x diag(1/a_x) J_x even though Jxdiag(1/ax)Jx=JxJ_x diag(1/a_x) J_x = J_x holds identically, and dividing by near-zero attention weights ax≈0a_x \approx 0 risks severe numerical instability.

Reviewer_f3VvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents BELT, a new KV-cache compression method based on low-rank decomposition. Its main contributions are:

  • Deriving an upper bound on the attention-output reconstruction loss that decouples the key and value objectives, making them easier to optimize.
  • Introducing a learned rank allocation strategy that distributes the memory budget across layers and between keys and values, accounting for their different sensitivities to compression. The resulting method achieves strong compression performance while remaining computationally efficient.
Critical strengths
  • The method is mathematically well motivated, with rigorous derivations and theoretical guarantees for the proposed surrogate objectives
  • Strong empirical results, outperforming the evaluated low-rank methods in most experimental settings
  • Performance analysis demonstrates improvements in both memory efficiency and decoding throughput
  • Impressive compression ratios when combined with quantization
Critical weaknesses
  • The fairness of the throughput comparison is questionable, since the proposed method is evaluated at 70% compression while competitors use 50%, justified by approximately matched accuracy. However, accuracy is not consistently matched across benchmarks, making it difficult to isolate the actual efficiency gains. Comparisons at matched compression ratios and batch sizes would be more convincing.
  • The method is primarily compared against other low-rank decomposition approaches. Strong token-eviction methods, such as KVzip [1], are missing from the evaluation. Comparing against or combining with these approaches would help establish its competitiveness.

[1] KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction, Jang-Hyun Kim et al.

Questions for authors
  • Lines 325-338: Can 4K tokens still be considered a long context? It would be helpful to specify the context lengths used for each task. In particular, given LLaMA-2's native 4K context window, how were the RULER-16K and LongBench evaluations conducted when inputs exceeded this limit?

  • Line 372: While 50% compression is a natural choice, the motivation for 70% is less clear. Since accuracy barely drops at 70% in some settings (e.g., Qwen on RULER), it would be interesting to evaluate more aggressive compression ratios.

  • Lines 408-411: Comparing throughput at 70% compression for BELT against 50% for competing methods seems insufficiently justified, especially since accuracy is not consistently comparable. Could the authors provide comparisons at matched compression ratios and batch sizes?

Reviewer_XtUoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper aims to address two limitations of existing KV cache compression methods: (1) focusing on KV reconstruction error rather than preserving actual attention outputs, and (2) inefficient rank allocation across layers and heads. To address these limitations, the authors propose BELT, a low-rank KV cache compression framework. Specifically, they derive quadratic compression surrogates with closed-form weighted-SVD solutions and introduce a learned global rank allocator that enforces a relaxed budget during training and an exact component-count budget at deployment. Extensive experiments across various LLMs and benchmarks demonstrate that BELT substantially outperforms strong baselines while achieving significant speedups in attention computation and decoding throughput.

Critical strengths
  1. The problem and main motivation of the paper, which focus on preserving post-softmax attention distributions and improving rank allocation, are well-motivated and technically sound.

  2. The formulation, including the decomposition objective, quadratic surrogate derivation, and closed-form solution, appears technically sound and theoretically well-justified.

  3. The learnable rank allocation strategy is intuitive and demonstrates strong empirical effectiveness.

  4. The method demonstrates strong performance across various benchmarks at different compression ratios (50% and 70%), while achieving significant speedups in attention computation and decoding throughput.

Critical weaknesses
  1. Limited empirical analysis and ablation studies.
  • The empirical contributions of the proposed attention-aware key decomposition and learned global rank allocation are not sufficiently disentangled. While Table 6 demonstrates the effectiveness of rank allocation with fixed BELT bases, it remains unclear how much of the overall improvement can be attributed to the proposed key-side KL objective. I think a more direct ablation comparing different key decomposition methods (e.g., vanilla SVD, activation-aware SVD, A3-style logit reconstruction, and BELT-KL) under the same rank allocation (either uniform or fixed learned ranks) would be valuable. This would help isolate and better justify the contribution of the proposed KL-based objective.

  • While Table 6 demonstrates the effectiveness of the proposed learned rank allocation, I wonder whether the additional learning process is necessary compared to simpler, training-free alternatives. In particular, since BELT already computes weighted SVD for each block, a natural baseline would be to allocate ranks by globally ranking the weighted singular values under the same component budget. Such a comparison would help clarify the benefits of the learned allocator over simpler allocation strategies.

  1. The proposed closed-form solution relies on two approximations: a second-order Taylor expansion of the KL divergence and a rank-one Kronecker approximation of the resulting quadratic form. While the authors provide an approximation error analysis in Appendix, I wonder how robust these approximations are across different compression ratios, layers, and models.

  2. Although BELT employs closed-form SVD solutions, the overall compression preparation cost remains unclear. In particular, constructing the key-side metric requires computing query-dependent statistics and fitting the Kronecker approximation, while the global rank allocator involves additional optimization. I wonder how much computational overhead these steps introduce compared to existing methods. A breakdown of the preparation time would help clarify the practical efficiency of BELT.

  3. Calibration context length and generalization. Since the proposed key-side decomposition relies on attention statistics collected from calibration data, I wonder what context lengths are used during calibration. In particular, given the long-context evaluations (16K RULER), it would be great to see how robust the learned compression bases are across varying evaluation context lengths

Questions for authors

Wrote above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer XtUo (Part 5 / 5)

Q5 (W4): Calibration context length and generalization. Since the proposed key-side decomposition relies on attention statistics collected from calibration data, I wonder what context lengths are used during calibration. In particular, given the long-context evaluations (16K RULER), it would be great to see how robust the learned compression bases are across varying evaluation context lengths.

A5: Thank you for raising this point. Basis construction uses 256 WikiText-2 sequences of 2,048 tokens, the same calibration length as the baselines, while rank allocation uses 4,096-token sequences, the same training length as STAR-KV in our experiments.

Following Table 11 in Appendix H.4, we additionally evaluate RULER at a context length of 4K on LLaMA-3.1-8B-Instruct and Qwen3-32B (A3 does not apply to Qwen3-32B, as in Table 11). The results are as follows:

ModelComp.MethodRULER-4K Avg ↑RULER-16K Avg ↑
LLaMA-3.1-8B-Inst0%Uncompressed (FullKV)96.8596.23
50%Palu45.2335.68
STAR-KV45.4720.56
ASVD95.1391.92
SVD-LLM94.7892.88
ERC-SVD95.7392.84
ZS-SVD92.9790.86
A36.072.09
BELT (Ours)95.7995.20
70%Palu0.430.75
STAR-KV36.809.99
ASVD49.7925.42
SVD-LLM70.1250.22
ERC-SVD76.5956.36
ZS-SVD5.774.56
A30.500.13
BELT (Ours)87.8874.16
Qwen3-32B0%Uncompressed (FullKV)97.3296.97
50%Palu84.8282.18
STAR-KV86.0070.57
ASVD76.3563.36
SVD-LLM95.6593.36
ERC-SVD95.8494.04
ZS-SVD96.3195.21
BELT (Ours)96.4196.14
70%Palu49.4631.85
STAR-KV61.0345.53
ASVD50.2534.87
SVD-LLM82.8970.44
ERC-SVD88.3881.27
ZS-SVD83.9375.40
BELT (Ours)96.0995.05

The above results suggest that bases calibrated on 2,048-token sequences remain robust at a context length of 16K, since at 50% compression BELT loses no more accuracy from 4K to 16K than the uncompressed model. Because Algorithm 1 uses the same bases at both compression rates, the larger drop on LLaMA-3.1-8B-Instruct at 70% compression comes from retaining fewer of their components. We will state both calibration lengths in Section 4.1 and add the RULER-4K results to Table 11 in Appendix H.4.


We hope these responses and additional results resolve your concerns, and we respectfully ask if you would be comfortable reconsidering your rating.

Reviewer_8vf6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

BELT combines attention-aware low-rank KV compression with learned rank allocation under a global budget. Its key-side objective approximates the KL divergence between original and compressed attention distributions, using a quadratic surrogate and Kronecker factorization to enable weighted-SVD optimization. The method supports RoPE, GQA, and quantization. Experiments cover language modeling, zero-shot accuracy, long-context tasks, and decoding performance.

Critical strengths
  1. Well-motivated objective. Weighting compression errors by softmax curvature better reflects their effect on attention distributions than raw key or logit reconstruction. The weighted-SVD surrogate makes this approach practical.
  2. Strong long-context results. At 70% compression on Qwen3-32B, BELT achieves 40.23 on LongBench versus 16.89 for the strongest reported baseline, and 95.05 on RULER versus 96.97 for the uncompressed model. Quantization and decoding experiments add practical value.
Critical weaknesses
  1. Incomplete contribution attribution. The allocation ablation fixes BELT's bases, establishing the allocator's benefit but not the benefit of the proposed key-side objective. Comparisons against reconstruction and logit-based objectives under the same allocator and training setup are needed.
  2. Gaps in theoretical validation. Table 8 reports aggregate approximation errors without corresponding KL magnitudes or variation across compression rates. The derivations do not clearly account for Qwen3's key normalization. In Appendix I.1, ρ∗∈[0.5,0.72]\rho^*\in[0.5,0.72] gives log⁡((1+ρ∗)/(1−ρ∗))≈1.10\log((1+\rho^*)/(1-\rho^*))\approx1.10–1.821.82, not 0.10.1, weakening the threshold-concentration argument.
Questions for authors
  1. With allocation, calibration, and value-side compression held fixed, how much does the KL-derived key objective improve over key reconstruction and logit reconstruction?
  2. How is Qwen3's key normalization incorporated into the compression metric and implementation?
  3. What settings and normalization underlie Table 8? What are the relative approximation errors at 70% compression?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.