- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes BELT, a low-rank KV cache compression method to avoid KV cache bottleneck in long-context inference. It propose to compress the K side cache using attention-distribution KL rather than pre-softmax logit reconstruction. Specifically, the value projection is obtained by solving a in a closed form, and is approximated via a second-order Taylor expansion in logit space using the query-dependent softmax curvature, followed by a rank-1 Kronecker factorization and Cholesky decomposition to yield a closed-form solution. Also, to allocate a global rank budget, BELT assigns a learnable score per block and defines component-wise score . At deployment, selection is fixed considering the largest scores ti,j and the budget.
Critical strengths
1. Direct Minimization of an Upper Bound on the Final Attention Output Error
Unlike prior approaches that reconstruct the raw weight or activation matrices (W_K, W_V), BELT directly targets the post-attention output perturbation and cleanly decouples it into an additive theoretical upper bound consisting of the Key error and the Value reconstruction error.
2. Key Compression Accounting for the Shift-Invariance of Softmax
Although the softmax function is invariant to adding a constant shift across all logits, minimizing pre-softmax Euclidean logit error unnecessarily penalizes such harmless uniform shifts. By incorporating the softmax Jacobian as a weighting metric, BELT ignores constant logit shifts and focuses compression capacity strictly on directions where the resulting attention probabilities are truly sensitive.
Critical weaknesses
1. Questionable Tightness of the Upper Bound
While the paper motivates BELT by deriving an upper bound on the post-attention output error in Equation (2), the tightness of this bound is neither theoretically guaranteed nor empirically verified. LLMs are known to exhibit severe activation outliers, a single outlier token with near-zero attention weight can arbitrarily inflate , rendering the term extremely loose. This may explain why the theoretical bound is only used to justify intra-block SVD directions, while cross-block K vs. V rank allocation must be learned from scratch via knowledge distillation.
2. Lack of Component-Wise Ablation Studies
Although BELT combines behavior-aligned K/V decomposition with learned global rank allocation, Section 4 only evaluates the full end-to-end pipeline without isolating each module's contribution. Without step-by-step ablations, such as testing the proposed decomposition bases under uniform ranks or applying the learned allocator to existing bases, it is unclear whether the performance gains stem primarily from the decomposition objective or the rank allocator.
Questions for authors
-
Can the authors empirically evaluate the tightness of the upper bound in Equation (2) on calibration data? Additionally, how sensitive is the worst-case constant to activation outliers in practice, and could the bound be tightened enough to guide cross-block K/V rank allocation without relying entirely on knowledge distillation?
-
To disentangle the contribution of the behavior-aligned decomposition objectives from the learned rank allocator, could the authors provide a step-by-step ablation in the main text? Specifically, how do and perform compared to prior baselines under uniform static rank allocation, and how much does the proposed global rank allocator improve existing bases when applied on top of them?
-
Could the authors clean up and unify the mathematical notations? It seems like and index in Equation (2) are used without definition in the main text, and Equation (4) defines even though holds identically, and dividing by near-zero attention weights risks severe numerical instability.
Response to Reviewer XtUo (Part 5 / 5)
Q5 (W4): Calibration context length and generalization. Since the proposed key-side decomposition relies on attention statistics collected from calibration data, I wonder what context lengths are used during calibration. In particular, given the long-context evaluations (16K RULER), it would be great to see how robust the learned compression bases are across varying evaluation context lengths.
A5: Thank you for raising this point. Basis construction uses 256 WikiText-2 sequences of 2,048 tokens, the same calibration length as the baselines, while rank allocation uses 4,096-token sequences, the same training length as STAR-KV in our experiments.
Following Table 11 in Appendix H.4, we additionally evaluate RULER at a context length of 4K on LLaMA-3.1-8B-Instruct and Qwen3-32B (A3 does not apply to Qwen3-32B, as in Table 11). The results are as follows:
The above results suggest that bases calibrated on 2,048-token sequences remain robust at a context length of 16K, since at 50% compression BELT loses no more accuracy from 4K to 16K than the uncompressed model. Because Algorithm 1 uses the same bases at both compression rates, the larger drop on LLaMA-3.1-8B-Instruct at 70% compression comes from retaining fewer of their components. We will state both calibration lengths in Section 4.1 and add the RULER-4K results to Table 11 in Appendix H.4.
We hope these responses and additional results resolve your concerns, and we respectfully ask if you would be comfortable reconsidering your rating.