- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the two problems when applying residual vector quantization to KV cache compression: (1) the reconstruction error does not imply better prediction, and (2) the residual stages could vary across layers and Key/Value components by their sensitivity to compression. The proposed method, IDEAL-KV, addresses the two problems respectively: post-optimizing the codebook using a composite objective to align attention maps, attention outputs, and continuation distributions, and allocating stage depth by Lagrangian search under a given KV-payload budget.
Critical strengths
- This paper addresses the critical problem of reconstruction-based methods in KV cache compression, and the well-presented experiments show the positive results of the proposed approach.
- The controls in the ablation study are well-designed and demonstrate the performance gain from inference alignment and allocation search. The variants of the score used also strengthen the continuation-KL allocation approach.
Critical weaknesses
- The paper lacks justification for the weight coefficients for the proposed composite objective. The involved targets may have different numerical scales, and it is unclear whether the alignment results are sensitive to the setting. This raises a robustness concern for the approach.
- The effectiveness of inference alignment needs to be further verified. Although the ablation study shows the positive result of Align-Uni control, it is only conducted on one model, Mistral. It is insufficient to demonstrate that the alignment learning is generalizable to other models. Providing the results for Align-Uni control that isolate the allocation search effect in the benchmarking tables (Table 1, Table 2) would be helpful to reveal its impact.
- The motivation for using Lagrangian search to optimize the stage allocation is unclear. Given the assumption that the compression effects can be independently aggregated, the simpler alternative is a greedy search on the ordered score list normalized by cost. There is a lack of justification for the benefit of Lagrangian search in this problem, considering the optimization objective is a surrogate.
Questions for authors
- How is the weight coefficient of each item in the composite objective determined? The targets may have different numerical scales. Are the alignment results sensitive to weight coefficient choices?
- Can you report the results for Align-Uni control in Table1 and Table 2 to clarify the impact of inference-aligned codebooks?
- What is your motivation to use Lagrangian search? Can you clarify the insights and how this allocation problem benefits from it?
6 comments
Response to Reviewer EyHy (2/3)
W2 / Q2 — Align-Uni in Tables 1 and 2, on all models
Thank you for the suggestion. We have extended the evaluation of Align-Uni from Mistral to Qwen2.5-7B and Llama-3.1-8B at both 16× and 24× compression. We have added the Align-Uni results for all three models to Tables 1 and 2 to enable direct comparison of the effects of inference-aligned learning and stage allocation.
Align-Uni and Rec-Uni use the same K/V representation and uniform stage depths, differing only in codebook learning. At 16× and 24×, both use 42 and 28 stages, respectively, for each layer’s Key and Value components. Thus, Rec-Uni → Align-Uni directly evaluates the benefit of inference-aligned codebook learning. With the aligned codebooks fixed, Align-Uni → IDEAL-KV evaluates the additional benefit of calibrated stage allocation. All configurations use the same evaluation protocol and are compared at the same target compression ratios.
The tables below report the mean over eight RULER tasks and the mean over the evaluated LongBench tasks.
RULER results:
| Compression | Method | Qwen2.5-7B | Mistral-7B | Llama-3.1-8B |
|---|---|---|---|---|
| 16× | Rec-Uni | 69.09 | 61.05 | 75.44 |
| 16× | Align-Uni | 70.71 | 64.06 | 77.78 |
| 16× | IDEAL-KV | 73.75 | 66.66 | 78.20 |
| 24× | Rec-Uni | 60.01 | 53.06 | 61.38 |
| 24× | Align-Uni | 61.41 | 55.48 | 67.31 |
| 24× | IDEAL-KV | 66.70 | 60.43 | 72.61 |
LongBench results:
| Compression | Method | Qwen2.5-7B | Mistral-7B | Llama-3.1-8B |
|---|---|---|---|---|
| 16× | Rec-Uni | 49.77 | 48.64 | 50.88 |
| 16× | Align-Uni | 50.84 | 49.07 | 51.48 |
| 16× | IDEAL-KV | 51.00 | 49.44 | 51.61 |
| 24× | Rec-Uni | 48.77 | 47.39 | 49.40 |
| 24× | Align-Uni | 49.20 | 48.05 | 50.37 |
| 24× | IDEAL-KV | 49.92 | 48.33 | 50.95 |
Inference-aligned learning consistently improves performance across all three models. At 16×, Align-Uni improves over Rec-Uni by 1.62, 3.01, and 2.34 points on RULER and 1.07, 0.43, and 0.60 points on LongBench for Qwen, Mistral, and Llama, respectively. At 24×, the corresponding gains are 1.40, 2.42, and 5.93 points on RULER and 0.43, 0.66, and 0.97 points on LongBench. With the representation and uniform stage depths fixed, these results support the effectiveness of inference-aligned learning across the evaluated models and compression ratios.
Stage allocation further improves performance. IDEAL-KV outperforms Align-Uni in every setting above, showing that KL-calibrated allocation provides additional gains on top of the aligned codebooks. Thus, both codebook learning and stage allocation contribute to the improvements of the full method. We thank the reviewer again for the suggestion and have included Align-Uni in the main experimental tables to make these comparisons explicit.
Response to Reviewer EyHy (3/3)
W3 / Q3 — why Lagrangian search rather than a cost-normalised greedy
We thank the reviewer for this suggestion. Our Lagrangian search can also be understood from a greedy allocation perspective: it allocates capacity according to how much KL is reduced per additional bit, using a common threshold to organize these choices.
Specifically, let and denote the calibrated KL score and storage cost of component —the Key or Value cache at a particular layer—at depth . For a given , we select
\left[D_i(s)+\lambda C_i(s)\right].$$ Here, $\lambda$ acts as a common price per bit. If a depth upgrade reduces KL by $\Delta D$ and increases storage by $\Delta C$, it decreases the Lagrangian objective whenever $\Delta D/\Delta C > \lambda$. Thus, this choice follows the same benefit-to-cost criterion as the greedy approach suggested by the reviewer. Geometrically, the Lagrangian solutions as $\lambda$ varies correspond to candidates on the lower convex hull of each component’s KL-versus-bit curve. The switching thresholds between candidates are the corresponding hull edges’ $\Delta D/\Delta C$ ratios. Gradually decreasing $\lambda$ therefore corresponds to accepting upgrades with progressively lower KL reduction per bit. This explains the connection to greedy selection ordered by benefit-to-cost ratio along the lower convex hull. For the actual discrete budget, we then apply local refinement using the same benefit-to-cost criterion to utilize the remaining capacity. We also implemented a direct greedy approach following this principle: at each step, it considers all deeper candidates that fit within the remaining budget and selects the upgrade with the largest KL reduction per additional bit. At **16× compression on all three models**, this greedy approach produces exactly the same final policy as our method. For example, Qwen’s 28 layers involve 56 K/V depth choices, and all 56 choices match between the two methods. Likewise, all 64 choices match for both Mistral and Llama. We chose the Lagrangian formulation primarily because it provides a clear way to express the global budget constraint. A common $\lambda$ turns the budget-coupled allocation problem into independent comparisons over all candidate depths for each component, without requiring diminishing returns between adjacent depths. The same calibrated score table can also be reused for different budgets by adjusting $\lambda$ and performing the corresponding discrete-budget refinement, without remeasuring candidate scores. This makes the formulation straightforward to implement and reuse.
Response to Reviewer EyHy (1/3)
W1 / Q1 — how the weights were determined; scales; sensitivity
We thank the reviewer for raising the questions about loss scales and weight selection. We clarify the normalization of each loss, the rationale behind the default weights, and the corresponding sensitivity experiments below.
Loss scales and weight settings. Continuation KL and attention-map KL are averaged over prediction positions and attention rows, respectively. The attention-output error, K/V reconstruction error, and parameter-anchoring term are normalized by the squared norms of the dense model’s attention outputs, the original K/V vectors, and the initial parameters, respectively. These operations reduce scale differences arising from activation magnitudes and the number of summed terms.
We use the following fixed empirical weights:
\lambda_{\mathrm{map}}=\lambda_{\mathrm{out}}=0.5,\qquad \lambda_{\mathrm{rec}}=0.05,\qquad \lambda_{\mathrm{anchor}}=10^{-4}.$$ This configuration uses continuation KL as the primary objective for predictive consistency, while the two attention terms provide auxiliary supervision for intermediate computations. The reconstruction term maintains K/V reconstruction quality across different stage depths, and the anchoring term limits codebook-parameter drift from initialization. **Weight sensitivity experiments.** On Mistral-7B and Llama-3.1-8B, we jointly halved or doubled the two attention weights while keeping the other loss coefficients unchanged. Within each model, evaluation uses the same fixed KL-based stage-allocation policy, corresponding to approximately 16× compression. We evaluate continuation KL on 70 held-out validation sequences per model, using 64 teacher-forced positions per sequence. The table reports the **percentage change in mean continuation KL relative to the original reconstruction-trained codebooks**; negative values indicate lower KL. | Attention-map / attention-output weights | Mistral | Llama | |:---|---:|---:| | 0.25 / 0.25 | −39.12% | −29.50% | | **0.5 / 0.5 (default)** | **−40.60%** | **−30.76%** | | 1.0 / 1.0 | −40.29% | −29.26% | Jointly halving or doubling the attention weights yields similar improvements in continuation KL: reductions range from **39.12% to 40.60%** on Mistral and from **29.26% to 30.76%** on Llama. These results indicate limited observed sensitivity to the specific values of the two attention coefficients **at this 16× operating point and within the tested joint-scaling range**. Appendix F of the revised paper reports the complete 16× results under both uniform depths and the fixed KL-based policy. We have also clarified the rationale behind the weight settings and the scope of the sensitivity conclusions.