acceptodds
Under review as a conference paper at ICLR 2027

IDEAL-KV: Inference-Aligned Codebook Learning and Calibrated Stage Allocation for KV-Cache Compression

Abstract

Residual-codebook quantization reduces key–value (KV) cache storage, but preserving prediction quality at high compression ratios depends on both the learned codewords and the allocation of residual stages. Reconstruction error alone does not fully characterize the predictive effects of these decisions. We propose IDEAL-KV, which combines inference-aligned codebook learning with continuation-calibrated global allocation. With the language-model backbone frozen, IDEAL-KV learns structured Key and Value codebooks by aligning attention maps, attention outputs, and continuation distributions with the dense model. It then measures the continuation KL induced by each layer–component–depth choice in an otherwise dense cache and uses the resulting additive scores to allocate residual stages under a global KV-payload budget. The selected offline policy retains all historical tokens and supports packed-code attention without materializing the full historical KV cache during decoding. Across three LLM backbones, IDEAL-KV achieves near-BF16 performance on LongBench, RULER-32K, and GSM8K at approximately historical-KV payload compression. At approximately , LongBench and RULER-32K averages remain within 0.67 and 0.80 points of BF16, respectively. Code is available at https://anonymous.4open.science/status/IDEAL-KV-7562.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large language models, KV-cache compression, residual vector quantization, multi-rate codebooks, rate allocation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_EyHyOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the two problems when applying residual vector quantization to KV cache compression: (1) the reconstruction error does not imply better prediction, and (2) the residual stages could vary across layers and Key/Value components by their sensitivity to compression. The proposed method, IDEAL-KV, addresses the two problems respectively: post-optimizing the codebook using a composite objective to align attention maps, attention outputs, and continuation distributions, and allocating stage depth by Lagrangian search under a given KV-payload budget.

Critical strengths
  • This paper addresses the critical problem of reconstruction-based methods in KV cache compression, and the well-presented experiments show the positive results of the proposed approach.
  • The controls in the ablation study are well-designed and demonstrate the performance gain from inference alignment and allocation search. The variants of the score used also strengthen the continuation-KL allocation approach.
Critical weaknesses
  • The paper lacks justification for the weight coefficients for the proposed composite objective. The involved targets may have different numerical scales, and it is unclear whether the alignment results are sensitive to the setting. This raises a robustness concern for the approach.
  • The effectiveness of inference alignment needs to be further verified. Although the ablation study shows the positive result of Align-Uni control, it is only conducted on one model, Mistral. It is insufficient to demonstrate that the alignment learning is generalizable to other models. Providing the results for Align-Uni control that isolate the allocation search effect in the benchmarking tables (Table 1, Table 2) would be helpful to reveal its impact.
  • The motivation for using Lagrangian search to optimize the stage allocation is unclear. Given the assumption that the compression effects can be independently aggregated, the simpler alternative is a greedy search on the ordered score list normalized by cost. There is a lack of justification for the benefit of Lagrangian search in this problem, considering the optimization objective is a surrogate.
Questions for authors
  • How is the weight coefficient of each item in the composite objective determined? The targets may have different numerical scales. Are the alignment results sensitive to weight coefficient choices?
  • Can you report the results for Align-Uni control in Table1 and Table 2 to clarify the impact of inference-aligned codebooks?
  • What is your motivation to use Lagrangian search? Can you clarify the insights and how this allocation problem benefits from it?
6 comments
AuthorsOpenReview ↗

Response to Reviewer EyHy (1/3)

W1 / Q1 — how the weights were determined; scales; sensitivity

We thank the reviewer for raising the questions about loss scales and weight selection. We clarify the normalization of each loss, the rationale behind the default weights, and the corresponding sensitivity experiments below.

Loss scales and weight settings. Continuation KL and attention-map KL are averaged over prediction positions and attention rows, respectively. The attention-output error, K/V reconstruction error, and parameter-anchoring term are normalized by the squared norms of the dense model’s attention outputs, the original K/V vectors, and the initial parameters, respectively. These operations reduce scale differences arising from activation magnitudes and the number of summed terms.

We use the following fixed empirical weights:

\lambda_{\mathrm{map}}=\lambda_{\mathrm{out}}=0.5,\qquad \lambda_{\mathrm{rec}}=0.05,\qquad \lambda_{\mathrm{anchor}}=10^{-4}.$$ This configuration uses continuation KL as the primary objective for predictive consistency, while the two attention terms provide auxiliary supervision for intermediate computations. The reconstruction term maintains K/V reconstruction quality across different stage depths, and the anchoring term limits codebook-parameter drift from initialization. **Weight sensitivity experiments.** On Mistral-7B and Llama-3.1-8B, we jointly halved or doubled the two attention weights while keeping the other loss coefficients unchanged. Within each model, evaluation uses the same fixed KL-based stage-allocation policy, corresponding to approximately 16× compression. We evaluate continuation KL on 70 held-out validation sequences per model, using 64 teacher-forced positions per sequence. The table reports the **percentage change in mean continuation KL relative to the original reconstruction-trained codebooks**; negative values indicate lower KL. | Attention-map / attention-output weights | Mistral | Llama | |:---|---:|---:| | 0.25 / 0.25 | −39.12% | −29.50% | | **0.5 / 0.5 (default)** | **−40.60%** | **−30.76%** | | 1.0 / 1.0 | −40.29% | −29.26% | Jointly halving or doubling the attention weights yields similar improvements in continuation KL: reductions range from **39.12% to 40.60%** on Mistral and from **29.26% to 30.76%** on Llama. These results indicate limited observed sensitivity to the specific values of the two attention coefficients **at this 16× operating point and within the tested joint-scaling range**. Appendix F of the revised paper reports the complete 16× results under both uniform depths and the fixed KL-based policy. We have also clarified the rationale behind the weight settings and the scope of the sensitivity conclusions.
AuthorsOpenReview ↗

Response to Reviewer EyHy (2/3)

W2 / Q2 — Align-Uni in Tables 1 and 2, on all models

Thank you for the suggestion. We have extended the evaluation of Align-Uni from Mistral to Qwen2.5-7B and Llama-3.1-8B at both 16× and 24× compression. We have added the Align-Uni results for all three models to Tables 1 and 2 to enable direct comparison of the effects of inference-aligned learning and stage allocation.

Align-Uni and Rec-Uni use the same K/V representation and uniform stage depths, differing only in codebook learning. At 16× and 24×, both use 42 and 28 stages, respectively, for each layer’s Key and Value components. Thus, Rec-Uni → Align-Uni directly evaluates the benefit of inference-aligned codebook learning. With the aligned codebooks fixed, Align-Uni → IDEAL-KV evaluates the additional benefit of calibrated stage allocation. All configurations use the same evaluation protocol and are compared at the same target compression ratios.

The tables below report the mean over eight RULER tasks and the mean over the evaluated LongBench tasks.

RULER results:

CompressionMethodQwen2.5-7BMistral-7BLlama-3.1-8B
16×Rec-Uni69.0961.0575.44
16×Align-Uni70.7164.0677.78
16×IDEAL-KV73.7566.6678.20
24×Rec-Uni60.0153.0661.38
24×Align-Uni61.4155.4867.31
24×IDEAL-KV66.7060.4372.61

LongBench results:

CompressionMethodQwen2.5-7BMistral-7BLlama-3.1-8B
16×Rec-Uni49.7748.6450.88
16×Align-Uni50.8449.0751.48
16×IDEAL-KV51.0049.4451.61
24×Rec-Uni48.7747.3949.40
24×Align-Uni49.2048.0550.37
24×IDEAL-KV49.9248.3350.95

Inference-aligned learning consistently improves performance across all three models. At 16×, Align-Uni improves over Rec-Uni by 1.62, 3.01, and 2.34 points on RULER and 1.07, 0.43, and 0.60 points on LongBench for Qwen, Mistral, and Llama, respectively. At 24×, the corresponding gains are 1.40, 2.42, and 5.93 points on RULER and 0.43, 0.66, and 0.97 points on LongBench. With the representation and uniform stage depths fixed, these results support the effectiveness of inference-aligned learning across the evaluated models and compression ratios.

Stage allocation further improves performance. IDEAL-KV outperforms Align-Uni in every setting above, showing that KL-calibrated allocation provides additional gains on top of the aligned codebooks. Thus, both codebook learning and stage allocation contribute to the improvements of the full method. We thank the reviewer again for the suggestion and have included Align-Uni in the main experimental tables to make these comparisons explicit.

AuthorsOpenReview ↗

Response to Reviewer EyHy (3/3)

W3 / Q3 — why Lagrangian search rather than a cost-normalised greedy

We thank the reviewer for this suggestion. Our Lagrangian search can also be understood from a greedy allocation perspective: it allocates capacity according to how much KL is reduced per additional bit, using a common threshold to organize these choices.

Specifically, let Di(s)D_i(s) and Ci(s)C_i(s) denote the calibrated KL score and storage cost of component ii—the Key or Value cache at a particular layer—at depth ss. For a given λ\lambda, we select

\left[D_i(s)+\lambda C_i(s)\right].$$ Here, $\lambda$ acts as a common price per bit. If a depth upgrade reduces KL by $\Delta D$ and increases storage by $\Delta C$, it decreases the Lagrangian objective whenever $\Delta D/\Delta C > \lambda$. Thus, this choice follows the same benefit-to-cost criterion as the greedy approach suggested by the reviewer. Geometrically, the Lagrangian solutions as $\lambda$ varies correspond to candidates on the lower convex hull of each component’s KL-versus-bit curve. The switching thresholds between candidates are the corresponding hull edges’ $\Delta D/\Delta C$ ratios. Gradually decreasing $\lambda$ therefore corresponds to accepting upgrades with progressively lower KL reduction per bit. This explains the connection to greedy selection ordered by benefit-to-cost ratio along the lower convex hull. For the actual discrete budget, we then apply local refinement using the same benefit-to-cost criterion to utilize the remaining capacity. We also implemented a direct greedy approach following this principle: at each step, it considers all deeper candidates that fit within the remaining budget and selects the upgrade with the largest KL reduction per additional bit. At **16× compression on all three models**, this greedy approach produces exactly the same final policy as our method. For example, Qwen’s 28 layers involve 56 K/V depth choices, and all 56 choices match between the two methods. Likewise, all 64 choices match for both Mistral and Llama. We chose the Lagrangian formulation primarily because it provides a clear way to express the global budget constraint. A common $\lambda$ turns the budget-coupled allocation problem into independent comparisons over all candidate depths for each component, without requiring diminishing returns between adjacent depths. The same calibrated score table can also be reused for different budgets by adjusting $\lambda$ and performing the corresponding discrete-budget refinement, without remeasuring candidate scores. This makes the formulation straightforward to implement and reuse.
Reviewer_2HeCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper works on KV cache compression with residual codebooks, following the representation of CommVQ. It argues that low reconstruction error of the compressed K and V vectors does not mean the model’s predictions are preserved, because key errors change the attention weights and value errors are then aggregated under those changed weights. Based on this, the paper proposes two things. First, the codebooks are trained with the LLM frozen, using a loss that matches the dense model’s attention maps, attention outputs, and next-token distribution, on top of a reconstruction loss and an anchor loss. Second, the number of residual stages for each layer and each of K and V is chosen by measuring how much the continuation KL goes up when only that one component is compressed, and then solving a budgeted allocation problem over these per-component scores. The selected policy is fixed offline and works with the packed-code attention of CommVQ. Experiments on Qwen2.5-7B, Mistral-7B, and Llama-3.1-8B show results close to BF16 at about 8x and better results than CommVQ and other baselines at about 16x on LongBench, RULER-32K, and GSM8K. There is also an ablation on Mistral and a check that the additive KL score ranks full policies well.

Critical strengths
  • I agree with the basic point of the paper. Reducing reconstruction error alone is not enough, and the objective should look at how the K and V errors move through the attention computation. Eq. 2 makes this concrete.
  • The method keeps the hard residual representation and the compressed-domain attention from CommVQ, so the training and allocation changes do not cost anything at inference.
  • Section 5.5 checks the additive KL surrogate against held-out NLL over 50 equal-cost policies and reports where it stops being able to tell policies apart. I do not see this often.
  • The paper includes Rec-Uni as a control that shares the same K and V representation, instead of comparing only against CommVQ’s original configuration.
  • At 16x, RULER and GSM8K results are consistently better than CommVQ and CommVQ-G512 across all three backbones.
Critical weaknesses
  • The gains are not cleanly attributed to the two stated contributions. On 16x RULER (Table 2), going from CommVQ to CommVQ-G512 gives 3 to 8.5 points, from CommVQ-G512 to Rec-Uni gives 2.5 to 7 points, and from Rec-Uni to IDEAL-KV gives only 1.3 to 2.9 points. The same pattern holds on LongBench. So most of the improvement over CommVQ comes from changing the representation (512-dim groups, 12-bit indices, and a structured Value codebook), which the paper does not present as a contribution and does not analyze. The alignment and allocation add a small margin on top. GSM8K at 16x is the exception, where Rec-Uni to IDEAL-KV is 2.6 to 11 points. The paper should either claim the representation change and study it, or be more careful about what the alignment and allocation actually deliver.
  • The ablation in Table 4 is only for Mistral at 16x. It also does not separate the attention-map term from the attention-output term. Since the attention-output loss already includes the routing error, I cannot tell from the paper whether the attention-map term is needed. There is no sensitivity study for the loss weights either.
  • Several things needed to reproduce the method are missing. I cannot find the maximum trained stage depth R_max, the legal depth set S_{l,c}, or the actual selected policy for any backbone or rate. The only concrete stage count is 42 for Rec-Uni at 16x. Without the selected policy, the reader also cannot see what the allocation learned, for example whether it puts more stages on K or on V, or on which layers.
  • The paper shows that the alignment objective helps, but not how. There is no analysis of what changes in the model after training, such as per-layer attention-map KL or attention-output error before and after.
  • The anonymous repository could not be opened when I checked (Oct 20, 2026).
  • At about 8x the benchmarks look saturated. CommVQ, CommVQ-G512, NOVA-KV, KIVI, and IDEAL-KV are all within about one point of BF16 on LongBench and RULER. This is not a problem with IDEAL-KV, but the 8x tables do not rank the methods. The useful comparison is at 16x, and there the paper has fewer baselines.
  • Only 5 of the 13 RULER tasks are evaluated, and the reason is not given. The aggregation tasks (CWE, FWE) and the UUID-based task (MK3) are not included. The largest 16x drop in the current results is on K2, which is the most retrieval-heavy of the five, so I would expect the harder tasks to show larger gaps.
Questions for authors
  • Can you ablate the attention-map and attention-output terms separately, report sensitivity to the loss weights, and run the ablation on at least one backbone other than Mistral and one rate other than 16x?
  • What are R_max, the legal depth set S_{l,c}, and the selected policy for each backbone and rate? A plot of stages per layer for K and V would be enough.
  • Since the objective targets attention maps and outputs directly, can you show what changes after alignment training, for example per-layer attention-map KL or attention-output error before and after?
  • Why were these 5 RULER tasks chosen? Can you report MK3, CWE, and FWE?
  • Can you report results at 24x or 32x, so that the methods are compared at a rate where they actually separate?
  • Out of curiosity I want to hear the authors’ thoght on how this work differs from Lexico[1] which resembles the CommVQ and IDEAL-KV partially. It also compresses the KV cache with a greedy multi-step procedure over a learned basis, which is close to the residual codebook setup here, although it uses sparse coding with real coefficients rather than a codebook. [1] Kim et al., “Lexico: Extreme KV Cache Compression via Sparse Coding over Universal Dictionaries,” ICML 2025.

If the authors can show what the alignment objective actually changes in the model (for example, per-layer attention-map KL or attention-output error of IDEAL-KV compared with Rec-Uni or CommVQ), and show that IDEAL-KV holds up on MK3, CWE, and FWE of RULER, I am willing to raise my score.

9 comments
AuthorsOpenReview ↗

Response to Reviewer 2HeC (1/6)

W1 — attributing the gains

Thank you for the suggestion on gain attribution. We compare at two levels: first, the configuration differences and performance changes among CommVQ, CommVQ-G512, and Rec-Uni; then, a complete 2×22\times2 ablation with a shared K/V representation to evaluate the contributions of inference-aligned codebook learning and stage allocation separately.

Representation and configuration comparison. CommVQ-G512 retains CommVQ’s Value compression module and changes the Key group size to 512 dimensions. The original configuration uses 256-dimensional groups for Qwen and 128-dimensional groups for Mistral and Llama; both configurations use 12-bit indices per Key stage. Rec-Uni uses our full K/V representation, with nested residual codebooks for both Key and Value, allowing one codebook family to support multiple compression ratios through different stage depths.

Controlled comparisons of the two components. With this shared representation, we compare four configurations:

  • Rec-Uni: reconstruction-trained codebooks + uniform stage allocation;
  • Align-Uni: inference-aligned codebooks + uniform stage allocation;
  • Rec-KL: reconstruction-trained codebooks + KL-calibrated allocation;
  • IDEAL-KV: inference-aligned codebooks + KL-calibrated allocation.

These four configurations share the same K/V representation and evaluation protocol and are compared at the same target compression ratios. At 16× and 24×, uniform allocation uses 42 and 28 stages, respectively, for both Key and Value at every layer. KL-based allocation is calibrated for the corresponding codebooks. The tables report averages over eight RULER tasks and the evaluated LongBench tasks.

RULER results:

CompressionMethodQwen2.5-7BMistral-7BLlama-3.1-8B
16×CommVQ53.2453.2760.89
16×CommVQ-G51264.9357.7765.09
16×Rec-Uni69.0961.0575.44
16×Align-Uni70.7164.0677.78
16×Rec-KL73.5864.2577.24
16×IDEAL-KV73.7566.6678.20
24×CommVQ37.0036.6326.93
24×CommVQ-G51253.1549.8929.08
24×Rec-Uni60.0153.0661.38
24×Align-Uni61.4155.4867.31
24×Rec-KL65.6956.5868.55
24×IDEAL-KV66.7060.4372.61

LongBench results:

CompressionMethodQwen2.5-7BMistral-7BLlama-3.1-8B
16×CommVQ49.1248.4050.34
16×CommVQ-G51249.9748.9150.54
16×Rec-Uni49.7748.6450.88
16×Align-Uni50.8449.0751.48
16×Rec-KL50.8449.1451.30
16×IDEAL-KV51.0049.4451.61
24×CommVQ46.0845.5846.55
24×CommVQ-G51248.3147.6346.28
24×Rec-Uni48.7747.3949.40
24×Align-Uni49.2048.0550.37
24×Rec-KL49.5447.9750.40
24×IDEAL-KV49.9248.3350.95

CommVQ-G512’s configuration change itself provides gains, whose magnitude depends on the model, task, and compression ratio. At 16×, compared with CommVQ, it improves RULER by 11.69, 4.50, and 4.20 points and LongBench by 0.85, 0.51, and 0.20 points on Qwen, Mistral, and Llama, respectively. At 24×, it also improves RULER across all three models.

Both proposed components contribute under the shared representation. With the representation and uniform stage depths fixed, Rec-Uni → Align-Uni improves performance across all three models, both benchmarks, and both compression ratios, supporting the effectiveness of inference-aligned learning. With the aligned codebooks fixed, Align-Uni → IDEAL-KV further improves performance, demonstrating additional benefits from calibrated allocation. Rec-Uni → Rec-KL also shows that allocation benefits reconstruction-trained codebooks. Relative to Rec-Uni, the two components together improve RULER by 4.66, 5.61, and 2.76 points and LongBench by 1.23, 0.80, and 0.73 points at 16× on Qwen, Mistral, and Llama, respectively. These gains are obtained with the same representation.

The benefits persist at higher compression. At 24×, full IDEAL-KV improves over Rec-Uni by 6.69, 7.37, and 11.23 points on RULER and 1.15, 0.94, and 1.55 points on LongBench. Thus, both components remain beneficial at higher compression, with larger combined gains on RULER.

We have clarified the representation design and the two methodological contributions separately in the revised manuscript, and added these configuration comparisons and representation-matched ablations to make the sources of improvement explicit.

AuthorsOpenReview ↗

Response to Reviewer 2HeC (2/6)

W2 / Q1 — separate ablation of the attention-map and attention-output terms; weight sensitivity; another backbone and another rate

We thank the reviewer for the thoughtful suggestions. We have added loss-term and weight ablations on Mistral-7B and Llama-3.1-8B to clarify the role of attention-map supervision and examine sensitivity to the loss weights.

Experimental setup. All variants retain the continuation-KL, reconstruction, and anchoring terms, varying only the weights of the two attention-related losses. Within each backbone, all variants use the same fixed KL-based stage-allocation policy at approximately 16× compression, isolating the effect of the training objective. We evaluate continuation KL on 70 held-out validation sequences per model, using 64 teacher-forced positions per sequence. The table reports the percentage change in mean continuation KL relative to the original reconstruction codebooks; lower is better.

Training objective (map / output weights)MistralLlama
(a) No attention terms (0 / 0)−35.09%−21.58%
(b) Output only (0 / 0.5)−38.35%−28.64%
(c) Both, default (0.5 / 0.5)−40.60%−30.76%
(d) Both, halved weights (0.25 / 0.25)−39.12%−29.50%
(e) Both, doubled weights (1.0 / 1.0)−40.29%−29.26%

Role of the attention-map term. We agree with the reviewer that attention-output error already reflects the effect of routing errors on the aggregated output. The attention-map term directly supervises the routing weights, providing an auxiliary optimization signal. In the observed training trajectories, the full objective first reaches the output-only variant’s 1,200-step continuation KL after 700/800 steps on Mistral/Llama, respectively. Thus, although its endpoint gains are modest in these runs, we retain the map term to help reach comparable validation quality in fewer training updates.

Joint contribution of the two attention terms. Compared with removing both attention terms in (a), the default objective (c) further reduces continuation KL by 8.49%/11.71% on Mistral/Llama. These results support the contribution of attention-level auxiliary supervision to preserving the final predictive distribution, even when continuation-KL supervision is already present.

Weight sensitivity. Jointly halving or doubling the two attention weights changes continuation KL by only +0.52% to +2.49% relative to the default. This indicates limited observed sensitivity to the precise weights at the tested 16× operating point and within this joint weight sweep.

AuthorsOpenReview ↗

Response to Reviewer 2HeC (3/6)

W3 / Q2 — Rmax⁡R_{\max}, legal depth set, selected policies

Thank you for pointing out these missing details needed for reproducibility. We clarify the codebook depth limit, legal depth set, and selected stage-allocation policies below.

Codebook depth limit. The codebook depth used in our experiments and the stage-allocation depth are both capped at 200. Each selected depth specifies the number of residual stages retained for a layer’s Key or Value component. Each stage uses one 12-bit index per 512-dimensional group.

Legal depth set. We use the same candidate depth set for every layer and for both Key and Value:

\lbrace 22,24,\ldots,60\rbrace \cup \lbrace 64,68,\ldots,120\rbrace \cup \lbrace 128,136,\ldots,200\rbrace.$$ This set contains 45 candidate depths. The candidate table therefore contains $32\times2\times45=2{,}880$ entries for Mistral and Llama, and $28\times2\times45=2{,}520$ entries for Qwen. **Selected policies.** The table below summarizes the mean and range of the selected Key and Value depths across layers for each backbone at 8×, 16×, and the additional 24× compression setting. | Backbone | Compression | Mean Key depth | Mean Value depth | Key range | Value range | |---|---:|---:|---:|---:|---:| | Qwen2.5-7B | 8× | 78.9 | 90.3 | 30–128 | 30–176 | | Qwen2.5-7B | 16× | 40.9 | 42.9 | 22–76 | 22–76 | | Qwen2.5-7B | 24× | 25.8 | 29.6 | 22–44 | 22–56 | | Mistral-7B | 8× | 80.0 | 89.8 | 22–128 | 26–136 | | Mistral-7B | 16× | 38.7 | 45.8 | 22–68 | 22–76 | | Mistral-7B | 24× | 26.4 | 29.6 | 22–46 | 22–54 | | Llama-3.1-8B | 8× | 82.7 | 87.1 | 28–112 | 56–144 | | Llama-3.1-8B | 16× | 39.6 | 44.9 | 22–64 | 22–68 | | Llama-3.1-8B | 24× | 27.2 | 28.8 | 22–44 | 22–50 | These results reveal two allocation patterns. First, **Value receives more stages on average than Key** in all three models at each reported compression ratio. This preference does not hold for every layer: at 16×, Value receives more stages than Key in 14/28 Qwen layers, 24/32 Mistral layers, and 19/32 Llama layers. Second, **the allocation varies substantially across layers, and the layers receiving the greatest depths depend on the backbone and K/V component**. For example, at 16×, Mistral assigns its maximum Key and Value depths to layers 12 and 15, respectively, while Qwen assigns them to layers 14 and 19. Llama assigns its maximum Value depth to layer 13 but its maximum Key depth to layer 30. All layer indices are zero-based. Following the reviewer’s suggestion, we have added these specifications, complete per-layer Key/Value stage plots for each backbone and compression ratio, and the corresponding analysis to the revised manuscript. The selected policies are available as JSON files in the policies/ directory of the anonymous repository.
Reviewer_wXirOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes IDEAL-KV, which improves residual-codebook KV cache compression through inference-aligned codebook learning and budget-aware stage allocation. Building on the structured representation of CommVQ, the authors train the KV codebooks by aligning attention maps, attention outputs, and continuation distributions with the uncompressed model, while keeping the model weights frozen. They then measure the continuation KL caused by compressing each individual layer’s Key or Value cache and use these measurements to allocate residual stages under a global storage budget. The authors also implement efficient fused kernels to compute attention directly from the compressed cache. Experiments on three models show near-BF16 performance at approximately 8× historical-KV compression and improved accuracy over the evaluated baselines at approximately 16× compression.

Critical strengths
  1. The proposed method is natural and makes sense. Reconstruction error alone does not directly reflect how KV compression affects attention and final predictions. The authors address this issue in both codebook learning and stage allocation. The ablation results show that both the codebook training and stage allocation improve accuracy.

  2. The authors implement efficient kernels to convert the KV cache compression savings into actual decoding benefits. The comparison with CommVQ using the same kernels helps separate the gains from the implementation and the method itself. This makes both the engineering and algorithm contributions more convincing.

Critical weaknesses
  1. The ablation study is limited to the Mistral model. Since different models may respond differently to codebook learning and stage allocation, ablations on more models would help verify whether the benefits of both components are consistent.

  2. The efficiency evaluation is limited to batch size 1 on a B200 GPU. Throughput results at larger batch sizes and comparisons with methods such as KIVI would help clarify the practical efficiency benefits.

Questions for authors
  1. Can the authors provide ablation results on other models to verify whether both inference-aligned codebook learning and KL-based stage allocation consistently improve performance?

  2. How does IDEAL-KV compare with CommVQ and KIVI in decoding throughput at larger batch sizes?

2 comments
AuthorsOpenReview ↗

Response to Reviewer wXir

W1 / Q1 — ablations on other models

We thank the reviewer for suggesting additional cross-model validation. We have extended the original Mistral ablations to Qwen2.5-7B and Llama-3.1-8B. Following Reviewer 2HeC’s suggestion, we also added three challenging tasks beyond the five RULER tasks in the submission: UUID-based multi-key retrieval (MK3) and the aggregation tasks CWE and FWE. The RULER results below report the equal-weight mean over these eight tasks at 32K.

Since IDEAL-KV aims to preserve prediction quality at high KV-cache compression ratios, we conducted additional ablations at approximately 24×, alongside 16× historical-KV payload compression. At each rate, we compare the four variants defined in Table 4: Rec-Uni (reconstruction-trained codebooks + uniform allocation), Align-Uni (inference-aligned codebooks + uniform allocation), Rec-KL (reconstruction-trained codebooks + KL-based allocation), and IDEAL-KV (inference-aligned codebooks + KL-based allocation).

All variants share the K/V representation and are compared at matched target compression ratios. Uniform allocation uses 42/28 residual stages at 16×/24×, respectively, and KL-based allocation is calibrated on the corresponding codebooks.

RULER

CompressionMethodQwenMistralLlama
16×Rec-Uni69.0961.0575.44
16×Align-Uni70.7164.0677.78
16×Rec-KL73.5864.2577.24
16×IDEAL-KV73.7566.6678.20
24×Rec-Uni60.0153.0661.38
24×Align-Uni61.4155.4867.31
24×Rec-KL65.6956.5868.55
24×IDEAL-KV66.7060.4372.61

LongBench

CompressionMethodQwenMistralLlama
16×Rec-Uni49.7748.6450.88
16×Align-Uni50.8449.0751.48
16×Rec-KL50.8449.1451.30
16×IDEAL-KV51.0049.4451.61
24×Rec-Uni48.7747.3949.40
24×Align-Uni49.2048.0550.37
24×Rec-KL49.5447.9750.40
24×IDEAL-KV49.9248.3350.95

These results support the contributions of both components:

  • Codebook learning is effective: with uniform stage depths fixed, Align-Uni outperforms Rec-Uni on both benchmarks across all three models and both compression ratios. For example, at 16×, alignment improves RULER by 1.62/3.01/2.34 points on Qwen/Mistral/Llama.
  • Stage allocation provides additional gains: with the aligned codebooks fixed, IDEAL-KV outperforms Align-Uni in every evaluated setting, adding 3.04/2.60/0.42 RULER points at 16×. Allocation also benefits reconstruction-trained codebooks: Rec-KL consistently outperforms Rec-Uni.
  • The benefits persist at higher compression: the combined RULER gain over Rec-Uni increases from 4.66/5.61/2.76 points at 16× to 6.69/7.37/11.23 points at 24×. The combined improvement is therefore larger at 24× on all three models.

These findings support the effectiveness of both inference-aligned codebook learning and KL-based stage allocation across the three evaluated backbones and both high-compression settings. We have added these results and their analysis to Section 5.3 and Table 5, while retaining the original ablations in Table 4.


W2 / Q2 — efficiency beyond batch 1; KIVI and CommVQ

Thank you for the suggestion. We have added decoding throughput comparisons at a larger batch size. The efficiency experiments in the original submission used a B200 GPU. Since we no longer have access to that GPU, we remeasured all methods on an RTX PRO 6000 (Blackwell, 96 GB) using a consistent measurement protocol. All experiments use Mistral-7B-Instruct-v0.3 with a 32K context length. CommVQ uses our optimized kernels. For batch size 1, we report the median per-token decoding latency over five runs; for batch size 16, we report the measured decoding throughput from a complete run.

MethodBatch size 1: latency (ms/token) ↓Batch size 16: throughput (tokens/s) ↑
BF1644.5429.1
IDEAL-KV26.18103.3
CommVQ27.8878.4
KIVI25.7698.1

At batch size 1, IDEAL-KV achieves decoding latency comparable to KIVI. At batch size 16, IDEAL-KV achieves 5.3% higher throughput than KIVI. We included these results in the revised manuscript.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer wXir,

Thank you again for your positive and constructive review. As the discussion period ends soon, we would like to briefly follow up on our response:

  • Ablations on other models (W1/Q1): we extended the ablations to Qwen2.5-7B and Llama-3.1-8B at both 16× and 24×. Inference-aligned codebook learning and KL-based allocation each improve RULER-8 and LongBench on all three backbones at both rates.
  • Efficiency beyond batch size 1 (W2/Q2): at batch size 16, IDEAL-KV reaches 103.3 tokens/s, 5.3% higher than KIVI.

We would be grateful to know whether these results address your concerns, and we are happy to clarify anything further before the discussion closes.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.