acceptodds
Under review as a conference paper at ICLR 2027

KV-Kaizen: Learning Context-Adaptive Cache Compression Choices

Abstract

As the context size of text processed with an LLM grows, the size of KV caches can outstrip the memory allocated for the original model weights. This impacts LLM throughput negatively, since decoding is memory-bound and decode cost grows with cache size. Recent work has proposed to alleviate this bottleneck by discarding some of these tokens, keeping the most relevant bits and dropping the rest. *Eviction* introduces a tension, since a one-off decision to discard content may prove detrimental later. Instead, we focus on alternative choices that can lead to cache compression **without** evicting tokens. We achieve this by learning a selector that is able to produce, based on context, a per-layer cache configuration towards an overall compression budget. The selector operates along three axes: sharing one cache across layers (*depth*), caching at fewer bits (*precision*), or truncating the low-rank latent cache representations (*rank*). We call the resulting method KV-Kaizen, for the many small per-layer choices it compounds. We observe that these interventions taken independently and uniformly over all layers limit achievable compression because they degrade accuracy. Crucially, we observe that composing them *locally* and *adaptively* to the context can instead preserve accuracy while achieving large memory savings. At train time, a lightweight selector module is given access to the embeddings of the context sequence, and trained to pick *at each layer* a combination of these three axes, with a language modeling loss that is regularized by a budget constraint satisfaction loss. At inference, the selector runs once, before pre-fill. In evaluations on instruction following and reasoning tasks, our selectors reach the Pareto frontier of accuracy against cache size, against learning-free and *post-hoc* baselines. On long-context tasks, KV-Kaizen improves on eviction and can be composed with it, reaching a smaller decode-time cache on a B model while preserving accuracy. A cache size reduction incurs no accuracy degradation from B parameters up, and a compressed model is more accurate than a smaller uncompressed one with the same cache size. Together, these findings support pre-training large models and compressing them only afterwards.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Efficient inference, KV cache compression

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_5RHeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

KV-Kaizen formulates KV-cache compression as a per-layer allocation problem over quantization precision, latent rank, and cross-layer sharing under a common memory budget. The method jointly fine-tunes the backbone and a lightweight prompt-conditioned selector, which chooses a configuration before prefill and keeps it fixed throughout generation. The paper evaluates accuracy–compression trade-offs across multiple model families and tasks, and explores combining representation compression with token eviction.

Critical strengths
  1. Comprehensive evaluation across different tasks and models.
  2. Promising quality–memory trade-offs in several settings.
  3. Low overhead for the selection step itself. The selector operates on prompt embeddings and runs only once before prefill.
Critical weaknesses
  1. The incremental benefit over strong fixed-bit quantization is not sufficiently established, particularly around 4× compression.
  2. The necessity of prompt-conditioned selection remains unclear. Table 20 reports 224 prompts of different areas shares the same configuration. A comparison between static tasks-aware per-layer selection is needed to prove the necessity of selector.
  3. The practical benefits under realistic serving workloads are unvalidated. If the effectiveness of prompt adaptiveness is proved, how to batch different requests will be a issue, where the benefit can be covered by the overheads of processing distinct quantization/sharing/SVD patterns.
Questions for authors

See weaknesses.

3 comments

We thank the reviewer for their careful reading and feedback.

W1. Benefit over fixed-bit quantization at 4x. We do not claim to dominate every point of the Pareto front, and at 4x, precision alone is already strong as the reviewer pointed out. Indeed, the selector finds this and uses precision. The method targets the range fixed bits cannot reach:

  • two-bit quantization stops at 8x;
  • KV-Kaizen reaches 16x, and 32x with eviction (Table 13).

We also note that, at matched 4x, the learned configuration runs as fast as uniform 4-bit on device, so learning the allocation costs nothing at inference.

W2. Necessity of prompt-conditioned selection. We believe Table 20 may have been read as evidence against the selector. It instead shows that some selectors settle on one configuration, which the paper reports and which lets the selector be dropped at test time (Section 3.6). Other selectors vary, and the variation matters. For instance, on 7B precision + depth at 8x, replacing the selector by its most common configuration costs ~4% accuracy on RULER-16k at the same budget.

We ran the static task-aware baseline the reviewer suggests, using for each task the configuration the selector chose most often on it. It recovers most of the gap: 0.447 on RULER-16k against 0.460 for the selector and 0.414 for a single configuration. However, a task-aware configuration needs the task label at inference. The trained selector reaches the same accuracy from the prompt alone, in 3 ms.

W3. Batching under serving workloads. Our main target is on-device inference, where batch size is 1, which is the setting of Section 4.

For batched serving, the configurations one selector chooses between cost the same within 3% in our measurements, so batching them does not mix costs. When a selector settles on one configuration, every request shares it.

We also measured batched serving on an H100 (general response 3). At a fixed memory budget, the learned configurations fit batches that grow with their compression, and aggregate decode throughput reaches 1.38x to 2.07x of bf16 for precision and precision + depth configurations, and 1.46x to 1.84x for the 14B all-three 16x ones.

We hope these answers and the revised manuscript address your concerns. We are also happy to continue the discussion and provide further details or clarification.

Thanks for the response.

  1. For W2, I was asking for training the per-layer quantized model based on task without the selector, instead of using the most frequently used config. But I recognize the reasoning here that selector can choose the right prompt-conditioned quantization configuration with one time effort.
  2. It is still unclear how you batched requests with different input/output length and different compression selection per layer into the same batch, and how you manage the storage of KVCache with different quantization configurations. My understanding of the added experiments is that you choose the same config for the whole batch, and some "all three" settings essentially adding more overheads despite the higher compression rate. Despite that, the baseline is BF16 but your accuracy ties on 4x compression baseline, so I think this baseline is too weak and a direct comparison with 4-bit makes more sense.
  3. Single request serving on the device cannot explain your motivation. A 14B model takes approximately 12-16GB memory for bf16 KV cache at 64K context length (about the max length in your experiments), contributing to 3-4GB on 4-bit compression. This size is hard to support your argument of necessity for further compression, not to mention the latency overheads during decoding, and the memory-bound of weight loading. Based on these, I would like to keep my scoring.

Thank you for engaging with our response and for the follow-up.

1. Per-task configurations. We are glad the reviewer agrees that a selector trained once can choose the configuration from the prompt, without training a model per task.

2. Batching, storage and the 4-bit baseline. We clarify that our target is on-device inference at batch size 1, where the selector gives each request its own configuration at a cost of about 3 ms. We note that the newly added Appendix H is a side experiment, to check that the savings also carry over to batched serving. There, as the reviewer correctly understood, each measurement batches one configuration. For input-dependent selectors (rows marked † in Table 22), each of their configurations is measured as its own batch and weighted by how often it is picked, which amounts to grouping requests by configuration. This keeps prompt dependence. Fixing one configuration per batch is simpler, at some cost in accuracy where the selector varies. Grouping stays cheap, since a selector picks among a few configurations, which differ in 1 to 6 of 48 layers and cost the same within 3%. As for storage, each layer keeps a packed cache at its own bit width or latent width and a sharing layer stores none, so within a configuration each layer's cache is a regular tensor.

On the baseline, Table 22 includes uniform 4-bit. At 4x, the learned configuration matches its throughput, as it matches its accuracy. Past 4x, precision + depth at 8x fits 1.8 to 1.9 times as many sequences and reaches 1.07 to 1.22x the aggregate throughput of 4-bit on Qwen2.5-14B, and 1.04 to 1.42x on Qwen2.5-7B. Configurations with rank are slower than bf16 at 4x and 8x (0.83 to 0.96x) and gain at 16x (1.46 to 1.84x).

3. Motivation on device. We remind the reviewer that decoding is memory-bound for the cache as well as for the weights: every decoded token reads the whole cache, so reducing its size enables throughput gains, even at batch size 1. On Qwen2.5-14B with 4-bit weights at 28k tokens, a 4-bit cache decodes 1.28x faster than bf16 (42.5 against 33.3 tokens/s), and our learned precision and precision + depth configurations keep that speed at up to 6.4x compression. We agree that at these lengths, going beyond 4-bit adds memory savings rather than speed. Configurations with rank are slower on device because MLX has no fused kernel for attention over a quantized latent, which reflects the available kernels, not the cache.

We hope this clarifies these points and we are happy to provide additional clarifications.

Reviewer_XeqwOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes KV-Kaizen, a framework that jointly optimizes three KV cache compression axes: depth, rank and precision. A lightweight selector predicts per-layer compression configurations from the input context and is jointly trained with the backbone LLM under a target memory budget. Experiments demonstrate that combining compression mechanisms can preserve accuracy at substantial compression ratios, achieving up to 32x cache reduction when combined with token eviction.

Critical strengths
  • The paper provides a unified framework for jointly optimizing three complementary KV cache compression mechanisms under a shared memory budget.
  • The extensive ablations across compression axes, model sizes, and model families provide useful insights into their trade-offs, particularly the limitations of aggressive cross-layer sharing.
  • The method demonstrates promising accuracy retention at high compression ratios and can be combined with existing token eviction methods for further memory savings.
Critical weaknesses
  • Unclear contribution of context adaptivity. Although the selector is designed to produce context-dependent configurations, some selectors produce identical configurations across prompts. The paper provides limited analysis of how compression decisions vary across layers and tasks, making it difficult to determine whether the improvements arise from context adaptivity or simply from learning effective static configurations.
  • Insufficient discussion of related learned compression methods. While the paper discusses existing compression approaches, it does not clearly explain how its learned compression strategy differs from closely related methods such as MatryoshkaKV, beyond combining multiple compression axes.
  • Unclear practical benefit of MLA conversion. Enabling rank compression requires converting non-MLA backbones using CARE, which introduces substantial accuracy degradation in some models. In particular, the main results show that configurations using all three axes do not consistently outperform simpler approaches using precision and depth. This raises questions about when the additional flexibility of rank compression justifies the conversion cost.
  • Limited end-to-end efficiency validation. Section 4 evaluates uniform compression configurations rather than the layer-specific configurations produced by the learned selector. Consequently, the reported throughput and memory improvements might not directly demonstrate the practical efficiency of the proposed adaptive framework.

Minor Suggestions

  • Figure 2. The colors of the pre-fill and selector call curves are difficult to distinguish.
Questions for authors
  • Given that some selectors produce identical configurations across prompts, could the authors clarify when context adaptivity is beneficial? A comparison against learned static per-layer configurations, together with an analysis of layer-wise compression patterns across tasks, would help distinguish context-dependent selection from learned static allocation.
  • Could the authors clarify how KV-Kaizen differs from existing learned compression methods such as MatryoshkaKV, particularly in terms of the compression selection strategy and its advantages?
  • Under what compression regimes does introducing the rank axis through MLA conversion provide a net benefit over preserving the original GQA architecture and using precision and depth alone?
  • Could the authors provide end-to-end inference throughput and peak memory using the actual per-layer configurations generated by the learned selector, including its execution overhead?
4 comments

We thank the reviewer for the positive remarks and the clear questions.

Q1. When is context adaptivity beneficial? Our claim is that per-layer allocation across three axes should be learned under one budget. A selector that returns one configuration for every prompt has learned that configuration. It is the method's output, and the selector can then be dropped at test time (Section 3.6).

When configurations vary, they matter. On 7B precision + depth at 8x, replacing the selector by its most common configuration costs 0.046 on RULER-16k at the same budget. A static configuration chosen per task, using the selector's own choices for that task, recovers most of this (0.447 against 0.460 for the selector and 0.414 for the single configuration). The selector gets there without being told the task. We added a figure of layer-wise choices across tasks (new Figure 7, Appendix F.9).

Q2. MatryoshkaKV.

  • MatryoshkaKV trains orthogonal projections of keys and values with a nested schedule, so one model works at several ranks, then searches a rank per layer and per head after training. It acts on one axis.
  • KV-Kaizen accounts for rank, precision and cross-layer sharing in one cost, and learns the allocation jointly with the backbone. The allocation can depend on the prompt and, with a budget-conditioned selector, on the requested budget.
  • A Matryoshka-style model could serve as our rank axis, so the two are complementary.

Section 5 and Appendix A discuss MatryoshkaKV in these terms. We make the contrast more explicit in Section 5.

Q3. When does rank through MLA conversion pay off? Past 8x.

  • Precision alone stops at 8x (two bits), and depth beyond that loses reasoning accuracy (Figure 4).
  • At 16x, with the same budget multiplier, adding rank to precision and depth raises accuracy from 0.563 to 0.635 at 14B and from 0.388 to 0.458 at 7B (Table 7). At 14B the converted model without compression scores 0.632, so at that ratio the conversion costs nothing.
  • At 4x, precision + depth on the original model is enough, and the paper reports this (Table 1).
  • The conversion cost itself depends on the model (+0.005 on Mistral-7B, −0.040 on OLMo-3-7B).
  • The cost of compressing on top of MLA falls with size: 0.145 at 7B, 0.022 at 14B and 0.008 at 32B below the MLA control at 16x (new 16x block of Table 17).

Q4. End-to-end efficiency with learned configurations. We benchmarked 13 configurations produced by trained selectors, including input-dependent ones (general response 1, new Appendix G.2).

  • Memory falls in every case as one would expect.
  • Precision and precision + depth configurations decode faster than bf16 from 8k tokens.
  • The selector incurs only about 3 ms per request.
  • In batched serving on GPU, aggregate throughput reaches 1.38x to 2.07x of bf16 for precision and precision + depth configurations and 1.46x to 1.84x for the 14B all-three 16x ones (general response 3).

We clarify that Section 4 measured each axis alone, to show that every option in the action space yields a gain with existing kernels; the learned configurations confirm that picture.

Minor. Figure 2 colors are fixed.

We hope these answers and the revised manuscript address your concerns, and would be happy to address any follow-up questions and concerns.

Follow-up on Q3 (when rank pays off), now that an additional run on Qwen2.5-32B has finished.

Rank pays off past 8x. At 16x, adding it raises accuracy at every model size, now including 32B.

That run is precision + depth at 16x, so rank can now be compared at three sizes.

Accuracy at 16x, without rank (precision + depth) and with rank (all three):

Multiplier7B14B32B
Adaptive0.162 → 0.4290.322 → 0.6100.455 → 0.666
Tuned, constant0.388 → 0.4580.563 → 0.635--

Adding rank raises accuracy at every size and with either multiplier. The gain is smaller when the multiplier is tuned (+0.07), because precision + depth recovers much more from tuning (+0.24 at 14B) than all three does (+0.03). At 32B, precision + depth is still ahead at 8x (0.690 against 0.653) and behind at 16x, so rank pays past 8x, as stated above.

We hope this addresses the question.

Thanks for your response and clarification on how KV-Kaizen differs from relevant prior works.

My remaining concern is about the novelty of this work, considering that each quantization axis has been independently studied in prior works. I think the novelty is somewhat incremental, as the selector determines how to utilize each quantization axis rather than leveraging context-dependent information.

This is also why I raised the lack of analysis on how the resulting configurations differ across tasks as a weakness. I believe such an analysis could provide further insights into future research directions, although designing task-specific quantization objectives is beyond the scope of this paper.

Still, I think the extensive experimental results provide valuable insights into various quantization axes, and I have updated my score accordingly.

Thank you for engaging with our response and for raising your score. We provide additional details below to address the reviewer's remaining concerns.

Novelty. Each axis has indeed been studied before, but on its own and mostly applied uniformly across layers, which limits how far it can go without losing accuracy (Figure 3, Table 9). What is new is choosing between them per layer. We price sharing, rank and precision in one cost, bits per token, and the selector learns the per-layer choice during fine-tuning, under that budget. This is what lets the combination reach ratios none reaches alone. We reach 16x (and 32x with eviction) smaller caches, where precision stops at 8x, for instance. Learning the choice during fine-tuning also matters as an offline search on the same model needs a fine-tuned uncompressed model, a calibration per budget and a second fine-tuning to match the selector (Appendix E.2).

We agree that the selector's main role is deciding how to use each axis in each layer, and this per-layer combination is the contribution we claim. The selector does read its context, the prompt and, when budget-conditioned, the requested budget (Section 3.5), and half of the selectors in Table 20 return several configurations.

Differences across tasks. The revision includes this analysis (Figure 7, Appendix F.9): for the 7B precision + depth selector at 8x, configurations are largely shared across IFEval, GSM8K and RULER, and the dataset groups differ only at layers 1, 6, 8 and 10 of 28. The other input-dependent selectors of Appendix G.2 show the same pattern: each configuration they return differs from their most common one in only 1 to 6 of 48 layers, in bit width alone for the 14B precision selector, and also in sharing and rank otherwise. Across selectors, precision and rank are spent first and sharing last (Section 3.6). For task-specific objectives, this suggests that adaptation can be confined to a few layers, with precision and rank as the first levers.

We hope this addresses the remaining concerns.

Reviewer_PSR2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces KV-Kaizen, a framework for learning per-layer KV-cache compression configurations under a shared memory budget. The action space combines cache quantization, truncation of MLA latent representations, and cross-layer cache sharing. A lightweight selector reads the prompt embeddings and chooses a configuration before prefill; this configuration remains fixed throughout generation. The selector and backbone are jointly fine-tuned using a language-modeling objective and a cache-budget penalty. The paper reports approximately 4× cache compression with comparable accuracy to matched fine-tuned controls in several settings. It also demonstrates composition with token eviction, achieving 32× decode-time cache compression on Qwen2.5-14B at 16k context length.

Critical strengths
  1. KV-Kaizen Expresses precision, rank, and sharing decisions in a common bits-per-token cost, which makes their trade-offs explicit and supports a consistent budget objective.

  2. KV-Kaizen evaluates several model sizes and families and reports important limitations, including MLA conversion losses, configuration collapse, and sensitivity to budget optimization.

  3. The 7B precision-and-depth configuration matches the fine-tuned control at 4× compression. The 14B experiment combining representation compression with eviction also demonstrates a useful way to distribute a larger compression target across mechanisms.

Critical weaknesses
  1. The benefit of context-dependent, three-axis selection is insufficiently established. Table 20 shows that several selectors return the same configuration across all sampled prompts, while Table 19 reports no cross-layer sharing in the evaluated fixed-budget, all-three-axis settings.

  2. Optimization sensitivity complicates the comparison between compression mechanisms. In Table 7, tuning the budget multiplier improves 7B precision-and-depth accuracy at 8× from 0.255 to 0.562. Such sensitivity suggests that some differences between action subsets may arise from optimization rather than their inherent capabilities.

  3. Accuracy-preservation claims require clearer qualification. For configurations involving rank compression, comparisons against the converted MLA control exclude potentially substantial conversion losses.

  4. The practical efficiency of the learned configurations remains unclear. Consequently, accuracy and system gains are not demonstrated together. Although selector latency appears small, the execution and batching costs of learned configurations remain unverified, and the additional backbone fine-tuning cost limits deployment flexibility.

Questions for authors
  1. Do the relative advantages of the compression subsets persist under comparable budget-multiplier tuning? What downstream accuracy does the offline searched configuration achieve, before and after matched fine-tuning?

  2. Can the main results consistently report both the matched architectural control and the original GQA backbone, together with uncertainty estimates for the central accuracy-preservation claims?

5 comments

We thank the reviewer for their detailed review. We answer each point in order.

W1. Context-dependent, three-axis selection. We believe the reviewer may have misread Table 19. "No sharing" in the 14B all-three settings is a choice the selector makes, not a sign that the depth axis is unused. At that budget, rank and precision save the same bytes at a smaller accuracy cost, so the selector prices depth out; Appendix F.8 shows the selector treats depth as the approach of last resort, and Section 3.6 explains why. When the budget pushes harder, it does use depth: with precision and depth at 16x, half or more of the layers inherit (Table 19: 0.500 to 0.528).

On context dependence: when a selector's configuration varies, the variation matters. On 7B precision + depth at 8x, replacing the selector by its most common configuration incurs 0.046 accuracy degradation on RULER-16k at the same budget.

W2/Q1. Optimization sensitivity. Table 7 reports this sensitivity because the multiplier sets how hard the budget binds. Under the same tuning, the conclusions hold:

  • At 14B and 16x, with the same fixed multiplier, all three axes reach 0.635 and precision + depth 0.563 (Table 7).
  • At 7B and 8x, with the multiplier tuned, the two subsets are within the seed range (0.569 and 0.546, Table 7).

On the offline-searched configuration, we may not have stated our goal clearly enough. Good cache configurations are not unique, and we do not claim the selector finds the only one. What we propose is a way to find a good one while a pre-trained model is being fine-tuned, in that same run.

The searched configuration, fine-tuned with the same recipe, reaches accuracies on par with learned selectors at 4x and 8x on 7B models. However, getting there takes more:

  • a fine-tuned uncompressed model to search on;
  • per-layer calibration for every budget;
  • a second fine-tuning run.

Both results support the paper's point that per-layer allocation is what preserves accuracy. The uniform configuration at the same budget loses far more (Table 9). Learned selectors however are a simpler and more practical alternative. We added this result to Appendix E.2.

W3. Conversion losses. We believe the reviewer may have missed that the conversion loss is reported in its own row of Table 1 ("MLA conversion only"), and that Table 17 lists the original model at every size. We agree it should sit next to each rank result, and the revision adds the MLA control to Table 1 and Table 17.

Against the original model, rank more than pays for the conversion at high compression: at 16x and under the same multiplier, all three axes beat precision + depth on the original architecture at 14B (0.635 against 0.563) and at 7B (0.458 against 0.388, Table 7). The conversion cost is also model specific (+0.005 on Mistral-7B, −0.040 on OLMo-3-7B). We added the 14B comparison to §3.3.

W4. Efficiency of the learned configurations. We benchmarked the actual per-layer configurations (general response 1).

  • They reduce memory in every case, and precision and precision + depth configurations decode faster than bf16 from 8k tokens.
  • The selector runs once per request in about 3 ms (Figure 2).
  • On batching: the configurations one selector chooses between cost the same within 3%, so a batch does not mix costs, and when a selector settles on one configuration every request shares it.
  • Batched serving on GPU: aggregate decode throughput reaches 1.38x to 2.07x of bf16 at a fixed memory budget for precision and precision + depth configurations (general response 3).
  • Fine-tuning is a one-off cost per budget, or per range of budgets with a budget-conditioned selector (Section 3.5).

Q2. Both controls and uncertainty. Table 17 already lists the original model ("Untuned model") at every size, and the revision adds the MLA control to Table 1.

For uncertainty, Table 16 reports seeds. The largest range we observed is 0.053, which Tables 1 and 17 use to mark ties, and two evaluations of one checkpoint differ by at most 0.013.

We hope these answers and the revised manuscript address your concerns.

Follow-up on W2/Q1, now that the remaining multiplier runs for all three approaches have finished.

The sensitivity the reviewer refers to comes from the restricted precision + depth subset, not from KV-Kaizen with all three approaches. With all three, the default adaptive multiplier needs no tuning: it lands within randomness of the best of seven tuned multipliers, while precision + depth drops to 0.255 without tuning. Under the same tuning, all three is also at least as accurate.

In our previous answer we compared precision + depth and all three at 7B and 8x with a tuned multiplier (0.569 for precision + depth, 0.546 for all three). The all-three value came from only two multipliers, so we ran the other five. Both subsets now have the same seven multipliers, and the pair is 0.569 and 0.578. The updated PDF shows this in Table 7.

Multiplier12.5510204080BestAdaptive
All three0.5340.5580.5390.5700.5460.5720.5780.5780.541
Precision + depth0.5230.5340.5410.5620.5690.5530.5410.5690.255
Difference+0.012+0.024-0.002+0.008-0.023+0.019+0.037+0.009+0.286

All three is ahead at 5 of 7 multipliers, every difference at a fixed multiplier is within randomness, and the best run of each is 0.578 against 0.569. The adaptive rule gives 0.541 for all three, so the all-three entry of Table 7 now reads 0.578 (+0.037). The same holds at 14B and 16x: the adaptive rule gives all three 0.610 against 0.635 at its best tuned multiplier, and precision + depth 0.322 against 0.563 (Table 7).

As you point out, the multiplier matters: across the seven constant multipliers it moves each subset by about 0.045. Where the two subsets differ is the adaptive rule we use by default, which suits all three approaches and not precision + depth alone.

We hope this addresses the question.

Reviewer_PSR2OpenReview ↗

Thanks for the rebuttal

Thank you for the additional experiments. The matched multiplier sweep addresses my concern about tuning fairness, and the benchmarks of the learned configurations address my concern that the original system evaluation used hand-designed configurations. The offline search comparison also clarifies the benefit of finding a cache configuration during backbone fine-tuning.

The 4.6% improvement on RULER-16k is evidence that prompt-dependent selection helps in the reported setting. My main remaining question is whether this gain persists when the fixed per-layer configuration is jointly optimized with the backbone from the start, using the same training recipe and cache budget. Replacing the selector after training measures how well the existing checkpoint tolerates a fixed configuration. Training the fixed configuration jointly would test whether prompt conditioning offers an advantage over a competitive static policy.

Reviewer_SQcZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors proposed KV-Kaizen - a new method for model KV cache compression, which is based on adding learnable selector which operates along three approaches: (1) sharing one cache across layers, (2) caching at fewer bits (precision), (3) using the low-rank latent cache representations (rank). Key idea is to train a light selector which composes these 3 methods locally form each layer based on train context. At inference, selector runs once before pre-fill.

Critical strengths

Original, practical idea to combine KV$ sharing between layers, MLA compression and low-precision quantization in one per-layer selector

Critical weaknesses

No clear benefits comparing to tuning-free KIVI quantization for 4x compression. The proposed method requires additional model fine-tuning if the compression target is > 5x. The question is how will KV-Kaizen generalize? Will selector work out of domain used for selector fine-tuning?

Questions for authors

How will KV-Kaizen generalize. Will selector work out of domain used for selector fine-tuning?

2 comments

We thank the reviewer for the positive assessment.

W1. Benefit over tuning-free KIVI at 4x. We agree that tuning-free quantization is strong at 4x, and do not claim otherwise. Note though that on device, our learned 4x configuration runs as fast as uniform 4-bit. Also, fine-tuning is what enables going beyond 8x, the limit of two-bit quantization: to 16x, and to 32x with eviction. So in summary learned selectors are as good as post-hoc approaches where post-hoc approaches work well, but cover ranges that post-hoc methods don't, at negligible inference cost.

W2. Generalization out of the fine-tuning domain. We believe there may be a misunderstanding about how the selector is trained as it is not trained for a specific domain. It is fine-tuned along with the backbone on a general-purpose instruction mixture (the Dolci Instruct SFT mix in our case), and the paper evaluates the same selector on instruction following (IFEval), math reasoning (GSM8K) and long-context retrieval (RULER).

We note that we do evaluate the learned selectors in situations that differ meaningfully from what was observed during training. Length for instance is a relevant shift since training sequences are at most 4,096 tokens, and RULER runs at up to 16k. At 16k, a 14B configuration combined with eviction stays on par with the uncompressed control at 32x (Table 13). We make the training and evaluation setup more explicit in Section 3.1.

We hope the clarifications further support your assessment.

We thank the reviewer again for their positive assessment, and remain available for any further discussion or clarification.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Note to all reviewers

We thank the reviewers for their careful assessment of our work.

We are glad they found that composing cache sharing, rank and precision in one per-layer selector an original and practical idea (SQcZ), the shared bits-per-token budget a clear way to make the trade-offs explicit (PSR2, Xeqw), the evaluation extensive across tasks, model sizes and families (PSR2, Xeqw, 5RHe), the selector's overhead low (5RHe), and the composition with eviction a useful way to reach higher compression (PSR2, Xeqw). KV-Kaizen makes the per-layer choice among three cache reduction approaches part of the fine-tuning a model undergoes anyway, under one budget, and reaches 16x, and 32x with eviction, where fixed-bit quantization stops at 8x.

Two points came up in most reviews, the benchmarking and the MLA conversion, and we address them here and in the revised manuscript, where changes ate highlighted in red.

1. Benchmarking. Section 4 measures each axis on its own, on device, to show that each one turns into a real gain with standard kernels; this is why they belong in the action space. On Qwen2.5-14B with long generations (Figure 5):

  • decode throughput rises from 30.3 tokens/s with a bf16 cache to 40.4 with a 4-bit cache and 42.0 with an MLA latent;
  • peak decode memory falls by up to 12 GB.

Reviewers asked for the same measurement on the configurations the selector actually learns. We benchmarked 13 learned per-layer configurations from Qwen2.5-14B and 7B selectors, including every configuration an input-dependent selector picks for at least 5% of prompts, at batch size 1 from 2k to 28k tokens. The learned configurations behave as the uniform ones in Section 4 predict:

  • Memory falls for every configuration: to between 0.63 and 0.73x of the bf16 cache at 28k on 14B, weights included.
  • Precision and precision + depth configurations decode faster than bf16 from 8k tokens: 1.14x to 1.28x at 28k, and as fast as a uniform 4-bit cache at matched compression.
  • The configurations one input-dependent selector switches between cost the same, so adapting to the input adds no decoding cost.

These results are in the new Appendix G.2.

2. MLA conversion. We convert since few open models are natively MLA. The method only needs a latent cache, and on a native MLA model the rank axis comes at no conversion cost.

  • The conversion cost depends on the model. It is +0.005 on Mistral-7B and −0.040 on OLMo-3-7B, against −0.09 to −0.18 on Qwen2.5 (Table 1).
  • At 16x, rank more than recovers the conversion cost. With the same budget multiplier for both, adding rank to precision and depth raises accuracy from 0.563 to 0.635 on Qwen2.5-14B and from 0.388 to 0.458 on Qwen2.5-7B (Table 7). At 14B this is on par with the converted model without compression (0.632), so nothing of the conversion is lost.
  • The cost of compressing on top of MLA falls with model size. All three axes at 16x sit 0.145 below the MLA control at 7B, 0.022 at 14B and 0.008 at 32B (Table 17).

3. Additional results.

  • Scale. We extended Table 17 to Qwen2.5-1.5B and 32B for every approach. The pattern of Table 1 holds at both ends. At 32B, every approach except depth alone is on par with its control at 4x, and all three axes sit 0.052 above the MLA control. At 1.5B, the approaches using rank lose 0.17 to 0.22 against it. Larger models tolerate compression better, as we argue in the paper.

  • Batched serving on GPU. In addition to the on-device results at batch size 1, we measured batched serving on one H100, in a decode benchmark with the real Qwen2.5 shapes. We again observe an improvement in throughput. At a fixed 75 GB memory budget, the batch the learned configurations fit grows with their compression: 1,401 sequences against 111 for bf16 at 2k, for the 14B 16x configurations. Aggregate decode throughput then rises above bf16 (Table 22, Appendix H):

    • 1.46x to 1.84x for the 14B all-three 16x configurations, with FlashInfer's MLA kernel;
    • 1.38x to 2.07x for every precision and precision + depth configuration, with a low-bit attention kernel we wrote.

    These results are in the new Appendix H.

Correction. While extending Table 17 to 32B we found that resuming an interrupted training run restarted the data order and reset the budget multiplier. We fixed both and retrained the affected runs. Two changes:

  • the 32B conversion cost in Appendix F.6, from 0.131 to 0.119;
  • one point of Figure 4, from 0.393 to 0.378, within randomness range.

No conclusion changes after corrections.

AuthorsOpenReview ↗

Summary of changes in the revised manuscript

We summarize the changes in the revised PDF. New and modified text is in red and tables and figures are updated without highlighting.

Added

  • §3.3: at high compression, adding rank to precision and depth under the same multiplier raises accuracy from 0.563 to 0.635 at 14B and 16x, on par with the MLA control (0.632).
  • Table 1 and Table 17: an MLA control row, so every rank result can be read against both references.
  • Table 17: every approach at 1.5B and 32B, all three approaches at 8x for every size, and a new 16x block.
  • Appendix G.2 (new): on-device benchmark of 13 learned per-layer configurations (Figures 11 and 12).
  • Appendix H (new): batched decode throughput of the learned configurations on an H100 (Table 22).
  • Appendix F.9: per-layer choices of an input-dependent selector across datasets (new Figure 7), and the task-aware static baseline.
  • Appendix E.2: downstream accuracy of the offline-searched configuration after fine-tuning.
  • §3.1: RULER runs at four times the training length. §4: pointers to Appendices G.2 and H. §5: MatryoshkaKV acts on rank alone and could serve as our rank axis. Figure 2: distinguishable colors.

Corrected

  • Table 7: each adaptive run is now paired with the best constant-multiplier run of its cell within 2% compression, and four rows change; the conclusions drawn from the table are unchanged.
  • Training-resume fix (see the general response): the 32B conversion cost in Appendix F.6 (0.131 to 0.119) and one point of Figure 4 (0.393 to 0.378). The sentence in Appendix F.6 about language-modeling loss at 32B relied on the affected run and is removed.
  • Table 1, OLMo-3-7B precision + rank: updated evaluation (+0.014, previously +0.009).
  • Table 21: the hidden-state selector row from a re-evaluation with per-prompt features (0.675 / 0.593).
  • Tables 18 and 19: missing values are filled in.
  • Smaller fixes: a typo in §6 ("with no significant accuracy difference"), hardware (B200 nodes for 14B and 32B), the model of one constant-multiplier run in Appendix D, the Table 13 caption, weight precision in two Appendix G captions, and grammar in §4 and Appendix G.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.