- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
KV-Kaizen formulates KV-cache compression as a per-layer allocation problem over quantization precision, latent rank, and cross-layer sharing under a common memory budget. The method jointly fine-tunes the backbone and a lightweight prompt-conditioned selector, which chooses a configuration before prefill and keeps it fixed throughout generation. The paper evaluates accuracy–compression trade-offs across multiple model families and tasks, and explores combining representation compression with token eviction.
Critical strengths
- Comprehensive evaluation across different tasks and models.
- Promising quality–memory trade-offs in several settings.
- Low overhead for the selection step itself. The selector operates on prompt embeddings and runs only once before prefill.
Critical weaknesses
- The incremental benefit over strong fixed-bit quantization is not sufficiently established, particularly around 4× compression.
- The necessity of prompt-conditioned selection remains unclear. Table 20 reports 224 prompts of different areas shares the same configuration. A comparison between static tasks-aware per-layer selection is needed to prove the necessity of selector.
- The practical benefits under realistic serving workloads are unvalidated. If the effectiveness of prompt adaptiveness is proved, how to batch different requests will be a issue, where the benefit can be covered by the overheads of processing distinct quantization/sharing/SVD patterns.
Questions for authors
See weaknesses.
3 comments
Thanks for the response.
- For W2, I was asking for training the per-layer quantized model based on task without the selector, instead of using the most frequently used config. But I recognize the reasoning here that selector can choose the right prompt-conditioned quantization configuration with one time effort.
- It is still unclear how you batched requests with different input/output length and different compression selection per layer into the same batch, and how you manage the storage of KVCache with different quantization configurations. My understanding of the added experiments is that you choose the same config for the whole batch, and some "all three" settings essentially adding more overheads despite the higher compression rate. Despite that, the baseline is BF16 but your accuracy ties on 4x compression baseline, so I think this baseline is too weak and a direct comparison with 4-bit makes more sense.
- Single request serving on the device cannot explain your motivation. A 14B model takes approximately 12-16GB memory for bf16 KV cache at 64K context length (about the max length in your experiments), contributing to 3-4GB on 4-bit compression. This size is hard to support your argument of necessity for further compression, not to mention the latency overheads during decoding, and the memory-bound of weight loading. Based on these, I would like to keep my scoring.
Thank you for engaging with our response and for the follow-up.
1. Per-task configurations. We are glad the reviewer agrees that a selector trained once can choose the configuration from the prompt, without training a model per task.
2. Batching, storage and the 4-bit baseline. We clarify that our target is on-device inference at batch size 1, where the selector gives each request its own configuration at a cost of about 3 ms. We note that the newly added Appendix H is a side experiment, to check that the savings also carry over to batched serving. There, as the reviewer correctly understood, each measurement batches one configuration. For input-dependent selectors (rows marked † in Table 22), each of their configurations is measured as its own batch and weighted by how often it is picked, which amounts to grouping requests by configuration. This keeps prompt dependence. Fixing one configuration per batch is simpler, at some cost in accuracy where the selector varies. Grouping stays cheap, since a selector picks among a few configurations, which differ in 1 to 6 of 48 layers and cost the same within 3%. As for storage, each layer keeps a packed cache at its own bit width or latent width and a sharing layer stores none, so within a configuration each layer's cache is a regular tensor.
On the baseline, Table 22 includes uniform 4-bit. At 4x, the learned configuration matches its throughput, as it matches its accuracy. Past 4x, precision + depth at 8x fits 1.8 to 1.9 times as many sequences and reaches 1.07 to 1.22x the aggregate throughput of 4-bit on Qwen2.5-14B, and 1.04 to 1.42x on Qwen2.5-7B. Configurations with rank are slower than bf16 at 4x and 8x (0.83 to 0.96x) and gain at 16x (1.46 to 1.84x).
3. Motivation on device. We remind the reviewer that decoding is memory-bound for the cache as well as for the weights: every decoded token reads the whole cache, so reducing its size enables throughput gains, even at batch size 1. On Qwen2.5-14B with 4-bit weights at 28k tokens, a 4-bit cache decodes 1.28x faster than bf16 (42.5 against 33.3 tokens/s), and our learned precision and precision + depth configurations keep that speed at up to 6.4x compression. We agree that at these lengths, going beyond 4-bit adds memory savings rather than speed. Configurations with rank are slower on device because MLX has no fused kernel for attention over a quantized latent, which reflects the available kernels, not the cache.
We hope this clarifies these points and we are happy to provide additional clarifications.
We thank the reviewer for their careful reading and feedback.
W1. Benefit over fixed-bit quantization at 4x. We do not claim to dominate every point of the Pareto front, and at 4x, precision alone is already strong as the reviewer pointed out. Indeed, the selector finds this and uses precision. The method targets the range fixed bits cannot reach:
We also note that, at matched 4x, the learned configuration runs as fast as uniform 4-bit on device, so learning the allocation costs nothing at inference.
W2. Necessity of prompt-conditioned selection. We believe Table 20 may have been read as evidence against the selector. It instead shows that some selectors settle on one configuration, which the paper reports and which lets the selector be dropped at test time (Section 3.6). Other selectors vary, and the variation matters. For instance, on 7B precision + depth at 8x, replacing the selector by its most common configuration costs ~4% accuracy on RULER-16k at the same budget.
We ran the static task-aware baseline the reviewer suggests, using for each task the configuration the selector chose most often on it. It recovers most of the gap: 0.447 on RULER-16k against 0.460 for the selector and 0.414 for a single configuration. However, a task-aware configuration needs the task label at inference. The trained selector reaches the same accuracy from the prompt alone, in 3 ms.
W3. Batching under serving workloads. Our main target is on-device inference, where batch size is 1, which is the setting of Section 4.
For batched serving, the configurations one selector chooses between cost the same within 3% in our measurements, so batching them does not mix costs. When a selector settles on one configuration, every request shares it.
We also measured batched serving on an H100 (general response 3). At a fixed memory budget, the learned configurations fit batches that grow with their compression, and aggregate decode throughput reaches 1.38x to 2.07x of bf16 for precision and precision + depth configurations, and 1.46x to 1.84x for the 14B all-three 16x ones.
We hope these answers and the revised manuscript address your concerns. We are also happy to continue the discussion and provide further details or clarification.