- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how to mitigate the performance loss caused by compressing the key–value (KV) cache of large language models. The authors propose compression-aware fine-tuning (CAFT), which keeps the low-rank projection basis fixed while adapting the model weights to compressed attention computations during training. Across the tested model scales and compression operators, CAFT generally achieves lower perplexity than a baseline that applies compression after fine-tuning.
Critical strengths
-
The research question is important and well defined. The paper addresses the performance degradation caused by KV-cache compression. Rather than applying compression only at deployment, it proposes adapting the model during fine-tuning to a fixed low-rank projection. This makes the motivation and the proposed approach easy to understand.
-
The paper is clearly written and well organized. It moves logically from the motivation and method to the experimental setup and results. The main argument is easy to follow, and the presentation helps readers understand how CAFT differs from post-hoc compression.
-
The main perplexity results cover a broad range of settings. The paper evaluates three model scales, two principal compression budgets, and six low-rank compression operators. CAFT outperforms PH-FT across these reported settings, suggesting that the observed benefit is not limited to a single model size or operator.
Critical weaknesses
-
Evaluation is limited to the Llama model family. Although the paper tests several model sizes, it does not establish whether CAFT generalizes to other architectures or scales effectively to larger models.
-
The comparison with other compression methods does not support a broad competitiveness claim. In Table 11, some methods perform substantially better than CAFT on C4. However, differences in model checkpoints and training conditions also prevent a controlled comparison. The table therefore establishes neither CAFT’s superiority nor a reliable ranking of the methods.
-
The paper does not directly test whether STE is necessary. The authors use a straight-through estimator (STE) to propagate gradients, but do not compare it with the exact gradient of the projection under otherwise identical conditions. The results show that the overall CAFT approach is effective, but do not establish STE as the key reason for its effectiveness.
-
The real-world serving benefits are not fully measured. The paper reports cache size and some prefill overhead, but provides limited evidence on decoding speed, end-to-end latency, and actual memory use at longer context lengths. It would help to distinguish theoretical cache savings from measured improvements in serving speed and request capacity.
Questions for authors
Please refer to the Critical Weaknesses section. If the authors can address these concerns satisfactorily, I would consider raising my score.
3 comments
Thank you for your response. I have one remaining question. As you noted, In a controlled comparison on the same 8B model and cache size, GEAR beats CAFT. Given this, could you clarify why CAFT is needed?
Thank you for the question. We see CAFT as a complementary method rather than a replacement for GEAR, for three reasons.
-
CAFT comes close to GEAR at the operating point we recommend. On the same 8B fine-tuned model at 2x, CAFT reaches WikiText perplexity 5.96 against GEAR’s 5.50, and downstream accuracy 0.679 against 0.721. It does this by reducing the dimension of the cache rather than its precision. A method acting on a different axis arrives near a strong quantization baseline.
-
CAFT works on a different axis, so it adds to quantization rather than competing with it. GEAR lowers precision; CAFT lowers dimension. Once a cache is already quantized, further savings must come from another axis, and that is where CAFT applies. When CAFT’s low-rank codes are additionally quantized to 4 bits (8x effective compression), CAFT stays usable: WikiText perplexity 7.46 at 8B and 12.89 at 1B. The post-hoc low-rank baseline collapses at the same budget (67.92 and 402.86).
-
Training the model against a frozen cache compressor has not been studied before. Its contribution is less in beating any single method than in showing this is a viable approach in low-rank compression that comes close to a state-of-the-art quantizer while covering an axis quantization does not.
Thank you for the constructive review and for the offer to reconsider the score. We address each weakness below.
Only the Llama family. We added a second model family, Qwen3, at 4B and 8B with the same recipe. CAFT beats PH-FT at both sizes and both compression levels we tested, as it does on Llama. We have not tested models above 8B.
Comparison with other compression methods. We agree. In a controlled comparison on the same 8B model and cache size, GEAR beats CAFT; CAFT's contribution is within low-rank compression.
STE vs the exact gradient. We have now run this comparison at 1B, 3B and 8B under identical conditions. The exact gradient is better than the STE at 3B and 8B and close at 1B, and with either gradient CAFT beats PH-FT by a wide margin. The STE is therefore not the key reason; the gain comes from training with the compression in the forward pass.
Serving benefits. We have now measured serving with a real compressed cache on an A100 at 8B, from 4k to 32k context. Memory falls and more concurrent requests fit, but our prototype is slower per step, so throughput stays below the uncompressed model.
We will revise the paper to reflect all of the above.