acceptodds
Under review as a conference paper at ICLR 2027

Compression-Aware Fine-Tuning: Adapting Language Model Weights to a Frozen Cache Bottleneck

Abstract

The low-rank projection of the key-value (KV) cache is a natural strategy to reduce inference memory. However, compressing a model whose weights were never trained to tolerate the resulting information loss leads to severe quality collapse at moderate compression ratios. Compression-aware fine-tuning (CAFT) closes this gap by adapting the model weights to a frozen cache compressor rather than compressing weights that were never trained for the loss. At a 2x cache budget on Llama-3.2-1B, CAFT reduces WikiText-103 perplexity by 11.5x relative to the strongest post-hoc baseline (a fine-tuned checkpoint compressed under a basis freshly calibrated on its own activations). The advantage holds across six compression operators, three model scales (1B to 8B), and two training domains, and persists when stacked with 4-bit code quantization. At 2x and 3x compression, CAFT preserves single-needle retrieval and maintains a large advantage on LongBench where the post-hoc baseline collapses. Mechanistically, the gain is functional robustness: the network routes around the discarded dimensions rather than concentrating cache energy into the retained subspace, as shown by spectral probes and basis-transfer controls that falsify induced compressibility. Through this paper, we demonstrate that CAFT recovers the capability that low-rank KV cache compression destroys.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache compression, compression-aware fine-tuning, low-rank projection, efficient LLM inference, long-context inference, straight-through estimator, large language models, memory-efficient inference, model compression, fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8dbnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to mitigate the performance loss caused by compressing the key–value (KV) cache of large language models. The authors propose compression-aware fine-tuning (CAFT), which keeps the low-rank projection basis fixed while adapting the model weights to compressed attention computations during training. Across the tested model scales and compression operators, CAFT generally achieves lower perplexity than a baseline that applies compression after fine-tuning.

Critical strengths
  1. The research question is important and well defined. The paper addresses the performance degradation caused by KV-cache compression. Rather than applying compression only at deployment, it proposes adapting the model during fine-tuning to a fixed low-rank projection. This makes the motivation and the proposed approach easy to understand.

  2. The paper is clearly written and well organized. It moves logically from the motivation and method to the experimental setup and results. The main argument is easy to follow, and the presentation helps readers understand how CAFT differs from post-hoc compression.

  3. The main perplexity results cover a broad range of settings. The paper evaluates three model scales, two principal compression budgets, and six low-rank compression operators. CAFT outperforms PH-FT across these reported settings, suggesting that the observed benefit is not limited to a single model size or operator.

Critical weaknesses
  1. Evaluation is limited to the Llama model family. Although the paper tests several model sizes, it does not establish whether CAFT generalizes to other architectures or scales effectively to larger models.

  2. The comparison with other compression methods does not support a broad competitiveness claim. In Table 11, some methods perform substantially better than CAFT on C4. However, differences in model checkpoints and training conditions also prevent a controlled comparison. The table therefore establishes neither CAFT’s superiority nor a reliable ranking of the methods.

  3. The paper does not directly test whether STE is necessary. The authors use a straight-through estimator (STE) to propagate gradients, but do not compare it with the exact gradient of the projection under otherwise identical conditions. The results show that the overall CAFT approach is effective, but do not establish STE as the key reason for its effectiveness.

  4. The real-world serving benefits are not fully measured. The paper reports cache size and some prefill overhead, but provides limited evidence on decoding speed, end-to-end latency, and actual memory use at longer context lengths. It would help to distinguish theoretical cache savings from measured improvements in serving speed and request capacity.

Questions for authors

Please refer to the Critical Weaknesses section. If the authors can address these concerns satisfactorily, I would consider raising my score.

3 comments

Thank you for the constructive review and for the offer to reconsider the score. We address each weakness below.

Only the Llama family. We added a second model family, Qwen3, at 4B and 8B with the same recipe. CAFT beats PH-FT at both sizes and both compression levels we tested, as it does on Llama. We have not tested models above 8B.

Comparison with other compression methods. We agree. In a controlled comparison on the same 8B model and cache size, GEAR beats CAFT; CAFT's contribution is within low-rank compression.

STE vs the exact gradient. We have now run this comparison at 1B, 3B and 8B under identical conditions. The exact gradient is better than the STE at 3B and 8B and close at 1B, and with either gradient CAFT beats PH-FT by a wide margin. The STE is therefore not the key reason; the gain comes from training with the compression in the forward pass.

Serving benefits. We have now measured serving with a real compressed cache on an A100 at 8B, from 4k to 32k context. Memory falls and more concurrent requests fit, but our prototype is slower per step, so throughput stays below the uncompressed model.

We will revise the paper to reflect all of the above.

Thank you for your response. I have one remaining question. As you noted, In a controlled comparison on the same 8B model and cache size, GEAR beats CAFT. Given this, could you clarify why CAFT is needed?

Thank you for the question. We see CAFT as a complementary method rather than a replacement for GEAR, for three reasons.

  1. CAFT comes close to GEAR at the operating point we recommend. On the same 8B fine-tuned model at 2x, CAFT reaches WikiText perplexity 5.96 against GEAR’s 5.50, and downstream accuracy 0.679 against 0.721. It does this by reducing the dimension of the cache rather than its precision. A method acting on a different axis arrives near a strong quantization baseline.

  2. CAFT works on a different axis, so it adds to quantization rather than competing with it. GEAR lowers precision; CAFT lowers dimension. Once a cache is already quantized, further savings must come from another axis, and that is where CAFT applies. When CAFT’s low-rank codes are additionally quantized to 4 bits (8x effective compression), CAFT stays usable: WikiText perplexity 7.46 at 8B and 12.89 at 1B. The post-hoc low-rank baseline collapses at the same budget (67.92 and 402.86).

  3. Training the model against a frozen cache compressor has not been studied before. Its contribution is less in beating any single method than in showing this is a viable approach in low-rank compression that comes close to a state-of-the-art quantizer while covering an axis quantization does not.

Reviewer_crWDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CAFT trains model weights through a frozen low-rank KV-cache projection using a straight-through estimator. It improves over ordinary fine-tuning followed by recalibration and post-hoc low-rank compression.

Critical strengths

The PH-FT control recalibrates on the fine-tuned model and matches the total retained-rank budget. Gains over this control are substantial and extend across several projection operators. The basis-transfer tests also expose the resulting checkpoint's dependence on its compression configuration.

Critical weaknesses
  1. Absolute performance remains weak despite gains over PH-FT. At 8B/4×, C4 perplexity is 110.21 for uniform CAFT and 54.73 for ReCalKV-style CAFT, versus 10.44 for GEAR (Tables 8, 11; CAFT uses three seeds, GEAR one). At 8B/2×, LongBench falls from uncompressed FT's 30.79 to 15.61. A practical advantage over stronger compression alternatives is not established.

  2. The central gradient choice is untested. Equation 4 replaces the exact projection gradient with the identity. This intentional bias has no descent guarantee, and the available exact-gradient implementation is not compared experimentally. The STE's contribution is therefore unclear.

  3. The proposed mechanism exceeds the evidence. Spectral measurements do not locate task-relevant information, and basis-transfer failures show weight–basis dependence. Neither establishes that the model “routes around” discarded dimensions.

  4. Decoding efficiency is unverified. Table 13 reports cache savings and prefill costs, but the reader reconstructs full-dimensional historical keys and values. Peak decoding memory and throughput are needed to establish deployment benefits.

Questions for authors
  1. How does exact-gradient training compare with the STE under the same basis and recipe?
  2. At the intended 2× operating point, what downstream advantage over GEAR remains with matched starting models, training data, and budgets, including measured decoding costs?
2 comments

Thank you for the detailed review. We answer your two questions first, then the remaining weaknesses.

Q1. Exact gradient vs STE under the same basis and recipe. We have now run this comparison at all three model sizes, training with the exact gradient under the same basis, recipe and budgets as the paper. At 3B and 8B the exact gradient is better than the STE, and its advantage grows with compression. At 1B the two are close, with the STE slightly ahead on WikiText. In every case, CAFT trained with the exact gradient still beats PH-FT by a wide margin. The gain therefore comes from having the compression in the forward pass during training, not from the STE.

Q2. Downstream advantage over GEAR at 2x, with decoding costs. None on accuracy: on the same 8B fine-tuned model, data and cache size, GEAR beats CAFT at 2x. On decoding, measured on an A100 at 8B, CAFT's compressed cache lowers memory and fits more concurrent requests, but our prototype is slower per step, so throughput stays below the uncompressed model. We did not measure GEAR in this setup. CAFT is not a replacement for quantization but complementary to it: CAFT with 4-bit codes stays usable where post-hoc compression collapses.

Absolute performance (C4, LongBench). The weak C4 numbers come mostly from fine-tuning on WikiText only, not from the compression. Fine-tuning on a WikiText and C4 mix lowers CAFT's C4 perplexity at 8B and 4x from 110 to 26, and CAFT still clearly beats PH-FT. The LongBench score uses the Uniform operator, our weakest; we are now running LongBench with ReCalKV-style, our strongest operator.

We will revise the paper to reflect all of the above.

Thanks for the additional experiments and clarifications. The new results make a clearer case for compression-aware fine-tuning. I’m still unsure when it offers a practical advantage over stronger compression baselines, so I’ll keep my score.

Reviewer_se4aOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

CAFT is a proposed framework to improve compression-aware-finetuning. The authors propose to insert a non-learnable projection in the forward pass of every attention layer. The model is trained with this projections in place, exposing the model to compressed input. The idea is that the quality of the compressed model will improve due to being exposed to compressed inputs during training.

Critical strengths

The accuracy related results Table 3 and score ratios Table 4 suggest that the framework can provide some performance improvement for compressed LLMs

Critical weaknesses

The contribution of the paper seems to be only incremental. It is not clear that the fixed projection (before RoPE) is not equivalent to restricting WkW_k to a fixed rr-dimensional subspace.

Questions for authors

Can you elaborate what is the advantage of using a fixed versus a learnable projection? ``Learned Subspace Compression for Communication-Efficient Pipeline Parallelism'' (https://arxiv.org/pdf/2606.05484) suggests potential drawbacks that your work seems to overcome.

It was not clear to me how is your projection placed. If it is placed after a learnable linear map, before RoPE, it can be absorbed during training. How is this different from using WkW_k in a smaller dimension subspace?

Regarding the results on Table 3, is the ``Downstream macro acc'' the average accuracy accross the accuracies for each ARC-Easy, ARC-Challenge, PIQA and HellaSwag? Or, is it a balanced accuracy accounting for class imbalance? Each of the benchmark contains different number of questions?

On \S 5.4 the authors report an 8×8\times peplexity reduction. Does this perplexity reduction translates to accuracy gain? Reducing peplexity, does not guarantee an improvement on the model's quality.

1 comment

Thank you for the careful review and the precise questions.

Is the fixed projection equivalent to restricting W_k to a subspace? In the forward pass, yes. The projection sits directly after the key and value projections, before RoPE, so CAFT is equivalent to restricting W_k and W_v to a fixed low-rank subspace. We chose this simple form on purpose, and the equivalence cannot explain our results: the post-hoc baseline (PH-FT) applies the same kind of restriction at the same cache budget. The only difference is that CAFT applies it during fine-tuning, while PH-FT applies it afterwards, and that difference alone accounts for the gap between them (at 8B and 4x compression, WikiText perplexity 10.1 vs 54.2). Fixing the subspace is also what makes the cache smaller: because it is known in advance, the cache only needs to store the low-rank coefficients. Our contribution is therefore not a new compression operator, but showing that applying this constraint during fine-tuning, rather than after it, closes most of the perplexity gap left by post-hoc compression, across six operators and two model families.

Why a fixed rather than a learnable projection? (MAPL) A fixed projection adds no trainable parameters, serves every budget from one calibration, and keeps the comparison with PH-FT clean, since only the training differs. Two new 1B experiments show it costs little. Learning the projection jointly with the weights changes results only slightly, in both directions. CAFT with a random projection still beats PH-FT, so most of the gain comes from adapting the weights. These experiments also show that a learned projection alone is competitive at mild (2x) compression. MAPL finds that learned projections must stay orthonormal, or they do worse than a fixed one; a fixed projection avoids this by construction, and constrained learned projections are a natural extension.

What is the downstream macro accuracy? It is the plain average of the four task accuracies, not a balanced accuracy. The tasks differ in size (from about 1,200 to 10,000 questions), and each counts equally in the average.

Does the perplexity reduction translate into accuracy? Only partly: perplexity gains overstate accuracy gains. In a new downstream comparison with PH-FT at 8B, CAFT is more accurate in every setting, recovering between a third and two thirds of the accuracy that PH-FT loses to compression. On retrieval (RULER) and long-context tasks (LongBench), CAFT also clearly beats PH-FT (Tables 3 and 4).

We will revise the paper to reflect all of the above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.