acceptodds
Under review as a conference paper at ICLR 2027

Freeze the Decoder, Heal the Encoder: Parameter-Efficient Adaptation for SVD-Based KV-Cache Compression

Abstract

The KV cache dominates inference memory, especially for vision-language models, which must cache image and text tokens alike. A pretrained model can be retrofitted with a compact, multi-head-latent-attention-style cache by SVD-factorizing its key/value projections into a down-projection ("encoder") and an up-projection ("decoder"), followed by a short fine-tune ("healing") that recovers the accuracy lost to truncation. Existing conversions heal both factors, without asking whether both need it. We freeze the decoder at its SVD initialization and heal only the encoder, which trains 1/3 of the new parameters. This keeps keys and values in the original top-r output subspace, but leaves the model free to choose what the cache stores. Under per-arm learning-rate tuning and three seeds, encoder-only healing is competitive with decoder-only and full healing on Qwen2.5-VL-3B-Instruct (MME, TextVQA) and on average accuracy over six zero-shot text tasks across two 7B backbones, including a setting with 3.5-6.3 points of headroom left. No difference is statistically significant, and equivalence tests bound any difference to within 1.2-5.4% of the reference score per metric; with three seeds, smaller deficits cannot be ruled out. Encoder-only healing uses 3x fewer trainable parameters and 3x less optimizer-state memory (0.57 vs. 1.72 GiB), producing an identical served model. We also find that comparing healing modes at one shared learning rate makes encoder-only healing look far better than it is, with gaps that pass a 3-seed t-test (t up to 8.78) but vanish under per-arm tuning, a pitfall for comparing any fine-tuning recipes of unequal size.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV-cache compression, parameter-efficient fine-tuning, low-rank adaptation, singular value decomposition, efficient language models, multi-head latent attention, multimodal language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_kFbROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates converting the K/V projections of pretrained models into a low-rank cache architecture via SVD, then freezing the decoder and training only the encoder during post-conversion fine-tuning. Across one vision-language model and two text models, the authors find that, after tuning the learning rate separately for each method, encoder-only healing performs comparably to training both factors on the reported aggregate metrics while substantially reducing the number of trainable parameters.

Critical strengths
  1. The paper addresses a practically relevant cost–performance trade-off. When converting an existing model to a low-rank KV cache, the subsequent healing step incurs additional training costs. The paper examines whether fewer parameters can be updated while maintaining aggregate task performance close to that of full healing.

  2. The text experiments make the results more informative. The vision-language model already performs close to its uncompressed baseline, making it difficult to judge from that setting alone whether the healing methods are genuinely comparable. By adding two text models with room for improvement, the paper shows that its main observation does not rely entirely on a near-baseline setting, although the evidence remains limited to three seeds per configuration.

  3. The paper is clearly structured and easy to follow. Its motivation, method, experimental comparisons, and limitations are presented in a logical order.

Critical weaknesses
  1. The main conclusion is evaluated at only one compression rank. All multi-seed comparisons use d_c=512. The other ranks in the appendix were tested with a shared learning rate and a single seed—a protocol the paper itself shows can be misleading. A per-method-tuned comparison at a more aggressive rank would clarify when freezing the decoder remains effective.

  2. Aggregate text-task accuracy may hide important regressions. The text experiments report the average accuracy over six tasks but not the result for each task. Because encoder-only healing has the lowest mean on both text backbones, per-task and per-seed results are needed to determine whether a similar average masks a substantial loss on a particular benchmark.

  3. The practical memory benefit is not fully demonstrated. The reported reduction concerns trainable parameters and optimizer-related memory, whereas the motivation emphasizes adaptation under limited GPU memory. Frozen weights, activations, and backward computation still consume resources. End-to-end peak GPU memory under matched training settings would show whether the savings change which configurations can actually be trained.

  4. The paper does not test the proposed explanation for why freezing works. Equation (4) identifies the output subspace fixed by freezing the decoder, but the experiments do not measure whether full healing moves the decoder substantially away from that subspace. Comparing the initial and fully healed decoder subspaces would help distinguish the restriction does not matter here from the current training setup does not exploit the extra flexibility.

5.The paper should distinguish its contribution more clearly from prior work. Converting pretrained models to MLA and recovering performance through subsequent fine-tuning have been studied previously [1, 2], and freezing one factor in low-rank adaptation is not a new idea [3]. However, these works do not directly answer the specific question investigated here: whether the SVD-initialized decoder can be frozen while only the encoder is trained after SVD-based KV-cache conversion, without a substantial loss in performance. The paper would benefit from positioning this specific question and its empirical evidence as the main contribution.

Reference:

[1] Meng, Fanxu, et al. "Transmla: Multi-head latent attention is all you need." arXiv preprint arXiv:2502.07864 (2025).

[2] Ji, Tao, et al. "Towards economical inference: Enabling deepseek’s multi-head latent attention in any transformer-based llms." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

[3] Zhang, Longteng, et al. "Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning." arXiv e-prints (2023): arXiv-2308.

Questions for authors

Please refer to the Critical Weaknesses section. If the authors can address these concerns satisfactorily, I would consider raising my score.

Reviewer_xv78OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper targets MLA-style latent cache onto a pretrained model by SVD-factorizing each layer's stacked K/V projection into an encoder and a decoder, then running a short fine-tune called healing to recover what truncation cost. Traditionally papers heals both factors by default. This paper asks whether both actually need it. The proposal is to freeze the decoder at its SVD initialization and train only the encoder, which is 1/3 of the new parameters.

Critical strengths

1- It is very clear and honest reporting of the results. E.g. Section 5.1 reports that encoder-only has the lowest mean on both text backbones and is the noisiest arm on Qwen2.5-7B. Takeaway 4 is titled "the saving is not speed or stability."

Critical weaknesses

1- The results on the llm side is not convincing. Results are only reported on multichoice tasks such as MMLU, ARC-C, PIQA, HellaSwag, OpenBookQA, and Winogrande which are not generative. The quality of their solution on generative tasks such as GSM8K, Math-500 is unclear. VL experiments are also not very challenging tasks.

2- Several parts in the results section is difficult to follow.
Two backbones, one ratio, three seeds, and a six-task average could hide per-task regressions. The tuned three-seed protocol is only run at d_c = 512.

3- The improvement is marginal. What encoder-only healing buys is 51M trainable parameters instead of 154M, and 0.57 GiB of AdamW state instead of 1.72, on a fine-tune that takes 27 minutes.

4- Missing baselines: PALU, and X-EcoMLA Why we do not compare with these papers? How does your solution work compared with PALU for example?

Questions for authors

1- Can we address this by adding an experiment: "we have not measured how far full healing moves the decoder away from col(U_r). Principal angles between the initial and healed decoder subspaces, and the healed decoder's singular spectrum, would show whether the restriction in Eq. (4) is ever close to binding."

2- Can you rerun one aggressive setting under the per-arm-tuned, three-seed protocol like d_c = 256, or d_c = 512 on OpenLLaMA where it's 16x rather than 4x? At d_c = 256 every arm collapsed in 300 steps under the old protocol. Is that a budget problem rather than a ratio problem, i.e. does a longer schedule separate the arms?

3- All three compressed arms beat the uncompressed baseline on MME Cognition (625.6 / 622.1 / 619.4 vs 616.4). Given that, what does the 5.4% equivalence bound on that metric tell a reader?

4- Can you add more challenging tasks such as GSM8K, and other generative tasks?

5- Could you report the six zero-shot tasks individually with per-seed values rather than just the average? Your limitations note that an average could hide a regression offset by gains, and you already have the data.

Reviewer_o3XtOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes an encoder-only fine-tuning strategy for adapting pretrained LLMs and VLMs to SVD-based KV-cache compression. By freezing the decoder and fine-tuning only the encoder, the proposed method achieves comparable model quality to full-module and decoder-only fine-tuning in the evaluated settings, while reducing optimizer-state memory by approximately 67%.

Critical strengths
  • Simplicity and parameter efficiency: The proposed method is simple and straightforward, achieving comparable performance to decoder-only and full healing under the evaluated settings while requiring substantially less optimizer-state memory.

  • Relative robustness to learning-rate selection: The proposed encoder-only healing method appears less sensitive to learning-rate selection in the vision-language experiments. As shown in Section 5.3, encoder-only healing maintains relatively stable performance across the tested learning rates, whereas decoder-only and full healing exhibit substantial performance degradation and higher variance when using the shared learning rate. This suggests that encoder-only healing may be easier to optimize in practice.

Critical weaknesses
  • Higher variance across training runs: As shown in Table 3, encoder-only healing exhibits higher accuracy variance across random seeds than decoder-only and full healing on both text benchmarks, particularly Qwen2.5-7B. This raises concerns about the stability and reproducibility of the proposed method. Additional training runs or hyperparameter tuning may be required to consistently obtain a high-quality model, potentially offsetting some of the claimed resource savings.

  • Limited improvement in fine-tuning speed: Despite reducing the number of trainable parameters, the proposed method provides only a marginal improvement in fine-tuning speed (approximately 3.5%, as reported in Section 5.2). Therefore, its efficiency benefits are primarily limited to memory savings rather than computational speed.

  • Unclear end-to-end training memory savings: While the proposed method significantly reduces optimizer-state memory, optimizer states account for only a portion of the overall memory footprint during fine-tuning. The paper does not report peak GPU memory consumption or total training memory savings, making it difficult to assess the practical memory benefits of the proposed method.

  • Limited experimental validation across compression ratios: The main experiments evaluate only a single latent dimension (d_c = 512) for each model. Although the authors provide exploratory results with different compression ratios, these experiments use only one seed and a shared learning rate, which the authors acknowledge can lead to misleading comparisons. Therefore, it remains unclear whether encoder-only healing can maintain comparable performance to full healing under more aggressive compression settings, where freezing the decoder may impose stronger representational constraints.

  • Somewhat specialized applicability (minor): The proposed method primarily targets the healing stage following SVD-based MLA conversion. While this is a meaningful application, its practical benefits are tied to this specific model conversion workflow. It would be interesting to understand whether the proposed encoder-only adaptation strategy could generalize to more low-rank compression or fine-tuning settings.

Overall, while the proposed approach is simple and potentially useful, I find the experimental evidence insufficient to establish its robustness across different compression settings and its practical end-to-end memory benefits. These limitations reduce my confidence in the practical significance and generalizability of the proposed method.

Questions for authors

What are the global memory saving when comparing encoder-only between decoder-only and full model?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.