- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates converting the K/V projections of pretrained models into a low-rank cache architecture via SVD, then freezing the decoder and training only the encoder during post-conversion fine-tuning. Across one vision-language model and two text models, the authors find that, after tuning the learning rate separately for each method, encoder-only healing performs comparably to training both factors on the reported aggregate metrics while substantially reducing the number of trainable parameters.
Critical strengths
-
The paper addresses a practically relevant cost–performance trade-off. When converting an existing model to a low-rank KV cache, the subsequent healing step incurs additional training costs. The paper examines whether fewer parameters can be updated while maintaining aggregate task performance close to that of full healing.
-
The text experiments make the results more informative. The vision-language model already performs close to its uncompressed baseline, making it difficult to judge from that setting alone whether the healing methods are genuinely comparable. By adding two text models with room for improvement, the paper shows that its main observation does not rely entirely on a near-baseline setting, although the evidence remains limited to three seeds per configuration.
-
The paper is clearly structured and easy to follow. Its motivation, method, experimental comparisons, and limitations are presented in a logical order.
Critical weaknesses
-
The main conclusion is evaluated at only one compression rank. All multi-seed comparisons use d_c=512. The other ranks in the appendix were tested with a shared learning rate and a single seed—a protocol the paper itself shows can be misleading. A per-method-tuned comparison at a more aggressive rank would clarify when freezing the decoder remains effective.
-
Aggregate text-task accuracy may hide important regressions. The text experiments report the average accuracy over six tasks but not the result for each task. Because encoder-only healing has the lowest mean on both text backbones, per-task and per-seed results are needed to determine whether a similar average masks a substantial loss on a particular benchmark.
-
The practical memory benefit is not fully demonstrated. The reported reduction concerns trainable parameters and optimizer-related memory, whereas the motivation emphasizes adaptation under limited GPU memory. Frozen weights, activations, and backward computation still consume resources. End-to-end peak GPU memory under matched training settings would show whether the savings change which configurations can actually be trained.
-
The paper does not test the proposed explanation for why freezing works. Equation (4) identifies the output subspace fixed by freezing the decoder, but the experiments do not measure whether full healing moves the decoder substantially away from that subspace. Comparing the initial and fully healed decoder subspaces would help distinguish the restriction does not matter here from the current training setup does not exploit the extra flexibility.
5.The paper should distinguish its contribution more clearly from prior work. Converting pretrained models to MLA and recovering performance through subsequent fine-tuning have been studied previously [1, 2], and freezing one factor in low-rank adaptation is not a new idea [3]. However, these works do not directly answer the specific question investigated here: whether the SVD-initialized decoder can be frozen while only the encoder is trained after SVD-based KV-cache conversion, without a substantial loss in performance. The paper would benefit from positioning this specific question and its empirical evidence as the main contribution.
Reference:
[1] Meng, Fanxu, et al. "Transmla: Multi-head latent attention is all you need." arXiv preprint arXiv:2502.07864 (2025).
[2] Ji, Tao, et al. "Towards economical inference: Enabling deepseek’s multi-head latent attention in any transformer-based llms." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.
[3] Zhang, Longteng, et al. "Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning." arXiv e-prints (2023): arXiv-2308.
Questions for authors
Please refer to the Critical Weaknesses section. If the authors can address these concerns satisfactorily, I would consider raising my score.