- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Cheap LoRA (cLA), which fixes one LoRA factor as a column selector and only trains the other factor, thereby restricting the update to selected columns. Its chained variant, c3LA, shifts the selected columns across training stages to expand cumulative coverage while keeping the number of simultaneously trainable parameters fixed. Experiments on 10 pretrained models and 14 datasets show that substantial restriction to the adaptation space can often retain most of LoRA’s performance with lower training time and memory.
Critical strengths
- The paper evaluates 11 fine-tuning methods across 10 pretrained models and 14 datasets. This provides a systematic evaluation of the empirical applicability of the proposed method.
- The method is straightforward and easy to implement. On suitable tasks, it reduces training time and peak GPU memory while retaining competitive performance, and the paper also provides information-theoretic generalization bounds.
Critical weaknesses
- Limited new insight. The paper shows that LoRA’s adaptation space can often be strongly restricted, but it does not provide a transferable minimum space or a rule for different tasks. As acknowledged in Section 5, “the bounds therefore do not distinguish when a particular column restriction will help or hurt empirically.”
- Lack of empirical guarantees or guidance. The performance of the proposed method is not guaranteed, with substantial drops observed on some tasks, such as TinyLlama/OpenBookQA in Table 16. Since the trade-off varies substantially across tasks, the paper provides limited guidance on whether Cheap LoRA will be effective for a new task.
- No task-aware subspace selection. The columns are selected in advance, randomly sampled, or shifted according to a fixed schedule. A task-aware method for selecting important subspaces could be useful, but this is not considered or discussed in the proposed method.
Questions for authors
Section 4.2 reports the gap between the training and test losses but does not report the two losses separately. Could the authors report both losses, since a small gap may also result from underfitting or failed optimization?
3 comments
References
[1] Woo et al., PaCA: Partial Connection Adaptation for Efficient Fine-Tuning, ICLR 2025.
[2] Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, ICLR 2022.
[3] Valipour et al., DyLoRA: Parameter-Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation, EACL 2023.
[4] Zhang et al., AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning, ICLR 2023.
Can Reviewer NuiG kindly let us know what else we need to do to get the reviewer's attention to our rebuttal and revised version?
Dear Reviewer NuiG,
We would greatly appreciate it if the reviewer could provide any comments on our work after we submitted the rebuttal and the revised version of the paper, as the author-reviewer discussion period will end in less than 24 hours.
We believe we have clarified all questions related to novelty and the task-specific subspace selection protocol. The other issues that the reviewer asked for further clarification on, that is,
- Large drop on TinyLlama/OpenBookQA.
- Report training and test losses separately,
are provided with objective details.
We would again like to reiterate that the paper’s scientific contribution therefore lies in the analytical connections, controlled extensions, and verified performance–cost consequences of restricting adaptation—not in claiming that the selector’s identity creates a new update class. We respectfully ask that novelty be assessed against these contributions, which the equivalence alone neither establishes nor explains.
If these responses address the reviewer's concerns, we kindly ask the reviewer to reconsider their score.
Sincerely,
The Authors
We thank the reviewer for their assessment and respond to each point below. We have revised the paper in response to the reviewers’ comments and marked the relevant changes in blue.
W1. No transferable minimum space or selection rule.
This is an excellent question. However, if this question were answered, there would be no further research for the field of parameter-efficient fine-tuning. Finding the task-dependent, minimal subspace to adapt any model to any task is the goal of PEFT as a whole. This paper takes a small step towards it by assessing the ability of a common PEFT method (LoRA) to adapt to tasks under heavily restricted adaptation spaces, and analyzes that ability theoretically, in terms of diagnostic tools (loss landscape, intruder dimensions), and empirically. We also highlight that leaving the minimum transferable space open is also common in the LoRA literature, where the rank is typically chosen high enough to ensure that the space can adapt to the task. The original LoRA paper calls the relationship between model size and the optimal rank an open question ([2], §H.2), and later work notes that optimizing the rank requires an exhaustive search (DyLoRA [3]) or allocates the rank budget across layers during training (AdaLoRA [4]).
The paper offers the following insights into LoRA's capacity under heavily restricted adaptation spaces:
(1) A column-restricted LoRA family that naturally arises from this goal and connects LoRA to another PEFT family (PaCA [1]; §2.3, p. 4, lines 185–211 and Appendix B, pp. 16–17, lines 848–884; Table 4, p. 17, lines 886–907).
(2) Information-theoretic generalization bounds for all considered parameterizations, showing that the column restriction does not enlarge the bound of its asymmetric parent (§3.2, p. 6, lines 295–299; Table 1, p. 5, lines 216–232).
(3) A matched-budget comparison of 11 methods on 10 models and 14 datasets covering performance, training time, memory, and empirical generalization (§4, pp. 6–9, lines 301–475) to assess the performance of the column-restricted variants and highlight a benefit in studying this direction: that the column restrictions can be used to increase compute efficiency. The restricted variants retain 97.3–98.1% of LoRA's performance at 6.3–7.8% lower step time (§4.1, p. 7, final paragraph beginning at line 375), where efficiency is an ad-hoc benefit.
The paper’s scientific contribution therefore lies in the analytical connections, controlled extensions, and verified performance–cost consequences of restricting adaptation—not in claiming that the selector’s identity creates a new update class. We respectfully ask that novelty be assessed against these contributions, which the equivalence alone neither establishes nor explains.
W2. No guarantee; large drop on TinyLlama/OpenBookQA.
The drop for adapting TinyLlama on OpenBookQA is not unique to column restriction, rather, it is shared among all variants that freeze the A matrix. Every method that freezes A loses to LoRA by roughly 5-7% on this task, as does full fine-tuning (FFT). For example, LoRA reaches 56.72% compared with 51.88% for Asymmetric LoRA and 51.12% for FFT. cLA and its variants achieve 50.04-50.80%, staying within 1.84% of Asymmetric LoRA here (Table 22). Of cLA's 6.68-point gap to LoRA, 4.84 points are already present in the Asymmetric LoRA baseline.
This gives the guidance the reviewer asks for. These results support using Asymmetric LoRA's performance in tasks as a practical guide to when column restriction is promising. cLA is within 2.2% of Asymmetric LoRA on 21 of the 23 non-perplexity model–dataset pairs (Table 22, p. 45, lines 2381–2419). On the same OpenBookQA task with LLaMA-3-8B, Asymmetric LoRA reaches 87.84% versus 87.64% for LoRA, and cLA reaches 88.32% (Table 22). Generally: the column-restricted variants serve to amplify the trade-off of Asymmetric LoRA relative to LoRA, giving more training speed for a small amount of performance while sometimes still being the best (§4.1).
W3. No task-aware subspace selection.
Please see our response to W1: adaptive selection is discussed as the main open direction (§5) and is outside the question the paper asks.
Question. Report training and test losses separately.
This is a great question. We now report the requested training and test losses in Appendix E.9 (Table 21, p. 44, lines 2322–2372), the corresponding generalization errors in Table 20 (p. 43, lines 2278–2317), and task performance in Appendix E.10 (Table 22, p. 45, lines 2381–2419). These results cover all 24 model–dataset pairs and all ten methods of Table 20.
If these responses address the reviewer's concerns, we kindly ask the reviewer to reconsider their score.