- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work proposes Curvature-Guided LoRA (CG-LoRA), an initialization method for LoRA. It aims to align LoRA outputs with those of full fine-tuning after the first update and at convergence. Its two variants initialize the adapter column spaces from the dominant singular subspaces of curvature-colored and curvature-whitened gradients, respectively. Experiments on T5-base and LLaMA-2-7B show competitive performance with strong baselines, with lower initialization cost on LLaMA-2-7B.
Critical strengths
- The motivation is clear. It focuses on aligning model outputs rather than parameter updates, accounting for how the Jacobian maps weight updates to output changes.
- The experiments show competitive performance and faster initialization than the compared non-zero initialization baselines on a large model, LLaMA-2-7B. The supplementary code also supports reproducibility.
Critical weaknesses
-
The theoretical objective does not fully explain the practical design. Theorem 5.1 favors covering the top 2r modes, but CG-LoRA-W uses paired top-r modes, which perform better on CoLA in Appendix F.4. This leaves the connection between optimal prediction alignment and downstream performance unclear.
-
In Table 1, the gains are small on most tasks, and plain LoRA remains competitive on SST-2, QNLI, and STS-B. This raises questions about whether the proposed initialization consistently benefits LoRA training. In Table 2, both variants have lower mean performance than LoRA-One on HumanEval, while CG-LoRA-C shows only a small gain on GSM8K. Overall, the performance advantage of the proposed method is not sufficiently clear.
Questions for authors
- Beyond the controlled network in Appendix F.1, can you directly measure output alignment with full fine-tuning on T5-base and LLaMA-2-7B after the first update and after training? Does better alignment correspond to better downstream performance in the existing experiments?
- Given that plain LoRA remains competitive on several tasks in Table 1 and LoRA-One achieves a higher mean score on HumanEval in Table 2, can you provide stronger evidence that the proposed initialization meaningfully improves LoRA training and clarify when it offers an advantage over these baselines?
- How much of the initialization-speed advantage remains when PiSSA uses randomized SVD and LoRA-One uses a smaller sketch and fewer power iterations? The reported comparison uses exact SVD for PiSSA and a sketch size of 512 with 16 power iterations for LoRA-One.
- Is there a practical rule for choosing between CG-LoRA-C and CG-LoRA-W without evaluating both?
4 comments
Thank you for the response and additional experiments, which I appreciate. They address some of my concerns. However, I find that better alignment does not consistently lead to better downstream performance across the evaluated methods. I will therefore maintain my original score, and I encourage the authors to explore this relationship further.
Thank you for the follow-up. We are glad that our response and the additional experiments addressed some of your concerns. On the remaining one, we would like to clarify the scope of our claim using elements in the paper and in our first response.
is the distance between the first-order changes of the predictions under one LoRA step and one full fine-tuning step, defined on the network itself. The paper uses it to construct the initialization and then evaluates the result on real models. We therefore claim that minimizing prediction misalignment yields a strong initialization, not that misalignment predicts accuracy for arbitrary methods. As noted in our response to Q1, alone does not rank accuracy across the six methods. We agree with your reading and will state it in the paper.
What Table R6 does show is that the construction achieves its objective. It measures directly on the network, with no approximation, and CG-LoRA-C attains the lowest value on every task. It is also the best of these methods on MRPC and GSM8K, and within points of the best on CoLA, which is CG-LoRA-W. We do not claim the converse: other methods can reach good accuracy by other routes. A zero initialization, for instance, is misaligned at the first step by construction but can still align during training. CG-LoRA-W targets the end-of-training objective, not , and remains competitive on our tested models.
As stated in our general response, an important contribution of our work is theoretical. The experiments further establish that the resulting initialization achieves its objective on real networks and is competitive at a low initialization cost. We agree that characterizing when alignment is also necessary for accuracy is an open question for this line of work, which our formulation makes measurable, and we will include Table R6 and this discussion in the revision. We thank you again for your valuable feedback.
Thank you for the reply. After careful consideration, I decide to increase my rating.
We thank the reviewer for their insightful questions. We address your concerns below.
W1 – paired versus disjoint index sets. We agree that the paper reports the comparison (Appendix F.4) without its explanation, which follows from the horizon of Theorem 5.1. By Theorem 5.1(ii), E∞=(∑i∈/I∪Jsi2(F))1/2: at convergence, a mode i of F is captured as soon as one factor reaches it (i∈I or i∈J). Covering 2r modes is then optimal, and the disjoint construction attains the bound of Theorem 5.1(i) in the controlled network of Appendix F.1. Fine-tuning, however, stops after a finite number of steps, so how fast each mode is learned matters too. Paired index sets correct this far-sightedness of E∞. One LoRA step is a full fine-tuning step preconditioned by P (Eq. 7), Pu=η2B0B0⊤u+u η2A0⊤A0. At the flat point, η2B0B0⊤ and η2A0⊤A0 are 32 times the projectors onto col(B0) and col(A0⊤), so for a mode u=ab⊤, Pu=34u when both factors reach it (a∈col(B0) and b∈col(A0⊤)) and Pu=32u when only one does. With small steps, paired sets, I=J=[r], thus learn the r dominant modes twice as fast, at the price of modes r+1,…,2r. Table 14 follows this order on CoLA: the r dominant modes reached by both factors (paired, 50.1), r/2 of them (half-overlapping, 48.3), none (disjoint, 47.5), and the dominant modes not reached at all (shifted, 44.6). We will add this explanation to Section 5.3 and Appendix F.4.
Q1 – Output alignment with full fine-tuning. We thank the reviewer for this question, which we answer with direct measurements on both models.
After the first update. The first-step objective E1 needs no full fine-tuning run: E1=∥J vec(D1)∥ is computed exactly without the Kronecker approximation. Table R6 reports it relative to ∥J vec(∇)∥, the change of the predictions under the first step of full fine-tuning.
Table R6. First-step misalignment E1/∥J vec(∇)∥ on 32 held-out examples, all adapted layers jointly. Lower is better.
CG-LoRA-C has the lowest misalignment on every task, 1.2 to 2.2 times below the next method. CG-LoRA-C is also the best of these methods on MRPC and GSM8K and within 0.4 points of the best on CoLA. Initializations whose first step overshoots that of full fine-tuning, such as LoRA-GA (50 to 130-fold in our setting), are among the weakest. Among the others, E1 alone does not rank accuracy, which is why CG-LoRA-C keeps the flat point rather than the first-step scale of Proposition 5.2(ii).
After training. This measurement does not distinguish the methods. On T5-base, at the end of the epoch, the predictions of every adapter are at a similar distance from those of full fine-tuning (relative distance of the label-word scores, 0.19 to 0.28 on CoLA and MRPC). Our end-of-training objective serves to select subspaces, not to describe the trained model, and on LLaMA-2-7B full fine-tuning is beyond our budget.
W2 and Q2 – Gains. We thank the reviewer for these points, which Reviewer otSn also raises. We answer them in the general response ("Size and consistency of the gains"), which also says when CG-LoRA offers an advantage over plain LoRA and LoRA-One.
Q3 – Initialization speed against faster baselines. On LLaMA-2-7B in the setting of Table 3 (three new runs), LoRA-One with the sketch of LoRA-GA (q=4r, four power iterations) returns the same subspaces as its released settings (overlap 1 on all 224 layers) in 46 s instead of 88 s (85 s in the earlier runs of Table 3). CG-LoRA, at 39 s (36 s in Table 3), remains the fastest gradient-based initialization and the only one that holds no host memory, against 12.6 GiB for LoRA-GA and LoRA-One. PiSSA uses no data, and a randomized SVD brings it to 2 s, at the cost of subspaces that differ from those of the exact SVD (median overlap 0.63). We will add both variants to Table 3.
Q4 – Choice of variant. We thank the reviewer for this question, which we answer in the general response ("Choosing between CG-LoRA-C and CG-LoRA-W").