acceptodds
Under review as a conference paper at ICLR 2027

Curvature-Aware LoRA Initialization in Function Space

Abstract

Parameter-efficient fine-tuning methods such as LoRA enable efficient adaptation of large pretrained models, but often lag behind full fine-tuning in both convergence speed and final performance. Recent approaches aim to reduce this gap by aligning LoRA parameter updates with those of full fine-tuning, but such parameter-space alignment only indirectly controls model predictions. Instead, we adopt a function-space perspective and formulate the prediction alignment problem, with the objective of matching the outputs of LoRA fine-tuning to those of full fine-tuning, both after the first training step and at convergence. We show that these two alignment objectives are governed by the curvature of the loss: first-step alignment leads to a curvature-colored gradient, whereas alignment at convergence leads to a Newton-like, curvature-whitened gradient. Building on these characterizations, we introduce Curvature-Guided LoRA (CG-LoRA), with two variants that initialize the adapter column spaces using the dominant singular subspaces of the colored and whitened gradients, respectively. We further derive the initialization spectrum from conditioning considerations, eliminating the need for scale tuning. Across natural language understanding, mathematical reasoning, and code generation benchmarks, CG-LoRA matches or exceeds state-of-the-art non-zero-initialization LoRA variants at a lower initialization cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
parameter-efficient fine-tuning, function-space alignment, Kronecker-factored curvature, linearized training dynamics.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_bxpmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes Curvature-Guided LoRA (CG-LoRA), an initialization method for LoRA. It aims to align LoRA outputs with those of full fine-tuning after the first update and at convergence. Its two variants initialize the adapter column spaces from the dominant singular subspaces of curvature-colored and curvature-whitened gradients, respectively. Experiments on T5-base and LLaMA-2-7B show competitive performance with strong baselines, with lower initialization cost on LLaMA-2-7B.

Critical strengths
  • The motivation is clear. It focuses on aligning model outputs rather than parameter updates, accounting for how the Jacobian maps weight updates to output changes.
  • The experiments show competitive performance and faster initialization than the compared non-zero initialization baselines on a large model, LLaMA-2-7B. The supplementary code also supports reproducibility.
Critical weaknesses
  • The theoretical objective does not fully explain the practical design. Theorem 5.1 favors covering the top 2r modes, but CG-LoRA-W uses paired top-r modes, which perform better on CoLA in Appendix F.4. This leaves the connection between optimal prediction alignment and downstream performance unclear.

  • In Table 1, the gains are small on most tasks, and plain LoRA remains competitive on SST-2, QNLI, and STS-B. This raises questions about whether the proposed initialization consistently benefits LoRA training. In Table 2, both variants have lower mean performance than LoRA-One on HumanEval, while CG-LoRA-C shows only a small gain on GSM8K. Overall, the performance advantage of the proposed method is not sufficiently clear.

Questions for authors
  1. Beyond the controlled network in Appendix F.1, can you directly measure output alignment with full fine-tuning on T5-base and LLaMA-2-7B after the first update and after training? Does better alignment correspond to better downstream performance in the existing experiments?
  2. Given that plain LoRA remains competitive on several tasks in Table 1 and LoRA-One achieves a higher mean score on HumanEval in Table 2, can you provide stronger evidence that the proposed initialization meaningfully improves LoRA training and clarify when it offers an advantage over these baselines?
  3. How much of the initialization-speed advantage remains when PiSSA uses randomized SVD and LoRA-One uses a smaller sketch and fewer power iterations? The reported comparison uses exact SVD for PiSSA and a sketch size of 512 with 16 power iterations for LoRA-One.
  4. Is there a practical rule for choosing between CG-LoRA-C and CG-LoRA-W without evaluating both?
4 comments

We thank the reviewer for their insightful questions. We address your concerns below.

W1 – paired versus disjoint index sets. We agree that the paper reports the comparison (Appendix F.4) without its explanation, which follows from the horizon of Theorem 5.1. By Theorem 5.1(ii), E∞=(∑i∉I∪Jsi2(F))1/2E_\infty=\big(\sum_{i\notin\mathcal I\cup\mathcal J}s_i^2(F)\big)^{1/2}: at convergence, a mode ii of FF is captured as soon as one factor reaches it (i∈Ii\in\mathcal I or i∈Ji\in\mathcal J). Covering 2r2r modes is then optimal, and the disjoint construction attains the bound of Theorem 5.1(i) in the controlled network of Appendix F.1. Fine-tuning, however, stops after a finite number of steps, so how fast each mode is learned matters too. Paired index sets correct this far-sightedness of E∞E_\infty. One LoRA step is a full fine-tuning step preconditioned by PP (Eq. 7), Pu=η2B0B0⊤u+u η2A0⊤A0Pu=\eta^2B_0B_0^\top u+u\ \eta^2A_0^\top A_0. At the flat point, η2B0B0⊤\eta^2B_0B_0^\top and η2A0⊤A0\eta^2A_0^\top A_0 are 23\tfrac23 times the projectors onto col⁡(B0)\operatorname{col}(B_0) and col⁡(A0⊤)\operatorname{col}(A_0^\top), so for a mode u=ab⊤u=ab^\top, Pu=43uPu=\tfrac43u when both factors reach it (a∈col⁡(B0)a\in\operatorname{col}(B_0) and b∈col⁡(A0⊤)b\in\operatorname{col}(A_0^\top)) and Pu=23uPu=\tfrac23u when only one does. With small steps, paired sets, I=J=[r]\mathcal I=\mathcal J=[r], thus learn the rr dominant modes twice as fast, at the price of modes r+1,…,2rr+1,\dots,2r. Table 14 follows this order on CoLA: the rr dominant modes reached by both factors (paired, 50.150.1), r/2r/2 of them (half-overlapping, 48.348.3), none (disjoint, 47.547.5), and the dominant modes not reached at all (shifted, 44.644.6). We will add this explanation to Section 5.3 and Appendix F.4.

Q1 – Output alignment with full fine-tuning. We thank the reviewer for this question, which we answer with direct measurements on both models.

After the first update. The first-step objective E1E_1 needs no full fine-tuning run: E1=∥J vec(D1)∥E_1 = \Vert J\ \mathrm{vec}(D_1)\Vert is computed exactly without the Kronecker approximation. Table R6 reports it relative to ∥J vec(∇)∥\Vert J\ \mathrm{vec}(\nabla)\Vert, the change of the predictions under the first step of full fine-tuning.

Table R6. First-step misalignment E1/∥J vec(∇)∥E_1/\Vert J\ \mathrm{vec}(\nabla)\Vert on 3232 held-out examples, all adapted layers jointly. Lower is better.

CoLAMRPCGSM8K
rsLoRA0.89 (48.6)0.88 (84.4)0.99 (59.0)
PiSSA0.58 (44.5)0.46 (85.1)0.51 (58.3)
LoRA-GA53.2 (44.0)57.5 (84.6)128.8 (57.1)
LoRA-One0.79 (47.3)0.45 (84.8)0.60 (58.9)
CG-LoRA-W0.72 (49.8)0.86 (84.6)0.92 (58.7)
CG-LoRA-C0.26 (49.4)0.39 (85.6)0.35 (59.4)

CG-LoRA-C has the lowest misalignment on every task, 1.21.2 to 2.22.2 times below the next method. CG-LoRA-C is also the best of these methods on MRPC and GSM8K and within 0.40.4 points of the best on CoLA. Initializations whose first step overshoots that of full fine-tuning, such as LoRA-GA (5050 to 130130-fold in our setting), are among the weakest. Among the others, E1E_1 alone does not rank accuracy, which is why CG-LoRA-C keeps the flat point rather than the first-step scale of Proposition 5.2(ii).

After training. This measurement does not distinguish the methods. On T5-base, at the end of the epoch, the predictions of every adapter are at a similar distance from those of full fine-tuning (relative distance of the label-word scores, 0.190.19 to 0.280.28 on CoLA and MRPC). Our end-of-training objective serves to select subspaces, not to describe the trained model, and on LLaMA-2-7B full fine-tuning is beyond our budget.

W2 and Q2 – Gains. We thank the reviewer for these points, which Reviewer otSn also raises. We answer them in the general response ("Size and consistency of the gains"), which also says when CG-LoRA offers an advantage over plain LoRA and LoRA-One.

Q3 – Initialization speed against faster baselines. On LLaMA-2-7B in the setting of Table 3 (three new runs), LoRA-One with the sketch of LoRA-GA (q=4rq = 4r, four power iterations) returns the same subspaces as its released settings (overlap 11 on all 224224 layers) in 4646 s instead of 8888 s (8585 s in the earlier runs of Table 3). CG-LoRA, at 3939 s (3636 s in Table 3), remains the fastest gradient-based initialization and the only one that holds no host memory, against 12.612.6 GiB for LoRA-GA and LoRA-One. PiSSA uses no data, and a randomized SVD brings it to 22 s, at the cost of subspaces that differ from those of the exact SVD (median overlap 0.630.63). We will add both variants to Table 3.

Q4 – Choice of variant. We thank the reviewer for this question, which we answer in the general response ("Choosing between CG-LoRA-C and CG-LoRA-W").

Thank you for the response and additional experiments, which I appreciate. They address some of my concerns. However, I find that better alignment does not consistently lead to better downstream performance across the evaluated methods. I will therefore maintain my original score, and I encourage the authors to explore this relationship further.

Thank you for the follow-up. We are glad that our response and the additional experiments addressed some of your concerns. On the remaining one, we would like to clarify the scope of our claim using elements in the paper and in our first response.

E1E_1 is the distance between the first-order changes of the predictions under one LoRA step and one full fine-tuning step, defined on the network itself. The paper uses it to construct the initialization and then evaluates the result on real models. We therefore claim that minimizing prediction misalignment yields a strong initialization, not that misalignment predicts accuracy for arbitrary methods. As noted in our response to Q1, E1E_1 alone does not rank accuracy across the six methods. We agree with your reading and will state it in the paper.

What Table R6 does show is that the construction achieves its objective. It measures E1E_1 directly on the network, with no approximation, and CG-LoRA-C attains the lowest value on every task. It is also the best of these methods on MRPC and GSM8K, and within 0.40.4 points of the best on CoLA, which is CG-LoRA-W. We do not claim the converse: other methods can reach good accuracy by other routes. A zero initialization, for instance, is misaligned at the first step by construction but can still align during training. CG-LoRA-W targets the end-of-training objective, not E1E_1, and remains competitive on our tested models.

As stated in our general response, an important contribution of our work is theoretical. The experiments further establish that the resulting initialization achieves its objective on real networks and is competitive at a low initialization cost. We agree that characterizing when alignment is also necessary for accuracy is an open question for this line of work, which our formulation makes measurable, and we will include Table R6 and this discussion in the revision. We thank you again for your valuable feedback.

Thank you for the reply. After careful consideration, I decide to increase my rating.

Reviewer_otSnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes CG-LoRA, a curvature-guided LoRA initialization that aligns LoRA predictions with full fine-tuning at the first step and at convergence. Under a linearized model and K-FAC approximation M≈S⊗T, it derives whitened-gradient subspaces for convergence alignment and colored-gradient subspaces for first-step alignment, plus a flat-point scaling. Experiments use T5-base/GLUE and LLaMA-2-7B.

Critical strengths
  1. Prediction-space alignment is a clean and well-motivated framing.
  2. Theorem 5.1 (exact Eckart–Young solution) and Proposition 5.3 (condition-number bound / flat point) are elegant.
  3. The sketch-based implementation avoids large matrices and has low measured initialization cost.
Critical weaknesses
  1. Gains are not convincing. On LLaMA-2-7B, GSM8K is +0.4 over rsLoRA with 3 seeds, and HumanEval mean is below LoRA-One. “Matches or slightly exceeds” is not enough.

  2. Outdated evaluation. Only T5-base and LLaMA-2-7B; no current open-source model (Llama-3, Qwen-3.5). Practical relevance is limited for a 2027 submission.

  3. Strong assumptions, unclear payoff. End-of-training alignment relies on linearization and M=S⊗T; Proposition C.4’s error grows with update size, and controlled experiments show order-one error at long horizons. The implemented colored variant uses the flat-point scale rather than the scale from Proposition 5.2(ii), so it does not exactly optimize the stated objective.

  4. High complexity relative to benefit. Probe batches, hooks, random-sign probes, randomized range finding, per-layer SVDs, and rank-deficient completion add substantial machinery over LoRA-GA/LoRA-One without demonstrated robust gains.

  5. No composition experiments. The paper argues initialization is orthogonal to DoRA/AdaLoRA/LoRA-Pro but does not test composition, so the practical value of another initialization is unclear.

Questions for authors
  1. Results on at least one current open-source model?
  2. Direct measurement of how close the LLaMA run is to the linearized/K-FAC regime?
  3. Does CG-LoRA compose with LoRA-Pro or DoRA?
3 comments

We thank the reviewer for their detailed assessment. Below, we address the concerns about our theoretical assumptions and practical benefits, and report the requested additional experiments.

W1 – Gains. We thank the reviewer for raising this point, also noted by Reviewer bxpm. We address it in the general response under "Size and consistency of the gains." In the revision, we will replace "matches or exceeds" with precise, task-specific statements. CG-LoRA-C stays within 0.40.4 points of the best adapter on every task in Table 1. CG-LoRA achieves strong worst-seed performance on the LLM benchmarks.

W2/Q1 – Outdated evaluation. We thank the reviewer for this suggestion. We have evaluated Llama-3.1-8B (base), a newer model used in related work including LoRA-Pro, GoRA, and EVA, following the protocol of Table 2. Given the time available for the rebuttal, all four methods use a fixed learning rate of 5×10−45\times10^{-4}, the rate selected for each on LLaMA-2-7B.

Table R4. Llama-3.1-8B, GSM8K accuracy, three seeds. Bold: best adapter.

meanworst
rsLoRA72.1 ± 0.771.3
LoRA-One72.2 ± 0.172.0
CG-LoRA-W72.8 ± 0.872.0
CG-LoRA-C72.1 ± 0.571.5

CG-LoRA-W achieves the highest mean accuracy and outperforms rsLoRA on all three seeds, with an average gain of 0.70.7 points (paired tt-test, p=0.03p=0.03). CG-LoRA-C performs comparably to rsLoRA and LoRA-One. These results extend the evidence for CG-LoRA's competitiveness to a newer model. We will add Table R4 to Section 6.

W3 – Strong assumptions, unclear payoff. We thank the reviewer for this careful reading. As in related work, our theory uses idealized models to derive principled initialization rules whose practical value we then evaluate on nonlinear networks.

The CG-LoRA-C objective is defined directly on the network (Section 3.1) and does not require linearized training dynamics. Its closed-form construction (Corollary 4.3) uses the Kronecker assumption M=S⊗TM=S\otimes T. LoRA-One's theoretical analysis implicitly uses the isotropic special case of this structure. Propositions 5.2(iii) and B.1 show that CG-LoRA-C recovers LoRA-One's subspaces under these isotropic assumptions, and otherwise generalizes it with consistent empirical benefits detailed in our general response. In implementation, the algorithm uses estimates of SS and TT even when the KFAC is approximate, as LoRA-DA does.

The convergence objective E∞E_\infty, used by CG-LoRA-W, additionally assumes a linearized model. In return, Theorem 5.1 solves this objective exactly. We agree that this is a stronger assumption. Malladi et al. (2023) and Jang et al. (2024) study fine-tuning and LoRA in this regime, and Q2 below measures it on LLaMA-2-7B.

Proposition C.4 applies only to CG-LoRA-W and bounds the gap to the linearization by a term quadratic in the update. In the controlled network of Appendix F.1, the closed form tracks the network within 4% over the two shortest horizons, then departs from it.

W3 – The scale of CG-LoRA-C. We thank the reviewer for this remark. The chosen scale deliberately compensates for the limited horizon of E1E_1: the objective considers only the first step, whereas subsequent gradients can explore additional directions.

A LoRA step corresponds to a full fine-tuning step preconditioned by PP (Eq. 7). A full fine-tuning direction uu becomes PuPu, giving a discrepancy ∥(I−P)u∥\Vert(I-P)u\Vert. As in Section 5.2, write η2B0B0⊤=UΣBU⊤,η2A0⊤A0=VΣAV⊤,\eta^2B_0B_0^\top=U\Sigma_BU^\top, \qquad \eta^2A_0^\top A_0=V\Sigma_AV^\top, with flat spectra ΣB=σBI\Sigma_B=\sigma_B I and ΣA=σAI\Sigma_A=\sigma_A I.

Consider rank-one directions u=ab⊤u=ab^\top, where aa and bb lie either in the retained output and input subspaces or in their orthogonal complements. If both lie in the retained subspaces, then Pu=(σA+σB)uPu=(\sigma_A+\sigma_B)u. If only aa lies in its retained subspace, then Pu=σBuPu=\sigma_Bu. If only bb does, then Pu=σAuPu=\sigma_Au. These three classes span the directions accessible to the linearized LoRA update.

The scale σA=σB=12\sigma_A=\sigma_B=\tfrac12 in Proposition 5.2(ii) exactly matches full fine-tuning along directions where both factors act. By construction, these include the retained modes of the first gradient, which is what E1E_1 rewards. However, this scale leaves a relative discrepancy of 12\tfrac12 in directions where only one factor acts, which later gradients may contain.

By contrast, σA=σB=23\sigma_A=\sigma_B=\tfrac23 leaves a relative discrepancy of 13\tfrac13 in each of the three classes, minimizing the worst-case discrepancy over accessible directions (Proposition 5.3). Replacing 23\tfrac23 with 12\tfrac12 in CG-LoRA-C, with everything else unchanged, reduces performance by 2.22.2 points on CoLA and 2.12.1 points on MRPC across ten seeds. We will clarify this rationale in the revision.

Q2 – Measuring the linearization and K-FAC approximations on LLaMA.

Linearization. Let ΔW\Delta W denote the final trained update for each method in Table 2, and let ff denote the logits at the answer tokens of 3232 held-out MetaMathQA examples. We measure ρ(t)=∥f(W0+tΔW)−f(W0)−t J vec(ΔW)∥∥f(W0+tΔW)−f(W0)∥,\rho(t)=\frac{\Vert f(W_0+t\Delta W)-f(W_0)-t\ J\ \mathrm{vec}(\Delta W)\Vert}{\Vert f(W_0+t\Delta W)-f(W_0)\Vert}, computing J vec(ΔW)J\ \mathrm{vec}(\Delta W) exactly by forward-mode differentiation.

At t=1t=1, ρ\rho is 2.52.5 for CG-LoRA-C, 2.82.8 for CG-LoRA-W, and 3.33.3 for rsLoRA. For small displacements along this direction, however, ρ\rho remains below 0.10.1 up to t=0.01t=0.01 and grows approximately linearly with tt, consistent with a quadratic remainder as in Proposition C.4. This measurement supports local accuracy near initialization. It does not establish that the full training trajectory remains in the linearized regime. CG-LoRA-C targets first-step alignment directly, while CG-LoRA-W uses convergence in the linearized model as a design principle for selecting subspaces whose practical performance is evaluated in Tables 1 and 2.

Kronecker structure. For a layer and update direction Δ\Delta, we compare the exact quadratic form q=vec(Δ)⊤M vec(Δ)=∥J vec(Δ)∥2q=\mathrm{vec}(\Delta)^\top M\ \mathrm{vec}(\Delta) =\Vert J\ \mathrm{vec}(\Delta)\Vert^2 with its Kronecker approximation q^=tr(TΔSΔ⊤).\hat q=\mathrm{tr}(T\Delta S\Delta^\top). We evaluate 5656 layers using the trained update directions from Table 2 and a random direction. With expand factors, q^/q\hat q/q has a geometric mean of 0.790.79 and is typically within a factor of 33 of one. The approximation also preserves the ranking of directions reasonably well, with a Spearman correlation of 0.770.77. Reduce factors, as used in our sketch, yield a Spearman correlation of 0.690.69.

W4 – Complexity relative to benefit. The additional implementation components arise from generalizing beyond isotropic curvature. When SS and TT are multiples of the identity, the colored gradient reduces, up to scaling, to the plain gradient used by LoRA-GA and LoRA-One to select their subspaces. Generalizing this construction requires estimating the two Kronecker factors. The estimation uses standard components, many of which are shared with existing baselines:

Probe batch: data-dependent initializations such as LoRA-GA, LoRA-One, CorDA, EVA, and LoRA-DA also require one.

Hooks: these collect layer inputs and output derivatives, following standard practice in K-FAC and related initialization methods, including EVA, CorDA, and LoRA-DA.

Random-sign probes: Hutchinson's estimator provides an estimate of TT without requiring one backward pass per output.

Randomized range finding: LoRA-GA and LoRA-One also use randomized SVD, with 44 and 1616 power iterations, respectively. Our method uses 00 on T5-base and 11 on LLaMA-2-7B.

Per-layer SVDs: our final decompositions operate on m×mm\times m matrices with m=16m=16, whereas LoRA-GA and LoRA-One target the dout×dind_{\mathrm{out}}\times d_{\mathrm{in}} gradient and PiSSA targets the weight matrix.

These components add neither measured cost nor tuning. On LLaMA-2-7B, initialization takes 3636 s, compared with 4949 s for LoRA-GA, 8585 s for LoRA-One, and 239239 s for PiSSA. CG-LoRA requires no additional host-memory storage, whereas LoRA-GA and LoRA-One use 12.612.6 GiB, and its peak GPU memory remains below that of LoRA training (Table 3). It requires no tuned constant, whereas LoRA-GA and LoRA-One use a stability constant γ\gamma. Our sketch settings are fixed across tasks for each model, and Table 8 shows robustness to these choices. We discuss the size and consistency of the gains in the general response.

W5/Q3 – Composition with DoRA. We thank the reviewer for this suggestion. CG-LoRA initializes the low-rank factors and can therefore be combined with methods that retain this parameterization. We tested this combination with DoRA, which decomposes each adapted weight into a magnitude and a direction. DoRA + CG-LoRA-W uses the CG-LoRA-W factors and initializes DoRA's magnitudes from W0+ηB0A0W_0+\eta B_0A_0. Both configurations train for one epoch using their individually selected learning rates, across five seeds.

Table R5. MCC on CoLA, accuracy on MRPC. RoBERTa-base: classification head trained jointly, no probe.

T5-base CoLAT5-base MRPCRoBERTa-base CoLARoBERTa-base MRPC
DoRA45.9 ± 1.285.1 ± 0.729.9 ± 27.481.6 ± 3.7
DoRA + CG-LoRA-W46.9 ± 0.884.9 ± 0.450.2 ± 1.982.4 ± 1.0

On T5-base, CG-LoRA-W improves DoRA's mean performance on CoLA and performs comparably on MRPC. On RoBERTa-base, it also improves stability: two of DoRA's five runs collapse to an MCC of 00 on CoLA, whereas none collapse with CG-LoRA-W initialization. On MRPC, the standard deviation decreases from 3.73.7 to 1.01.0. We will add these results to the appendix.

Dear Reviewer otSn,

The authors have submitted their rebuttal and we are closing in on the end of the discussion period. Please take some time to carefully read over the rebuttal and respond whether your concerns have been addressed.

Best, AC

Reviewer_BiRSOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces Curvature Guided Low Rank Adaptation (CG-LoRA), a gradient-based initialization for LoRA adapters. Unlike LoRA's weight updates which are aligned with those of full finetuning in parameter space, the paper formulates the alignment in function space. They consider two different horizons, where one is at the first gradient update step and the other is at the end of the training. The end of training horizon is analyzed is analyzed under a linearized model with squared loss. By approximating the Gauss-Newton curvature, they show that first step alignment is goverened by a 'colored' gradient while the end of training alignment is governed by a whitened gradient. The adapter factors AA and BB are initialized from the top-r singular subspaces of their gradients that are mapped through the curvature. The singular values of the factors are fixed at a “flat point” which are derived from the conditioning of the LoRA preconditioner.

Critical strengths
  • Principled reframing of LoRA initialization. Moving alignment from parameter space to function space is a natural step. The paper explains why weight directions should not be treated equally, and the discussion of existing gradient-based initializations is well motivated.
  • Curvature-guided subspaces. The K-FAC factors weight the directions of the gradient by how much they actually move the predictions, a criterion that gradient-based methods such as LoRA-GA and LoRA-One lack.
  • No scale tuning. The flat-point analysis (in Propostion 5.3) removes the tuned scale constants of other gradient-based initializations, and shows that the LoRA preconditioner has condition number κ≥2\kappa \geq 2.
Critical weaknesses
  • No full-fine tuning reference on the generation task: The paper motivates the gap between LoRA and full finetuning yet FFT has not been reported on GSM8K and HumanEval in Table 5. Most other baselines which are used for the Section 6.1 Natural Language Understanding have been dropped in the Section 6.2 Natural Language Generation.
  • The evidence for the convergence speed, which is a key motivation, yet only the final epoch metrics are reported.
  • The method is initialization only. But the Appendix D.2 indicates that ηB0A0\eta B_0 A_0 are subtracted in the forward pass. This adds additional rank-r product per layer with adapters at each training step.
Questions for authors
  • Why were full finetuning and the other methods, not run on LLaMA-2-7B? The paper uses them on T5 to argue that "curvature statistics alone” don’t explain the gains. Can that claim be tested at scale?
  • The paper does not provide guidance on choosing between the CG-LoRA-C and CG-LoRA-W variants. Is there a principled criterion for their selection as we do not see any task specific winner between these two?
  • The paper motivates the work by slower convergence of LoRA. Could you show training-loss curves for the CG-LoRA vs other gradient methods as this would directly show weather the flat point speeds up convergence?
2 comments

We thank the reviewer for the encouraging assessment. Below, we address your concerns and provide the requested additional experiments. These results strengthen the submission and will be included in the revision.

W1/Q1 – FFT on LLaMA-2-7B. We evaluated a smaller set of baselines on LLaMA-2-7B because of the substantially higher computational cost relative to fine-tuning T5. In particular, the bf16 weights and gradients, together with the fp32 Adam moments and master weights, require approximately 108108 GB before storing any activations, exceeding the capacity of our single-GPU setup. We therefore focused on the closest methodological baselines that informed CG-LoRA: rsLoRA, PiSSA, LoRA-GA, and LoRA-One. We reran these methods under our protocol to ensure a controlled comparison, rather than importing results from other studies. We will clarify this choice in the revision.

W1/Q1 – CG-LoRA gains from curvature statistics. On T5-base, EVA and LoRA-DA also construct their adapters using second-order layer statistics, yet both remain approximately 55 points below the two CG-LoRA variants on CoLA. We agree that this comparison should also be made at scale, and have therefore evaluated EVA and LoRA-DA on LLaMA-2-7B with three seeds, following the protocol of Table 2 except for learning-rate selection. Given the time available for the rebuttal, we used a fixed learning rate for each task rather than searching over seven rates. For LoRA-DA, we used 2×10−42\times10^{-4}, the sole learning rate in its released LLaMA-2-7B recipe. For EVA, we used the rate selected for rsLoRA on each task (5×10−45\times10^{-4} on GSM8K and 2×10−42\times10^{-4} on HumanEval), since both methods initialize with B0=0B_0=0. Both methods construct their initialization from the first 256256 training examples, following their released code.

Table R1. LLaMA-2-7B. EVA and LoRA-DA (last two rows) are new. The other rows are Table 2 of the paper. Bold: best adapter.

GSM8K meanGSM8K worstHumanEval meanHumanEval worst
rsLoRA59.0 ± 0.458.627.2 ± 0.926.2
PiSSA58.3 ± 0.757.525.2 ± 1.424.4
LoRA-GA57.1 ± 1.455.623.0 ± 1.921.3
LoRA-One58.9 ± 0.857.929.5 ± 3.126.2
CG-LoRA-W58.7 ± 0.957.728.7 ± 1.128.0
CG-LoRA-C59.4 ± 0.659.028.9 ± 1.327.4
EVA57.4 ± 0.456.925.4 ± 0.425.0
LoRA-DA59.0 ± 1.357.525.4 ± 0.924.4

On GSM8K, CG-LoRA-C retains the best mean and worst-seed performance. EVA, which constructs the adapter using only each layer's input covariance, falls 2.02.0 points below CG-LoRA-C and 1.31.3 points below CG-LoRA-W. LoRA-DA, which combines the gradient with a K-FAC estimate of the Fisher as CG-LoRA does, achieves 59.059.0, matching rsLoRA and close to LoRA-One. Its variation across seeds is approximately twice that of CG-LoRA-C, and its worst-seed performance is 1.51.5 points lower. On HumanEval, both the mean and worst-seed performance of EVA and LoRA-DA are substantially below those of both CG-LoRA variants. These results support the effectiveness of CG-LoRA's construction.

W3 – Subtraction of ηB0A0\eta B_0A_0. Performing the subtraction of ηB0A0\eta B_0A_0 in the forward pass is an implementation choice, rather than a requirement of the method. As stated in Section 3, the offset can instead be subtracted once from the frozen weights at initialization: W0←W0−ηB0A0.W_0\leftarrow W_0-\eta B_0A_0. Training then uses a standard LoRA layer at the same per-step cost. On LLaMA-2-7B under the setup of Table 2, we measured 1.771.77 s per step for both LoRA and CG-LoRA with the offset folded into the frozen weights. At inference, neither implementation adds overhead, since the trained adapter can be merged with the original pretrained weights to give W0+η(BA−B0A0)W_0+\eta(BA-B_0A_0).

In our experiments, both CG-LoRA variants and LoRA-GA used the forward-pass subtraction, while PiSSA used its released residual implementation. The results in Table 2 therefore remain unchanged. The forward-pass implementation preserves the pretrained function at initialization, as assumed in our analysis, even when the base weights are stored in bf16. We will describe both implementation options in Appendix D.2.

W2/Q3 – Convergence speed and training/validation curves. We agree that reporting performance throughout training, particularly the validation metric, provides useful additional evidence. Table R2 reports validation results averaged over the seeds used in Table 1. The final column averages all ten evaluations during the epoch, summarizing performance across training.

Table R2. Validation metric during the epoch, T5-base, same runs as Table 1. Bold: best adapter.

CoLA (MCC)

fraction of the epoch10%20%30%50%100%average
FFT, query/value3.319.538.144.847.438.1 ± 1.6
rsLoRA2.815.335.443.748.637.8 ± 1.7
PiSSA4.19.536.340.644.534.4 ± 2.5
LoRA-GA3.212.930.639.344.033.9 ± 2.5
LoRA-One3.922.634.039.147.336.6 ± 5.8
CG-LoRA-W4.622.134.242.949.838.9 ± 2.4
CG-LoRA-C6.328.340.445.249.440.1 ± 3.3

MRPC (accuracy)

fraction of the epoch10%20%30%50%100%average
FFT, query/value71.576.480.485.186.282.6 ± 0.9
rsLoRA68.070.576.481.584.479.6 ± 1.1
PiSSA68.771.879.883.285.181.0 ± 0.8
LoRA-GA69.075.178.783.284.680.9 ± 1.3
LoRA-One69.674.178.982.684.880.7 ± 0.9
CG-LoRA-W69.571.077.083.084.680.2 ± 1.3
CG-LoRA-C70.075.078.583.285.681.6 ± 1.0

CG-LoRA-C achieves the highest average validation metric among the adapters on both CoLA and MRPC. On CoLA, it leads at every evaluation through the midpoint of the epoch. Its average exceeds those of LoRA-One (+3.5+3.5, p=0.04p=0.04, paired tt-test across ten seeds) and rsLoRA (+2.4+2.4, p=0.07p=0.07). Both CG-LoRA variants exceed LoRA-GA and PiSSA by at least 4.54.5 points (p<0.01p<0.01). On MRPC, CG-LoRA-C improves over rsLoRA by 2.02.0 points (p<0.01p<0.01) and LoRA-One by 0.80.8 points (p=0.02p=0.02). On SST-2 and QNLI, where performance is close to saturation, all methods remain within 0.70.7 points of one another at every evaluation.

Table R3. Training loss, mean over windows of the epoch: T5-base (same runs as Table R2) and LLaMA-2-7B (Table 2).

windowCoLA 0–10%CoLA 10–25%CoLA last 10%MRPC 0–10%MRPC 10–25%MRPC last 10%GSM8K 0–10%GSM8K 10–25%GSM8K last 10%
FFT, query/value5.700.6360.4652.750.3130.162–––
rsLoRA5.860.6390.4553.820.3680.1740.2330.1810.118
PiSSA5.390.6550.4783.360.3770.1740.2330.1770.121
LoRA-GA5.590.6530.4882.740.3270.1790.2390.1860.131
LoRA-One3.670.6140.4661.680.3300.1720.2350.1800.118
CG-LoRA-W5.300.6260.4553.380.3580.1760.2300.1780.118
CG-LoRA-C4.870.6060.4553.030.3220.1660.2300.1790.118

Table R3 reports training loss. On LLaMA-2-7B, rsLoRA, LoRA-One, and both CG-LoRA variants use the same learning rate and follow nearly identical training-loss curves. After the first 2% of the epoch, their losses remain within 3% of one another, and they finish at the same loss.

On CoLA and MRPC, LoRA-One achieves the lowest training loss early in training. This reflects its shifted initialization: training starts at W0+ηB0A0W_0+\eta B_0A_0, after a step along the negative gradient, whereas CG-LoRA, rsLoRA, PiSSA, and LoRA-GA start at the pretrained function. This lower training loss does not translate into better validation performance: LoRA-One has a lower average validation metric than CG-LoRA-C on both CoLA and MRPC (Table R2). Both CG-LoRA variants achieve lower early training loss than rsLoRA on every task reported in Table R3, with values comparable to those of PiSSA and LoRA-GA.

Q2 – Choice of variant. We thank the reviewer for this question and address it in our general response, under "Choosing between CG-LoRA-C and CG-LoRA-W."

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response

We thank all reviewers for their thoughtful feedback and for recognizing the motivation and theoretical effort behind our analysis of LoRA at the prediction level. To our knowledge, our contribution is the first end-to-end theory of LoRA initialization based on aligning its predictions with those of full fine-tuning. We formulate alignment objectives after the first step and at convergence (Section 3). Under Kronecker-factored curvature, we solve the convergence objective exactly (Theorem 5.1), derive a bound for the first-step objective (Proposition 5.2), and determine the spectra through the conditioning of the LoRA preconditioner (Proposition 5.3). Every component of the algorithm follows from these results, without tuned constants. LoRA-One's analysis is recovered as the isotropic special case (Proposition B.1). Our rebuttal further develops the theory by explaining how the two departures from the optima account for the horizon of each objective (responses to Reviewer otSn, W3, and Reviewer bxpm, W1). On real networks, CG-LoRA-C also achieves the lowest first-step misalignment among the six methods evaluated (Table R6).

Choosing between CG-LoRA-C and CG-LoRA-W. We thank Reviewers BiRS and bxpm for this question. CG-LoRA-C is our default recommendation. Its objective, prediction alignment after the first step, is defined directly on the network and requires only a Kronecker-factored metric (response to Reviewer otSn, W3). It achieves the highest average validation metric among LoRA variants on CoLA and MRPC (Table R2), the best mean and worst-seed performance on GSM8K with LLaMA-2-7B (Table 2), and the lowest first-step misalignment on every evaluated task (Table R6, response to Reviewer bxpm).

CG-LoRA-W is a principled extension that additionally assumes linearized training dynamics (Theorem 5.1). Its performance remains within one point of CG-LoRA-C on every task in Tables 1 and 2. We recommend it when the output layer is newly initialized, where it shows greater robustness. For RoBERTa-base (Appendix F.3), CG-LoRA-W has no collapsed runs on MRPC, either with or without a probed head, compared with four and two, respectively, for CG-LoRA-C. On CoLA with a probed head, it has only one collapsed run out of twenty, compared with six for CG-LoRA-C.

Size and consistency of the gains. We thank Reviewers otSn and bxpm for raising this point. The modest absolute gains on some tasks reflect the limited room for improvement over standard LoRA. Within this range, CG-LoRA delivers consistently strong performance across tasks.

Limited headroom, substantial gains where improvement remains possible. Full fine-tuning (FFT) is a natural reference for evaluating LoRA initialization. On SST-2, QNLI, and STS-B, FFT exceeds plain LoRA by at most 0.60.6 points (Table 1), indicating limited headroom relative to this reference. On smaller datasets, however, our methods achieve larger gains: CG-LoRA-W improves over plain LoRA by 5.55.5 points on CoLA, and CG-LoRA-C by 3.83.8 points on MRPC. On MRPC, CG-LoRA-C also closes more than half the gap between rsLoRA and FFT, improving from 84.484.4 to 85.685.6, compared with 86.686.6 for FFT. The advantage is greatest early in training, the regime targeted by CG-LoRA-C (Table R2).

CG-LoRA therefore provides the largest benefits where fine-tuning has more room to improve and can sometimes outperform FFT itself. For LLMs, FFT is too costly under our experimental protocol (response to Reviewer BiRS, W1), so we compare against the strongest adapter baselines. These gains require no additional per-step training cost and no constant to tune, whereas LoRA-GA and LoRA-One rely on a stability constant γ\gamma. On LLaMA-2-7B, CG-LoRA initialization is also 1.41.4–6.66.6 times faster than LoRA-GA, LoRA-One, and PiSSA in their released settings (Table 3).

Consistently competitive across tasks. When performance differences are small, consistency across tasks is particularly informative. CG-LoRA-C is within 0.40.4 points of the best adapter on all five tasks in Table 1. Every baseline, including rsLoRA, falls at least 1.21.2 points behind on at least one task. On GSM8K, a CG-LoRA variant achieves the highest mean performance on both LLaMA-2-7B and Llama-3.1-8B (Table R4, response to Reviewer otSn, W2). On HumanEval, LoRA-One's higher mean is driven by a single seed, whereas both CG-LoRA variants achieve better worst-seed performance than every evaluated baseline, including the newly evaluated EVA and LoRA-DA (Table R1, response to Reviewer BiRS). We will include these additional results in the revision.

Dear reviewers,

The authors have submitted their rebuttal. Please read it carefully, along with the other reviews, and respond whether it adequately addresses your concerns and whether you plan to update your score.

If concerns remain, please state the open points clearly and specifically so the authors can follow up during the discussion period.

Thank you for your engagement!

Best, AC

Dear reviewers,

Thank you again for reviewing our paper. We believe we have addressed all your questions and concerns by providing additional experiments demonstrating the strong advantages of our algorithms and clarifying our contributions (a first end-to-end perspective on LoRA). We would appreciate if you could let us know what you think!

Many thanks!

The authors

AuthorsOpenReview ↗

Closing comments

We thank the reviewers and the AC for this productive discussion. As it nears its end, we give a short summary. Two of the three reviewers responded and acknowledged that our response addressed concerns they had raised.

  • Reviewer BiRS confirmed that their questions were adequately addressed.
  • Reviewer bxpm raised one remaining point: better alignment does not consistently yield better accuracy across methods. We agree and will state it in the paper, but our claim is purely constructive. We introduce a theoretical framework for optimizing prediction alignment, yielding an initialization that attains in practice the lowest E1E_1 on every task (Table R6) and is consistently competitive across the evaluated tasks. We believe these findings establish the value of the approach.
  • Reviewer otSn has not yet responded. All experiments they requested are reported (Tables R4 and R5, response to Q2). They support the practicality of our initialization on a newer model and in combination with DoRA, and we believe our response addresses their concerns.

To our knowledge, our work is the first end-to-end theory of LoRA initialization based on aligning its predictions with those of full fine-tuning. This theory recovers previous results (those from LoRA-One’s paper), and yields an efficient algorithm without tuned constants. That is the core contribution and we are glad that all three reviews list the prediction-level formulation among the paper’s strengths, with Reviewers BiRS and otSn highlighting the theoretical results. On real networks, the algorithm achieves its objective (Table R6) at low cost. The main shared reservation concerns the size of the empirical gains, and the additional experiments of the rebuttal (Tables R1, R2 and R4) show that the algorithm is consistently competitive.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.