- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies the geometry of non-zero LoRA initializations and its impact on learning rate scaling. The authors use a first order factor update to charachterize updates that remain the initial tangent space versus updates corresponding to departure from that initial tangent space. They show that scaling initialization by a factor of leads to a factor update size in features, where p depends on the optimiser. Based on this, they derive some rules for learning rate scaling with initialization scale a.
Critical strengths
- The assumptions are explicit in the paper, and the authors made the scope of their result very clear.
- The tangent space approach is useful for first order analysis of the training dynamics near initialization. It reminds me somewhat of the neural tangent kernel, where features of not move significantly from initialization.
Critical weaknesses
- Theory: the analysis is restricted to the lazy training regime (the tangent space near initialization). It is well known that LoRA, especially on hard tasks, does not operate in the lazy training regime, and significant feature learning (as measured from the cross herm) occurs. Also, the proof’s constant grows like , and therefore for T large enough (realistic in the experiments), the bound is not meaningful (likely vacuous).
- Significance and practical implications: it is unclear to me what the implications of these results are. The paper studies non-zero initialization of LoRA adapters, although the default is zero initialization. The authors do not discuss or validate why we should even consider these non-zero initializations. Even there empirical results in Fig 2 suggest that as initialization scale gets smaller and smaller, training loss improves.
- Learning rate transfer: I suggest the authors remove this from the title. Learning rate transfer is a well-established notion and it generally refers to how learning rate should scale with model size or training horizon (HP scaling laws). Here, the learning rate scaling is with respect to initialization scale . The authors did not mention in which case this is useful. Why would one consider scaling initialization scale a? Are you suggesting one should tune a? If that’s the case, would the increase in tuning cost justify the gains (if any)? None of this was discussed in the paper.
Questions for authors
See weaknesses.
7 comments
[W1] Lazy Training and the Time Dependence of the Bound
Scope of the lazy-training analysis
We appreciate the reviewer’s perspective on lazy training and would like to clarify the scope of our theoretical results. Theorem 3 provides asymptotic sufficient conditions for optimal LR transfer in a lazy-training limit. The LR scaling exponent follows from Theorem 1 under the stated response-order conditions, without requiring the full training trajectory to remain in the initial tangent space . In Theorem 3, lazy-training behavior emerges as the initialization amplitude increases, under the stated regularity and LR scaling conditions.
The initialization scale also matters when characterizing LoRA’s training regime. With zero initialization, , the contribution of changes in to the weight displacement appears through the cumulative cross term . The importance of this term alone does not determine the regime across all LoRA configurations. For homogeneous models, initialization scale is known to influence the transition between lazy and rich regimes [R7, R8]. The LoRA factor product is also 2-homogeneous, motivating an examination of its behavior across initialization scales.
Time dependence of the bound
We acknowledge that the factor can make the bound numerically loose at large . The numerical tightness of this bound and the accuracy of the approximation in practice should be evaluated separately.
Theorem 3 concerns at fixed . Its proof accumulates possible error amplification at each step using a common Lipschitz constant. Exponential time dependence also appears in the general finite-time result of Chizat et al. [R7]. That work improves the dependence for square loss and establishes time-uniform results under strong convexity and additional conditions.
In the normalized coordinates of Lemma B.6, the cumulative cross term has an explicit coefficient of exactly . To assess the approximation at finite initialization amplitudes and practical training horizons, we directly compared LoRA with its linearized counterpart. We separately evaluated LR prediction across practical initialization methods.
Direct comparison with linearized LoRA
We compared LoRA and linearized LoRA at matched gains using the fixed-basis initializations on Qwen3-0.6B. The linearized model excludes the cumulative cross term . Both models used the same optimizer and minibatch order, with FP32 adapter branches.
Across the two normalized learning rates and , the absolute difference in mean training loss was at most 0.0007 in every reported window after step 100. The maximum before rounding was 0.00068, and the difference over the final 100 steps was at most 0.0002 (Table R1). These results show that linearized LoRA closely reproduced LoRA’s training loss in the tested finite-amplitude settings.
LR prediction beyond the fixed-shape amplitude family
We next used a reference LR from DST and the magnitude of each initialization to predict search starting points for five initialization methods on Bengali adaptation with Llama-3.1-8B. For four methods, the interpolated optimal LR was within a factor of approximately 1.78 of the prediction. For three methods, the loss at the grid point nearest the predicted LR was within 0.001 of the minimum over the search grid. For LoRA-GA, the observed optimal LR was 0.30 times the prediction, indicating that lower LRs needed to be explored (Table R2).
We also examined vanilla LoRA, which falls outside the direct guarantees of Theorem 3. Its departure from the initial input subspace reached 68.0%. Training diverged at the predicted LR, while the observed optimal LR was 0.60 times the prediction. For rsLoRA, which also uses zero-product initialization, the observed optimal LR was 0.85 times the prediction.
Nevertheless, these results suggest that the LR prediction based on our scaling rule can serve as a starting point for LR search beyond the fixed-shape amplitude families studied in Theorem 3, including settings with substantial departure from the initial first-order update space.
References
[R7] Lénaïc Chizat et al., On Lazy Training in Differentiable Programming, NeurIPS, 2019.
[R8] Blake Woodworth et al., Kernel and Rich Regimes in Overparametrized Models, COLT, 2020.
Table R1. LoRA–linearized LoRA training-loss differences.
Each entry is the difference in mean training loss, , over the indicated window. Results use one training seed.
| Steps 100–200 | 200–400 | 400–800 | 800–1200 | Final 100 steps | ||
|---|---|---|---|---|---|---|
| 0.1 | −0.0003 | −0.0001 | −0.0001 | −0.0002 | 0.0000 | |
| 1 | −0.0001 | −0.0001 | 0.0000 | −0.0001 | 0.0000 | |
| 10 | −0.0001 | −0.0002 | 0.0000 | 0.0000 | +0.0001 | |
| 0.1 | −0.0004 | −0.0007 | −0.0003 | −0.0005 | −0.0002 | |
| 1 | −0.0004 | +0.0001 | 0.0000 | −0.0001 | −0.0001 | |
| 10 | +0.0001 | +0.0001 | +0.0001 | 0.0000 | 0.0000 |
Table R2. LR transfer on Bengali adaptation.
The observed optimal LR is estimated by interpolation. The loss gap is the difference in mean training loss over the final 100 steps between the grid point nearest the predicted LR and the best grid point. For zero-product initializations, (row) denotes departure measured relative to the initial input subspace .
| Initialization | Observed / predicted LR | (%) | Loss gap at predicted grid point | |
|---|---|---|---|---|
| MiLoRA | 0.88 | 25.4 | 0.0000 | |
| PiSSA | 0.63 | 0.19 | 0.0017 | |
| OLoRA | 0.77 | 0.44 | 0.0000 | |
| LoRA-GA | 0.30 | 0.010 | 0.0048 | |
| vanilla LoRA | 0.60 | 68.0 (row) | 1.437 | |
| rsLoRA | 0.85 | 2.98 (row) | 0.0004 |
We thank reviewer aE3s for the detailed feedback on the theoretical scope and practical implications of our work. We provide clarifications and supporting results in response.
[W2] Significance and Practical Implications of Nonzero Initialization
Although vanilla LoRA remains the default configuration, nonzero initialization has also been studied in published work and implemented in widely used libraries. Our goal is to understand the gains reported for these methods and establish fair conditions for comparing them. At the same nominal learning rate, initialization scale and the optimizer can change the actual weight-update magnitude. Observed performance differences may therefore reflect both initialization direction and effective update size.
As discussed in our Introduction, several published studies report improved accuracy, faster convergence, or greater learning-rate robustness with nonzero initialization. PiSSA [R1] and LoRA-GA [R2], both published at NeurIPS 2024, report performance gains or faster convergence over vanilla LoRA in some settings. Beyond Zero Initialization [R3] studies nonzero initialization dynamics and reports greater robustness to suboptimal learning rates, particularly small ones. Hugging Face PEFT [R4] also supports several such methods, including PiSSA, LoRA-GA, OLoRA [R5], and CorDA [R6]. Understanding when these methods offer benefits requires accounting for how initialization and the optimizer affect suitable learning rates.
Figure 2 isolates this relationship by fixing the initialization direction and varying only its scale. In the left panel, the optimal learning rate and the range of suitable learning rates shift with initialization scale. In the right panel, the optima cluster within a similar range after applying our predicted learning-rate normalization. Thus, the lower loss at smaller gain under a fixed learning rate should be interpreted alongside this shift. These results support comparing initialization methods after selecting an appropriate learning rate for each.
All tested values of ginit in Figure 2 are positive, and reducing the gain scales down both factors together. The figure therefore does not compare nonzero initialization against zero initialization. Instead, it shows why learning-rate differences induced by initialization scale and the optimizer must be considered when interpreting the reported benefits of nonzero initialization.
References
[R1] Fanxu Meng et al., PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models, NeurIPS, 2024.
[R2] Shaowen Wang et al., LoRA-GA: Low-Rank Adaptation with Gradient Approximation, NeurIPS, 2024.
[R3] Shiwei Li et al., Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics, ICML, 2025.
[R4] Hugging Face, PEFT Documentation: LoRA, Version 0.21.0, https://huggingface.co/docs/peft/v0.21.0/package_reference/lora
[R5] Kerim Büyükakyüz, OLoRA: Orthonormal Low-Rank Adaptation of Large Language Models, arXiv:2406.01775, 2024.
[R6] Yibo Yang et al., CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning, NeurIPS, 2024.