acceptodds
Under review as a conference paper at ICLR 2027

LoRA Initialization Geometry and Learning-Rate Transfer

Abstract

LoRA parameterizes weight updates as a product of low-rank matrices, making the geometric constraints imposed by this structure important for understanding its training dynamics. Initialization determines the tangent space from which learning begins and the local curvature, constraining the directions of weight changes and their departure from the initial tangent space. We analyze how this geometry, which varies with the initialization, interacts with the learning rate to shape training. First, we use the initial spectrum and cumulative movement to analyze subspace rotation and local departure from the initial tangent space. Our analysis indicates that geometric differences relative to the initialization alone do not readily explain the performance attained. We further investigate whether that weight update magnitudes provide a more direct explanation for the performance attained. From this analysis, we derive a learning-rate scaling rule that matches the magnitudes of first-order weight updates across initialization scales. We establish a criterion for assessing learning-rate ranges over which both training loss trajectories and final evaluation performance are comparable across settings. Using this criterion to guide learning-rate selection, we show that existing LoRA initialization methods can achieve similar evaluation performance when their learning rates are tuned suitably. We also identify the potential for learning-rate transfer across different initialization methods.

Primary area:
learning theory
Keywords:
Low Rank Adaptation, Training Optimizer, LoRA Initialization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aE3sOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies the geometry of non-zero LoRA initializations and its impact on learning rate scaling. The authors use a first order factor update to charachterize updates that remain the initial tangent space versus updates corresponding to departure from that initial tangent space. They show that scaling initialization by a factor of aa leads to a factor ap+1a^{p+1} update size in features, where p depends on the optimiser. Based on this, they derive some rules for learning rate scaling with initialization scale a.

Critical strengths
  • The assumptions are explicit in the paper, and the authors made the scope of their result very clear.
  • The tangent space approach is useful for first order analysis of the training dynamics near initialization. It reminds me somewhat of the neural tangent kernel, where features of not move significantly from initialization.
Critical weaknesses
  • Theory: the analysis is restricted to the lazy training regime (the tangent space near initialization). It is well known that LoRA, especially on hard tasks, does not operate in the lazy training regime, and significant feature learning (as measured from the cross herm) occurs. Also, the proof’s constant grows like LipTTLip_T^T, and therefore for T large enough (realistic in the experiments), the O(a−2)O(a^{-2}) bound is not meaningful (likely vacuous).
  • Significance and practical implications: it is unclear to me what the implications of these results are. The paper studies non-zero initialization of LoRA adapters, although the default is zero initialization. The authors do not discuss or validate why we should even consider these non-zero initializations. Even there empirical results in Fig 2 suggest that as initialization scale gets smaller and smaller, training loss improves.
  • Learning rate transfer: I suggest the authors remove this from the title. Learning rate transfer is a well-established notion and it generally refers to how learning rate should scale with model size or training horizon (HP scaling laws). Here, the learning rate scaling is with respect to initialization scale aa. The authors did not mention in which case this is useful. Why would one consider scaling initialization scale a? Are you suggesting one should tune a? If that’s the case, would the increase in tuning cost justify the gains (if any)? None of this was discussed in the paper.
Questions for authors

See weaknesses.

7 comments

We thank reviewer aE3s for the detailed feedback on the theoretical scope and practical implications of our work. We provide clarifications and supporting results in response.

[W2] Significance and Practical Implications of Nonzero Initialization

Although vanilla LoRA remains the default configuration, nonzero initialization has also been studied in published work and implemented in widely used libraries. Our goal is to understand the gains reported for these methods and establish fair conditions for comparing them. At the same nominal learning rate, initialization scale and the optimizer can change the actual weight-update magnitude. Observed performance differences may therefore reflect both initialization direction and effective update size.

As discussed in our Introduction, several published studies report improved accuracy, faster convergence, or greater learning-rate robustness with nonzero initialization. PiSSA [R1] and LoRA-GA [R2], both published at NeurIPS 2024, report performance gains or faster convergence over vanilla LoRA in some settings. Beyond Zero Initialization [R3] studies nonzero initialization dynamics and reports greater robustness to suboptimal learning rates, particularly small ones. Hugging Face PEFT [R4] also supports several such methods, including PiSSA, LoRA-GA, OLoRA [R5], and CorDA [R6]. Understanding when these methods offer benefits requires accounting for how initialization and the optimizer affect suitable learning rates.

Figure 2 isolates this relationship by fixing the initialization direction and varying only its scale. In the left panel, the optimal learning rate and the range of suitable learning rates shift with initialization scale. In the right panel, the optima cluster within a similar range after applying our predicted learning-rate normalization. Thus, the lower loss at smaller gain under a fixed learning rate should be interpreted alongside this shift. These results support comparing initialization methods after selecting an appropriate learning rate for each.

All tested values of ginitg_{\mathrm{init}} in Figure 2 are positive, and reducing the gain scales down both factors together. The figure therefore does not compare nonzero initialization against zero initialization. Instead, it shows why learning-rate differences induced by initialization scale and the optimizer must be considered when interpreting the reported benefits of nonzero initialization.

References

[R1] Fanxu Meng et al., PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models, NeurIPS, 2024.

[R2] Shaowen Wang et al., LoRA-GA: Low-Rank Adaptation with Gradient Approximation, NeurIPS, 2024.

[R3] Shiwei Li et al., Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics, ICML, 2025.

[R4] Hugging Face, PEFT Documentation: LoRA, Version 0.21.0, https://huggingface.co/docs/peft/v0.21.0/package_reference/lora

[R5] Kerim Büyükakyüz, OLoRA: Orthonormal Low-Rank Adaptation of Large Language Models, arXiv:2406.01775, 2024.

[R6] Yibo Yang et al., CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning, NeurIPS, 2024.

[W1] Lazy Training and the Time Dependence of the Bound

Scope of the lazy-training analysis

We appreciate the reviewer’s perspective on lazy training and would like to clarify the scope of our theoretical results. Theorem 3 provides asymptotic sufficient conditions for optimal LR transfer in a lazy-training limit. The LR scaling exponent follows from Theorem 1 under the stated response-order conditions, without requiring the full training trajectory to remain in the initial tangent space T0T_0. In Theorem 3, lazy-training behavior emerges as the initialization amplitude increases, under the stated regularity and LR scaling conditions.

The initialization scale also matters when characterizing LoRA’s training regime. With zero initialization, B0=0B_0=0, the contribution of changes in AA to the weight displacement appears through the cumulative cross term sΔBΔAs\Delta B\Delta A. The importance of this term alone does not determine the regime across all LoRA configurations. For homogeneous models, initialization scale is known to influence the transition between lazy and rich regimes [R7, R8]. The LoRA factor product sBAsBA is also 2-homogeneous, motivating an examination of its behavior across initialization scales.

Time dependence of the bound

We acknowledge that the factor LipTT\mathrm{Lip}_{\mathcal T}^{T} can make the bound numerically loose at large TT. The numerical tightness of this bound and the accuracy of the approximation in practice should be evaluated separately.

Theorem 3 concerns a→∞a\to\infty at fixed TT. Its proof accumulates possible error amplification at each step using a common Lipschitz constant. Exponential time dependence also appears in the general finite-time result of Chizat et al. [R7]. That work improves the dependence for square loss and establishes time-uniform results under strong convexity and additional conditions.

In the normalized coordinates of Lemma B.6, the cumulative cross term has an explicit coefficient of exactly a−2a^{-2}. To assess the approximation at finite initialization amplitudes and practical training horizons, we directly compared LoRA with its linearized counterpart. We separately evaluated LR prediction across practical initialization methods.

Direct comparison with linearized LoRA

We compared LoRA and linearized LoRA at matched gains using the fixed-basis initializations on Qwen3-0.6B. The linearized model excludes the cumulative cross term ΔBΔA\Delta B\Delta A. Both models used the same optimizer and minibatch order, with FP32 adapter branches.

Across the two normalized learning rates and ginit∈{0.1,1,10}g_{\mathrm{init}}\in\lbrace 0.1,1,10\rbrace, the absolute difference in mean training loss was at most 0.0007 in every reported window after step 100. The maximum before rounding was 0.00068, and the difference over the final 100 steps was at most 0.0002 (Table R1). These results show that linearized LoRA closely reproduced LoRA’s training loss in the tested finite-amplitude settings.

LR prediction beyond the fixed-shape amplitude family

We next used a reference LR from DST and the magnitude of each initialization to predict search starting points for five initialization methods on Bengali adaptation with Llama-3.1-8B. For four methods, the interpolated optimal LR was within a factor of approximately 1.78 of the prediction. For three methods, the loss at the grid point nearest the predicted LR was within 0.001 of the minimum over the search grid. For LoRA-GA, the observed optimal LR was 0.30 times the prediction, indicating that lower LRs needed to be explored (Table R2).

We also examined vanilla LoRA, which falls outside the direct guarantees of Theorem 3. Its departure from the initial input subspace reached 68.0%. Training diverged at the predicted LR, while the observed optimal LR was 0.60 times the prediction. For rsLoRA, which also uses zero-product initialization, the observed optimal LR was 0.85 times the prediction.

Nevertheless, these results suggest that the LR prediction based on our scaling rule can serve as a starting point for LR search beyond the fixed-shape amplitude families studied in Theorem 3, including settings with substantial departure from the initial first-order update space.

References

[R7] Lénaïc Chizat et al., On Lazy Training in Differentiable Programming, NeurIPS, 2019.

[R8] Blake Woodworth et al., Kernel and Rich Regimes in Overparametrized Models, COLT, 2020.

Table R1. LoRA–linearized LoRA training-loss differences.

Each entry is the difference in mean training loss, LLoRA−Llin\mathcal L_{\mathrm{LoRA}}-\mathcal L_{\mathrm{lin}}, over the indicated window. Results use one training seed.

βeffη\beta_{\mathrm{eff}}\etaginitg_{\mathrm{init}}Steps 100–200200–400400–800800–1200Final 100 steps
3.6×10−63.6\times10^{-6}0.1−0.0003−0.0001−0.0001−0.00020.0000
3.6×10−63.6\times10^{-6}1−0.0001−0.00010.0000−0.00010.0000
3.6×10−63.6\times10^{-6}10−0.0001−0.00020.00000.0000+0.0001
6.4×10−66.4\times10^{-6}0.1−0.0004−0.0007−0.0003−0.0005−0.0002
6.4×10−66.4\times10^{-6}1−0.0004+0.00010.0000−0.0001−0.0001
6.4×10−66.4\times10^{-6}10+0.0001+0.0001+0.00010.00000.0000

Table R2. LR transfer on Bengali adaptation.

The observed optimal LR is estimated by interpolation. The loss gap is the difference in mean training loss over the final 100 steps between the grid point nearest the predicted LR and the best grid point. For zero-product initializations, (row) denotes departure measured relative to the initial input subspace row⁡(A0)\operatorname{row}(A_0).

InitializationB0B_0Observed / predicted LRε\varepsilon (%)Loss gap at predicted grid point
MiLoRA≠0\ne 00.8825.40.0000
PiSSA≠0\ne 00.630.190.0017
OLoRA≠0\ne 00.770.440.0000
LoRA-GA≠0\ne 00.300.0100.0048
vanilla LoRA=0=00.6068.0 (row)1.437
rsLoRA=0=00.852.98 (row)0.0004
Reviewer_3tCwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes how LoRA initialization affects the geometry and magnitude of training updates. For nonzero rank-r initial products, it characterizes the initial tangent space and bounds support rotation and departure from that space in terms of the initialization spectrum and cumulative movement; vanilla zero-product initialization is treated separately. It then derives how factor rescaling and optimizer response change first-order update magnitude, proposes a learning-rate scaling rule, and proves finite-horizon transfer results under stated regularity conditions.

Critical strengths
  1. The paper makes a useful distinction between the geometry induced by an initialization and the effective update magnitude induced by the initialization/optimizer/learning-rate combination. This is an important confound in comparisons of LoRA initialization methods.
  2. The factor-update expansion explicitly separates first-order product response from the cross term. The resulting tangent-space and support-rotation analysis provides a principled language for discussing how LoRA training departs from its starting geometry.
  3. The learning-rate transfer rule is practical in spirit: it aims to guide hyperparameter search rather than claim that a particular initialization is universally superior.
Critical weaknesses
  1. While using a relatively small model may be reasonable for conducting extensive controlled experiments, the empirical evaluation is limited to Qwen3-0.6B-Base and a single fine-tuning task. Since the paper draws broader conclusions about the relationship between initialization geometry, learning-rate tuning, and final performance, it remains unclear whether these observations generalize to larger models and more diverse downstream tasks. I encourage the authors to provide additional evidence supporting the generality of their findings.
  2. The experiments are conducted on a single relatively small model, without examining how the findings change as model size increases. Since larger models may exhibit different optimization behaviors and initialization sensitivities, it is unclear whether the proposed learning-rate transfer rules and conclusions about initialization geometry remain valid at larger scales.
  3. The paper does not provide systematic evaluations across multiple random seeds or report the mean and standard deviation of the results.
  4. The proposed learning-rate transfer framework primarily focuses on changes in initialization magnitude while keeping the initialization directions fixed. However, practical initialization methods such as PiSSA and LoRA-GA construct their directions using different information. Although the paper provides a directional correction analysis, its effectiveness for transferring learning rates across different initialization methods remains insufficiently validated. It is unclear whether the proposed framework can reliably predict suitable learning rates across these methods without extensive additional tuning.
  5. The proposed correction for changes in initialization direction is derived from a one-step analysis under local curvature assumptions. However, it is unclear whether this result extends to multi-step training, where gradients and optimization directions continuously change. The paper would benefit from further theoretical analysis explaining how the one-step correction relates to the optimal learning rate over the full training process.
Questions for authors
  1. Do the authors expect the observed relationship between initialization geometry, learning-rate tuning, and final performance to hold for larger models and more diverse downstream tasks?
  2. How do the findings change across different model sizes?
  3. How reliably can the proposed framework transfer learning rates between initialization methods with different directions, such as PiSSA and LoRA-GA, without requiring extensive additional tuning?
  4. Can the authors provide further analysis explaining whether and how the proposed one-step directional correction extends to multi-step training, where gradients and optimization directions change throughout the training process?
6 comments

Thank you for your constructive feedback. We present additional experimental results and analysis in response.

[W1,2] Experiments Across Diverse Models and Tasks

Across model sizes, model families, and tasks, the scaling of the optimal learning rate with initial factor amplitude was close to the AdamW prediction. We tested larger model sizes using Qwen3-0.6B, 4B, and 8B on MetaMath. We also extended the experiments across model families and tasks using Qwen3-0.6B and Llama-3.2-1B on MetaMath and Commonsense170K [R1].

In each setting, we fixed the initialization directions and scaled both initial LoRA factors by the same multiplier aa. We then found the learning rate η∗\eta^\ast that minimized the loss at each amplitude. The slope assessed whether the rate of change matched the prediction. The corrected max/min ratio measured the remaining variation across amplitudes after applying the rule.

The slope of log⁡η∗\log \eta^\ast against log⁡a\log a ranged from -1.180 to -0.983, close to the AdamW prediction of -1. We corrected η∗\eta^\ast by multiplying it by the relative factor amplitude a/arefa/a_{\mathrm{ref}}. The corrected max/min ratios ranged from 1.11 to 1.54. These results support the predicted scaling trend and show that the same correction aligns the optima within a narrow range in each setting.

Table R1. LR scaling across models and tasks (AdamW)

ModelTaskSlope of log⁡η∗\log \eta^\ast vs. log⁡a\log a (prediction: −1-1)Corrected η∗\eta^\ast max/min
Qwen3-0.6BMetaMath-1.1801.50–1.54
Qwen3-4BMetaMath-1.141.39
Qwen3-8BMetaMath-1.141.37
Llama-3.2-1BMetaMath-0.9831.48
Qwen3-0.6BCommonsense170K-1.1111.36
Llama-3.2-1BCommonsense170K-1.0151.11

References

[R1] Zhiqiang Hu et al., LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models, EMNLP, 2023.

[W3] Runs with Multiple Training Seeds

The main findings remained consistent across three training seeds. We now report the mean and standard deviation (SD) across seeds in our paper.

First, the observation that different initialization geometries can yield similar tuned performance held across all three seeds (Figure 1). Restricting training to T0T_0 also continued to incur only a small mean CE increase of 0.00238–0.00330 (Table 2). Second, the optimizer-dependent response to initialization amplitude and the effect of LR normalization remained consistent. Reducing the amplitude decreased AdamW’s update magnitude but left LoRA-Muon’s nearly unchanged (Section 5.2). In Figure 2, the normalized optimal LRs remained within a factor of 1.50–1.54 in each seed, close to the originally reported factor of about 1.5. Finally, both LR transfer rules maintained evaluation performance close to the reference LR across all three seeds. The LoRA-Muon pilot ratio and AdamW norm ratio met the paper’s CE-gap tolerance in every seed (Appendix Table 4).

[W4] Validation of the Reliability of LR Transfer Across Initialization Methods with Different Directions

Theorem 3 establishes LR transfer for amplitude changes with fixed initialization directions. In practice, the size rule can also help identify where to focus the LR search, even when initialization directions differ.

To test this, we used Llama-3.1-8B on MetaMath and the Bengali subset of Aya Collection [R2]. Using DST as the reference, we predicted LRs for MiLoRA, OLoRA, PiSSA, LoRA-GA, and rsLoRA. On both tasks, four of the five initializations achieved their lowest loss at the LR candidate closest to the predicted LR or at a neighboring candidate (Table R2). For LoRA-GA, the lowest loss occurred two grid steps below the candidate closest to the predicted LR on both tasks.

Specifically, we first swept the LR for DST on each task to obtain the reference LR. We then applied the size rule to predict an LR for each target initialization before training. We measured initialization size as m=s[rms(A0)+rms(B0)]m=s[\mathrm{rms}(A_0)+\mathrm{rms}(B_0)], where ss is the LoRA scale. With ss fixed, multiplying both factors by a>0a>0 gives m↦amm\mapsto am. For the target LR sweeps, we used a predefined log-spaced grid with a spacing of 0.25 decades. Neighboring LRs therefore differed by a factor of 100.25≈1.7810^{0.25}\approx1.78. We then evaluated 3–4 LR candidates around each predicted LR. These runs formed the target LR sweep.

Without a starting estimate, finding a good LR over a wide range with only 3–4 trials is difficult. These results show that the size rule helps focus a limited search budget on promising LRs, even when initialization directions differ. Moreover, the prediction errors were similar on the two tasks, supporting this approach across different training datasets. We will include these validation results in the final version.

Table R2. LR prediction, search, and loss across initializations

InitializationMetaMath: observed/predicted LRGrid offsetLossAya Col. (Bengali): observed/predicted LRGrid offsetLoss
DST (reference)—Reference0.1509—Reference0.3580
MiLoRA0.811 step lower0.15140.88Same0.3588
OLoRA0.561 step lower0.15570.77Same0.3671
PiSSA0.681 step lower0.15240.631 step lower0.3615
LoRA-GA0.282 steps lower0.15320.302 steps lower0.3629
rsLoRA1.02Same0.14950.851 step lower0.3553

References

[R2] Shivalika Singh et al., Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning, ACL, 2024.

[W5] Connection Between One-Step Directional Correction and the Optimal LR for Multi-Step Training

Lemma 3 concerns one-step directional correction. We tested whether this initial information remains useful for choosing the optimal LR over full training, where gradients and update directions change. We found that the one-step optimal LR ratios predicted the full-training LR ratios more accurately than the size rule in both settings. Thus, the initial diagnostics helped estimate how much to adjust the LR when switching initializations.

Specifically, we measured ηi†\eta_i^\dagger by one-step line search with a freshly initialized Adam state. Using DST’s full-training optimal LR η0∗\eta_0^\ast, we predicted ηpred,i=η0∗(ηi†/η0†)\eta_{\mathrm{pred},i}=\eta_0^\ast(\eta_i^\dagger/\eta_0^\dagger) and compared it with existing full-training results. Across five initializations on Llama-3.1-8B, mean fold error fell from 1.65× to 1.30× on MetaMath and from 1.55× to 1.35× on Aya Collection (Bengali). Maximum errors also decreased in both settings (Table R3).

We also fixed predictions before the LR sweeps for size-matched pairs. The predicted and observed LR ratios were 0.58 versus 0.65 for PiSSA/Fixed-PiSSA, and 0.37 versus 0.30 for LoRA-GA/Fixed-GA. The size rule predicts 1 in both cases. These comparisons show that the one-step measurements captured LR differences beyond initialization size. We will include these validation results in the final version.

Table R3. Full-training LR prediction ratios (ηpred/η∗\eta_{\mathrm{pred}}/\eta^\ast) and fold errors

DatasetRuleMiLoRAOLoRAPiSSALoRA-GArsLoRAMean fold errorMax. fold error
MetaMathSize ratio1.241.781.473.610.981.65×3.61×
MetaMathOne-step optimal LR ratio0.640.680.921.330.901.30×1.57×
Aya Col. (Bengali)Size ratio1.131.301.593.311.171.55×3.31×
Aya Col. (Bengali)One-step optimal LR ratio0.710.491.011.311.201.35×2.04×

We thank the reviewer again for the constructive feedback.

In brief, our response provides:

  • Broader evaluation across model sizes, families, and tasks, with consistent LR scaling trends up to Qwen3-8B.
  • Results across three training seeds, supporting the main findings with means and standard deviations.
  • LR prediction across initialization methods, showing that the size rule helps narrow the LR search: for four of five methods on both tasks, the lowest loss was at the candidate nearest the prediction or a neighboring candidate.
  • Evidence connecting one-step diagnostics to full-training LR selection. Lemma 3 is a one-step result and we do not claim a multi-step guarantee; empirically, however, the one-step optimal LR ratio gave lower mean fold errors than the size rule alone on both tasks.

As the discussion period comes to an end, we would appreciate any further feedback on our response and would be glad to clarify any remaining concerns.

Reviewer_BuhSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the effects of LoRA initialization on training geometry and learning-rate selection. The authors analyze how initialization scale affects the weight update and the rotation of the input and output subspaces. They derive a learning-rate scaling rule for different optimizers. The numerical results show that different initializations can achieve similar performance after learning-rate tuning.

Critical strengths
  1. The problem studied by this work is interesting and has practical value. The results help choose the learning-rate range when changing the LoRA initialization scale. Different LoRA initializations may produce different weight-update sizes under the same learning rate. The authors also explain why using the same learning rate for different initializations may lead to an unfair comparison.

  2. The theoretical proofs are clear and easy to follow. The authors bound subspace rotation and departure from the initial tangent space. They also derive optimizer-dependent learning-rate scaling rules and prove a conditional learning-rate transfer result over a fixed training horizon.

Critical weaknesses
  1. The suitability of the theoretical assumptions needs more discussion. For example, Theorem 3 requires the normalized states and responses are bounded in a neighborhood of a common limiting trajectory. The authors should provide more detailed discussions on such assumptions about whether they are valid in specific cases or how they are guaranteed experimentally, especially when projection or clipping techniques are needed in experiments.

  2. Some of the theoretical statements are not directly checked in the experiments. I think the assumptions and quantitative error bound in Theorem 3, as well as the geometric bounds in Proposition 2, need more direct experimental evidence.

Questions for authors
  1. See 'critical weakness'.
  2. Minor issue: The definitions of some symbols are hard to find. Although the authors summarize the notation in Appendix E, some symbols are missing, such as βeff\beta_{\mathrm{eff}} in Table 2. Adding a brief explanation when these symbols first appear would make the paper easier to follow.
5 comments

We appreciate your thoughtful assessment and comments. We address the concerns raised through further clarification and additional experimental results.

[W1] Validity of the Assumptions in Theorem 3 and the Role of Clipping

For Adam, coordinatewise updates are bounded by a constant multiple of the learning rate under standard conditions on the moment coefficients. With the prescribed scaling $\eta\propto a^{-1}$, the normalized factor displacements can therefore be bounded uniformly in aa over a fixed, finite training horizon. Regularity of the normalized update maps additionally requires nondegenerate normalization denominators. LoRA-Muon requires further nondegeneracy conditions on the factor Gram matrices and the inputs to the polar operation. We will clarify these optimizer-specific sufficient conditions. Our experiments do not apply projection to the iterates.

As shown in Table R1, Consistent with this scaling, at a matched normalized learning rate of $\beta_{\mathrm{eff}}\eta\approx2.0\times10^{-6}$, the final relative weight displacement $\|\Delta W\|/\|W\|$ ranged from 0.0113 to 0.0137 across initialization gains from 0.001 to 10. Meanwhile, the relative factor displacement $\|A-A_0\|/\|A_0\|$ decreased from 0.235 to 0.0029. The values Table R1 are arithmetic means over the 196 adapters at step 1,563, using the default BF16 configuration.

We also trained LoRA linearized at the initial factors using the same optimizer and minibatch order as the corresponding nonlinear model. For the fixed-basis initializations with gains $g_{\mathrm{init}}\in\{0.1,1,10\}$, the absolute difference in mean training loss between the paired models was below 0.0007 in every reported window after step 100, at both the loss-optimal normalized learning rate, $3.6\times10^{-6}$, and the adjacent value, $6.4\times10^{-6}$. Both models used FP32 adapter branches to control for numerical precision. Table R2 reports the paired mean training loss difference, computed as $L_{\mathrm{LoRA},g}-L_{\mathrm{lin},g}$.

The native factor magnitudes of PiSSA and LoRA-GA fall between those of the fixed-basis initializations (DST-I) at $g_{\mathrm{init}}=0.1$ and 1. The paired linearized experiments reported in Table R2 use the fixed-basis initializations and provide empirical evidence of agreement in loss between nonlinear and linearized training over the tested amplitude range.

We used gradient clipping in the main experiments to reflect standard fine-tuning practice.
Fixed-threshold gradient clipping can break the scale covariance used in the theoretical analysis. We therefore repeated the linearized experiments with clipping disabled. Table R3 reports the difference in mean training loss between the linearized models at $g_{\mathrm{init}}=10$ and 0.1, computed as $L_{\mathrm{lin},g=10}-L_{\mathrm{lin},g=0.1}$, separately for runs with and without clipping.

During steps 100-400, disabling clipping reduced the difference between amplitudes from 0.0026-0.0050 to 0.0002-0.0006. Over the final 100 steps, the differences were already small under both settings, ranging from 0.0001 to 0.0007 with clipping and equaling approximately 0.0004 without clipping.

For nonlinear LoRA, we also disabled clipping while retaining the learning rates selected in the original sweep. Table R4 reports the change in mean training loss over the final 100 steps, computed as $L_{\text{no-clip}}-L_{\text{clip}}$.

At the loss-optimal normalized learning rate of $3.6\times10^{-6}$, disabling clipping increased the final mean loss by 0.0008 and 0.0013 for $g_{\mathrm{init}}=1$ and 10, respectively. The latter run had clipping active at every training step. At $6.4\times10^{-6}$, the tested configurations changed by -0.0004 to +0.0001. The effect was larger at the lowest tested normalized learning rate, $6.4\times10^{-7}$, where the loss increased by 0.0030-0.0038. Across the tested loss-optimal and higher-learning-rate settings, the final mean losses remained similar after clipping was disabled.

Table R1. Relative weight and factor displacements across initialization gains.

ginitg_{\mathrm{init}}Learning rate η\eta∥ΔW∥/∥B0A0∥\lVert\Delta W\rVert/\lVert B_0A_0\rVert∥ΔW∥/∥W∥\lVert\Delta W\rVert/\lVert W\rVert∥A−A0∥/∥A0∥\lVert A-A_0\rVert/\lVert A_0\rVert
0.0011.12×10−41.12\times10^{-4}0.3580.01130.235
0.016.32×10−56.32\times10^{-5}0.1160.01160.078
0.13.56×10−53.56\times10^{-5}0.03820.01210.0257
12.00×10−52.00\times10^{-5}0.01320.01320.0089
101.12×10−51.12\times10^{-5}0.004340.01370.0029

Table R2. Training-loss differences between LoRA and linearized LoRA.

βeffη\beta_{\mathrm{eff}}\etaginitg_{\mathrm{init}}Steps 100–200Steps 200–400Steps 400–800Steps 800–1200Final 100 steps
3.6×10−63.6\times10^{-6}0.1−0.0003−0.0001−0.0001−0.0002−0.0000
3.6×10−63.6\times10^{-6}1−0.0001−0.0001+0.0000−0.0001+0.0000
3.6×10−63.6\times10^{-6}10−0.0001−0.0002+0.0000+0.0000+0.0001
6.4×10−66.4\times10^{-6}0.1−0.0004−0.0007−0.0003−0.0005−0.0002
6.4×10−66.4\times10^{-6}1−0.0004+0.0001+0.0000−0.0001−0.0001
6.4×10−66.4\times10^{-6}10+0.0001+0.0001+0.0001−0.0000+0.0000

Table R3. Effect of clipping on cross-amplitude loss differences in linearized LoRA.

βeffη\beta_{\mathrm{eff}}\etaClipping thresholdSteps 50–100Steps 100–200Steps 200–400Steps 400–800Steps 800–1200Final 100 steps
3.6×10−63.6\times10^{-6}1.0+0.0024+0.0026+0.0026+0.0016+0.0005+0.0001
3.6×10−63.6\times10^{-6}Disabled+0.0007+0.0003+0.0004+0.0003+0.0003+0.0004
6.4×10−66.4\times10^{-6}1.0+0.0044+0.0050+0.0045+0.0035+0.0014+0.0007
6.4×10−66.4\times10^{-6}Disabled+0.0009+0.0006+0.0002+0.0002+0.0002+0.0004

Table R4. Changes in LoRA training loss after disabling clipping.

ginitg_{\mathrm{init}}βeffη\beta_{\mathrm{eff}}\etaFraction of steps clipped in the original runLoss change over the final 100 steps
103.6×10−63.6\times10^{-6} (loss-optimal)100%+0.0013
13.6×10−63.6\times10^{-6} (loss-optimal)21%+0.0008
106.4×10−66.4\times10^{-6} (near the loss cliff)100%−0.0004
16.4×10−66.4\times10^{-6} (near the loss cliff)10%−0.0003
0.016.4×10−66.4\times10^{-6} (control)0.4%+0.0001
106.4×10−76.4\times10^{-7} (lowest normalized learning rate)100%+0.0038
16.4×10−76.4\times10^{-7} (lowest normalized learning rate)99.9%+0.0030

[W2] Experimental Evaluation of the Theoretical Results

Evaluation of Theorem 3

We evaluated the learning-rate scaling rule (Theorem 1 and Eq. 11), whose transfer of optimality Theorem 3 establishes in the large-amplitude limit, across model sizes and training seeds. We also checked whether the geometric inequalities in Proposition 2 hold during training and assessed their tightness.

We tested the predicted scaling η∗(a)∝a−(1+p)\eta^\ast(a)\propto a^{-(1+p)} at three model sizes, where η∗\eta^\ast denotes the loss-minimizing learning rate within the searched range. In each setting, we fixed the initialization directions, varied the common factor amplitude, and estimated the regression slope of log⁡η∗\log\eta^\ast against log⁡a\log a. As shown in Table R5, AdamW slopes were close to the predicted −1-1 and LoRA-Muon slopes close to the predicted 00 at all three sizes.

Table R5. Learning-rate scaling slopes across model sizes.

ModelAdamW slope (prediction: −1-1)LoRA-Muon slope (prediction: 00)
Qwen3-0.6B−1.180−0.060
Qwen3-4B−1.143−0.002
Qwen3-8B−1.135+0.004

We also evaluated AdamW on Qwen3-0.6B across three seeds, fixing the initial factors while varying the data order and training RNG. Table R6 reports slopes for learning rates selected by training loss and by held-out CE. For both criteria, the slopes were consistent across seeds, with similar deviations from −1-1.

Table R6. Learning-rate scaling slopes across training seeds for AdamW on Qwen3-0.6B.

Selection criterionSeed 42Seed 43Seed 44Mean ± SD
Training loss−1.182−1.177−1.181−1.180 ± 0.003
Held-out CE−1.183−1.172−1.175−1.177 ± 0.006

Evaluation of the geometric inequalities in Proposition 2

The geometric inequalities held at all saved states where their applicability conditions were satisfied. The displacement-based rotation bound was close to the observed rotation: among configurations with a nontrivial final rotation bound, the median observed-to-bound ratio across adapters ranged from 0.72 to 0.94, and across all saved states with ∥ΔW∥2/σr(X0)<1\lVert\Delta W\rVert_2/\sigma_r(X_0)<1 it was 0.87.

Table R7 summarizes the measurements at the final training step. Here, ξ\xi is the normalized cumulative path length, xx the normalized displacement, and ε\varepsilon the departure ratio. The displacement-based bounds are ∥ΔW∥2/σr(X0)\lVert\Delta W\rVert_2/\sigma_r(X_0) for rotation and x2/(4γˉ2)x^2/(4\bar{\gamma}^2) for departure, the latter requiring γˉ>0\bar{\gamma}>0.

Table R7. Final geometric measurements and observed-to-bound ratios across initializations.

InitializationLRξ\xixxξ/x\xi/x∥sin⁡Θ∥2\lVert\sin\Theta\rVert_2Rotation / boundε\varepsilonDeparture / bound
ginit=0.001g_{\mathrm{init}}=0.0012.00×10−42.00\times10^{-4}59.96.648.90.7700.458.04%0.17%
ginit=0.1g_{\mathrm{init}}=0.16.32×10−56.32\times10^{-5}6.130.688.90.1710.930.072%0.46%
ginit=10g_{\mathrm{init}}=102.00×10−52.00\times10^{-5}0.690.0778.90.0200.940.001%0.49%
PiSSA3.56×10−53.56\times10^{-5}3.380.398.60.0690.720.014%0.33%
LoRA-GA1.12×10−51.12\times10^{-5}0.840.1088.00.0150.770.001%0.31%
LoRA-GA6.32×10−56.32\times10^{-5}4.790.5379.30.0750.860.027%0.33%

The path-length-based bound is conservative in magnitude, but expresses how initialization amplitude and learning rate control the scale of the geometric bounds. With fixed initialization directions and matched normalized learning rates, ξ\xi decreased from 59.93 to 6.13 to 0.69 as ginitg_{\mathrm{init}} increased 100-fold at each step. Since a∝ginit1/4a\propto g_{\mathrm{init}}^{1/4}, this matches the theoretical a−2a^{-2} scaling. At the final step, ξ/x\xi/x ranged from 8.0 to 9.3. Cumulative path length exceeds net displacement because it ignores cancellation between successive updates, but their ratio was similar across settings.

The displacement-based departure bound, ε≤x2/(4γˉ2)\varepsilon\leq x^2/(4\bar{\gamma}^2), also held at all saved states satisfying the positive-gap condition. Between the two LoRA-GA learning rates, a 4.97-fold increase in xx came with a 26.4-fold increase in ε\varepsilon, close to the quadratic growth suggested by x2x^2.

In the revision, we will retain the inequalities and reorganize the presentation so that the displacement-based bounds form the main proposition and the path-length-based bounds follow as a corollary, with the applicability conditions and experimental results presented alongside the corresponding bounds.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Contributions and Rebuttal Results

Summary of Contributions and Rebuttal Results

We thank the AC and reviewers for their feedback. To address concerns about the connection between theory and practical training, we expanded our evaluation across model sizes, tasks, training seeds, and initialization methods.

Contributions

This paper aims to distinguish the effects of initialization direction and update magnitude when interpreting performance differences across LoRA initializations.

  • Geometry (Props. 1–2). We quantify subspace rotation and departure from the initial tangent space. Initializations with substantially different departure levels achieved similar performance after LR tuning.
  • Update magnitude (Thm. 1). At the same nominal LR, the first-order update magnitude depends on both initialization scale and the optimizer. Shared-LR comparisons can therefore conflate directional effects with update-magnitude effects.
  • LR criterion (Eq. 11, Thm. 3). We derive optimizer-dependent LR scaling relations that match first-order update magnitudes. Theorem 3 establishes asymptotic optimal-LR transfer under the stated conditions. The scaling relation provides a starting point for LR search when changing initialization.

Strengths Noted by Reviewers

The reviewers highlighted the confound in shared-LR comparisons (3tCw, BuhS), the principled analysis, clear proofs, and explicit assumptions (3tCw, BuhS, aE3s), and the practical goal of guiding LR search (3tCw, BuhS).

Additional Results and Clarifications

1. Direct evaluation of geometric bounds and finite-amplitude behavior. [BuhS W1–2, aE3s W1]
The geometric inequalities held at all saved states satisfying their applicability conditions. Displacement-based rotation bounds closely tracked observed rotations, and linearized LoRA closely reproduced standard LoRA’s training loss at the finite amplitudes tested.

2. Consistent scaling trends across models, tasks, and seeds. [3tCw W1–3, BuhS W2]
Optimizer-dependent LR scaling trends remained consistent across larger models, different model families and tasks, and multiple training seeds. We also reported the deviations from the ideal scaling predictions. In response to reviewer 3tCw’s follow-up, we also evaluated final downstream performance after selecting LRs separately for each initialization. Performance differences were generally small in the model–task settings we directly compared.

3. LR prediction across initialization directions. [3tCw W4–5, aE3s W1]
Across both tasks, most initialization methods attained their lowest tested loss near the predicted LR. One-step directional diagnostics further improved full-training LR predictions over size-only scaling. Cases requiring additional adjustment were also reported.

4. Clarified scope and practical use. [aE3s W1–3]
Our goal is to account for initialization-dependent update magnitudes when comparing methods and guiding LR search. We clarified the role of amplitude as an analysis variable, Theorem 3’s asymptotic conditions and time-dependent bound, and the need for further LR adjustment in some settings.

These results support accounting for both initialization scale and the optimizer when comparing initialization methods. We will incorporate the additional evidence and clarifications into the final manuscript, together with a clearer statement of Theorem 3’s conditions and a revised presentation of Proposition 2. We thank the AC and reviewers again for their time and feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.