acceptodds
Under review as a conference paper at ICLR 2027

On the Convergence of LoRA under Generalized Smoothness

Abstract

Low-rank adaptation (LoRA) has become a popularly used parameter-efficient approach for fine-tuning large pretrained models via replacing the full weight update with a factorized update . Despite its practical utility, existing convergence theory for LoRA remains limited: some prior analyses handle this factorized optimization problem by imposing boundedness assumptions on the LoRA factors, an additional assumption not guaranteed by LoRA updates. More recent results remove this assumption but still rely on -smoothness with respect to the weight variable . In practice, empirical studies of neural network training show that the Hessian norm can grow with the gradient norm, violating the -smoothness assumption on the bounded Hessian norm. Therefore, we analyze LoRA-GD and LoRA-SGD under generalized smoothness without assuming the bounded factor norms. For LoRA-GD, we prove an upper bound and the first lower bound, even when the stepsize is chosen adaptively from the optimization history. For LoRA-SGD with unbiased stochastic gradients of bounded variance, we obtain an high-probability guarantee. We also prove an lower bound at fixed confidence for adaptive stepsize policies that may depend on the observed optimization history and the current stochastic gradient. Experiments support our theoretical analysis: the Hessian norm grows with the gradient norm, the factor norms grow during optimization, and this growth amplifies stochastic factor-gradient noise.

Primary area:
optimization
Keywords:
LoRA, Optimization, Generalized Smooth

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vNBoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the convergence theory of LoRA (low-rank adaptation, ΔW = BA) under generalized smoothness, removing two assumptions common in prior work: bounded LoRA factor norms and global L-smoothness of the weight-space objective F. The latter is motivated empirically by the observation that Hessian norms tend to grow with gradient norms during neural network training. The analysis must therefore control the weight-gradient norm and the factor norms jointly (gradient–factor coupling), and, in the stochastic case, handle factor-amplified noise: a bounded-variance weight-gradient oracle induces factor-gradient noise with conditional second moment up to σ²R_t, where R_t is the factor scale.

Critical strengths

The theoretical analysis is sound and well-founded, and the extensive experiments are conducted with rigor.

Critical weaknesses

None

Questions for authors

None

1 comment

We sincerely thank the reviewer for the careful assessment and positive evaluation of our work. We appreciate your recognition of the soundness of the theoretical analysis and the rigor of the experiments. We also acknowledge your assessment of the presentation and will further improve the explanation of the main proof ideas and the theoretical results. Thank you again for your thoughtful review. We would be happy to address any additional questions or provide further clarification.

Reviewer_oH85OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how bilinear factorization affects LoRA convergence under generalized smoothness. It proves convergence under weaker smoothness assumptions without assuming bounded factors, and gives matching worst-case lower bounds for certain types of simultaneous factor updates. The results measure convergence to stationary points in factor space. Experiments show how the main quantities in the analysis change during training.

Critical strengths

The deterministic convergence guarantee holds under weaker smoothness assumptions while keeping the known O(ϵ−4)O(\epsilon^{-4}) complexity. It also avoids an extra bounded-factor assumption. This is a useful extension because the result applies to objectives outside the globally smooth class without worsening the dependence on the stationarity tolerance.

Critical weaknesses
  1. I think the deterministic upper-bound contribution modest. The broader smoothness assumptions extend where the known O(ϵ−4)O(\epsilon^{-4}) rate applies, but the rate itself is unchanged. The main gain is a less restrictive setting, without an improvement in the complexity's dependence on the stationarity tolerance.
  2. The matching lower bounds add new information, but their value is tied to a limited class of updates. They allow adaptive scalar stepsizes, yet still concern simultaneous factor updates and exclude momentum or separate factor stepsizes. This limits how much their conclusions tell us about the wider range of methods used for practical LoRA training.
Questions for authors
  1. What is the main new understanding of LoRA optimization gained from the weaker-smoothness analysis and matching lower bounds? How does this change the understanding provided by earlier LoRA convergence theory, given the differences in assumptions and update rules?
  2. What is the main contribution of proving tight complexity for the shared-scalar update class to the broader study of LoRA optimization? Which important conclusion follows from this tightness that was not supported by earlier upper-bound results?
3 comments

We sincerely thank the reviewer for the thoughtful assessment and constructive feedback. We appreciate the recognition of the theoretical results and agree that the scope and implications of our contributions deserve further clarification. In response, we have strengthened the lower-bound analysis and added decoder-only LLM experiments. We have added a multiple-matrix theoretical analysis in Appendix I and two decoder-only LLM fine-tuning experiments in Appendix K. We address the concerns and questions point by point below.

W1. Significance of the deterministic upper bound

We agree that our deterministic upper bound does not improve the existing O(ϵ−4)O(\epsilon^{-4}) convergence rate. Its contribution is to extend this guarantee beyond the globally smooth setting.

Why generalized smoothness? Classical convergence analyses typically assume global LL-smoothness, which requires the Hessian norm to be uniformly bounded over the entire parameter space. However, Zhang et al. [1] observed that local smoothness varies along neural network training trajectories and tends to increase with the gradient norm, motivating the (L0,L1)(L_0,L_1)-smoothness condition. Li et al. [2] further generalized this condition to the following gradient-dependent smoothness framework:

\leq \ell(\lVert\nabla F(X)\rVert_F), \qquad \ell(u)=o(u^2).$$ This framework includes both global $L$-smoothness and $(L_0,L_1)$-smoothness as special cases, while allowing a broader range of gradient-dependent curvature. **Why is this relevant to LoRA?** For LoRA, the objective is parameterized as $J(B,A)=F(BA)$. Its factor gradients depend on both the weight gradient and the current factors: $$\nabla_B J=\nabla F(BA)A^\top,\qquad \nabla_A J=B^\top\nabla F(BA).$$ Consequently, analyzing convergence without assuming bounded factor norms requires controlling the weight-gradient norm and factor norms jointly. This becomes particularly important under generalized smoothness, where the Hessian norm is itself gradient-dependent. Our Theorem 4.1 establishes an $O(\epsilon^{-4})$ convergence guarantee under sub-quadratic generalized smoothness, without assuming bounded factor norms. The full proof is provided in Appendix B. The guarantee also extends to simultaneously adapted multiple weight matrices, as established by Proposition I.1 and Theorem I.2 in Appendix I (in rebuttal version PDF). Thus, our contribution is not a better convergence exponent, but an extension of LoRA convergence theory to a broader smoothness regime without an additional bounded-factor assumption. ## W2. The matching lower bounds We agree that the original deterministic lower bound, which considers a shared scalar stepsize for simultaneous factor updates, leaves open whether independently adapting the two factor learning rates can improve the worst-case complexity. **To directly address this concern, we have added a new appendix, Appendix J, dedicated to establishing a stronger lower bound for separate factor learning rates.** Specifically, we consider simultaneous LoRA updates of the form $$\begin{aligned} B_{t+1}&=B_t-\eta_t^B\nabla_BJ(B_t,A_t),\\\\ A_{t+1}&=A_t-\eta_t^A\nabla_AJ(B_t,A_t), \end{aligned}$$ where $\eta_t^A$ and $\eta_t^B$ are independently selected finite nonnegative scalars and may depend on the complete optimization history and current first-order information. **Theorem J.1 (Appendix J)** establishes that, for every such policy and every target accuracy, there exists a rank-one hard instance that prevents the method from reaching an $\epsilon$-stationary point within $\lceil\epsilon^{-4}\rceil$ iterations. The constructed objective is globally $1$-smooth, lower bounded, and has a uniformly bounded initialization gap. **Corollary J.2 (Appendix J)** further extends this lower bound to any prescribed factor rank through zero-padding, which preserves the original rank-one hard trajectory while keeping the additional factor directions inactive. Combining this new lower bound with the arbitrary-rank upper bound in Theorem 4.1 establishes the tight deterministic complexity $$T_\epsilon=\Theta(\epsilon^{-4})$$ for simultaneous LoRA updates with separate nonnegative scalar learning rates. **References** [1] Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. *Why Gradient Clipping Accelerates Training: A Theoretical Justification for Adaptivity.* ICLR, 2020. [2] Haochuan Li, Jian Qian, Yi Tian, Alexander Rakhlin, and Ali Jadbabaie. *Convex and Non-convex Optimization Under Generalized Smoothness.* NeurIPS, 2023.

Q1. What new understanding of LoRA optimization do the results provide?

The main new understanding is that the bilinear factorization introduces an intrinsic coupling between weight-space optimization and factor-space dynamics, which affects both deterministic convergence and stochastic optimization.

For J(B,A)=F(BA)J(B,A)=F(BA), the factor gradients satisfy

\qquad \nabla_AJ=B^\top\nabla F(BA).$$ Consequently, the factor-gradient norms depend not only on the weight-space gradient but also on the current factor magnitudes. This coupling has two important implications. First, generalized smoothness must be analyzed together with factor growth. A bound on the weight-space Hessian in terms of the weight-gradient norm alone does not directly characterize the factor-space optimization dynamics. Our deterministic analysis establishes convergence without requiring the factors to remain bounded. Second, the same coupling amplifies stochastic gradient noise. Let $E_t$ denote the weight-gradient noise and define $\zeta_t=(E_tA_t^\top,B_t^\top E_t)$. Then $$\mathbb{E}\\!\left[\lVert\zeta_t\rVert_F^2\mid\mathcal{F}_t\right] \leq \sigma^2 R_t, \qquad R_t=\lVert A_t\rVert_F^2+\lVert B_t\rVert_F^2.$$ This factor-dependent variance bound is established in the main stochastic analysis and extended to multiple adapted matrices in **Lemma I.3 of Appendix I**. Thus, even when the weight-gradient noise has uniformly bounded conditional variance, the corresponding factor-gradient noise bound depends on the evolving factor scale. Our matching lower bounds further show that the deterministic and stochastic complexity barriers already arise for globally smooth weight-space objectives under the respective update classes. Hence, these barriers are not only consequences of generalized smoothness; they are related to the bilinear LoRA parameterization. To complement this theoretical understanding, we have added two decoder-only LLM fine-tuning experiments in **Appendix K**: - **AdamW (Appendix K.1, Figure 7):** TinyLlama-1.1B-Chat fine-tuned on Alpaca with rank-32 LoRA across 88 adapted matrices. Across three seeds, the aggregate factor scale increases from $5.769$ to $2057$ (approximately $357\times$), while factor-gradient noise increases from $0.00115$ to $0.205$. Meanwhile, weight-gradient noise decreases from $36.3$ to $16.1$. - **Plain SGD (Appendix K.2, Figure 8):** Qwen2.5-1.5B fine-tuned on Dolly-15k with rank-32 LoRA across 112 adapted matrices. The aggregate factor scale increases from $5.508$ to $18.24$. Factor-gradient noise increases over training, while weight-gradient noise decreases initially and then remains relatively stable. The AdamW experiment is presented in **Appendix K.1 (Figure 7)**, and the plain-SGD experiment is presented in **Appendix K.2 (Figure 8)**. Together, these experiments provide complementary empirical evidence of factor growth and factor-gradient noise amplification under two different optimizers in decoder-only LLM fine-tuning. They support the practical relevance of the mechanism identified by our analysis. ## Q2. What does tight complexity establish beyond an upper bound? An upper bound alone establishes that a particular update rule can achieve a given convergence rate, but it does not determine whether this rate is optimal or whether a more sophisticated stepsize policy can improve it. The matching lower bound provides this missing characterization. The original lower bound shows that history-dependent shared scalar stepsizes cannot improve the $O(\epsilon^{-4})$ worst-case exponent. The new result in **Theorem J.1 of Appendix J** further establishes that allowing the two LoRA factors to use independently adaptive scalar learning rates does not improve this exponent either. **Corollary J.2** shows that this limitation persists at any prescribed factor rank. Together with the arbitrary-rank upper bound in Theorem 4.1, these results establish a tight worst-case complexity characterization for the considered class of simultaneous scalar-rate LoRA updates. This provides a theoretical benchmark for future LoRA optimizer design: improving the worst-case exponent requires going beyond the update class covered by our lower bound, or introducing additional structural assumptions. Possible directions include momentum, preconditioning, or alternative update structures. We hope that the strengthened lower-bound analysis and the additional decoder-only LLM experiments clarify the theoretical significance and practical relevance of our work.

Thank you again for your careful review and constructive feedback. We have provided point-by-point responses to your concerns, supported by additional analyses and experiments. We would be happy to provide further clarification or address any additional questions.

Reviewer_LFfwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Prior convergence analyses of LoRA assume bounded factors or global L-smoothness, which conflicts with the observation that the Hessian norm grows with the gradient norm. The paper analyses LoRA-GD and LoRA-SGD under sub-quadratic generalized smoothness without bounded factors. It proves an upper bound for GD and a high-probability bound for SGD, with matching lower bounds for adaptive stepsizes. Experiments check the Hessian–gradient relation, factor growth, and factor-amplified noise.

Critical strengths

S1. The motivation is clear. It removes two restrictive assumptions from earlier LoRA theory, and the Hessian-growth observation is plausible.

S2. The theoretical contribution is solid. It covers both GD and SGD, handles the factor–gradient coupling and the factor-amplified noise.

S3. The experiments directly probe the quantities the analysis relies on (Hessian v.s. gradient norm, factor growth, noise amplification).

Critical weaknesses

W1. The practical implication is unclear. The stepsize conditions depend on constants that are unknown in practice. The experiments do not show that the prescribed schedules help in real fine-tuning. The analysis covers plain GD/SGD, not Adam, which is what LoRA uses in practice. W2. The evidence for generalized smoothness is modest. The data come from two settings and three seeds, measured along trajectories on one diagnostic batch, while the assumption is global. On ViT the improvement over a constant fit is not clear-cut. W3. The text model appears only in the Hessian test. Factor growth and noise amplification are shown only on a linear classifier over frozen ResNet-18 features and on a ViT. There is no decoder-only LLM, generation task, or other modality, which narrows the scope of the empirical analysis. W4. The theory covers a single factorized matrix. It is unclear whether, or how, the results extend to many adapted layers with different factor norms, although the experiments adapt 12–48 matrices.

Questions for authors

Q1. What practical guidance does the theory give, for example for choosing the stepsize? Can the prescribed schedules be compared with standard choices (constant stepsize, Adam) in real fine-tuning?

Q2. Do the results extend to multiple LoRA modules? How do factor norms and noise amplification behave across layers?

4 comments

We thank the reviewer for the careful review and constructive feedback. We appreciate the reviewer’s comments and questions, which have helped us clarify the theoretical contributions, better explain the practical implications, and strengthen the empirical evaluation. Below, we address each concern and question point by point. In response, we have added a multiple-matrix theoretical analysis in Appendix I and two decoder-only LLM fine-tuning experiments in Appendix K. We address each concern below.

W1. Practical implications of the theoretical stepsize

We agree that the stepsize prescribed by our convergence theorem depends on problem-dependent quantities that may not be known exactly in practical fine-tuning. Our result should therefore be interpreted as a sufficient theoretical schedule, rather than an automatic stepsize-selection procedure. Specifically, Theorem 4.1 establishes convergence for the schedule ηt=γ/t+1\eta_t=\gamma/\sqrt{t+1}, where the admissible value of γ\gamma depends on the initial objective gap, the initial factor scale, and the generalized-smoothness function. This identifies a theoretically sufficient decay rate.

To broaden the empirical evaluation beyond plain SGD, we have added a decoder-only LLM experiment using AdamW. In Appendix K.1 (Figure 7), TinyLlama-1.1B-Chat is fine-tuned on Alpaca with rank-32 LoRA applied to 88 attention-projection matrices. Across three seeds, the aggregate factor scale increases from 5.7695.769 to 20572057, while factor-gradient noise increases from 0.001150.00115 to 0.2050.205. Meanwhile, weight-gradient noise decreases from 36.336.3 to 16.116.1.

We also report a plain-SGD decoder-only experiment in Appendix K.2 (Figure 8), using Qwen2.5-1.5B and Dolly-15k. The factor scale increases from 5.5085.508 to 18.2418.24, accompanied by an increase in factor-gradient noise, while weight-gradient noise decreases initially and then remains relatively stable.

These experiments show that factor growth and factor-gradient noise amplification are also observed in practical decoder-only fine-tuning with AdamW, in addition to plain SGD. They provide empirical evidence about the optimization dynamics.

W2. Empirical evidence for generalized smoothness

Generalized smoothness is motivated by empirical observations that local smoothness can vary substantially during neural network training and tends to increase with the gradient norm. Zhang et al. [1] documented this behavior and introduced the (L0,L1)(L_0,L_1)-smoothness condition. Li et al. [2] subsequently extended this framework beyond an affine dependence on the gradient norm to a broader class of generalized-smooth objectives.

In our work, we examine whether similar behavior arises in LoRA fine-tuning. Section 6.1 studies the relationship between the Hessian norm and the weight-gradient norm for DistilBERT on IMDb and ViT-B/16 on CIFAR-100. The fitting procedure and additional results are provided in Appendix G.1. For each evaluation trajectory, the constant and gradient-dependent fits are estimated using the other two seeds. Across both settings, the Hessian norm varies substantially during training and tends to be larger at points with larger gradient norms. The gradient-dependent fit achieves a lower average normalized fitting error in both settings.

We agree that these experiments characterize selected training trajectories rather than establish the global generalized-smoothness condition over the entire parameter space. Our goal is to provide empirical motivation for gradient-dependent curvature in LoRA fine-tuning, consistent with the observations in prior work.

References

[1] Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why Gradient Clipping Accelerates Training: A Theoretical Justification for Adaptivity. ICLR, 2020.

[2] Haochuan Li, Jian Qian, Yi Tian, Alexander Rakhlin, and Ali Jadbabaie. Convex and Non-convex Optimization Under Generalized Smoothness. NeurIPS, 2023.

W3. Decoder-only LLM and generation experiments

We agree that the original experiments did not examine factor growth and factor-gradient noise in decoder-only language-model generation. To address this gap, we have added two decoder-only instruction fine-tuning experiments in Appendix K.

  • AdamW — Appendix K.1, Figure 7: TinyLlama-1.1B-Chat is fine-tuned on Alpaca using rank-32 LoRA on 88 attention-projection matrices. Across three seeds, the factor scale grows by approximately 357×357\times (from 5.7695.769 to 20572057), and factor-gradient noise increases from 0.001150.00115 to 0.2050.205, while weight-gradient noise decreases from 36.336.3 to 16.116.1.
  • Plain SGD — Appendix K.2, Figure 8: Qwen2.5-1.5B is fine-tuned on Dolly-15k using rank-32 LoRA on 112 attention-projection matrices. The factor scale increases from 5.5085.508 to 18.2418.24, and factor-gradient noise increases during training, whereas weight-gradient noise decreases initially and then remains relatively stable.

Together, these experiments extend the empirical analysis to decoder-only LLM instruction generation and show the same qualitative factor-growth and factor-gradient-noise patterns under two different optimizers.

W4. Extension to multiple LoRA matrices

We agree that the single-matrix formulation in the main analysis does not by itself explain how the results extend to multiple simultaneously adapted matrices. We have therefore added a dedicated theoretical section, Appendix I: Multiple LoRA-Adapted Matrices under Generalized Smoothness.

The extension considers a joint objective over MM adapted matrices, Wj=Wj0+BjAjW_j=W_j^0+B_jA_j, and allows arbitrary cross-matrix interactions in the objective. The key result is an exact block-diagonal reduction to a single-matrix LoRA problem:

  • Proposition I.1 establishes that the reduction preserves the objective, full factor-gradient norm, total factor scale, and the joint generalized-smoothness condition, including cross-matrix Hessian interactions.
  • Theorem I.2 establishes the deterministic LoRA-GD upper bound for multiple adapted matrices.
  • Lemma I.3 establishes the factor-gradient noise bound in terms of the aggregate factor scale.
  • Theorem I.4 establishes the high-probability LoRA-SGD upper bound for multiple adapted matrices.

Q1. Practical guidance for stepsize selection

Our theory provides a sufficient stepsize schedule rather than an automatic procedure for selecting the optimal learning rate. Specifically, Theorem 4.1 establishes convergence for ηt=γ/t+1\eta_t=\gamma/\sqrt{t+1}, where the admissible γ\gamma depends on the initial objective gap, the initial factor scale, and the generalized-smoothness function. Since these quantities are generally unknown in practical fine-tuning, directly evaluating the theoretical stepsize bound may not be feasible.

As a practical heuristic, we suggest a short pilot run to calibrate the initial learning-rate scale. One can test a small set of candidate initial learning rates, select a stable value based on the training loss and gradient behavior, and then apply the inverse-square-root decay ηt=ηpilot/t+1\eta_t=\eta_{\mathrm{pilot}}/\sqrt{t+1}. During training, the factor scale and factor-gradient noise can be monitored alongside the loss to assess optimization stability. This procedure preserves the decay form suggested by our theory.

For stochastic fine-tuning, our analysis further motivates monitoring factor-gradient noise because its conditional second moment scales with the factor norms. Finally, our additional AdamW experiment in Appendix K.1 (Figure 7) shows that factor growth and factor-gradient noise amplification also occur under AdamW.

Q2. Multiple LoRA modules and layer-wise behavior

The theoretical extension is given in Appendix I. Proposition I.1 and Theorems I.2 and I.4 show that the deterministic and stochastic upper-bound analyses extend to a joint objective over multiple adapted matrices, while Lemma I.3 controls the aggregate factor-gradient noise by the sum of the factor scales.

The decoder-only experiments in Appendix K.1 (Figure 7) and Appendix K.2 (Figure 8) adapt 88 and 112 matrices, respectively, and report the aggregate factor scale and gradient-noise diagnostics. Thus, they demonstrate the phenomena in models with many simultaneously adapted matrices.

Thank you again for your careful review and constructive feedback. We have provided point-by-point responses to your concerns, supported by additional analyses and experiments. We would be happy to provide further clarification or address any additional questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.