acceptodds
Under review as a conference paper at ICLR 2027

On the Convergence of Stochastic Low-Rank Adaptation

Abstract

Low-rank adaptation (LoRA) minimizes the training loss after adding a trainable low-rank adapters to a frozen pretrained weight matrix. Previous analysis of gradient descent for LoRA (LoRA-GD) gives a sufficient complexity of iterations to find an -stationary point in the deterministic setting. We sharpen this analysis and show that iterations suffice for the same first-order criterion. We further study stochastic LoRA under unbiased stochastic gradients with finite variance. We propose LoRA-NSGDM, which finds an -stationary point with stochastic oracle calls. Under an additional mean-square smoothness condition, we propose LoRA-STORM, which improves the stochastic oracle complexity to .

Primary area:
optimization
Keywords:
LoRA, stochastic optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_kjaiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this manuscript, the authors study the convergence of LoRA optimization when the original loss is smooth and lower bounded, but the factorized objective is not globally smooth. It improves the deterministic LoRA-GD complexity to (O(\epsilon^{-4})), provides a finite-variance counterexample for plain LoRA-SGD, and establishes stochastic oracle complexities of (O(\epsilon^{-8})) for normalized momentum and (O(\epsilon^{-6})) for a STORM variant.

Critical strengths
  1. The deterministic analysis improves the previous exponential sufficient bound to a polynomial bound for the same algorithm and stationarity criterion.

  2. The scalar counterexample explains why finite-variance product-space gradients do not suffice for ordinary simultaneous factor updates.

  3. The stochastic proofs account for factor-dependent smoothness and variance without assuming uniformly bounded factors. I did not identify a specific error in the main arguments.

Critical weaknesses
  1. The stochastic algorithms adapt established methods. Further discussion of rate tightness would help clarify the theoretical contribution.

  2. The experiments use tuned parameters and schedules that differ from those analyzed, so they do not directly validate the convergence rates.

  3. Comparisons mainly involve prior adaptive LoRA-GD rules. Tuned SGD with momentum and AdamW, repeated runs, and comparisons under matched computational budgets would strengthen the evaluation.

  4. Factor-space stationarity need not imply a useful solution or stationarity in the product variable. This limitation deserves more discussion.

Questions for authors
  1. Could sharper trajectory control improve the stochastic rates, or are the current bounds expected to be tight?

  2. Does STORM’s advantage remain when comparing gradient evaluations or runtime rather than iterations?

  3. In the multilayer experiments, is normalization performed jointly across adapters or separately by layer? How does this correspond to the analysis?

  4. Can a controlled experiment measure the full factor-gradient norm under the theoretical schedules?

Reviewer_X1A6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the convergence of LoRA optimization. The authors improve the existing convergence for deterministic LoRA gradient descent from exponential to polynomial. They also show that standard stochastic LoRA may fail to converge under bounded variance assumptions. The authors propose two methods, LoRA-NSGDM and LoRA-STORM, and provide convergence guarantees for both. The authors test their methods on CIFAR-10 with logistic regression and ResNet-18, and on Alpaca with TinyLlama-1.1B. The results show faster training loss reduction compared to the considered LoRA-GD baselines. Proposed stochastic methods are adaptations of existing NSGDM and STORM algorithms.

Critical strengths

The authors improve the convergence bound for deterministic LoRA from exponential to polynomial under the same assumptions. Convergence guarantees for stochastic LoRA have relatively mild assumptions. This helps to understand LoRA optimization.

Critical weaknesses

Convergence guarantees are established for stationary points of the factorized objective. Such points do not necessarily correspond to good solutions of the original optimization problem. I do not think the paper sufficiently demonstrates the significance of these theoretical results for practical LoRA training.

The experiments mainly compare the proposed methods with LoRA-GD variants. LoRA-STORM is evaluated only on logistic regression. The authors do not demonstrate its advantages on ResNet-18 or TinyLlama-1.1B.

Experiments lack comparisons under equal computational budgets and across multiple random seeds. And do not evaluate the stationary criterion from theory. I do not think the experimental results are strong enough.

The paper does not sufficiently discuss existing convergence guarantees for stochastic low-rank adaptation. In particular, RAC-LoRA already provides convergence results for SGD, Random Reshuffling and FL for smooth nonconvex objectives.

Questions for authors

I do not have specific clarification questions. My main concerns are about the significance of the theory and limited experiments.

Reviewer_iPw7OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies convergence of LoRA for the factorized objective J(V)=f(BA)J(V)=f(BA), whose gradient is generally not globally Lipschitz. It improves the deterministic convergence bound and gives stochastic convergence bounds under finite-variance noise, with a better bound under mean-square smoothness. Experiments compare the methods' behavior in training.

Critical strengths

By establishing polynomial bounds for first-order stationarity without assuming uniformly bounded factors, the paper gives useful guarantees under smoothness and lower boundedness of the original loss, with additional oracle conditions in the stochastic setting. This matters because factor growth can invalidate a direct use of standard global smoothness arguments. The results therefore improve the understanding of convergence for the factorized LoRA objective under the stated conditions.

Critical weaknesses
  1. The polynomial bounds under the stated assumptions are useful, but the contribution mainly sharpens existing analysis and extends established stochastic optimization tools to factorized LoRA. Controlling factor growth is a real part of the analysis, yet I find the broader insight beyond establishing these guarantees limited. The work strengthens what can be proved under these conditions, while offering less guidance on what this teaches us about LoRA optimization more generally. On this basis, the theoretical advance is a limited step beyond prior work.
  2. The narrow baseline choice and the smaller gains in the reported larger-initialization setting limit the evidence for a general optimizer advantage. Faster loss reduction also does not establish faster convergence to stationarity, and the reported minibatch gradient norm is not the full-objective gradient norm used in the theory. Although the paper states STORM's extra oracle cost, its experimental comparison does not match methods by total oracle calls. These gaps leave it unclear whether the gains extend beyond the selected baselines and whether STORM offers a computational benefit at equal evaluation budgets.
Questions for authors
  1. What is the main new idea in the analysis of factor growth and stochastic error, and how does it change our understanding of factorized optimization beyond the improved bounds and the use of established optimization tools under the stated assumptions?
  2. On an existing task, what evidence from the reported experiments is available for an advantage over standard optimizers when tuning effort and total gradient evaluations are comparable? Please clarify how the initialization affects that advantage and whether the comparison supports faster convergence to stationarity as well as lower training loss, including whether STORM remains beneficial after accounting for its extra evaluations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.