acceptodds
Under review as a conference paper at ICLR 2027

Gradient-Flow Framework for LoRA: Finite-Stepsize Guarantees and Rank-Dependent Implicit Bias

Abstract

Previous empirical studies have shown that LoRA achieves model quality comparable to full-parameter methods on downstream fine-tuning tasks, even for rank-1 updates. By contrast, the theoretical underpinnings of LoRA remain relatively unexplored. In this work, we analyze the behavior of LoRA from a gradient flow perspective. We first rigorously derive the gradient-flow equations governing LoRA-adapted gradient descent in the limit that finetuning stepsize . In particular, we show that their form is agnostic to whether LoRA parameter updates are applied sequentially or simultaneously. We then show that for finite stepsize , the discrete LoRA iterates track the gradient-flow trajectory to within uniformly on finite finetuning horizons. Finally, we show how the gradient flow equations can be used to analyze the dependence of LoRA's implicit bias on rank parameter . We do so by demonstrating the relative approximation error between the low-rank and full-rank minimizers on two analytically tractable objectives, obtaining closed-form, rank-explicit characterizations of the approximation error in each setting. The objective functions themselves serve as illustrative testbeds with well-behaved, closed-form solutions; the contribution is the gradient-flow framework and methodology, which extends in principle to other fine-tuning objectives.

Primary area:
optimization
Keywords:
LoRA, gradient flow

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_EGnQOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper uses gradient flow to study the optimization dynamics and rank-dependent implicit bias of LoRA. Mainly, it derives the gradient-flow equations from discrete updates and gives exact limiting solutions for a trace-squared objective and low-rank approximation under specific initializations.

Critical strengths

The paper finds exact gradient-flow trajectories and limiting solutions for the two objectives under the given initializations. This lets us directly see how rank and initialization affect the optimization.

Critical weaknesses
  1. The technical novelty of Theorem 2.5 is limited. The analysis appears to follow standard techniques for first-order ODE approximation. The paper does not clearly identify the additional technical or conceptual contribution beyond these standard techniques.

  2. The motivation for the two examples and their relevance to LoRA are insufficiently developed. The authors choose these two objectives since their gradient-flow dynamics can be solved explicitly. However, the paper does not sufficiently explain what these examples reveal about LoRA.

In addition, the spectral initialization assumption in the low-rank approximation setting requires exact spectral alignment and therefore seems too strong, although the authors acknowledge this limitation.

Questions for authors
  1. Since Eqs. (26) and (32) already give the exact limiting solutions, Eq. (33) can be obtained directly by simple algebra. Could the authors explain what extra insights into LoRA we get from the rank-dependent results in Theorem 2.8?

  2. Why did the authors choose the trace-squared and low-rank approximation problems, other than the fact that their gradient flows have closed-form solutions? Specifically, what properties of LoRA do these two examples capture, and what do the results tell us about real fine-tuning tasks?

  3. Theorem 2.5 only holds on finite time intervals, while Theorems 2.8 and 2.11 study the limit as t→∞t \to \infty. Could the authors explain how these results connect to each other? Also, what do they imply for the discrete algorithm when using a fixed, small step size?

  4. The authors state that their contribution is methodological. Could you clarify what is new in the proposed method itself, and what theoretical insights or practical guidance it gives for using LoRA?

Reviewer_MhJ2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies deterministic gradient descent under the LoRA parameterization through a gradient-flow framework. The authors first establish a common continuous-time limit for a family of simultaneous, sequential, and repeated block-update schemes, together with an O(kα)O(k\alpha) trajectory approximation bound over fixed finite time intervals. They then analyze two tractable objectives. For the trace-squared objective, they derive initialization-dependent closed-form solutions and an exact rank-dependent expression for the expected squared relative discrepancy between the low-rank and full-parameter solutions. For the squared Frobenius objective, they solve the dynamics under spectrally aligned initialization and recover the optimal rank-rr truncated SVD solution.

Critical strengths
  1. For the simple trace-squared objective, the authors explicitly derive the solutions selected by LoRA and full-parameter training. Both achieve zero loss, but they produce different matrices, and LoRA’s solution depends on its initialization. This example clearly illustrates how the parameterization can affect the final solution even when the training objective is unchanged.

  2. The authors analyze simultaneous, sequential, and repeated block updates within one framework and estimate their differences from gradient flow. The appendix experiments further examine how these errors change with the learning rate and the number of updates. This makes the connection between the continuous-time analysis and the discrete training process more explicit.

Critical weaknesses
  1. The main analysis is closely related to existing work on gradient flow and matrix factorization. In the low-rank approximation example, the initialization already uses the target matrix’s leading singular directions.The authors should clarify what new difficulty their framework resolves beyond prior work.[1]

  2. The paper shows that discrete updates approximate gradient flow over a fixed time interval when the learning rate is sufficiently small. This does not directly guarantee that different update schemes select the same final solution after arbitrarily long training. Statements about the final solution being independent of the update scheme therefore need more careful qualification.

Reference: [1] Xu Z, Min H, MacDonald L E, et al. Understanding the learning dynamics of lora: A gradient flow perspective on low-rank adaptation in matrix factorization. AISTATS 2025

Questions for authors
  1. Compared with existing studies of gradient flow and matrix factorization, which important conclusions cannot be directly obtained from prior results?

  2. When the learning rate is kept at a small positive value, do simultaneous and sequential updates still reach the same final solution? If this is not established, should the corresponding claims be restricted to the gradient-flow limit?

Reviewer_wqy7OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies LoRA under deterministic, fixed-step gradient descent through a gradient-flow framework. It establishes a common continuous-time limit for a family of simultaneous, sequential, and interpolated block-update schemes, together with a first-order finite-horizon trajectory error bound whose stated dependence on the number of inner updates is linear. It then analyzes two tractable objectives. For the trace-squared objective, it derives the algorithm-selected limiting product and an exact expression for the expected squared relative Frobenius distance from the full-parameter solution under Gaussian initialization. For matrix low-rank approximation, a prescribed spectral initialization decouples the dynamics into scalar singular modes and recovers the classical truncated-SVD optimum and spectral-tail error. Numerical experiments on these objectives examine the step-size error, rank-dependent bound, and inner-update dependence.

Critical strengths
  1. The trace-squared example provides a concrete characterization of implicit bias. Full-parameter and factorized gradient flows can both attain zero objective value while selecting different matrices. The exact Gaussian-initialization moment calculation exposes how this difference depends on dimension and rank. This is the most informative specific result in the paper and deserves credit beyond a generic discussion of low-rank expressiveness.
  2. The finite-horizon comparison covers several explicitly defined update schemes within one analysis. Relating the discrete iterates to the continuous dynamics, rather than only postulating the differential equations, makes the scope of subsequent gradient-flow calculations easier to assess. The numerical studies provide supporting evidence for the stated first-order dependence on the learning rate.
Critical weaknesses
  1. The theoretical increment over prior work is insufficiently established. The low-rank approximation section uses spectral alignment to obtain the same two-variable mode dynamics as Saxe et al. (2014), Section 1.3, Eq. (8), with unequal-factor initialization also treated in that paper's Appendix A. Its limiting product and spectral-tail error are classical. The manuscript cites Saxe et al., but does not explain the additional mathematical contribution through a direct comparison. The general finite-horizon proof uses standard local truncation estimates and error recursion, so its LoRA-specific methodological novelty also needs sharper justification. The two analytic testbeds provide limited evidence for the claimed extension to broader fine-tuning objectives. This uncertainty about the contribution is the main reason for my weak-rejection recommendation.
  2. Several fixable issues weaken the claimed rigor. Remark B.5 infers global gradient-flow existence from continuity and boundedness of the gradient on bounded sets; that inference is insufficient without a non-explosion or continuation argument. A negative cubic scalar objective gives a counterexample to that inference. The inequality between Eqs. (274) and (275) omits the dimension factor in the bound relating squared trace to squared Frobenius norm; the identity matrix has these quantities equal to n squared and n, respectively. Theorem 2.11 should explicitly include the complete spectral initialization of Eq. (45), including zero off-diagonal entries and a zero initial B, since nonzero diagonal entries alone do not imply the conclusion. The preceding text specifies this initialization, so the concern is the self-contained theorem statement. These points appear repairable, but require corrected arguments and statements.
Questions for authors
  1. What precise result or proof technique in Section 2.3 is new relative to Saxe et al. (2014), especially its Eq. (8) and Appendix A? Please identify the additional assumptions, initial conditions, trajectory characterization, or guarantees and explain why they are not a specialization of the earlier analysis. This clarification could materially change the novelty assessment.
  2. Can you give a complete non-explosion and continuation argument for the general gradient flow used in Theorem 2.5, correct the trace bound in Eqs. (274)-(275), and state all initialization conditions directly in Theorem 2.11? Please indicate whether these corrections change any constants, assumptions, or main conclusions. A complete repair could improve the soundness assessment.
  3. Are the rank-dependent limiting-solution claims intended only for gradient flow, or also for converged discrete gradient descent at a fixed positive learning rate? If the latter is claimed, where is the argument that connects the finite-time approximation in Theorem 2.5 to the infinite-time limit? If the claims are restricted to gradient flow, please make that scope explicit in the contribution statements.
Reviewer_1bS4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper develops a gradient-flow framework to analyze LoRA's optimization dynamics and rank-dependent implicit bias. The authors rigorously derive the continuous-time gradient-flow equations for LoRA-adapted gradient descent and show that they are invariant to sequential or simultaneous parameter updates. They further establish an (O(k\α)) bound on the deviation between discrete LoRA iterates and gradient-flow trajectories for finite learning rates. Finally, the framework is applied to two analytically tractable objectives, yielding closed-form, rank-explicit characterizations of the approximation error between low-rank and full-rank minimizers. The main contribution lies in the theoretical framework and methodology rather than the specific objectives, with potential extensions to more general fine-tuning problems.

Critical strengths

The paper provides a unified theoretical framework that connects LoRA's discrete parameter updates, continuous-time gradient-flow dynamics, and the low-rank approximation error of the converged solutions. This represents a mathematically meaningful contribution.

Critical weaknesses

The paper's novelty appears somewhat limited, and several important questions remain unanswered.

1 why does the main result for low-rank approximation rely on a spectral initialization that requires prior knowledge of the target matrix's singular vectors? How would the theoretical results and convergence dynamics change under standard random LoRA initialization, which is more commonly used in practice?

2 can the proposed gradient-flow framework provide insights into early-stage optimization dynamics, rather than focusing primarily on the final approximation error? In particular, it would be valuable to investigate how initialization scale, gradient magnitude, and rank affect the effective weight-update direction and convergence speed.

3 prior studies such as PiSSA, MiLoRA, and NoRA have explored the relationship between initialization and optimization dynamics, suggesting that stronger or better-structured initial gradients can facilitate faster early-stage convergence. Can the proposed framework theoretically explain these observations and clarify the relationship between initial gradient magnitude, gradient alignment with full fine-tuning, and convergence speed?Addressing these questions would significantly strengthen the paper's novelty and practical relevance to LoRA fine-tuning.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.