Gradient-Flow Framework for LoRA: Finite-Stepsize Guarantees and Rank-Dependent Implicit Bias

Previous empirical studies have shown that LoRA achieves model quality comparable to full-parameter methods on downstream fine-tuning tasks, even for rank-1 updates. By contrast, the theoretical underpinnings of LoRA remain relatively unexplored. In this work, we analyze the behavior of LoRA from a gradient flow perspective. We first rigorously derive the gradient-flow equations governing LoRA-adapted gradient descent in the limit that finetuning stepsize $\alpha \to 0$. In particular, we show that their form is agnostic to whether LoRA parameter updates are applied sequentially or simultaneously. We then show that for finite stepsize $\alpha >0$, the discrete LoRA iterates track the gradient-flow trajectory to within $O(k\alpha )$ uniformly on finite finetuning horizons. Finally, we show how the gradient flow equations can be used to analyze the dependence of LoRA's implicit bias on rank parameter $r$. We do so by demonstrating the relative approximation error between the low-rank and full-rank minimizers on two analytically tractable objectives, obtaining closed-form, rank-explicit characterizations of the approximation error in each setting. The objective functions themselves serve as illustrative testbeds with well-behaved, closed-form solutions; the contribution is the gradient-flow framework and methodology, which extends in principle to other fine-tuning objectives.