On the Convergence of Stochastic Low-Rank Adaptation

Low-rank adaptation (LoRA) minimizes the training loss after adding a trainable low-rank adapters to a frozen pretrained weight matrix. Previous analysis of gradient descent for LoRA (LoRA-GD) gives a sufficient complexity of $\exp (\mathcal{O}({ϵ}^{-2}))$ iterations to find an $ϵ$-stationary point in the deterministic setting. We sharpen this analysis and show that $\mathcal{O}({ϵ}^{-4})$ iterations suffice for the same first-order criterion. We further study stochastic LoRA under unbiased stochastic gradients with finite variance. We propose LoRA-NSGDM, which finds an $ϵ$-stationary point with $\mathcal{O}({ϵ}^{-8})$ stochastic oracle calls. Under an additional mean-square smoothness condition, we propose LoRA-STORM, which improves the stochastic oracle complexity to $\mathcal{O}({ϵ}^{-6})$.