On the Convergence of LoRA under Generalized Smoothness

Low-rank adaptation (LoRA) has become a popularly used parameter-efficient approach for fine-tuning large pretrained models via replacing the full weight update with a factorized update $\Delta W=BA$. Despite its practical utility, existing convergence theory for LoRA remains limited: some prior analyses handle this factorized optimization problem by imposing boundedness assumptions on the LoRA factors, an additional assumption not guaranteed by LoRA updates. More recent results remove this assumption but still rely on $L$-smoothness with respect to the weight variable $W$. In practice, empirical studies of neural network training show that the Hessian norm can grow with the gradient norm, violating the $L$-smoothness assumption on the bounded Hessian norm. Therefore, we analyze LoRA-GD and LoRA-SGD under generalized smoothness without assuming the bounded factor norms. For LoRA-GD, we prove an $O({ϵ}^{-4})$ upper bound and the first $\Omega ({ϵ}^{-4})$ lower bound, even when the stepsize is chosen adaptively from the optimization history. For LoRA-SGD with unbiased stochastic gradients of bounded variance, we obtain an $O({ϵ}^{-8})$ high-probability guarantee. We also prove an $\Omega ({ϵ}^{-8})$ lower bound at fixed confidence for adaptive stepsize policies ${\eta }_t\in [0,1]$ that may depend on the observed optimization history and the current stochastic gradient. Experiments support our theoretical analysis: the Hessian norm grows with the gradient norm, the factor norms grow during optimization, and this growth amplifies stochastic factor-gradient noise.