Improving Low-Rank Adaptation by Adjusting its Dynamics

Low-Rank Adaptation (LoRA) efficiently fine-tunes pre-trained large language models with two trainable low-rank matrices B and A. Previous LoRA variants primarily improve performance by designing the initialization or update dynamics of $\mathbf{B}$ and $\mathbf{A}$ to preserve more information from the full parameter gradient matrix. Such information preservation depends on the alignment between the subspaces induced by ${\mathbf{B}}^t,{\mathbf{A}}^t$ and the current gradient, which is difficult to guarantee throughout training. Nevertheless, few works investigate the conditions on ${\mathbf{B}}^t$ and ${\mathbf{A}}^t$ that yield the most loss descent for a given current adapted weight ${\mathbf{W}}^t$ at an arbitrary training iterate. In this paper, we theoretically analyze the loss descent of LoRA from the $t$-th step to the $t+1$-th step and provide the conditions on ${\mathbf{B}}^t$ and ${\mathbf{A}}^t$ for the most loss descent: the largest and smallest singular values of ${\mathbf{B}}^t$ and ${\mathbf{A}}^t$ equal the square roots of the singular values of ${\mathbf{B}}^t{\mathbf{A}}^t$ with aligned internal singular subspaces. We reveal that the balance $({\mathbf{B}}^t{)}^{\top }{\mathbf{B}}^t={\mathbf{A}}^t({\mathbf{A}}^t{)}^{\top }$ is a simple and easy-to-implement case of the conditions on ${\mathbf{B}}^t$ and ${\mathbf{A}}^t$, and propose the method Balance-Preserving Dynamics Adjustment for LoRA (BDA-LoRA) by adjusting the dynamics to make ${\mathbf{B}}^t$ and ${\mathbf{A}}^t$ balance in each step. Specifically, BDA-LoRA obtains a gradient-approximate balanced initialization via LoRA-based randomized SVD without computing or storing the full parameter gradient matrix, and modifies raw optimizer update directions slightly to make ${\mathbf{B}}^{t+1}$ and ${\mathbf{A}}^{t+1}$ balance from ${\mathbf{B}}^t$ and ${\mathbf{A}}^t$ for all $t$. Experiments on NLU, NLG and commonsense reasoning tasks show the effectiveness and superiority of BDA-LoRA.