OReA: Orthogonal Re-Factorization for Low-Rank Adaptation

Low-rank adaptation is a prominent paradigm for the parameter-efficient fine-tuning of large-scale pre-trained models. Conventional LoRA updates weight as $\Delta W=BA$. Based on SVD, recent works parameterize the weight update in a three-factor form: $\Delta W=US{V}^{\top }$. Existing methods differ in how orthogonality is incorporated, ranging from initialization and regularization to manifold-based optimization. In this paper, we propose OReA, which restores orthogonality through a product-preserving QR–SVD re-factorization at designated training steps. Moreover, we propose a nonlinear variant algorithm, OReA+, to augment the representation with a nonlinear complementary branch. We establish the shared rank-$r$ approximation limit of the linear forms with a common frozen reference and bounds for the structured nonlinear extension. Extensive experiments across diverse downstream tasks validate that our method consistently outperforms state-of-the-art methods. Code is available at https://anonymous.4open.science/r/OReA-2BE9.