PSR-LoRA: Power-Similarity Regularized Low-Rank Adaptation

Low-Rank Adaptation (LoRA) has become a widely adopted approach for parameter-efficient fine-tuning of large language models. However, its rank components are typically optimized jointly without explicitly modeling their mutual dependencies, which may cause multiple components to learn highly correlated adaptation patterns and lead to inefficient utilization of the limited rank budget. In this work, we propose Power-Similarity Regularized LoRA (PSR-LoRA), a parameter-efficient adaptation framework that explicitly models and regulates the relationships among rank-wise update components. Specifically, PSR-LoRA first decomposes the LoRA update into rank-one components and maps them into a power-transformed space using a signed power function controlled by an exponent hyperparameter. Pairwise dependencies among the transformed components are then characterized through rank-wise similarity modeling. Based on the resulting similarity structure, we introduce a redundancy regularization objective to suppress excessive dependencies among rank components and further derive similarity-guided weights to adaptively adjust their contributions to the final low-rank update. To maintain efficiency, the proposed similarity computation is performed directly on the low-rank factors without explicitly constructing full rank-one update matrices. Extensive experiments across multiple downstream tasks demonstrate that PSR-LoRA consistently improves adaptation performance under constrained trainable parameter budgets while inducing less redundant rank-wise adaptation patterns. These results show that explicitly organizing the internal structure of a fixed low-rank space provides an effective alternative to simply increasing the LoRA rank.