RSLoRA: Dynamic Rank Allocation via Representational Similarity

Low-Rank Adaptation (LoRA) freezes the pretrained parameters and introduces additional trainable low-rank modules. Determining an appropriate rank for each LoRA module remains a nontrivial problem. Prior approaches typically assign module-wise ranks by estimating module importance using information derived from the module itself or from interactions among modules, but largely neglect the relationship between each module and its associated pretrained weights. As noted by Hu et al. (2021), LoRA modules effectively amplify task-relevant directions that are already present in the pretrained model but are insufficiently emphasized. Hence, rank allocation should more appropriately be guided by characterizing the relationship between each module and its associated pretrained weights. Motivated by this observation, we propose RSLoRA, which introduces a new evaluation mechanism for rank allocation based on representational similarity. Specifically, RSLoRA measures the similarity between representations produced by the LoRA branch and the pretrained branch using Centered Kernel Alignment (CKA). Under a fixed rank budget, we devise a rank allocation criterion based on the principle that modules exhibiting greater representational dissimilarity (i.e., lower CKA) should be assigned larger rank capacity, as they must model structures that are not adequately captured by the pretrained weights. Overall, our method offers a representation-centric perspective on rank allocation. Experiments across multiple pretrained language models and downstream tasks demonstrate that RSLoRA consistently outperforms state-of-the-art parameter-efficient fine-tuning (PEFT) baselines.