SLoRA: Learning Structured Low-Rank Updates for Cross-Domain Transfer

Parameter-efficient fine-tuning (PEFT) methods such as Low-Rank Adaptation (LoRA) have become standard tools for adapting large pre-trained models. However, conventional LoRA applies uniform low-rank updates across all adaptation directions, which can limit flexibility and lead to unstable optimization or degraded generalization under domain shift, particularly in multimodal settings. We propose SLoRA, a lightweight extension of LoRA that introduces structured basis-space reparameterization and adaptive gating into low-rank updates to improve stability and robustness during transfer. Specifically, SLoRA transforms the low-rank adaptation space into a fixed orthogonal basis representation, applies learnable basis scaling to redistribute adaptation capacity across structured directions, and employs local basis modulation together with a global input-adaptive gate to control update strength. These mechanisms are batch-stabilized, allowing SLoRA to operate reliably on mixed-modality data without requiring explicit modality labels. We further provide theoretical insights suggesting that the proposed design promotes smoother update dynamics and acts as a form of structured adaptation regularization. Empirically, we evaluate SLoRA across four diverse domains, radiology, pathology, remote sensing, and art, on six multimodal visual question answering (VQA) benchmarks and eleven image classification datasets using multiple general-purpose vision and vision-language backbones. SLoRA consistently outperforms standard LoRA, achieving average gains of over 2.2% on VQA tasks and 2.9% on classification tasks, while remaining fully plug-and-play and parameter-efficient. These results demonstrate that structured low-rank reparameterization with input-adaptive modulation provides an effective and practical solution for robust parameter-efficient transfer under domain shift.