Exploiting Symmetry in Low-Rank Decomposition for LoRA Fusion

Low-Rank Adaptation (LoRA) is widely used to fine-tune Large Language Models across diverse downstream tasks, motivating model fusion as a way to integrate multiple task-specific LoRA adapters into a single model without access to the original training data. Existing LoRA fusion methods, ranging from naive averaging to interference-resolution approaches, overlook a key geometric property of the LoRA parameter space: LoRA adapters admit multiple reparameterizations that are functionally equivalent, a property we refer to as low-rank decomposition (LRD) symmetry. Ignoring this symmetry, however, causes direct fusion of independently trained adapters to suffer from parameter-space misalignment, leading to degraded representations and performance. To address this issue, we propose a symmetry-aware parameter alignment framework that maps LoRA adapters into a shared latent basis prior to fusion, first through a closed-form rotational alignment and then through a gradient-based refinement that relaxes orthogonality to the wider space of invertible reparameterizations. Our method is a plug-and-play enhancement compatible with existing LoRA fusion techniques. Extensive experiments on real-world benchmarks show broad improvements across diverse fusion methods. Our implementation is available on https://anonymous.4open.science/r/GRA.