Collaboration through Diversity and Specialization via Orthogonality for Improved LoRA-based MoE

Combining the Mixture-of-Experts (MoE) paradigm with Low-Rank Adaptation (LoRA) enables parameter-efficient adaptation of large language models (LLMs) to downstream tasks. However, existing LoRA-based MoE approaches often suffer from expert interference and weak specialization, where experts produce correlated outputs, leading to overlapping representations and reduced effective capacity. A key challenge is to balance collaboration and specialization: experts jointly capture task-specific knowledge while reducing redundancy and conflicts across experts. To address this challenge, we propose a principled framework that characterizes expert interactions in a data-dependent manner. We show that redundancy arises from correlated expert outputs and propose an explicit regularization objective that encourages orthogonality, promoting effective specialization and mitigating interference. Furthermore, built on an asymmetric LoRA framework with shared low-rank projections, we introduce a complementary regularization that encourages diversity in the shared representations. Our experiments across diverse benchmarks demonstrate that our method encourages representational specialization and improves task performance while maintaining the efficiency of parameter-efficient fine-tuning. Our code is available at https://anonymous.4open.science/r/OD_MoLE-A5F2.