RAC-LoRA: Router-Aware Combinatorial LoRA for Adaptive Parameter-Efficient Fine-Tuning

Low-Rank Adaptation (LoRA) provides an efficient approach to parameter-efficient fine-tuning, while recent MoE-LoRA methods introduce multiple LoRA experts and input-dependent routing to improve adaptation capacity. However, existing designs commonly impose fixed relationships between the input-side and output-side low-rank factors, and the router mainly acts as an external selector rather than directly participating in expert computation. These constraints limit the flexibility of expert composition and weaken the coupling between routing and expert specialization. We propose RAC-LoRA (Router-Aware Combinatorial LoRA), which constructs $N_A{N}_B$ implicit experts by independently parameterizing $A$ and $B$ factors and dynamically routing among their combinations. In addition to expert selection, each router defines an input direction that conditions the corresponding expert while retaining the original representation. This enables the router to participate directly in expert computation without excessive information loss. We aggregate selected experts additively to allow complementary adaptation components to be jointly incorporated. Experiments on Llama-3.2-3B and Llama-3.1-8B show that RAC-LoRA achieves the highest average accuracy in the single-domain setting and the highest multi-task accuracy on both backbones among all compared methods.