Disentangled Mixture of Rank-1 Experts for Low-Rank Adaptation

The uniform rank allocation in Low-Rank Adaptation (LoRA) overlooks heterogeneous learning dynamics across modules and inherent semantic differences across inputs. We propose Disentangled Mixture of Rank-$1$ Experts (DMoRE), a principled framework for input-dependent rank allocation. To this end, we develop dynamic self-routing to activate a variable number of rank-$1$ components per input without an external router, along with gradient-derived budget estimation to avoid potentially misaligned manual budget specification. We further propose co-activation-aware rank disentanglement that encourages frequently co-activated components to learn distinct transformations. Extensive experiments on diverse language and vision tasks demonstrate that DMoRE consistently outperforms competitive LoRA baselines with negligible additional parameter overhead. It also exhibits structured rank allocation patterns at multiple granularities and reliably translates additional component activations into effective rank capacity.