DimLoRA: Dimension-Level LoRA Experts for Multi-Task Learning

Parameter-efficient fine-tuning (PEFT) has been widely adopted to reduce the computational cost of adapting large language models to downstream tasks. Among existing PEFT methods, Low-Rank Adaptation (LoRA) is widely recognized for its simplicity and strong empirical performance. However, most LoRA-based approaches are designed for single-task adaptation and often suffer from severe inter-task interference in multi-task scenarios. Recent LoRA-MoE methods mitigate this issue by integrating LoRA with mixture-of-experts (MoE) architectures, but introducing multiple full LoRA modules as experts incurs considerable parameter overhead. Through empirical analysis, we identify significant parameter redundancy in LoRA and task-specific activation patterns among feed-forward network neurons. Motivated by these observations, we propose DimLoRA, a fine-grained expert architecture for parameter-efficient multi-task adaptation. Instead of assigning a complete LoRA module to each expert, DimLoRA decomposes the parameter space along the intermediate dimension of feed-forward networks and introduces lightweight dimension-level experts for localized parameter updates. Extensive experiments demonstrate that DimLoRA achieves superior multi-task performance with substantially lower parameter overhead, highlighting the effectiveness of fine-grained experts for efficient multi-task learning. See the code at https://anonymous.4open.science/r/DimLoRA.