CAST-LoRA: Cross-Attentive Soft-Transition MoE-LoRA for Multi-Task Adaptation

Parameter-efficient multi-task fine-tuning is important for deploying large language models (LLMs) across diverse tasks under limited training, storage, and serving budgets. However, existing Low-Rank Adaptation (LoRA) methods for multi-task adaptation still have the following limitations that require further improvement in performance: (1) training a single adapter across all tasks induces severe task interference in multi-task scenarios. (2) routing over a flat expert pool neglects dedicated shared capacity, failing to capture common knowledge across tasks. To address these problems, this paper proposes CAST-LoRA, a novel multi-task Mixture-of-Experts Low-Rank Adaptation (MoE-LoRA) framework. By clustering training data to guide early routing and then annealing cluster-guided assignments into input-dependent soft routing, CAST-LoRA stabilizes task-to-expert specialization and mitigates task interference. CAST-LoRA further introduces role-separated task-specific, entropy-aligned shared, and floor-preserved shared experts, providing dedicated capacity for routing uncertainty and reusable cross-task knowledge. Additionally, a cross-attentive task encoder extracts compact task representations with $\mathcal{O}(Ld)$ rather than $\mathcal{O}(L^{2}d)$ attention interactions, where $L$ is the source-sequence length and $d$ is the hidden dimension. Extensive experiments with 10 baselines on 7 benchmarks demonstrate that CAST-LoRA achieves the optimal average accuracy, reduces gradient conflict to an optimal level of 1.09%, cuts task-encoder latency by 86.9%, cuts task-encoder peak memory by 23.8%, lowers expert-parameter redundancy by 26.8%, and increases effective expert directions by 26.8%.