FedEDS: Discovering and Specializing LoRA Experts for Heterogeneous Federated Fine-Tuning

In practical federated fine-tuning of large pretrained models, clients often pursue different downstream tasks, while clients working on the same task may exhibit skewed label distributions. The coexistence of task heterogeneity and intra-task label skew poses challenges to knowledge sharing among clients. In this setting, multi-expert LoRA enables cluster-specific collaboration, but it requires both reliable client grouping and experts with genuinely distinct low-rank updates. We find that intra-task non-IID data degrade the task-discriminative signal of conventional LoRA factors, making client clustering unreliable, and that even correctly grouped clients can produce experts with highly overlapping low-rank subspaces. We propose FedEDS, a federated multi-expert LoRA framework that jointly addresses these challenges. FedEDS builds task-discriminative client representations from centered query-$A$ factors, adaptively clusters clients without task labels or a predefined number of experts, initializes group-specific experts via effective-update aggregation and truncated SVD, and encourages expert specialization through subspace overlap regularization. Extensive experiments demonstrate that FedEDS outperforms existing federated LoRA methods while achieving reliable client clustering and effective expert specialization.