acceptodds
Under review as a conference paper at ICLR 2027

FedEDS: Discovering and Specializing LoRA Experts for Heterogeneous Federated Fine-Tuning

Abstract

In practical federated fine-tuning of large pretrained models, clients often pursue different downstream tasks, while clients working on the same task may exhibit skewed label distributions. The coexistence of task heterogeneity and intra-task label skew poses challenges to knowledge sharing among clients. In this setting, multi-expert LoRA enables cluster-specific collaboration, but it requires both reliable client grouping and experts with genuinely distinct low-rank updates. We find that intra-task non-IID data degrade the task-discriminative signal of conventional LoRA factors, making client clustering unreliable, and that even correctly grouped clients can produce experts with highly overlapping low-rank subspaces. We propose FedEDS, a federated multi-expert LoRA framework that jointly addresses these challenges. FedEDS builds task-discriminative client representations from centered query- factors, adaptively clusters clients without task labels or a predefined number of experts, initializes group-specific experts via effective-update aggregation and truncated SVD, and encourages expert specialization through subspace overlap regularization. Extensive experiments demonstrate that FedEDS outperforms existing federated LoRA methods while achieving reliable client clustering and effective expert specialization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Federated Learning, Federated Fine-Tuning, Low-Rank Adaptation, LoRA, Mixture of Experts, Parameter-Efficient Fine-Tuning, Client Clustering, Task Heterogeneity, Label Distribution Skew, Subspace Regularization

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_qWTHOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FedEDS studies federated LoRA fine-tuning with task heterogeneity and within-task label skew. It uses cross-client centered query-A factors and affinity propagation to discover client groups, constructs group-specific experts by aggregating effective LoRA updates followed by truncated SVD, and applies an input-subspace overlap regularizer to encourage expert specialization. Experiments on four GLUE classification tasks report improved average accuracy over federated LoRA baselines, accompanied by analyses of client clustering and expert subspace overlap.

Critical strengths
  1. Relevant problem setting and coherent design. The paper considers task differences alongside statistical heterogeneity within each task, where indiscriminate parameter sharing can be detrimental. Its discovery, initialization, and specialization stages form a clear procedure for organizing collaboration without requiring known task labels during grouping.

  2. Informative empirical diagnostics. The centered query-A representation achieves strong clustering results under the evaluated label-skew settings, and the overlap analysis shows that even task-grouped experts can share highly similar input subspaces. The main GLUE comparison also reports an average accuracy of 88.13%, compared with 87.12% for the strongest listed baseline. These observations provide a useful empirical motivation for studying client grouping and expert differentiation together.

Critical weaknesses

1. The independent contribution of centered query-A to client discovery and final accuracy remains unclear.

FedEDS extends earlier work on client grouping and federated LoRA experts, including FedLEASE and FedTreeLoRA. The use of a centered query-A representation is a potentially useful distinction, but the evidence currently focuses on clustering quality rather than its effect on the complete training procedure. Section 5.2.1 reports strong clustering results after the warmup phase, while the diagnostics use a single random seed and the end-to-end ablations in Table 3 keep centered query-A clustering unchanged.

A controlled end-to-end comparison with raw query-A, the displacement from the shared initialization (Ai−A0A_i-A_0), and oracle task grouping would help establish the benefit of cross-client centering. In particular, Ai−A0A_i-A_0 provides a relevant alternative reference for removing a shared component. Holding expert initialization and subsequent training fixed would make it possible to determine whether better grouping accounts for a meaningful portion of the final performance gain. Such evidence is important for distinguishing the proposed client representation from existing grouping-based approaches.

2. Reduced geometric overlap has not yet been linked convincingly to functional expert specialization.

Table 2 shows that A-only regularization reduces input-subspace overlap from 0.8643 to 0.0061 in the reported experiment. This directly supports the geometric effect of the penalty, but smaller overlap alone does not establish that experts develop complementary task-specific capabilities. Task-conditioned routing statistics, expert utilization, or the effect of removing individual experts would help assess whether the separated subspaces correspond to meaningful differences in model behavior.

The choice of A-only regularization also merits a more explicit trade-off analysis. Table 2 already compares it with joint A+B regularization; the latter reports 89.13% accuracy versus 88.69% for A-only regularization in a single-seed IID setting. This result does not establish a consistent advantage across seeds, but it makes the rationale for the input-only design worth explaining. Measured runtime or memory overhead for the two alternatives would help determine whether the simpler regularizer offers a practical benefit.

Additional experimental and presentation observations (minor). The primary evaluation is restricted to RoBERTa-Large, four fixed classification tasks, 16 clients, and full client participation. Unequal task-group sizes or partial participation would be useful tests of expert discovery, particularly because the clustering and main ablations rely on one seed even though the principal results use three. Reporting paired per-seed differences, along with empirical runtime and expert-bank storage relative to the closest multi-expert baselines, would improve the assessment of robustness and cost. Figure 5 has small labels and legends, and the closely placed tables and explanatory text on page 9 would benefit from more space.

Questions for authors
  1. Value of client centering. How does final model performance change when centered query-A clustering is replaced with raw query-A, Ai−A0A_i-A_0, or oracle task grouping, keeping expert initialization and later training fixed?

  2. Functional specialization. Does the substantial reduction in input-subspace overlap correspond to task-dependent expert utilization or distinct contributions to task accuracy? If available, routing statistics or expert-removal experiments would help answer this question.

  3. Choice of regularizer. What accounts for the choice of A-only over joint A+B regularization, given the accuracy difference reported in Table 2? Are there measurable computational savings, and how stable is the comparison across seeds?

Reviewer_qPfpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FedEDS targets federated LoRA fine-tuning in which clients work on different tasks and, within a task, have skewed label distributions. After Tw=5T_w = 5 warmup rounds from a shared LoRA initialization, the server centers each client's query-AA factor across clients and normalizes it per layer. It then clusters clients with affinity propagation (AP) using the median preference, so neither task labels nor the number of experts KK are given. Each group expert is initialized by averaging the clients' effective weight updates swBiAis_w B_i A_i and applying truncated SVD at the expert rank rer_e. In later rounds, each client trains its assigned expert and the group router, following the routing scheme of FedLEASE. A QR-based regularizer penalizes the input subspace overlap (the mean squared cosine of the principal angles between the row spaces of expert AA factors) with the other, frozen experts. Experiments use RoBERTa-Large with 16 clients on SST-2, QNLI, MRPC and QQP (four clients per task, Dirichlet α=0.5\alpha = 0.5). FedEDS reports 88.13% average accuracy over three seeds, 1.01 pp above the strongest of eight baselines (FedALT). In the seed-42 diagnostics, centered query-AA recovers the true groups (ARI = 1.0) at all label-skew levels, and AA-only regularization reduces AA-overlap from 0.8643 to 0.0061 under intra-task IID data.

Critical strengths
  1. The finding that cross-client centering makes query-AA task-discriminative under label skew is clear, cheap and well supported. Table A.2 gives the cleanest evidence, because pairwise AUC does not depend on the clustering algorithm.

    • The same-task pair AUC of centered query-AA is at least 0.9900 in all eight settings (warmup ranks 4 and 8; IID and α∈{1.0,0.5,0.1}\alpha \in \{1.0, 0.5, 0.1\}).
    • Under label skew, the AUC of All BB falls to between 0.5781 and 0.8003.
    • Raw query-AA has cross-task cosine similarity of 0.975 to 0.985.

    Figure 5 shows the same pattern in ARI. Appendix A.1.3 gives a simple reason: under the shared initialization, the common component compresses raw-factor cosine distances, and centering removes it. FedLEASE clusters warmup BB factors (Section 2.2), so this is a useful alternative. It is computed once after warmup (Appendix A.5.1), and with the median AP preference it needs neither task labels nor a preset KK. This finding is the main reason I see value in the paper.

  2. The complete method is compared with eight federated LoRA baselines under a shared protocol, with three seeds and standard deviations. The protocol fixes the backbone, data partitions, LoRA injection positions, participation, local epochs, communication rounds and seeds. FedEDS has the highest average accuracy in Table 1 (88.13 ± 1.12). It is 3.18 pp above FedLEASE (84.95 ± 2.34), whose routing scheme it adopts (Section 4.4), so the gain points to the parts FedEDS changes. Per-task results are reported in full, including SST-2 and QQP, where FedEDS is not the best. This supports the claim that the complete method is competitive in the tested setting.

Critical weaknesses
  1. The evidence for the second contribution, expert specialization through the subspace overlap regularizer, is weak. Three points lead me to this view.

    • The motivating observation may reflect the initialization rather than the grouping. Observation 2 reads the AA-overlap of 0.8643 (Figure 3, Table 2) as experts for different tasks retaining "highly similar input adaptation directions". However, all warmup adapters start from a shared A0A_0 with B0=0B_0 = 0, and AA starts to move only at the second local step (Proposition A.1). After warmup, raw query-AA of clients from different tasks still has cosine similarity of 0.975 to 0.985 (Table A.2). Experts initialized by Eq. 8–9 take their AA row spaces from the right singular vectors of averaged updates, and the row spaces of these updates are close to that of A0A_0. BB, which starts from zero, ends with an overlap of only 0.0401. The paper does not say how the Table 2 experts are initialized and does not report the overlap at initialization, so this simpler explanation is not ruled out.
    • Lower AA-overlap is a geometric property, not evidence of specialization. No experiment shows that the experts learn different functions. Possible tests include each expert's accuracy on each task, the routing weights per task, or the effect of removing one expert, each with and without the regularizer.
    • The accuracy evidence is small and comes from seed 42 only.
      • In Table 2 (intra-task IID), AA-only regularization adds 0.41 pp. The best configuration is A+BA+B regularization (89.13), which FedEDS does not use.
      • Section 4.3 justifies AA-only by the extra QR computations on the output side. These cost O((2K−1)dinre2+Kre3)O((2K-1)d_{in} r_e^2 + K r_e^3) per module (Eq. A.23), which is small next to a forward and backward pass of RoBERTa-Large.
      • In Table 3, the 1.03 pp gain over w/o Reg. comes entirely from QNLI (+5.20). FedEDS is lower on SST-2 (−0.77) and QQP (−0.75).
      • On QNLI, the three-seed standard deviation of FedEDS in Table 1 is 3.87, and the seed-42 value in Table 3 (90.48) lies about 4 pp above the three-seed mean (86.38). The ablation gap is therefore of the same size as the seed-to-seed variation.

    The regularizer is one of the two stated contributions, and the title promises specialization, so this point weighs most in my assessment.

  2. The main accuracy gain is modest, and the clustering result is shown in one narrow setting.

    • All accuracy results use RoBERTa-Large on four binary GLUE tasks with 16 clients and full participation, while the abstract and introduction frame the method for large pretrained models and LLMs.
    • FedEDS leads FedALT by 1.01 pp (88.13 ± 1.12 vs. 87.12 ± 0.68), and it is not the best on SST-2 or QQP (84.58 vs. 87.53 for FedALT on QQP).
    • The claim of "perfect clustering across all evaluated label-skew settings" rests on seed 42 (Table A.1, Appendix A.3.1) and on one task mix with four equal groups. ARI and KK are not reported for the main runs with seeds 37 and 64.
    • In this task mix, each task has its own input source and format, and clients of the same task differ only in their label proportions. By Proposition A.1, the rows of the displacement of AA lie in the span of the layer inputs, whereas BB is driven by the error signal, which depends on the labels. This suggests that centered query-AA separates clients by their input distribution, which here coincides with the task.

    The experiments do not test tasks that share an input distribution, unequal group sizes or a different number of tasks. They also do not test the sensitivity of AP to the preference ρ\rho and to TwT_w. As it stands, the generality of the first contribution is not shown.

Questions for authors
  1. What is the AA-overlap between the experts right after initialization, before any expert training? How are the experts in Table 2 initialized?

    • For the main runs at α=0.5\alpha = 0.5, please report FedEDS, w/o Reg. and w/o SVD separately for seeds 37, 42 and 64, and compare AA-only with A+BA+B regularization, including measured training time.
    • To test specialization directly, please report each expert's accuracy on each task (or the routing weights per task), and the change in accuracy when one expert is removed, with and without the regularizer.
    • What is "the best result for each configuration" in Table 2 selected over, and on which data? Section 5.1 suggests that the client validation sets also serve as the evaluation sets.

    If the overlap is already high at initialization and the w/o Reg. gap does not hold across seeds, I would read the regularizer as removing an initialization effect rather than producing specialization.

  2. With the median AP preference, does centered query-AA still recover the true groups and KK in the main runs with seeds 37 and 64? Does it also do so in at least one harder setting, for example tasks that share an input distribution, unequal group sizes or a different number of tasks? How sensitive is the result to ρ\rho and TwT_w, and how much accuracy is lost if AP returns a wrong KK? Positive answers would address my main doubt about the first contribution.

  3. (Clarification)

    • What is the total number of communication rounds TT? Table A.1 does not list it and Algorithm A.1 leaves it symbolic. Do the Tw=5T_w = 5 warmup rounds count toward it?
    • Is the FedLEASE routing dense or top-kk, and how are the classification heads handled across the four tasks?
    • For each baseline, please give the LoRA rank, the number of experts, the trainable parameters and the per-round communication.
    • In w/o SVD, how are rank-4 experts obtained by averaging rank-8 warmup factors (Table A.1)?

    These details decide whether the 1.01 pp margin over FedALT and the 1.00 pp gain over w/o SVD are like-for-like comparisons.

1 comment

Dear Reviewer,

We sincerely thank you again for your recognition of our work. As we noted in our previous response, we have addressed your suggestions in our point-by-point response. We hope that these responses have addressed your concerns satisfactorily. We are grateful for your insightful comments, which have helped us improve the clarity and quality of our manuscript. If you have any additional comments, we will do our best to address them.

Best regards,

The Authors

Comments on the paper

2 comments

Dear Reviewers,

Thank you to those who have already contributed to the discussion.

As the discussion period draws to a close, I’d especially appreciate hearing from Reviewers Reviewer ESCZ, Reviewer qPfp, and Reviewer qWTH, who have yet to join the discussion. Please post a brief reply to the authors indicating which concerns their response has addressed and which, if any, remain. If all your concerns have been resolved, a short confirmation is sufficient.

Thank you for your time and input.

Best,
Your AC

Dear Reviewer,

We sincerely thank you again for your recognition of our work. As we noted in our previous response, we have addressed your suggestions in our point-by-point response. We hope that these responses have addressed your concerns satisfactorily. We are grateful for your insightful comments, which have helped us improve the clarity and quality of our manuscript. If you have any additional comments, we will do our best to address them.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.