acceptodds
Under review as a conference paper at ICLR 2027

From Task Mixtures to Specialized Experts

Abstract

In collaborative foundation model fine-tuning, client data is rarely homogeneous. Instead, clients typically possess unknown mixtures of distinct data distributions, or tasks. Conventional federated learning primarily addresses heterogeneity across clients without explicitly resolving latent task mixtures within each client. We study this setting as compound heterogeneity, where data is heterogeneous both across and within clients. We study adaptation over a common frozen representation and show that, when tasks share the same feature geometry, the optimal model for a client's task mixture under squared loss is a convex combination of the optimal models for its underlying tasks. Thus, a single locally trained model represents the client's overall task mixture, while individual inputs may be drawn from different underlying task distributions. This motivates routing inputs to specialized experts, and we show that, when the task optima form a simplex, task-aligned routing achieves lower risk than any single adapted model for genuinely mixed clients. With access to a small set of task-labeled public samples, we derive a convex program to recover task experts and match them to their corresponding tasks. Our routing analysis shows that effective specialization requires input-dependent expert selection aligned with each client's task mixture. Motivated by this analysis, we propose FedSEE. Across our experiments, FedSEE avoids the negative transfer observed in the evaluated baselines and improves performance by 2.9 points overall and 3.7 points for the worst-served quartile.

Primary area:
learning theory
Keywords:
compound heterogeneity, latent task mixtures, mixture of experts, federated learning, expert recovery, input-dependent routing, identifiability, foundation model adaptation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_m9R2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work addresses compound heterogeneity in federated learning and proposes FedSEE for task-specific expert extraction and mixture-aware routing. Theoretical results characterize task-mixture geometry, expert recovery, and routing.

Critical strengths
  1. The work addresses the important problem of heterogeneity both across and within clients, providing a meaningful perspective on federated personalization.
  2. Authors provide a systematic theoretical analysis of task-mixture geometry, expert recovery, and identifiability, offering a principled foundation for the proposed method (Section 3). These theoretical results are very interesting.
Critical weaknesses
  1. The main theoretical results (Theorems 1, 2, and 5) rely on strong assumptions, including squared loss, shared feature second-order moments, and near-pure task clients, which may not hold practically. Although the authors provide extensions to general losses and deep models (Theorems 6,7, and Corollary 8), the theoretical guarantees remain insufficient to fully justify the practical FedSEE implementation, limiting the strength of the theoretical contributions.
  2. In Tables 1 and 2, FedSEE uses additional task-labeled public data that the baselines do not use, making it difficult to isolate the source of its performance gains.
Questions for authors
  1. Can the authors clarify whether the error bounds in Theorem 6 and Corollary 8 remain meaningful in practical deep-model settings?
  2. How much of FedSEE's performance improvement can be attributed to the additional task-labeled public data? Would the improvements persist if the baselines were given access to the same supervision?
2 comments
AuthorsOpenReview ↗

Theoretical Guarantees, Practical Evidence, and Public-Data Fairness

Theoretical Guarantees and Practical Relevance

We thank the reviewer for emphasizing the importance of connecting our theoretical analysis to the practical FedSEE framework. The assumptions in Theorems 1, 2, and 5 allow an exact analysis, but the conclusions are not limited to that setting. Our analysis extends beyond these assumptions, and empirical results in transformers trained with cross-entropy support its main predictions.

Near-pure client assumption. FedSEE does not require near-pure clients. The extended Theorem 2 characterizes expert-recovery error under impure anchors. Consistent with this, FedSEE outperforms the evaluated federated baselines at αDir=0.5\alpha_{\mathrm{Dir}}=0.5 and 1.01.0, where the mean maximum task shares are only 55% and 48%, respectively, with just one of 18 task–seed cases exceeding 80% purity in each setting (Appendix B.3.2).

Evidence in deep models. Theorem 6 and Corollary 8 extend the analysis beyond the quadratic setting, providing explicit bounds under curvature and local approximation conditions. While these bounds may be conservative for deep networks (Remarks 9 and 12), their value lies in characterizing when the theoretical geometry is approximately preserved. We further assess practical relevance by testing three predictions directly in transformers trained with cross-entropy (Appendix B.2):

1. Geometry (Theorem 1). In controlled deep-LoRA experiments (Appendix B.2, Table 7), learned client updates are reconstructed more accurately by task experts weighted by the client's true mixture (normalized residual 0.493 ± 0.023) than by a uniform mixture (0.579 ± 0.015) or the same proportions assigned to the wrong tasks (0.650 ± 0.012). The task-simplex structure predicted by Theorem 1 thus persists in deep models.

2. Mean expert usage (Proposition 1). The analysis predicts that uniform mean expert usage conflicts with specialization for nonuniform client mixtures. In the full RoBERTa-large pipeline, mixture-aware calibration achieves 82.65% accuracy, whereas a uniform target achieves 80.85%, below the 81.35% obtained without any calibration (Table 15).

3. Input-dependent routing (Theorem 5). Using the same frozen task experts, we compare input-dependent routing with module-specific constant routing trained on the same data, objective, and schedule. Input-dependent routing lowers held-out cross-entropy by 0.076 for edge clients and 0.089 for interior clients, and is better in all 18 mixed-client evaluations (Appendix B.2).

Public Data Fairness

We thank the reviewer for highlighting that the additional task-labeled public data used by FedSEE may affect the fairness of comparisons in Tables 1 and 2. FedSEE uses these data only to optimize server-side extraction coefficients, not to train model parameters (Section 5.1). This task-indexed supervision is essential to expert identification, as motivated by Theorems 3–4 (Section 3.2). To directly address the concern, we added a public-supervision ablation and an equal-access comparison in which baselines receive the same public examples (Appendix B.6, Tables 15–16). FedSEE retains overall accuracy advantages of 1.47 and 2.07 percentage points over the strongest warmed-up baselines on RoBERTa-large and DistilRoBERTa, respectively.

Equal-access comparison. Each baseline receives 250 public examples per task through a one-time pooled LoRA warm-up (1,500 examples, five epochs, AdamW, learning rate 10−410^{-4}), while FedSEE uses these examples solely for extraction. FedSEE outperforms every warmed-up baseline across all three seeds on both backbones, with worst-25% gains of 3.46 and 2.27 points, respectively.

RoBERTa-largeOverallWorst-25%DistilRoBERTaOverallWorst-25%
FedSoft80.87 ± 0.4168.90FedSoft73.47 ± 1.3261.29
FedSA-LoRA79.83 ± 1.3868.00FedSA-LoRA73.38 ± 1.0661.74
FedLEASE79.75 ± 1.9468.73FedLEASE72.25 ± 1.3161.93
IFCA+LoRA79.72 ± 0.3968.62IFCA+LoRA72.78 ± 0.9360.86
FedDPA-T79.00 ± 1.0968.08FedDPA-T72.56 ± 0.5759.44
FedSEE (L1)82.35 ± 0.9472.36FedSEE (L1)75.54 ± 1.0764.20

Accuracy (%). Overall: mean ± SD over three seeds; Worst-25%: mean across seeds. Only baselines receive public-data warm-up.

We acknowledge that these controls do not exhaust every possible task-aware baseline extension, but they directly address the concern about unequal access to public supervision.

Public-supervision ablation. Reducing public supervision tenfold to 100 examples per task decreases FedSEE's accuracy only from 82.65% to 81.27%, remaining above the strongest baseline in Table 1.

Thank you for the detailed response and additional experiments. The new empirical results provide useful evidence for the theoretical predictions in deep-model settings. However, the practical relevance of the error bounds in Theorem 6 and Corollary 8 is not fully established. The equal-access comparison also largely addresses my concern about the fairness of public-data usage.

Overall, the response has alleviated my main concerns. I will maintain my current rating for now but remain open to updating it based on further discussions with the other reviewers and the AC.

Reviewer_K1aZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper investigates federated learning scenarios where each client's local data consists of a mixture of multiple tasks. Under a squared-loss model, it shows that client optima can be expressed as convex combinations of task-specific optima and establish theoretical results on expert recovery and identifiability. Based on these insights, the paper proposes FedSEE, a mixture-of-experts framework that employs LoRA experts with a shared down-projection, task-specific up-projections, and a mixture-aware router. Experiments on six natural language understanding tasks demonstrate improvements in average and worst-client accuracy.

Critical strengths
  1. The problem formulation is meaningful and clearly motivated. The paper addresses a limitation of conventional federated learning, where each client may contain data from multiple tasks rather than a single homogeneous distribution. The motivation for recovering task-specific experts and performing input-level routing is clearly articulated.
  2. The paper provides a theoretical analysis of the relationship between client-specific and task-specific optima, offering useful insights into expert extraction.
Critical weaknesses
  1. A major concern is the novelty of the proposed FedSEE framework. Several of its key components, including MoE architectures, LoRA-based adaptation, and the decoupling of shared and task-specific parameters, have been extensively explored in federated learning and multi-task learning. Although the paper provides substantial theoretical analysis, it does not sufficiently clarify how the proposed method differs from existing approaches or what methodological advances it introduces beyond their combination. Furthermore, the underlying federated optimization still relies on FedAvg.
  2. The experimental evaluation is restricted to GLUE/SuperGLUE tasks using a single RoBERTa backbone. While the reported results are encouraging, this relatively narrow evaluation makes it difficult to assess the generalizability of the proposed framework. Experiments on additional domains, such as computer vision, and with more recent model architectures would provide stronger evidence of its effectiveness and broader applicability.
Questions for authors

Do compared baselines also use the same labeled public data used by FedSEE? If not, how does this difference affect the fairness of the comparisons and the reported performance gains?

5 comments
AuthorsOpenReview ↗

Methodological Novelty and Expert Extraction

We thank the reviewer for raising the concern about methodological novelty beyond existing MoE, LoRA, and FedAvg components. FedSEE addresses the inverse problem of recovering task-indexed experts from client adaptations that already mix information from multiple tasks.

We first characterize the geometry of these mixed adaptations (Theorem 1, Section 3.1) and establish two fundamental identifiability limits (Theorems 3–4, Section 3.2), showing that task identities cannot be recovered without task-indexed information and that interior upload pools may not uniquely determine the underlying expert set. These results motivate our task-supervised extraction framework, for which we establish recovery guarantees under explicit conditions (Theorem 2, Section 3.2).

FedSEE then instantiates this framework through supervised expert extraction, contribution-guided allocation, and mixture-aware routing (Section 4), rather than relying on FedAvg to construct task experts. These theoretical and methodological contributions are detailed in Sections 3–4 of the revised manuscript, with additional experimental validation in Section 5 and Appendix B.9.

Methodological distinction

Unlike clustering-based methods (IFCA, FedLEASE) or methods that jointly learn latent mixture components (FedEM, FedSoft), FedSEE constructs task-specific experts by optimizing task-dependent convex combinations of client uploads against public task losses (Eq. 14). The same mixed-client upload can contribute differently to multiple experts. The resulting contribution weights also guide expert allocation and mixture-aware routing. Thus, while shared parameters and routers are averaged, task experts are constructed through supervised, task-dependent optimization rather than FedAvg.

Theoretical motivation

Beyond the core recovery and identifiability results, Theorem 5 (Section 3.3) characterizes the roles of client-specific mean expert usage and input-dependent routing, while Proposition 1 establishes why uniform mean usage can prevent task-aligned specialization. Our analysis also extends to general losses, deep models, and balanced extraction under explicit assumptions (Theorem 7, Corollary 8, Proposition 12). Controlled deep-LoRA experiments further support approximate mixture geometry, showing that true task-mixture weights reconstruct client adaptations more accurately than uniform or shuffled weights (Appendix B.2, Table 7).

Direct evidence for extraction

To evaluate the benefit of optimized expert construction, we compare FedSEE against best-upload selection on six tasks with 36 clients and 50% participation. FedSEE improves overall accuracy by 5.87 ± 3.29 percentage points (paired across three seeds), with a 7.68-point gain for the worst-served 25% of clients. The overall improvement is positive in every seed. Full results are provided in Appendix B.9 of the revised manuscript.

Accuracy (%, mean ± SD)

MethodOverallWorst-50%Worst-25%
FedSEE69.49 ± 0.5964.56 ± 0.9061.41 ± 1.39
Selection63.62 ± 2.7157.35 ± 4.4653.73 ± 4.21
Local-only65.35 ± 0.6760.15 ± 0.0856.50 ± 0.41

The best-upload selection experiment isolates a specific expert-construction choice by comparing selection of the lowest-loss client upload with optimized convex combinations under the same task-labeled public supervision. This controlled comparison complements our evaluations against existing federated mixture-learning methods in their original formulations (Section 5). Together, these experiments assess both the effectiveness of FedSEE as a complete framework and the benefit of its optimized expert construction.

Overall empirical advantages.

On RoBERTa-large, FedSEE outperforms the strongest baseline on each metric by 1.57 points overall and 3.65 points for the worst-served quarter of clients (Table 1). Replacing mixture-aware mean-usage calibration with a uniform target, while preserving input-dependent routing, further reduces accuracy by 1.80 points (Table 15).

Taken together, these theoretical and empirical findings show that FedSEE's methodological contributions translate into improved personalized accuracy under compound heterogeneity.

AuthorsOpenReview ↗

Evaluation Beyond Language and Public-Data Fairness

Evaluation Beyond Language Tasks

We thank the reviewer for highlighting the importance of evaluation beyond language. Our Office-Home experiment shows that FedSEE achieves the highest mean accuracy across all reported metrics.

Compared with FedLEASE, the strongest overall baseline, FedSEE gains 1.03 points overall but 3.43 points on worst-25% accuracy. Compared with FedEM, the strongest worst-25% baseline, FedSEE gains 0.84 points on worst-25% accuracy while improving overall accuracy by 2.83 points. Thus, FedSEE achieves strong average and worst-client performance simultaneously, without the relative weaknesses observed in these baselines. These results are reported in Section 5.4 (Table 4), with additional experimental details in Appendix B.10.

Office-Home setup. We treat the four Office-Home domains (Art, Clipart, Product, and Real World) as separate tasks, with heterogeneous mixtures across 12 clients. We use a frozen DINOv3 ViT-S/16 backbone with rank-4 LoRA and 300 labeled public examples per task. Following Reviewer uHiV's suggestion, we also include FedEM and FedSoft among the baselines.

Broader evaluation. With this addition, FedSEE is evaluated across two modalities and three backbones, including RoBERTa-large, DistilRoBERTa, and DINOv3 ViT-S/16. Full results are provided in Section 5.4 and Appendix B.10. We also plan to evaluate additional architectures in the final version.

Office-Home results — accuracy (%, mean ± SD)

MethodTask Avg.OverallWorst-25%
FedSA-LoRA43.29 ± 1.2446.87 ± 0.4738.88 ± 1.10
FedIT59.63 ± 1.7260.24 ± 0.6455.29 ± 0.79
FedDPA-T60.33 ± 1.0760.89 ± 0.5554.82 ± 0.77
FedSoft47.07 ± 1.9850.19 ± 0.6441.20 ± 2.58
FedEM65.11 ± 1.8364.92 ± 0.9758.57 ± 1.64
FedLEASE63.37 ± 2.0866.72 ± 1.1055.98 ± 2.66
FedSEE66.28 ± 1.1067.75 ± 1.5159.41 ± 2.12

Accuracy (%, mean ± SD over three seeds). Task Avg. is the unweighted average of the four task accuracies.

Public Data Fairness

We thank the reviewer for raising the fairness of comparisons involving task-labeled public data. In the main comparisons, baselines do not use these data, while FedSEE uses them only to optimize server-side extraction coefficients rather than model parameters (Section 5.1).

Such task-indexed supervision is central to identifying experts from mixed client uploads (Theorems 3–4, Section 3.2). To directly address this concern, we have strengthened the revised manuscript with a public-data fairness comparison and a public-supervision ablation (Appendix B.6, Tables 15–16). In the equal-access comparison, FedSEE retains overall accuracy advantages of 1.47 and 2.07 percentage points on RoBERTa-large and DistilRoBERTa, respectively.

Equal-public-data comparison.

For the equal-access comparison, each baseline receives 250 public examples per task through a one-time pooled LoRA warm-up (1,500 examples, five epochs, AdamW, learning rate 10−410^{-4}), while FedSEE uses the same examples solely for extraction. FedSEE outperforms every warmed-up baseline across all three seeds, with additional worst-25% accuracy gains of 3.46 and 2.27 points on the two backbones, respectively.

RoBERTa-largeOverallWorst-25%DistilRoBERTaOverallWorst-25%
FedSoft80.87 ± 0.4168.90FedSoft73.47 ± 1.3261.29
FedSA-LoRA79.83 ± 1.3868.00FedSA-LoRA73.38 ± 1.0661.74
FedLEASE79.75 ± 1.9468.73FedLEASE72.25 ± 1.3161.93
IFCA+LoRA79.72 ± 0.3968.62IFCA+LoRA72.78 ± 0.9360.86
FedDPA-T79.00 ± 1.0968.08FedDPA-T72.56 ± 0.5759.44
FedSEE (L1)82.35 ± 0.9472.36FedSEE (L1)75.54 ± 1.0764.20

Accuracy (%). Overall: mean ± SD over three seeds; Worst-25%: mean across seeds. Only baselines receive public-data warm-up.

Public-supervision ablation

Specifically, reducing public supervision tenfold to 100 examples per task lowers FedSEE's accuracy only from 82.65% to 81.27%, remaining above the strongest baseline in Table 1.

Best-upload selection control.

Furthermore, best-upload Selection also uses task-labeled public data but trails FedSEE by 5.87 ± 3.29 points in paired overall accuracy (Appendix B.9, Table 20). This provides complementary evidence that public-data access alone does not explain FedSEE's performance advantage. We do not require identical public-data processing across fundamentally different algorithms, nor claim that these controls exhaust every possible task-aware baseline extension.

Thank the authors for the detailed response. The additional experiments on Office-Home address my previous concern regarding the evaluation of the method's applicability.

I still have some questions on the public reference data usage.

  1. How does the method perform when the number of reference examples is further reduced, e.g., to 20 or 10 examples per task?
  2. Regarding the response to Reviewer uHiV, could the authors explain why the overall accuracy improves when the reference data is replaced entirely with Yelp data (100% Yelp)?
Reviewer_uHiVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies federated foundation-model fine-tuning under compound heterogeneity, where each client’s data is an unknown mixture of tasks. The authors show that a single local model only learns a convex combination of the per-task optima, while task-aligned routing to experts achieves lower risk for mixed clients. They then use a small set of task-labeled public data to recover task experts via a convex program, and propose FedSEE. FedSEE avoids negative transfer and improves performance across datasets.

Critical strengths
  • Theoretical support. The paper provides a clear theoretical foundation by characterizing mixed-client optima, establishing conditional expert-recovery guarantees, and analyzing identifiability and routing requirements.

  • Robust positive transfer. FedSEE improves over local-only training across all five evaluated heterogeneity regimes, achieving the best federated accuracy in four and remaining competitive at the near-uniform endpoint.

Critical weaknesses
  • Restrictive anchor assumption and missing selection baseline. The expert-recovery guarantee assumes a near-pure client for each task, limiting its applicability to practical settings where all clients jointly optimize diverse tasks without near-pure participants. Moreover, under this assumption, a natural alternative is to evaluate uploaded client models on each task’s public reference set and select the model with the lowest task-specific loss. Comparing against this simple baseline would clarify whether convex-combination extraction provides additional benefits beyond identifying already specialized client models.

  • Limited scalability evaluation. The NLU experiments use only 10–12 fully participating clients, leaving scalability to larger federations and partial participation unclear. The synthetic extension to 60 clients examines expert identifiability rather than practical scalability.

  • Untested robustness to public-data mismatch. Public reference data comes from the same task datasets as client data. Since expert extraction relies on public task losses, distribution mismatch could bias both recovered experts and contribution-based personalization. Evaluating domain shifts between public and client data would clarify practical robustness.

  • Restrictive contribution balancing. Equal total contribution across clients can impair expert recovery under task imbalance. With nine math-only clients and one coding, only client, the balancing constraint limits the coding client’s contribution to 20%, forcing at least 80% of the coding expert’s extraction weight onto math-only uploads, even if an ideal coding expert is available.

  • Missing mixture-based federated learning baselines. Although FedLEASE provides a relevant recent comparison, the experiments omit FedEM and FedSoft, which explicitly model each client’s data as a mixture of latent distributions and are discussed in the related work.

Questions for authors

Please refer to the weakness section

4 comments
AuthorsOpenReview ↗

Restrictive anchor assumption and missing selection baseline

We thank the reviewer for these suggestions. In the revised manuscript, we have extended Theorem 2 to characterize expert recovery under impure anchors (Section 3.2), report empirical anchor-purity statistics (Appendix B.3.2), and introduce the suggested best-upload selection baseline (Appendix B.9, Table 20). The additional experiments demonstrate FedSEE's advantage over selection across different client populations and heterogeneity levels. All additions are highlighted in blue in the main manuscript.

Anchor assumption. FedSEE does not require a near-pure client for every task. It targets compound heterogeneity, where clients contain task mixtures with varying proportions. This extension establishes upper and lower bounds on recovery error in terms of each task's anchor impurity, showing that near-pure clients strengthen the guarantee but are not required for applying FedSEE. As client mixtures approach the same uniform distribution, their uploads become increasingly similar, fundamentally limiting exact expert recovery from these uploads. However, exact recovery is not necessary: even partially recovered, task-aligned experts can preserve useful task-specific information and improve collaborative learning.

Our experiments support this practical benefit. In the heterogeneity sweep, the client most concentrated on a task devotes on average only 55% of its data to that task at αDir=0.5\alpha_{\mathrm{Dir}}=0.5 and 48% at 1.01.0, and exceeds 80% in just 1 of 18 task–seed cases in each setting. FedSEE nevertheless achieves the highest accuracy among the federated methods in both. Even in our headline setting (αDir=0.1\alpha_{\mathrm{Dir}}=0.1), in every seed two of the six tasks have no client above 80%. The corresponding task-wise anchor-purity statistics are reported in Appendix B.3.2 of the revised manuscript.

Best-upload selection. We agree that selecting the best uploaded model per task is an important baseline. However, as characterized by Theorem 1, individual uploads may reflect client task mixtures rather than task-specific optima. Thus, selecting a single upload may not yield a well-specialized expert, whereas FedSEE can leverage complementary uploads through convex extraction. Moreover, selection identifies an upload for each task but does not directly provide client–expert relevance information. In contrast, FedSEE's extraction weights also guide client-specific expert selection for local training and router calibration.

Here, Selection chooses the client upload with the lowest task-specific public loss, as suggested by the reviewer, while Local-only trains each client independently without federated collaboration. Following the reviewer's suggestion, we evaluate best-upload selection on all six tasks with 36 clients, αDir=0.5\alpha_{\mathrm{Dir}}=0.5, and 50% participation, using DistilRoBERTa and three seeds. The results (accuracy %, mean ± SD)** are summarized below.

MethodOverallWorst-50%Worst-25%
FedSEE69.49 ± 0.5964.56 ± 0.9061.41 ± 1.39
Selection63.62 ± 2.7157.35 ± 4.4653.73 ± 4.21
Local-only65.35 ± 0.6760.15 ± 0.0856.50 ± 0.41

FedSEE outperforms Selection by 5.87 ± 3.29 percentage points in paired overall accuracy, with improvements in all three seeds. The gains extend to worst-50% and worst-25% accuracy (7.21 and 7.68 points, respectively). Selection also underperforms Local-only, highlighting the limitations of relying on a single client upload rather than extracting task experts from complementary uploads.

We further validate this advantage in a smaller setting with QNLI and QQP, 10 clients, and two heterogeneity levels. FedSEE consistently outperforms Selection across all seeds, with overall gains of 2.13 and 2.74 points at αDir=0.1\alpha_{\mathrm{Dir}}=0.1 and 0.50.5, respectively. Accuracy (%), mean ± SD:

Methodα=0.1 Overallα=0.1 Worst-25%α=0.5 Overallα=0.5 Worst-25%
FedSEE-L175.15 ± 0.6671.15 ± 2.0974.46 ± 1.5969.74 ± 2.39
Selection73.02 ± 0.9467.18 ± 1.7771.72 ± 3.1867.04 ± 4.08
Local-only73.07 ± 0.8668.80 ± 1.4367.68 ± 1.9059.97 ± 1.54
AuthorsOpenReview ↗

Scalability and robustness to public data mismatch

Scalability evaluation

We thank the reviewer for emphasizing scalability and partial participation. In the revised manuscript, we include a 36-client experiment with 50% participation after initialization, where FedSEE achieves the highest overall and worst-25% accuracy, outperforming FedLEASE by 2.08 percentage points (main-paper Table 3; Appendix B.9, Table 20). We will further extend the evaluation to larger client populations and varying participation rates in the final manuscript.

Specifically, we triple the original client population to 36, with all clients participating in initialization to construct the initial expert pool, followed by 50% (18 clients) participating per round. We use heterogeneous task mixtures (αDir=0.5\alpha_{\mathrm{Dir}}=0.5) and compare against FedLEASE, FedIT, and FedSA-LoRA under matched partitions and participation schedules across three seeds. Full results are provided in Appendix B.9.

Personalized accuracy (%, mean ± SD):

MethodOverallWorst-25%
FedSA-LoRA62.32 ± 1.1854.27 ± 0.65
FedIT63.58 ± 1.2955.46 ± 0.35
FedLEASE67.41 ± 1.1159.93 ± 1.74
FedSEE69.49 ± 0.5961.41 ± 1.39

Robustness to public-data mismatch

We thank the reviewer for raising the concern that differences between public reference and client distributions may affect expert extraction and contribution-based personalization. In the revised manuscript, we establish stability guarantees under bounded public-loss shifts (Proposition 5), evaluate cross-domain shifts by replacing SST-2 public data with Yelp Polarity samples (Appendix B.11, Table 22), and assess incomplete public task coverage (Appendix B.6, Table 15).

Theoretically, we establish Proposition 5, showing that bounded perturbations in public task losses yield bounded changes in extracted experts, while task identification is preserved when the discrepancy is sufficiently small relative to task separation.

Empirically, we replace 50% or 100% of the SST-2 public reference data with labeled Yelp Polarity examples, introducing a domain shift within binary sentiment classification. Client data, evaluation sets, and the other five public task sets remain unchanged. Accuracy remains comparable even under complete replacement, while mixture MAE increases only slightly, suggesting robustness to the evaluated shift. All conditions use a matched protocol, which differs from the main-paper setup in batch size, projection-bias handling, and checkpoint evaluation.

Public SST-2 dataOriginal50% Yelp100% Yelp
Overall accuracy (%)72.66 ± 0.8372.21 ± 1.8772.74 ± 1.90
Mixture MAE ↓0.113 ± 0.0030.115 ± 0.0140.117 ± 0.007

Under complete replacement, accuracy remains comparable (72.74% vs. 72.66%), with only a 0.004 increase in mixture-estimation MAE. We additionally evaluate incomplete public task coverage using only three public task anchors for six client tasks, achieving 82.01% accuracy compared with 82.65% under full coverage (Appendix B.6, Table 15).

Contribution balancing

We thank the reviewer for highlighting that equal client-contribution constraints may restrict expert recovery under uneven task representation. However, balancing prevents a few clients from repeatedly dominating the extraction of multiple experts, encouraging broader participation and cross-client knowledge sharing. We characterize this trade-off theoretically (Proposition 12, Appendix D.9) and experimentally with exact, soft, and unconstrained balancing, including the reviewer's 9:1 scenario (Appendix B.8, Tables 17–19). In this scenario, the rare-task client selects its corresponding expert in every round, while personalized accuracy remains comparable across variants. We also introduce FedSEE-Soft as a tunable alternative (Section 4). All additions are highlighted in blue.

Importantly, the contribution balancing constraint does not necessarily eliminate task specialization. In the reviewer's 9:1 example, the coding client can contribute at most 0.2 to the coding expert. Under a symmetric allocation of the remaining weight, each math client contributes 0.8/9, so the coding client's weight is 2.25 times that of each math client. Consequently, the extracted expert remains more task-aligned than uniform averaging in this illustrative setting.

Moreover, extraction contributions also guide client-to-expert assignment. In this illustrative allocation, the coding client's normalized relevance to the coding expert is 1, and each math client has greater relevance to the math expert. Thus, correct top-1 assignments are possible despite imperfect extraction. Subsequent task-specific local updates can further reinforce specialization, although server-side mixing remains.

We directly evaluate this concern using nine QNLI-only clients and one QQP-only client, reproducing the reviewer's M/N=0.2M/N=0.2 contribution limit.

BalancingQQP weightOverallQQP clientWorst-25%
Exact0.169 ± 0.00582.88 ± 0.5378.33 ± 2.7578.72 ± 0.95
Soft (λbal=1\lambda_{\mathrm{bal}}=1)0.196 ± 0.01282.70 ± 0.5278.00 ± 2.6078.72 ± 0.67
Soft (λbal=0.1\lambda_{\mathrm{bal}}=0.1)0.347 ± 0.18582.68 ± 0.5278.33 ± 1.4479.22 ± 1.21
None0.785 ± 0.37382.72 ± 0.8779.33 ± 1.2679.78 ± 1.55

Mean ± SD over three seeds. Accuracy in %, QQP weight is the QQP client's share of the extracted QQP expert.

QQP client's extraction weight is 0.169, and its normalized contribution profile still assigns approximately 84.3% relevance to the QQP expert. Consistently, the QQP client selects the QQP expert for local training in all 10 rounds across all three seeds under every balancing variant.

Personalized accuracy remains competitive despite the difference in extraction weights. Removing balancing modestly improves rare-client and worst-quartile accuracy through more task-aligned contributions, but may concentrate extraction on a small subset of clients. Soft balancing provides a tunable alternative: reducing λbal\lambda_{\mathrm{bal}} relaxes the contribution constraint while retaining a penalty against excessive concentration. Both the 9:1 and six-task imbalance experiments appear in Appendix B.8.

Mixture-based federated learning baselines

We thank the reviewer for suggesting FedEM and FedSoft as mixture-based baselines. In the revised manuscript, we evaluate both methods in the main RoBERTa-large experiment (Section 5.2, Table 1), across five heterogeneity regimes (Section 5.3, Table 2), and on the Office-Home vision benchmark (newly added to Section 5.4, Table 4; Appendix B.10). Across the five regimes, FedSEE achieves higher mean personalized accuracy than both methods in every regime, outperforming FedSoft and FedEM by 3.10 and 3.30 percentage points on average, respectively. FedSoft's deficit grows toward uniform mixtures, whereas FedEM's is largest for nearly pure clients. These findings support task-supervised expert extraction and mixture-aware routing.

As shown by Theorems 3–4, latent components are not generally identifiable by task without task-indexed information. FedSEE addresses this through task-supervised expert extraction and mixture-aware routing.

Methodα=0α=0.1α=0.5α=1.0α=50Avg.
FedSoft-0.39 ± 0.50-1.77 ± 0.66-2.91 ± 0.83-5.00 ± 0.31-5.45 ± 0.37-3.10
FedEM-8.67 ± 0.77-3.89 ± 0.38-2.47 ± 1.56-1.35 ± 1.08-0.13 ± 1.25-3.30

Paired differences in overall personalized accuracy (baseline minus FedSEE, in percentage points; negative favors FedSEE), averaged over three seeds.

AuthorsOpenReview ↗

Looking Forward to Your Feedback

Dear Reviewer uHiV,

We hope that our revised manuscript and the detailed responses above address your concerns. We would greatly appreciate your thoughts and would be happy to answer any further questions or clarify any remaining concerns.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.