- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work addresses compound heterogeneity in federated learning and proposes FedSEE for task-specific expert extraction and mixture-aware routing. Theoretical results characterize task-mixture geometry, expert recovery, and routing.
Critical strengths
- The work addresses the important problem of heterogeneity both across and within clients, providing a meaningful perspective on federated personalization.
- Authors provide a systematic theoretical analysis of task-mixture geometry, expert recovery, and identifiability, offering a principled foundation for the proposed method (Section 3). These theoretical results are very interesting.
Critical weaknesses
- The main theoretical results (Theorems 1, 2, and 5) rely on strong assumptions, including squared loss, shared feature second-order moments, and near-pure task clients, which may not hold practically. Although the authors provide extensions to general losses and deep models (Theorems 6,7, and Corollary 8), the theoretical guarantees remain insufficient to fully justify the practical FedSEE implementation, limiting the strength of the theoretical contributions.
- In Tables 1 and 2, FedSEE uses additional task-labeled public data that the baselines do not use, making it difficult to isolate the source of its performance gains.
Questions for authors
- Can the authors clarify whether the error bounds in Theorem 6 and Corollary 8 remain meaningful in practical deep-model settings?
- How much of FedSEE's performance improvement can be attributed to the additional task-labeled public data? Would the improvements persist if the baselines were given access to the same supervision?
2 comments
Thank you for the detailed response and additional experiments. The new empirical results provide useful evidence for the theoretical predictions in deep-model settings. However, the practical relevance of the error bounds in Theorem 6 and Corollary 8 is not fully established. The equal-access comparison also largely addresses my concern about the fairness of public-data usage.
Overall, the response has alleviated my main concerns. I will maintain my current rating for now but remain open to updating it based on further discussions with the other reviewers and the AC.
Theoretical Guarantees, Practical Evidence, and Public-Data Fairness
Theoretical Guarantees and Practical Relevance
We thank the reviewer for emphasizing the importance of connecting our theoretical analysis to the practical FedSEE framework. The assumptions in Theorems 1, 2, and 5 allow an exact analysis, but the conclusions are not limited to that setting. Our analysis extends beyond these assumptions, and empirical results in transformers trained with cross-entropy support its main predictions.
Near-pure client assumption. FedSEE does not require near-pure clients. The extended Theorem 2 characterizes expert-recovery error under impure anchors. Consistent with this, FedSEE outperforms the evaluated federated baselines at αDir=0.5 and 1.0, where the mean maximum task shares are only 55% and 48%, respectively, with just one of 18 task–seed cases exceeding 80% purity in each setting (Appendix B.3.2).
Evidence in deep models. Theorem 6 and Corollary 8 extend the analysis beyond the quadratic setting, providing explicit bounds under curvature and local approximation conditions. While these bounds may be conservative for deep networks (Remarks 9 and 12), their value lies in characterizing when the theoretical geometry is approximately preserved. We further assess practical relevance by testing three predictions directly in transformers trained with cross-entropy (Appendix B.2):
1. Geometry (Theorem 1). In controlled deep-LoRA experiments (Appendix B.2, Table 7), learned client updates are reconstructed more accurately by task experts weighted by the client's true mixture (normalized residual 0.493 ± 0.023) than by a uniform mixture (0.579 ± 0.015) or the same proportions assigned to the wrong tasks (0.650 ± 0.012). The task-simplex structure predicted by Theorem 1 thus persists in deep models.
2. Mean expert usage (Proposition 1). The analysis predicts that uniform mean expert usage conflicts with specialization for nonuniform client mixtures. In the full RoBERTa-large pipeline, mixture-aware calibration achieves 82.65% accuracy, whereas a uniform target achieves 80.85%, below the 81.35% obtained without any calibration (Table 15).
3. Input-dependent routing (Theorem 5). Using the same frozen task experts, we compare input-dependent routing with module-specific constant routing trained on the same data, objective, and schedule. Input-dependent routing lowers held-out cross-entropy by 0.076 for edge clients and 0.089 for interior clients, and is better in all 18 mixed-client evaluations (Appendix B.2).
Public Data Fairness
We thank the reviewer for highlighting that the additional task-labeled public data used by FedSEE may affect the fairness of comparisons in Tables 1 and 2. FedSEE uses these data only to optimize server-side extraction coefficients, not to train model parameters (Section 5.1). This task-indexed supervision is essential to expert identification, as motivated by Theorems 3–4 (Section 3.2). To directly address the concern, we added a public-supervision ablation and an equal-access comparison in which baselines receive the same public examples (Appendix B.6, Tables 15–16). FedSEE retains overall accuracy advantages of 1.47 and 2.07 percentage points over the strongest warmed-up baselines on RoBERTa-large and DistilRoBERTa, respectively.
Equal-access comparison. Each baseline receives 250 public examples per task through a one-time pooled LoRA warm-up (1,500 examples, five epochs, AdamW, learning rate 10−4), while FedSEE uses these examples solely for extraction. FedSEE outperforms every warmed-up baseline across all three seeds on both backbones, with worst-25% gains of 3.46 and 2.27 points, respectively.
Accuracy (%). Overall: mean ± SD over three seeds; Worst-25%: mean across seeds. Only baselines receive public-data warm-up.
We acknowledge that these controls do not exhaust every possible task-aware baseline extension, but they directly address the concern about unequal access to public supervision.
Public-supervision ablation. Reducing public supervision tenfold to 100 examples per task decreases FedSEE's accuracy only from 82.65% to 81.27%, remaining above the strongest baseline in Table 1.