- Rating
- 3(+2)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper presents an adaptive personalized FL method to address extreme non-IID environments.
Critical strengths
-
This paper has a good presentation, and the method is well defined.
-
The author has provide a decent appendix list.
Critical weaknesses
Weaknesses:
-
In the Introduction (Lines 43–44), the authors state that “Tab. 1 shows that in heterogeneous regimes both FL and even centralized training perform worse than local training.” I do not find this claim sufficiently reliable. The results in Table 1 are obtained under an extremely strong label distribution shift, which may partially explain why FedAvg performs substantially worse than even the centralized setup. Under such an extreme label-support mismatch, strong local performance and severe degradation of a single global model are largely expected. Therefore, I do not think Table 1 provides convincing evidence for the broader claim that FL or centralized training generally underperform local training in heterogeneous regimes. The severity of the non-IID setting, particularly the label distribution shift in this case, appears to be a key factor driving these performance gaps.
-
Personalized FL is commonly used to address client heterogeneity. However, another line of work, clustered FL, also targets the same problem. I am not suggesting that more baselines are necessarily required, but I think it would be important to compare the proposed method with recent clustered FL approaches, ideally from 2025–2026. Ultimately, from a practitioner's perspective, the question is which method should be chosen to address this type of heterogeneity. The empirical evaluation should therefore provide stronger evidence that the proposed pFL method performs better than clustered FL approaches for the problems considered in this work.
-
Related to the first comment, the partition used in this work (two classes per client) already represents an extremely strong non-IID setting. As a result, a substantial portion of the headline empirical evidence, particularly Table 1 and the pathological label-skew experiments, may have limited external validity.
-
Considering the comments above, I find that the current empirical evaluation is not sufficiently strong to support the paper's broad conclusions. In my view, the paper would require a substantially stronger and more systematic experimental setup to adequately support these claims, along with a stronger baseline selection.
Questions for authors
Please see Critical Weaknesses.
4 comments
Dear Reviewer sS9n,
As the rebuttal deadline approaches, we would like to briefly follow up. We have clarified the scope of our claims and highlighted that the original submission already includes practical Dirichlet label skew, pure feature shift, and hybrid settings, demonstrating gains beyond extreme non-IID scenarios.
We also clarified our distinction from clustered FL and highlighted the existing PerFed-CKT ( clustering approach ) comparison.
We hope these explanations address your concerns and would appreciate your reconsideration of the evaluation.
I thank the author for the rebuttal materials, and they have mostly addressed my concerns. I will update my score.
We sincerely appreciate your constructive engagement throughout the discussion and thank you for reconsidering and increasing your evaluation score. We are pleased that our clarifications addressed your main concerns.
We thank the reviewer for raising the question of external validity under severe heterogeneity.
Severity of the non-IID setting
We agree that the two-class-per-client partition represents extreme label heterogeneity. We also agree that the statement motivated by Table 1 is too broad. The table demonstrates the existence of a regime where centralized and conventional federated training can underperform local training. It does not imply that this behavior is typical of heterogeneous FL. We will clarify the wording.
However, the review appears to interpret this pathological partition as the main basis for our conclusions. This is not the case.
Section 4 evaluates three different heterogeneity scenarios and explicitly states that “each scenario stresses a different aspect of adaptive collaboration” (lines 318-323):
The pure feature-shift experiments are particularly relevant to the reviewer's concern, since they do not rely on the extreme label-support mismatch criticized in the review. Table 3 shows that FEDMOSAIC improves over both local and centralized training in these settings. Averaging the reported per-domain accuracies gives:
Thus, the improvement is not limited to the pathological label-skew setting. In fact, the gains are considerably more meaningful here than in the nearly saturated pathological Fashion-MNIST and CIFAR-10 experiments.
The practical Dirichlet experiments provide further evidence. On CIFAR-10, FEDMOSAIC obtains 86.15% compared with 83.91% for local training. On Tiny-ImageNet, it obtains 41.90% compared with 36.75%. Centralized training remains slightly better in these two cases.
This qualification is already made in the Discussion. We explicitly state that the gain over local training under pathological label skew is very small, question whether such settings are particularly informative for collaborative gain, and note that under the milder practical non-IID setting “centralized training performs best, as expected” (lines 460-476).
Our claim is therefore not that centralized training generally fails under heterogeneity, nor that FEDMOSAIC always outperforms centralized learning. We study the regime described in the paper where a single shared model is insufficient but collaboration can still improve personalized models.
Clustered FL
We agree that clustered FL is an interesting and relevant direction. We see it, however, as largely orthogonal to the problem studied here.
Client clustering determines which groups of clients should collaborate. Our expertise mechanism asks whose prediction should matter for this particular example within a collaborating set. These are different decisions. Two clients can be useful collaborators for one part of the input space and poor collaborators for another, even if they belong to the same cluster.
The two approaches are also naturally compatible. Clustering could restrict which clients participate in a consensus, while expertise weighting determines their relative influence for each example within that cluster. Alternatively, different clusters could construct separate consensuses. This would add another layer of personalization to the collaboration structure rather than replace the mechanism studied here.
Our evaluation already includes PerFed-CKT, which uses clustered knowledge transfer. In the hybrid experiments (Table 4), FEDMOSAIC-W achieves 87.44% on DomainNet and 89.06% on Office-10, compared with 79.24% and 82.49% for PerFed-CKT.
We agree that a broader study combining recent clustering approaches with example-level expertise would be interesting future work. It introduces an additional question, namely which clients should collaborate at all, whereas the present paper focuses on how collaboration should behave once a set of clients collaborates: when to trust the collaborative signal and whose predictions to trust at the example level.
In summary, pathological label skew alone would indeed provide limited evidence for general applicability. The submitted evaluation, however, also includes practical label skew, pure feature shift, and hybrid heterogeneity. The pure feature-shift results in particular demonstrate substantial collaborative gains without the extreme label-support mismatch on which the review focuses.