- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes FedCRF, a server-side aggregation method that infers client reliability with a conditional random field. It combines individual evidence and inter-client similarities to adjust FedAvg weights, with extensions to clustered and personalized FL. Experiments evaluate its performance under data heterogeneity, unreliable clients, and Byzantine attacks.
Critical strengths
- The paper proposes a server-side client aggregation framework that can be easily integrated into the current FL methods.
- The paper combines individual client evidence with inter-client relationships, providing an intuitive approach to perform client aggregation.
- The experiments cover diverse FL settings across three datasets.
Critical weaknesses
- The novelty is limited. Combining individual client evidence with inter-client relationships is well studied in FL, and the technical approach largely combines established components. While the CRF formulation offers a different perspective, its claimed benefits are not sufficiently supported.
- The added value of CRF inference in clustered and personalized FL remains unclear, as the comparisons evaluate complete pipelines without isolating its contribution.
- It remains unclear whether FedCRF distinguishes legitimate heterogeneity from poor data quality, since both can produce updates that deviate from the majority.
- The authors should compare FedCRF with client quality estimation and client selection methods such as [1-2].
[1] FedTrans: Client-Transparent Utility Estimation for Robust Federated Learning. ICLR, 2024.
[2] Robust Federated Learning With Noisy and Heterogeneous Clients. CVPR, 2022.
Questions for authors
Does FedCRF remain effective under naturally occurring feature skew, such as those in DomainNet or Office-Caltech, beyond the synthetic rotations and colour transformations?
2 comments
The same holds when heterogeneity is legitimate by construction. In personalized FL on CIFAR-10 with strong feature skew ( high), we additionally train 30% of the clients with 50% label noise. The affinities decide which clients share models, and the reliabilities decide how much each of them contributes. Using both is more accurate than either component alone:
| Personalized FedCRF$_{G+F}$ () | label noise 50% |
|---|---|
| pairwise only (, no reliabilities) | 78.2 |
| unary only () | 78.5 |
| full CRF () | 79.3 |
These results are reported in App. K (Tab. 17).
W4
We thank the reviewer for suggesting these works. We have now evaluated both FedTrans [1] and RHFL [2] under the same experimental setup as Tab. 1, adapting the latter's client-confidence reweighting component (RHFL-CCR) to FedAvg for a comparable setting to Tab. 1.
The additional results further support FedCRF's versatility: across all ten conditions, FedCRF$_G$ achieves the highest average accuracy (74.8% vs. 72.8% for FedAvg, 72.0% for RHFL-CCR, and 71.6% for FedTrans), without setting-specific hyperparameter tuning. While both suggested baselines perform well in individual settings, FedCRF remains consistently competitive across datasets and corruption types. Unlike independent client-quality estimation or reweighting, FedCRF jointly infers client reliabilities within a structured probabilistic framework.
We now include both methods in Tab. 1, report the label-noise comparison in App. N (Tab. 20), and discuss them in the Related Work (App. A).
Additional baselines for Tab. 1. Client test accuracy (%). LN denotes label noise applied to 30% of clients.
| Method | CIFAR-10 Clean | CIFAR-10 Gauss. | CIFAR-10 LN 50% | CIFAR-10 LN 80% | FEMNIST Clean | FEMNIST Gauss. | FEMNIST LN 50% | ISIC Clean | ISIC Gauss. | ISIC LN 50% |
|---|---|---|---|---|---|---|---|---|---|---|
| FedAvg | 84.5 | 77.5 | 82.9 | 74.4 | 74.0 | 72.4 | 72.5 | 68.3 | 53.3 | 68.1 |
| FedTrans | 83.6 | 75.7 | 82.2 | 71.9 | 73.6 | 71.7 | 72.3 | 66.0 | 52.6 | 66.1 |
| RHFL-CCR | 85.4 | 77.4 | 82.6 | 67.4 | 73.5 | 72.3 | 70.4 | 68.6 | 53.4 | 68.7 |
| FedCRF$_G$ | 84.6 | 80.8 | 83.3 | 82.1 | 73.3 | 71.4 | 73.1 | 69.7 | 62.0 | 68.1 |
| FedCRF$_F$ | 84.3 | 82.7 | 81.5 | 65.5 | 71.7 | 71.1 | 72.0 | 69.2 | 54.4 | 66.5 |
| FedCRF$_{G+F}$ | 84.7 | 82.0 | 82.9 | 69.9 | 73.7 | 72.9 | 73.3 | 69.1 | 55.5 | 68.1 |
Q1
We sincerely thank the reviewer for this suggestion. We agree on the importance of evaluating naturally occurring feature skew beyond synthetic transformations. This motivated our original inclusion of FEMNIST, with clients partitioned by writer, and Fed-ISIC2019, with clients derived from different acquisition centers, both preserving natural heterogeneity without introducing artificial domain transformations. To broaden this evaluation, we extended our experiments to the suggested domain-shift benchmarks: Office-Caltech10 (4 domains, 20 clients) and DomainNet (6 domains, 10 classes, 24 clients), partitioning each domain among multiple clients.
The additional results support FedCRF's effectiveness under naturally occurring feature skew. In global aggregation, FedCRF remains competitive on clean data and performs better under corruption: 93.9% vs. 91.4% on Office-Caltech10 and 80.3% vs. 79.3% on DomainNet, compared with the strongest baselines.
The clustered FL results are particularly encouraging. On DomainNet, FedCRF$_{G+F}$ recovers all six domains (ARI 1.00 vs. 0.48 for IFCA and 0.46 for FLUX), while achieving the highest accuracy (86.7% vs. 86.3%). On Office-Caltech10, FedCRF$G$ and FedCRF${G+F}$ also achieve the highest ARI (0.76 vs. 0.56 for IFCA), while matching its accuracy (93.7%). We now include these experiments in the revised paper (App. M, Tab. 19), and refer to them from Sec. 4.2.
| Method | Office Clean | Office Corr. | Office Cluster Acc. | Office ARI | DomainNet Clean | DomainNet Corr. | DomainNet Cluster Acc. | DomainNet ARI |
|---|---|---|---|---|---|---|---|---|
| FedAvg | 94.3 | 91.4 | — | — | 81.0 | 79.3 | — | — |
| FedProx | 94.3 | 91.4 | — | — | 81.2 | 79.0 | — | — |
| FedLWS | 94.3 | 91.4 | — | — | 80.8 | 78.9 | — | — |
| IFCA (oracle J) | — | — | 93.7 | 0.56 | — | — | 86.3 | 0.48 |
| FLUX | — | — | 92.6 | -0.03 | — | — | 84.1 | 0.46 |
| FedCRF$_G$ | 94.8 | 93.9 | 93.7 | 0.76 | 79.7 | 80.1 | 85.1 | 0.93 |
| FedCRF$_F$ | 93.7 | 93.9 | 92.4 | -0.06 | 80.7 | 80.3 | 81.2 | 0.16 |
| FedCRF$_{G+F}$ | 94.0 | 93.7 | 93.7 | 0.76 | 80.7 | 80.0 | 86.7 | 1.00 |
We thank the reviewer for the thoughtful review.
W1
We agree that client evidence, similarities and CRFs are established concepts. Our novelty does not lie in these components individually, but in formulating federated client assessment as a joint structured probabilistic inference. Rather than using client scores or similarities independently, FedCRF couples them through unary and pairwise potentials to jointly infer client reliabilities. These reliabilities and affinities support global aggregation, Byzantine robustness, clustered FL and personalized FL, with geometric, functional and mixed evidence as different instantiations.
What we found particularly striking is that the same inference framework remained competitive across such different FL objectives, without task-specific hyperparameter tuning. We deliberately kept hyperparameters fixed to assess its versatility beyond individual benchmarks, even though tuning them separately could potentially improve performance. We find this particularly encouraging because, in real federations, client heterogeneity, data quality and adversarial presence are often unknown, complicating the selection and tuning of specialized methods.
Our results support this versatility: FedCRF variants achieve the highest average accuracies in global aggregation (73.6% vs. 72.1%, Tab. 1), partial participation (65.5% vs. 63.5%, Tab. 2), clustered FL with known group count (73.4% vs. 69.9%, Tab. 4) and personalized FL (74.6% vs. 72.7%, Tab. 5), while matching the strongest Byzantine-robust baseline (83.4% vs. 83.1%, Tab. 3).
Additional experiments (see YCky, W5) illustrate the value of not requiring prior knowledge of the attacker fraction. With 30% ALIE attackers, Zeno achieves 82.4% in its best-performing configuration but drops to 71.4% and 60.6% when assuming 20% and 10% attackers, respectively. Multi-Krum ranges from 18.7% to 48.5%, while FedCRF$_F$ achieves 83.0% without this information. Moreover, our ablation (Tab. 14) directly supports the benefit of structured inference: pairwise coupling reduces attackers' aggregation weights and improves worst-seed accuracy from 63.4% to 69.3% over unary-only inference.
We clarity these aspects in the revision, Sec. 3.2 and App. A. Sec. 5 now discusses the practical motivation and App. L reports the attacker-fraction experiments.
W2
We isolate two CRF components in clustered FedCRF: pairwise affinities and inferred reliabilities. We compare three variants that use the same grouping procedure and the true number of groups:
The setting is clustered CIFAR-10 (P(y) strict) with 30% low-quality clients placed inside the groups. We report the test accuracy (%) of the clean clients over 3 seeds.
The results show that the two components are complementary. Neither one alone is enough: the full CRF is 3.5 and 0.6 points above grouping with plain averaging, and 5.2 and 6.7 points above reliability weighting without the pairwise term. These results are reported in App. K.
W3
We agree that legitimate heterogeneity does not necessarily imply poor data quality, although both can produce updates that deviate from the majority. FedCRF does not explicitly identify their cause; instead, it jointly infers client reliabilities from unary evidence and pairwise affinities. Notably, the evidence can help distinguish harmful contributions from benign heterogeneity: for example, functional evidence (FedCRF$_F$) measures improvement on a proxy set, providing information beyond update similarity alone.
Our experiments directly investigate this distinction (Fig. 2). On heterogeneous CIFAR-10, we additionally corrupt 30% of clients' training data with label or input noise. Despite the underlying heterogeneity, FedCRF generally assigns lower reliability to corrupted clients than legitimate heterogeneous clients. Functional evidence reduces the reliability of input-noisy clients to approximately 0.1, compared with 0.5 for uncorrupted clients. Geometric evidence drives severely label-noisy clients toward zero, while mixed evidence also distinguishes the groups.
This separation is consistent with improved aggregation: on corrupted CIFAR-10, FedCRF$_F$ achieves 82.7% accuracy vs. 78.1% for the strongest baseline (Tab. 1), while remaining competitive under clean heterogeneity. Nevertheless, we cannot exclude that legitimate distributional outliers may still be down-weighted when their contributions resemble low-quality ones under the adopted evidence metrics, particularly in global aggregation. We clarify this limitation in Sec. 5 and the interpretation of Fig. 2.