acceptodds
Under review as a conference paper at ICLR 2027

Federated Learning with Energy-Based Structured Probabilistic Inference

Abstract

Federated Learning (FL) enables multiple clients to collaboratively train machine learning models without sharing their local data. However, heterogeneity in client data and the presence of unreliable or adversarial updates make effective server aggregation challenging, motivating methods that adapt client contributions according to their reliability and relationships. In this work, we formulate server aggregation in FL as structured probabilistic inference using conditional random fields. We represent clients as nodes in a graph and define unary and pairwise energy potentials. Unary potentials capture individual client properties, while pairwise potentials encode similarities between clients. This formulation treats individual client evidence and pairwise relationships as coupled evidence in a joint inference problem, allowing client reliabilities to be inferred collectively and the resulting estimates to guide aggregation. Where most FL methods target a single challenge, FedCRF's inferred reliabilities and affinities provide one shared representation for global aggregation, Byzantine robustness, clustered FL, and personalized FL. With a single set of hyperparameters, it is competitive with or outperforms specialized methods on CIFAR-10, FEMNIST, and Fed-ISIC2019 across diverse forms of client heterogeneity and unreliability.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
federated learning, conditional random fields, structured inference, mean-field inference, robust aggregation, clustered federated learning, personalized federated learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qQcQOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FedCRF, a server-side aggregation method that infers client reliability with a conditional random field. It combines individual evidence and inter-client similarities to adjust FedAvg weights, with extensions to clustered and personalized FL. Experiments evaluate its performance under data heterogeneity, unreliable clients, and Byzantine attacks.

Critical strengths
  • The paper proposes a server-side client aggregation framework that can be easily integrated into the current FL methods.
  • The paper combines individual client evidence with inter-client relationships, providing an intuitive approach to perform client aggregation.
  • The experiments cover diverse FL settings across three datasets.
Critical weaknesses
  • The novelty is limited. Combining individual client evidence with inter-client relationships is well studied in FL, and the technical approach largely combines established components. While the CRF formulation offers a different perspective, its claimed benefits are not sufficiently supported.
  • The added value of CRF inference in clustered and personalized FL remains unclear, as the comparisons evaluate complete pipelines without isolating its contribution.
  • It remains unclear whether FedCRF distinguishes legitimate heterogeneity from poor data quality, since both can produce updates that deviate from the majority.
  • The authors should compare FedCRF with client quality estimation and client selection methods such as [1-2].

[1] FedTrans: Client-Transparent Utility Estimation for Robust Federated Learning. ICLR, 2024.

[2] Robust Federated Learning With Noisy and Heterogeneous Clients. CVPR, 2022.

Questions for authors

Does FedCRF remain effective under naturally occurring feature skew, such as those in DomainNet or Office-Caltech, beyond the synthetic rotations and colour transformations?

2 comments

We thank the reviewer for the thoughtful review.

W1

We agree that client evidence, similarities and CRFs are established concepts. Our novelty does not lie in these components individually, but in formulating federated client assessment as a joint structured probabilistic inference. Rather than using client scores or similarities independently, FedCRF couples them through unary and pairwise potentials to jointly infer client reliabilities. These reliabilities and affinities support global aggregation, Byzantine robustness, clustered FL and personalized FL, with geometric, functional and mixed evidence as different instantiations.

What we found particularly striking is that the same inference framework remained competitive across such different FL objectives, without task-specific hyperparameter tuning. We deliberately kept hyperparameters fixed to assess its versatility beyond individual benchmarks, even though tuning them separately could potentially improve performance. We find this particularly encouraging because, in real federations, client heterogeneity, data quality and adversarial presence are often unknown, complicating the selection and tuning of specialized methods.

Our results support this versatility: FedCRF variants achieve the highest average accuracies in global aggregation (73.6% vs. 72.1%, Tab. 1), partial participation (65.5% vs. 63.5%, Tab. 2), clustered FL with known group count (73.4% vs. 69.9%, Tab. 4) and personalized FL (74.6% vs. 72.7%, Tab. 5), while matching the strongest Byzantine-robust baseline (83.4% vs. 83.1%, Tab. 3).

Additional experiments (see YCky, W5) illustrate the value of not requiring prior knowledge of the attacker fraction. With 30% ALIE attackers, Zeno achieves 82.4% in its best-performing configuration but drops to 71.4% and 60.6% when assuming 20% and 10% attackers, respectively. Multi-Krum ranges from 18.7% to 48.5%, while FedCRF$_F$ achieves 83.0% without this information. Moreover, our ablation (Tab. 14) directly supports the benefit of structured inference: pairwise coupling reduces attackers' aggregation weights and improves worst-seed accuracy from 63.4% to 69.3% over unary-only inference.

We clarity these aspects in the revision, Sec. 3.2 and App. A. Sec. 5 now discusses the practical motivation and App. L reports the attacker-fraction experiments.

W2

We isolate two CRF components in clustered FedCRF: pairwise affinities and inferred reliabilities. We compare three variants that use the same grouping procedure and the true number of groups:

  • pairwise only: the CRF affinities form the groups, and each group is averaged with FedAvg weights
  • unary only: the reliabilities weight the clients inside each group, without the pairwise term (λ=0)
  • full CRF: both, as in the paper.

The setting is clustered CIFAR-10 (P(y) strict) with 30% low-quality clients placed inside the groups. We report the test accuracy (%) of the clean clients over 3 seeds.

Clustered FedCRF$_G$label noise 50%corrupted
pairwise only74.879.8
unary only73.173.7
full CRF78.380.4

The results show that the two components are complementary. Neither one alone is enough: the full CRF is 3.5 and 0.6 points above grouping with plain averaging, and 5.2 and 6.7 points above reliability weighting without the pairwise term. These results are reported in App. K.

W3

We agree that legitimate heterogeneity does not necessarily imply poor data quality, although both can produce updates that deviate from the majority. FedCRF does not explicitly identify their cause; instead, it jointly infers client reliabilities from unary evidence and pairwise affinities. Notably, the evidence can help distinguish harmful contributions from benign heterogeneity: for example, functional evidence (FedCRF$_F$) measures improvement on a proxy set, providing information beyond update similarity alone.

Our experiments directly investigate this distinction (Fig. 2). On heterogeneous CIFAR-10, we additionally corrupt 30% of clients' training data with label or input noise. Despite the underlying heterogeneity, FedCRF generally assigns lower reliability to corrupted clients than legitimate heterogeneous clients. Functional evidence reduces the reliability of input-noisy clients to approximately 0.1, compared with 0.5 for uncorrupted clients. Geometric evidence drives severely label-noisy clients toward zero, while mixed evidence also distinguishes the groups.

This separation is consistent with improved aggregation: on corrupted CIFAR-10, FedCRF$_F$ achieves 82.7% accuracy vs. 78.1% for the strongest baseline (Tab. 1), while remaining competitive under clean heterogeneity. Nevertheless, we cannot exclude that legitimate distributional outliers may still be down-weighted when their contributions resemble low-quality ones under the adopted evidence metrics, particularly in global aggregation. We clarify this limitation in Sec. 5 and the interpretation of Fig. 2.

The same holds when heterogeneity is legitimate by construction. In personalized FL on CIFAR-10 with strong feature skew (P(x∣y)P(x|y) high), we additionally train 30% of the clients with 50% label noise. The affinities decide which clients share models, and the reliabilities decide how much each of them contributes. Using both is more accurate than either component alone:

Personalized FedCRF$_{G+F}$ (s=10s=10)label noise 50%
pairwise only (Wij∝BijW_{ij}\propto B_{ij}, no reliabilities)78.2
unary only (λ=0\lambda=0)78.5
full CRF (λ=0.5\lambda=0.5)79.3

These results are reported in App. K (Tab. 17).

W4

We thank the reviewer for suggesting these works. We have now evaluated both FedTrans [1] and RHFL [2] under the same experimental setup as Tab. 1, adapting the latter's client-confidence reweighting component (RHFL-CCR) to FedAvg for a comparable setting to Tab. 1.

The additional results further support FedCRF's versatility: across all ten conditions, FedCRF$_G$ achieves the highest average accuracy (74.8% vs. 72.8% for FedAvg, 72.0% for RHFL-CCR, and 71.6% for FedTrans), without setting-specific hyperparameter tuning. While both suggested baselines perform well in individual settings, FedCRF remains consistently competitive across datasets and corruption types. Unlike independent client-quality estimation or reweighting, FedCRF jointly infers client reliabilities within a structured probabilistic framework.

We now include both methods in Tab. 1, report the label-noise comparison in App. N (Tab. 20), and discuss them in the Related Work (App. A).

Additional baselines for Tab. 1. Client test accuracy (%). LN denotes label noise applied to 30% of clients.

MethodCIFAR-10 CleanCIFAR-10 Gauss.CIFAR-10 LN 50%CIFAR-10 LN 80%FEMNIST CleanFEMNIST Gauss.FEMNIST LN 50%ISIC CleanISIC Gauss.ISIC LN 50%
FedAvg84.577.582.974.474.072.472.568.353.368.1
FedTrans83.675.782.271.973.671.772.366.052.666.1
RHFL-CCR85.477.482.667.473.572.370.468.653.468.7
FedCRF$_G$84.680.883.382.173.371.473.169.762.068.1
FedCRF$_F$84.382.781.565.571.771.172.069.254.466.5
FedCRF$_{G+F}$84.782.082.969.973.772.973.369.155.568.1

Q1

We sincerely thank the reviewer for this suggestion. We agree on the importance of evaluating naturally occurring feature skew beyond synthetic transformations. This motivated our original inclusion of FEMNIST, with clients partitioned by writer, and Fed-ISIC2019, with clients derived from different acquisition centers, both preserving natural heterogeneity without introducing artificial domain transformations. To broaden this evaluation, we extended our experiments to the suggested domain-shift benchmarks: Office-Caltech10 (4 domains, 20 clients) and DomainNet (6 domains, 10 classes, 24 clients), partitioning each domain among multiple clients.

The additional results support FedCRF's effectiveness under naturally occurring feature skew. In global aggregation, FedCRF remains competitive on clean data and performs better under corruption: 93.9% vs. 91.4% on Office-Caltech10 and 80.3% vs. 79.3% on DomainNet, compared with the strongest baselines.

The clustered FL results are particularly encouraging. On DomainNet, FedCRF$_{G+F}$ recovers all six domains (ARI 1.00 vs. 0.48 for IFCA and 0.46 for FLUX), while achieving the highest accuracy (86.7% vs. 86.3%). On Office-Caltech10, FedCRF$G$ and FedCRF${G+F}$ also achieve the highest ARI (0.76 vs. 0.56 for IFCA), while matching its accuracy (93.7%). We now include these experiments in the revised paper (App. M, Tab. 19), and refer to them from Sec. 4.2.

MethodOffice CleanOffice Corr.Office Cluster Acc.Office ARIDomainNet CleanDomainNet Corr.DomainNet Cluster Acc.DomainNet ARI
FedAvg94.391.4——81.079.3——
FedProx94.391.4——81.279.0——
FedLWS94.391.4——80.878.9——
IFCA (oracle J)——93.70.56——86.30.48
FLUX——92.6-0.03——84.10.46
FedCRF$_G$94.893.993.70.7679.780.185.10.93
FedCRF$_F$93.793.992.4-0.0680.780.381.20.16
FedCRF$_{G+F}$94.093.793.70.7680.780.086.71.00
Reviewer_f1ssOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the effective server-side aggregation problem in federated learning, where the server needs to account for both the reliability of individual client contributions and the relationships among clients. To this end, this paper formulates the server’s aggregation decision as a structured probabilistic inference by using a conditional random field (CRF), and proposes FedCRF. In the proposed FedCRF, participating clients are represented as the nodes in a fully connected graph, with each client associated with a latent reliability label that determines how its contribution should be treated during the aggregation. FedCRF defines a client CRF energy consisting of the unary and pairwise potentials, where the unary potentials encode evidence about individual client reliability and the pairwise potentials capture affinities between the clients. A joint inference over the CRF is then used to estimate the client reliabilities, and adjust their aggregation weights. Beyond the global aggregation, the same reliability and affinity representation is further extended to the Byzantine-robust aggregation, clustered FL and personalized FL. Experiments on CIFAR-10, FEMNIST, and Fed-ISIC2019 demonstrate the effectiveness of FedCRF under diverse settings of client heterogeneity and unreliability.

Critical strengths
  1. This paper provides a formulation of federated aggregation as the structured probabilistic inference by using a CRF. It jointly estimates the client reliabilities through both the unary and pairwise potentials, and uses them to adapt the standard FedAvg's aggregation weights.

  2. This paper provides a unified mechanism that can be reused across the global aggregation, Byzantine-robust aggregation, clustered FL and personalized FL. This is an appealing aspect of the framework, as these settings are typically handled by separate specialized methods, while FedCRF addresses them through a common client-assessment mechanism.

Critical weaknesses
  1. The empirical evaluation is limited to the relatively small-scale datasets,and all evaluated models are only CNN-based architectures. This paper lacks a validation on some larger-scale datasets, such as ImageNet, and more diverse model architectures, such as the Transformer-based ViT. Additional experiments in these settings would provide a stronger evidence for the scalability and generalizability of the proposed FedCRF.

  2. FedCRF constructs a fully connected graph among all the participating client,s and requires computing and storing the pairwise affinities between client contributions. Consequently, the computational and memory overhead associated with the client graph grows quadratically with the number of participating clients. This may limit the scalability of FedCRF in the large-scale federated learning scenarios with substantially more participating clients.

Questions for authors

In Section 3.2, what is the motivation or theoretical basis for the specific formulations of ρi\rho_i and AijA_{ij}? In particular, for the geometric reliability score ρiG\rho_i^G, how were the relative coefficients of the different geometric terms determined? Were these coefficients theoretically motivated, empirically tuned, or just heuristically chosen?

1 comment

We thank the reviewer for the detailed and constructive review. All changes in the revised paper are marked in blue.

W1

We thank the reviewer for the suggestion. To test a transformer at a larger scale, we added Tiny ImageNet (200 classes, 100k images, 64×64) with a ViT-S/8 (21.5M parameters) trained from scratch: 20 clients with Dirichlet(0.5) label skew, 100 rounds, and the same local training for all methods (AdamW, cosine decay, label smoothing, 2 local epochs). Details and the full comparison are in the new App. J of the revised paper.

Methodcleancorrupted
FedAvg26.221.1
FedCRF$_G$26.921.4
FedCRF$_F$26.722.4
FedCRF$_{G+F}$27.121.8

On clean data, all FedCRF variants match or slightly exceed FedAvg, so the reliability model does no harm on a transformer either. With 30% corrupted clients, FedAvg loses 5.1 points, FedCRF$_F$ recovers 1.3 of them. Federated training from scratch on the full ImageNet-1k, for all methods and seeds, is beyond our compute capacity for the rebuttal period, but we aim to complete comparison for the final version.

W2

We agree that FedCRF's dense client graph introduces quadratic computational and memory costs, which may limit scalability to very large participating cohorts. However, two distinctions are important.

First, the graph includes only clients participating in the current round, not the entire federation. Under partial participation, its size can therefore remain much smaller than the total client population.

Second, theoretical complexity does not directly reflect practical overhead. To investigate this, we additionally profiled FedCRF$_G$ on FEMNIST (6.6M-parameter CNN), increasing participating clients from 100 to 400. Although pairwise cosine computation scales as O(N²d), it accounts for only 0.6%, 1.5%, and 2.2% of server-step time, respectively. Mean-field inference, O(RN²|Y|²), contributes at most 0.05%. In contrast, collecting updates and computing their weighted average account for 87–91% of the server step. Thus, the quadratic graph operations are not the primary computational bottleneck at the evaluated scales.

Regarding memory, the affinity matrix requires O(N²) storage, corresponding to approximately 0.64 MB at N=400 using 32-bit floats. This is small compared with the approximately 10.6 GB needed to store 400 full updates of the same model. While the dense graph remains a scalability limitation for substantially larger cohorts, these results show that its additional cost is modest at the evaluated scale. We include this analysis and clarify the limitation in App. G.

Server-step componentN=100N=200N=400
Collecting updates (flatten, copy to GPU)48%46%48%
Weighted model aggregation39%42%43%
FedCRF evidence and inference
→ Coordinate-wise median, O(Nd log N)8.8%8.6%6.1%
→ Unary statistics, O(Nd)3.6%2.5%0.9%
→ Pairwise cosines, O(N²d)0.6%1.5%2.2%
→ Mean-field inference, O(RN²Y²)0.05%0.03%0.02%

Q1

In case of geometric evidence, the reliability score combines three quantities: the cosine alignment cic_i, the normalized distance did_i, and the norm anomaly hih_i. By construction, di=hi=1d_i=h_i=1 for the median client. We chose the coefficients so that the median client sits at the centre of the sigmoid: with a reference alignment of ci=0.75c_i=0.75, the log-odds 2ci−di−0.5hi2c_i-d_i-0.5 h_i equal zero, so ρi=0.5\rho_i=0.5 and the prior of Eq. 6 is uniform. Each coefficient is then the change in log-odds per typical unit of deviation (Eq. 16). The relative weights were chosen from a few candidate combinations. As Tab. 15 shows, the exact values matter little as long as all three terms are kept. Sec. 3.2 and the new App. I now discuss this.

Reviewer_YCkyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The work uses conditional random fields to aggregate server models in federated learning. It models clients as a graph structure and evaluates geometric and functional potential, as well as their combination, to determine client reliabilities. It also mentions the benefits of this setup beyond global model aggregation, including personalized FL, clustered FL, and Byzantine robustness. However, the results across different setups are somewhat mixed, rather than showing clean, substantial gains.

Critical strengths
  1. The paper proposes one method using conditional random fields for server-side aggregation in federated learning, along with its benefits in personalized FL, clustered FL, and Byzantine robustness.
  2. The idea of computing the reliability of client updates based on unary as well as pairwise potential is well motivated.
Critical weaknesses
  • The existing method appears to penalize clients if high heterogeneity is present between the clients, and it might treat the heterogeneous signal as noise, thereby reducing the reliability of the specified client. How does this method then deal with such scenarios?
  • One of the major drawbacks of the paper appears to be its results when compared against state-of-the-art methods as well as basic methods like FedAvg, and also when it is compared against the claims made in the paper. A few such instances are: (a) - In the Global aggregation results (Table 1), even FedAvg performs comparably to FedCRF on CIFAR10, FEMNIST, and Fed-ISIC2019 clean settings. Even in the corrupted settings, there is only a substantial improvement in the CIFAR10 and Fed-ISIC2019 datasets. (b) - In partial participation results (Table 2), K=100, 10% setting, FedCRF's best score of 32.9% is considerably lower than FedLAW's 62.1. What explains this difference? And in almost all the settings in this table, there isn't any considerable gain for FedCRF, and even FedAvg again performs comparably and sometimes even better. (c) - Similar results are seen in Byzantine robustness (Table 3), with little to no improvement over the existing state-of-the-art methods. (d) - In clustered FL (Table 4), some improvements can be seen on certain datasets, but the results are still inconsistent across different settings, for example, the FEMNIST (rotated) case. (e) - The high attack success rate on model replacement and neurotoxin, along with average clean accuracy compared to other methods, defeats the purpose in Byzantine Robustness results on FEMNIST (Table 6).
  • Mean reliability of benign clients is around 0.3-0.5 (Figure 2), instead of the 1.0 that the paper assumes.
  • The paper mentions FedCDA as a baseline but never uses it in the results.
  • The concept of using server-side loss updates as a measure of client reliability was already present in Zeno, which is quite similar to the functional variant of FedCRF; hence, the novelty of this variant appears to be questionable.
Questions for authors
  • What's the reasoning behind the specific formula and the coefficients for reliability and affinities in the geometric evidence? (Equation 8)
  • Is there any existing prior work that uses concepts like cosine similarity, normalized distance, norm anomaly, or empirical mean symmetric divergence in this line of work, or are these metrics explored for the first time?
  • What's the justification for having three pre-specified labels: {0, 0.25, 1}?
  • Instead of classifying labels into one of the three bins, can it be a regression function or a probability score between 0 and 1, which would then be converted into a reliability metric? Would that be a better approach?
3 comments

We thank the reviewer for the careful assessment of our work. We address the comments below.

W1

We agree that heterogeneity does not necessarily imply unreliability, as benign clients may produce substantially different updates that should not automatically be treated as noise. FedCRF does not penalize clients based on similarity alone: unary reliability and pairwise affinities are jointly considered through CRF inference, while functional evidence additionally measures improvements on a proxy set.

For global aggregation, the objective is to learn a single model that performs well across the federation. When heterogeneous contributions conflict, maximizing overall performance may require down-weighting some of them, even when locally useful. Our empirical results support this rationale: all three FedCRF variants achieve the three highest average accuracies across the six clean and corrupted heterogeneous settings (73.6%, 73.0%, and 72.2% vs. 72.1% for the strongest baseline, Tab. 1). Fig. 2 also shows that corrupted clients generally receive lower inferred reliability than clean clients, with improved accuracy under corruption (e.g., 82.7% vs. 78.1% on CIFAR-10).

When heterogeneity is too severe for a single global model, or when group- or client-specific performance is the objective, this motivates our clustered and personalized FL extensions. Rather than forcing all clients into a single model, FedCRF leverages client affinities to identify related groups or construct personalized model mixtures, allowing heterogeneous clients to contribute to models better suited to their distributions.

W2

We thank the reviewer for the detailed assessment. We agree that FedCRF does not outperform every specialized method in every setting, and have now clarified our claims in Sec. 5. Nevertheless, our contribution goes beyond achieving the highest accuracy on individual benchmarks: we propose a unified inference mechanism supporting global aggregation, Byzantine robustness, clustered FL, and personalized FL with fixed hyperparameters, while remaining competitive with methods specifically designed for each setting. This addresses an important practical challenge: in real federations, client heterogeneity, data quality, and adversarial presence are often unknown in advance, complicating the selection and tuning of specialized methods.

(a) Global aggregation (Tab. 1). Strong FedAvg performance on clean data is expected, since no clients are deliberately corrupted. FedCRF maintains comparable performance while improving substantially under corruption: +4.6 points on CIFAR-10 and +8.5 on Fed-ISIC2019 over the strongest baselines. On FEMNIST, corruption itself has limited impact on FedAvg (−1.6 points). Across all six settings, the three FedCRF variants achieve the highest average accuracies.

(b) Partial participation (Tab. 2). We acknowledge the gap against FedLAW at K=100, 10% participation (32.9% vs. 61.2%). All methods follow the same early-stopping protocol, which can be sensitive to validation-loss fluctuations under small client cohorts. In a preliminary experiment without early stopping, FedCRF$_F$ reaches 85.1%, vs. 85.0% for FedLAW and 83.8% for FedAvg, suggesting that stopping dynamics contribute substantially to the original gap. Nevertheless, FedCRF$_F$ achieves the highest average across all six settings (65.5% vs. 63.5%).

(c) Byzantine robustness (Tab. 3). FedCRF$_F$ matches Zeno in average clean accuracy and ASR. Importantly, Zeno and several robust baselines are configured with the true attacker fraction, generally unknown in practice. New experiments (see W5) illustrate the importance of this assumption: with 30% ALIE attackers but an assumed 20%, Zeno reaches 71.4% and Multi-Krum 20.4%, while FedCRF$_F$ maintains 83.0% without requiring the attacker fraction. These comparisons are now included in Sec. 4.3 and App. L, Tab. 18.

(d) Clustered FL (Tab. 4). We acknowledge in Secs. 4.4 and 5 that clustering remains sensitive to the affinity representation, particularly on rotated FEMNIST, where FedCRF fails to recover the underlying groups. Nevertheless, among methods provided with the true group count, FedCRF$G$ achieves the highest average accuracy (73.4% vs. 69.9%). Without this information, FedCRF${G+F}$ achieves ARI 0.95 on medium-severity CIFAR-10.

(e) FEMNIST robustness (Tab. 6). We acknowledge the limitations against model replacement and Neurotoxin. However, these failures are not specific to FedCRF: Zeno, RFA, and trimmed mean also exhibit ASR above 95% despite high clean accuracy. Methods that suppress these attacks incur substantial clean-accuracy losses, highlighting an unresolved trade-off under these conditions. FedCRF successfully suppresses targeted label flipping (ASR 0–1%). These results highlight the difficulty of detecting backdoors from whole-model statistics or clean proxy evaluations, which we discuss in Sec. 5.

W3

Our method does not assume benign clients to reach 1. The aggregation uses only the normalized weight

αi=αiFAyˉi/∑jαjFAyˉjα_i = α_i^{FA} \bar{y}_i / \sum_j α_j^{FA} \bar{y}_j (Eq. 13).

A common level of yˉ\bar{y} therefore cancels, and only the ratios between clients matter. The observed range is also what the design predicts. The scores are relative to the cohort and place a typical client near ρi=0.5\rho_i = 0.5, where the prior of Eq. 6 is uniform (see App. I for details). What matters is the gap between benign and unreliable clients. In Tab. 3, attackers make up 20% of the clients, yet under FedCRF$_F$ they hold only 0.2% (sign flip), 1.3% (ALIE), 3.2% (targeted label flipping), 0.2% (model replacement), and 0.3% (Neurotoxin) of the aggregation weight, while the benign yˉ\bar{y} is only 0.36–0.52.

W4

We thank the reviewer for pointing out this omission. FedCDA was initially included among our planned baselines, but its computational cost prevented us from completing the accuracy experiments before submission. We inadvertently retained its mention in the manuscript without reporting the corresponding results.

We have now completed the comparison, and report the results below compared to the three variants of our FedCRF (and include it in Tab. 1 of the revised paper). All FedCRF variants outperform FedCDA on average.

MethodCIFAR-10 cleanCIFAR-10 corruptedFEMNIST cleanFEMNIST corruptedFed-ISIC2019 cleanFed-ISIC2019 corruptedavg
FedCDA82.0±0.775.8±0.870.7±6.068.6±5.169.1±1.052.9±1.069.9
FedCRF$_G$84.6±0.980.8±0.773.3±4.571.4±5.169.7±0.962.0±3.473.6
FedCRF$_F$84.3±1.282.7±1.071.7±5.371.1±3.469.2±0.554.4±3.472.2
FedCRF$_{G+F}$84.7±0.582.0±1.873.7±4.372.9±3.369.1±0.555.5±2.473.0

W5

We agree that Zeno uses server-side loss improvement to assess client updates, and we do not claim this signal as novel (see also Q2). FedCRF's novelty lies instead in incorporating such signals into a structured probabilistic formulation, which is not tied to specific metrics, as illustrated by our geometric, functional, and mixed instantiations.

Zeno ranks updates by proxy-loss improvement penalized by update magnitude and selects the highest-scoring subset. In contrast, FedCRF$_F$ uses proxy-loss improvement as unary evidence and pairwise predictive agreement as affinity within a CRF. Mean-field inference jointly estimates client reliabilities, producing continuous aggregation corrections rather than hard selection. The inferred reliabilities and affinities also support global aggregation, Byzantine robustness, clustered FL, and personalized FL within the same framework.

Empirically, FedCRF$_F$ matches Zeno's Byzantine robustness (83.4% vs. 83.1% average clean accuracy; 1.7% vs. 1.4% ASR, Tab. 3). Importantly, unlike Zeno, FedCRF does not require an assumed attacker fraction. We investigate this distinction under different attacker fractions on CIFAR-10 under ALIE. As shown below, incorrectly specifying this fraction can substantially degrade specialized defenses, while FedCRF$_F$ maintains strong performance without requiring this information. We include these experiments and clarify the methodological distinction in the revised paper (App. L, Tab. 18; App. A).

MethodAssumed fractionTrue 10%True 20%True 30%True 40%
Multi-Krum10%84.066.448.526.6
Multi-Krum20%83.365.520.417.9
Multi-Krum40%82.962.818.721.4
Zeno10%83.679.160.655.4
Zeno20%83.683.771.463.8
Zeno40%84.483.382.481.2
Trimmedmeanβ=0.184.170.644.433.8
Trimmedmeanβ=0.284.462.142.416.8
Trimmedmeanβ=0.479.960.133.421.4
FedCRF$_G$None84.276.753.916.8
FedCRF$_F$None84.383.683.081.1
FedCRF$_{G+F}$None84.477.961.950.0

Q1

For geometric evidence, the reliability score combines three quantities: the cosine alignment cic_i, the normalized distance did_i, and the norm anomaly hih_i. By construction, di=hi=1d_i=h_i=1 for the median client. We chose the coefficients so that the median client sits at the centre of the sigmoid: with a reference alignment of ci=0.75c_i=0.75, the log-odds 2ci−di−0.5hi2c_i-d_i-0.5 h_i equal zero, so ρi=0.5\rho_i=0.5 and the prior of Eq. 6 is uniform. Each coefficient is then the change in log-odds per typical unit of deviation (Eq. 16). The relative weights were chosen from a few candidate combinations. As Tab. 15 shows, the exact values matter little as long as all three terms are kept. Sec. 3.2 and the new App. I now discuss this.

Q2

We thank the reviewer for the question and clarify this distinction in Sec. 3.2 and the Related Work. These metrics have indeed been explored in prior FL research. Cosine similarity is used by FoolsGold [1] and FLTrust [2]; distance-based measures underpin Krum [3]; and FLAME [4] uses update-norm statistics. Similarly, symmetric KL divergence has been explored by Cao et al. [5], while Zhu et al. [6] use JS divergence to compare client predictions on public data.

FedCRF's novelty lies instead in integrating these signals into a structured probabilistic formulation. Specifically, FedCRF incorporates them as unary and pairwise potentials of a CRF, using mean-field inference to jointly estimate client reliabilities by coupling individual evidence with inter-client relationships. Our geometric, functional, and mixed variants are specific instantiations; the formulation is not tied to these metrics and can accommodate alternative reliability and affinity signals. The resulting signals provide a unified framework for global aggregation, Byzantine robustness, clustered FL, and personalized FL, without separate client-assessment mechanisms.

References

[1] Fung et al., The Limitations of Federated Learning in Sybil Settings, RAID 2020

[2] Cao et al., FLTrust: Byzantine-robust Federated Learning via Trust Bootstrapping, NDSS 2021

[3] Blanchard et al., Machine Learning with Adversaries: Byzantine Tolerant Gradient Descent, NeurIPS 2017

[4] Nguyen et al., FLAME: Taming Backdoors in Federated Learning, USENIX Security 2022

[5] Cao et al., Knowledge-Aware Federated Active Learning with Non-IID Data, ICCV 2023

[6] Zhu et al., Federated Two-Stage Decoupling with Adaptive Personalization Layers, Complex Intell. Syst., 2024

Q3

The three labels represent distinct aggregation decisions: 1 retains the client's contribution, 0.25 partially down-weights it when its reliability is uncertain, and 0 excludes it entirely. This provides a simple distinction between reliable, potentially useful, and unreliable contributions. Importantly, these are latent states rather than fixed aggregation coefficients: mean-field inference computes their posterior expectations, allowing continuous corrections between 0 and 1.

We also evaluated alternative label configurations, including binary {0, 1}, five discrete labels, and a continuous Gaussian relaxation (App. F, Tab. 9). While performance is comparable under moderate heterogeneity, the default three-label configuration achieves higher mean accuracy under severe label skew. We therefore adopt {0, 0.25, 1} as a simple and empirically effective choice.

Q4

As described in Q3, FedCRF already produces continuous reliability coefficients in [0, 1], as aggregation uses the posterior expectation of the latent labels rather than a hard assignment (Eq. 13).

We also evaluated a fully continuous alternative (App. F, Tab. 9), replacing the discrete labels with a Gaussian relaxation and quadratic unary potential. This yields a closed-form expression for the posterior mean, avoiding iterative mean-field inference. However, while performance is comparable under moderate heterogeneity, the discrete formulation performs better under severe label skew, e.g., 85.7% versus 76.8% on MNIST and 18.8% versus 11.0% on CIFAR-10. Thus, the chosen formulation combines continuous reliability weighting with the empirical benefits of discrete latent states.

Comments on the paper

2 comments

We sincerely thank all reviewers for their time and for their thoughtful, constructive feedback. We appreciate the careful reading of our work, and the comments and suggestions have helped us substantially strengthen the paper.

We have uploaded a revised version of the manuscript that incorporates the reviewers’ feedback. All modifications are highlighted in blue.

Dear Reviewers,

As the discussion phase is coming to an end, we would be very happy to discuss our rebuttal and the revised paper. Below, concerns mapped to what changed in the revised PDF (all changes are in blue).

ConcernRaised byWhat changedWhere
Novelty and benefits of joint CRF inference isolatedqQcQ, YCkyComponent ablation: the full CRF beats affinities alone and reliabilities alone in clustered FL (78.3 vs. 74.8 / 73.1 with 50% label noise; 80.4 vs. 79.8 / 73.7 corrupted) and personalized FL (79.3 vs. 78.2 / 78.5). Under ALIE, pairwise coupling lowers the attackers' weight and raises worst-seed accuracy (63.4 → 69.3). Positioning against Zeno and prior metrics clarified.App. K (Tab. 17), App. H (Tab. 14), Sec. 3.2, App. A
Client-quality baselinesqQcQFedTrans and RHFL added, also under label noise. Over all 10 settings, FedCRF_G averages 74.8 vs. 72.8 FedAvg, 72.0 RHFL-CCR, 71.6 FedTrans.Tab. 1, App. N (Tab. 20)
Legitimate heterogeneity vs. poor qualityqQcQ, YCkyCorrupted clients receive lower reliability than benign heterogeneous ones (Fig. 2). In personalized FL, where heterogeneity is legitimate by construction, the full CRF is still best. Remaining limitation stated.App. H (Fig. 2), Tab. 17, Sec. 5
Natural feature skewqQcQOffice-Caltech10 and DomainNet: better under corruption (93.9 vs. 91.4; 80.3 vs. 79.3). Clustered FedCRF_G+F recovers all 6 DomainNet domains (ARI 1.00 vs. 0.48 IFCA).App. M (Tab. 19)
Scale and architecturesf1ssViT-S/8 on Tiny ImageNet: matches or exceeds FedAvg on clean data, and recovers part of the loss under corruption (22.4 vs. 21.1).App. J (Tab. 16)
Quadratic graph costf1ssProfiled with 100–400 participating clients: the quadratic steps take ≤2.3% of the server step; the affinity matrix is 0.64 MB at N=400.App. G (Tab. 13)
Design of ρ, coefficients motivation, discrete labelsf1ss, YCkyMore detailed explanation of the reliability ρ design with motivation for coefficients choice.Sec. 3.2, App. I (Tab. 15), App. F (Tab. 9)
Mixed results; similarity to ZenoYCkyClaims revised. Robust defenses with a misspecified attacker count: at 30% ALIE, Zeno drops to 71.4% and Multi-Krum to 20.4% when set to 20%, while FedCRF_F keeps 83.0% without this input.Sec. 4.3, Sec. 5, App. L (Tab. 18)
FedCDA missingYCkyAdded, all FedCRF variants are above it on average.Tab. 1

We would be glad to clarify anything further before the discussion closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.