Federated Learning with Energy-Based Structured Probabilistic Inference

Federated Learning (FL) enables multiple clients to collaboratively train machine learning models without sharing their local data. However, heterogeneity in client data and the presence of unreliable or adversarial updates make effective server aggregation challenging, motivating methods that adapt client contributions according to their reliability and relationships. In this work, we formulate server aggregation in FL as structured probabilistic inference using conditional random fields. We represent clients as nodes in a graph and define unary and pairwise energy potentials. Unary potentials capture individual client properties, while pairwise potentials encode similarities between clients. This formulation treats individual client evidence and pairwise relationships as coupled evidence in a joint inference problem, allowing client reliabilities to be inferred collectively and the resulting estimates to guide aggregation. Where most FL methods target a single challenge, FedCRF's inferred reliabilities and affinities provide one shared representation for global aggregation, Byzantine robustness, clustered FL, and personalized FL. With a single set of hyperparameters, it is competitive with or outperforms specialized methods on CIFAR-10, FEMNIST, and Fed-ISIC2019 across diverse forms of client heterogeneity and unreliability.