acceptodds
Under review as a conference paper at ICLR 2027

FedBHG: Bayesian Coalition Stability in Federated Learning

Abstract

At a federated learning checkpoint, clients must choose between local and shared continuation before either alternative is trained. We study this decision through posterior gradient-estimation risk under a Gaussian hierarchy. With known scalar variances, clients whose sampling variance is below the between-client variance remain local; the rest pool using inverse-total-variance weights. Under the specified summary disclosure, we prove that this partition almost surely admits no coalition that weakly reduces every member's risk and strictly reduces at least one. For at least three pooling-eligible clients, these are the unique fixed positive linear weights that all members accept over staying local for every disclosed summary configuration. When every covariance direction favors the same local-or-shared choice, the construction extends to general covariances; when directions favor conflicting choices, every partition can be unstable. Under shared quadratic curvature, the posterior-risk comparison also determines the expected-loss comparison between the corresponding Newton updates. FedBHG estimates the variances at a checkpoint and continues with ordinary training. Synthetic experiments isolate failures from route errors, directional conflict and aggregation weights. Across 20 runs in four selected task/clusterer settings with fixed client partitions, FedBHG beats both universal continuations in accuracy in 19 and loss in 17. Four fresh-partition observations improve accuracy in all four and loss in three. These experiments support checkpoint-based selective continuation in the tested settings, with the Bayesian routing rule's incremental benefit over simpler routes varying by task. The theory concerns oracle posterior-risk stability; the neural experiments evaluate predictive transfer under fitted routing and ordinary training.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
federated learning, Bayesian inference, coalition formation, hedonic games, coalition stability, individual rationality, model aggregation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kfc7OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies a checkpoint decision in federated learning, namely which clients should continue training locally and which should pool into a shared model, before either alternative is trained. It models client mean gradients through a Gaussian hierarchy that separates sampling variance from between-client variance, and uses posterior gradient-estimation risk as the decision criterion. With known scalar variances, clients with sampling variance below the between-client variance stay local and the rest pool with inverse-total-variance weights (Lemma 3.1 and Theorem 3.2). The paper proves this partition almost surely admits no blocking coalition, characterizes precision weights as the unique fixed weights preserving participation for at least three pooled clients (Theorem 3.3), extends stability to general covariances when all directions agree (Theorem 3.4), and gives a counterexample where conflicting covariance directions defeat every partition (Theorem 3.5). Under shared quadratic curvature the risk comparison also orders expected loss after Newton updates (Proposition 4.1). The implemented FedBHG estimates the variances once at a checkpoint and continues with ordinary SGD or Adam. Synthetic experiments isolate failure modes of fitted routing, and neural experiments on Fashion-MNIST, EMNIST-Letters, and Shakespeare compare against universal continuations, simpler routes, and personalization baselines.

Critical strengths
  1. The theory is tight and symmetric in both directions. The stable construction (Theorem 3.2), the weight characterization (Theorem 3.3), and the impossibility counterexample (Theorem 3.5, with exact gains verified in Table 1) together map out where stability can and cannot hold, which is rare for participation analyses in federated learning.
  2. The empirical work is unusually disciplined about its own limits. The paper states the claim boundary explicitly in Section 5, isolates route error, directional conflict, and weight error in controlled six-client simulations (93.0% no-deviation with correct routes versus 14.5% after one error in Figure 1), reports paired per-cell intervals rather than cherry-picked aggregates (Table 2), and includes misspecification checks showing the fitted spherical model is rejected yet the score retains useful ranking (Section 5.5).
Critical weaknesses
  1. The gap between the theory and the deployed procedure is substantial. The guarantees assume known variances, the Gaussian hierarchy, and quadratic shared curvature with Newton updates, while FedBHG estimates variances once and trains with ordinary optimizers, so no formal stability carries over to the neural setting. The practical edge of the full Bayesian score over simpler routes is correspondingly modest, since its routing matches sample-size routing in 7 of 20 repeats and variance-only routing in 5 of 20 (Section 5.3).
  2. The experimental scope is narrow. Only four cells selected from a ten-cell development sweep are studied in depth, with fixed client partitions of 100 clients on three datasets, and some headline gains are small or unresolved, for example +0.051 accuracy points on Shakespeare/CFL. Broader validation across more tasks and partition scales would strengthen the practical claims.
Questions for authors
  1. The gap between the theory and the deployed procedure is substantial. The guarantees assume known variances, the Gaussian hierarchy, and quadratic shared curvature with Newton updates, while FedBHG estimates variances once and trains with ordinary optimizers, so no formal stability carries over to the neural setting. The practical edge of the full Bayesian score over simpler routes is correspondingly modest, since its routing matches sample-size routing in 7 of 20 repeats and variance-only routing in 5 of 20 (Section 5.3).
  2. The experimental scope is narrow. Only four cells selected from a ten-cell development sweep are studied in depth, with fixed client partitions of 100 clients on three datasets, and some headline gains are small or unresolved, for example +0.051 accuracy points on Shakespeare/CFL. Broader validation across more tasks and partition scales would strengthen the practical claims.
1 comment
AuthorsOpenReview ↗

Client Selection and Aggregation Weights

Thank you for the questions. We address weaknesses W1–W2 and author questions Q1–Q2.

1. Client Selection and Aggregation Weights (W1, Q1)

We added a new four-arm experiment to measure the contributions of client selection and gradient aggregation separately. On Fashion-MNIST, selecting which clients share a model improves accuracy by 3.488 percentage points with sample-count aggregation and 3.103 points with inverse-total-variance aggregation. Holding the selected clients fixed, inverse-total-variance aggregation reduces loss by 0.0473 on Fashion and 0.0709 on EMNIST. The paired intervals for these loss reductions exclude zero (Section 5.3, Appendix G.2).

The experiment compares sharing among all clients in an upstream cluster with sharing only among clients selected by FedBHG. Clients excluded from the shared model train independently. Each selection is tested with both aggregation methods. All four branches use the same regenerated checkpoint, restored optimizer states, reduced learning rate, local training budget, and client shuffle streams. Thus, differences within each pair isolate client selection or aggregation under the tested training protocol.

Precision-based aggregation also improves the Fashion loss outcome of selective sharing. With sample-count aggregation, choosing the FedBHG-selected clients increases loss by 0.0144 (95% paired interval for the increase [0.0054, 0.0234]). With inverse-total-variance aggregation, the same selection reduces loss by 0.0089 (interval for the reduction [0.0028, 0.0151]). The corresponding weighting differences on Shakespeare have intervals including zero. These results show distinct contributions from choosing clients and combining their gradients.

FedBHG's Bayesian client-selection score gives an additional accuracy gain over simpler selection rules on Fashion. With the number of locally trained clients matched, it gains 0.901 points over sample-count selection and 1.108 points over gradient-variance selection. The Fashion loss difference against sample-count selection and the accuracy differences in the other three settings remain unresolved. The clearest score-specific gain is therefore Fashion accuracy; the four-arm results separately show the contribution of selective sharing and precision aggregation.

Theorem 3.2 explains why the Bayesian model selects clients for local or shared gradient estimation under known variances and union-conditioned summary disclosure. For groups of at least three pooling-eligible clients, Theorem 3.3 characterizes inverse-total-variance coefficients as the only fixed positive aggregation coefficients that preserve every member's willingness to share for every observed summary configuration. The theoretical no-deviation result concerns posterior estimation risk. The experiments evaluate prediction after estimated routing and multi-round SGD or Adam training.

2. Baseline Comparisons and Experimental Scope (W2, Q2)

We added a new experiment comparing FedBHG against CFL and IFCA continuing their original training procedures. FedBHG improves both accuracy and held-out loss in three of four settings. On Fashion/CFL and EMNIST/IFCA, its accuracy gains are 3.079 and 1.983 percentage points, with loss reductions of 0.0606 and 0.0547. Shakespeare/IFCA also has positive paired intervals for both outcomes. Both intervals include zero on Shakespeare/CFL (Table 3, Appendix G.2).

Each comparison starts from the same regenerated round-200 checkpoint and restores the model and optimizer states. Five paired seeds per setting run ten additional rounds, with one local epoch per client per round. Continuing CFL/IFCA retains its original learning rate, sample-count aggregation, and clustering or reassignment rule. FedBHG assigns clients to independent or shared training once, fixes the shared group's gradient-aggregation coefficients, and uses one-tenth of the original learning rate. These comparisons evaluate the full continuation procedures, so learning rate and computation are not matched. The four-arm experiment above keeps the reduced learning rate fixed to isolate client selection and gradient aggregation.

The additional comparisons cover four settings selected from a ten-cell development grid, using 100 clients and full participation. Four earlier fresh vision-partition observations test changed client populations. FedBHG improves accuracy in all four and loss in three; one Fashion partition gains 1.45 accuracy points while increasing loss by 0.013. Those results remain in the revised paper. Sections 5.2–5.3 and Appendix G.2 now distinguish the new baseline comparisons, the new four-arm experiment, and the earlier robustness observations. Broader client scales and additional tasks remain for future evaluation.

Reviewer_v9v4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a Bayesian framework where each client decides whether to continue training locally alone or share a model with other clients based on gradient variance. Training continues until the partition stable condition is met.

Critical strengths
  1. Formulation of the proposed framework is well-motivated and theoretically grounded.
Critical weaknesses
  1. Lack of discussion and empirical evaluation with relevant baselines and recent PFL methods [1, 2, 3, 4, 5].
  2. Lack of sensitivity analysis with different numbers of clients, levels of data heterogeneity, and partial client participation scenarios.
  3. Many important details are left out of the main paper and placed in the Appendix sections (results tables, discussion of recent relevant methods in the related work section, ..).

References:

  1. Incentive Mechanism Design for Federated Learning: Hedonic Game Approach (https://arxiv.org/pdf/2101.09673)
  2. An Overlapping Coalition Game for Individual Utility Maximization in Federated Learning (https://openreview.net/pdf?id=pL6ducu0wL)
  3. Clustered Federated Learning via Gradient-based Partitioning (https://openreview.net/pdf?id=J4HJUF70qm)
  4. Interaction-Aware Gaussian Weighting for Clustered Federated Learning (https://openreview.net/pdf?id=dZAQxNFKGg)
  5. Personalized Federated Learning via Feature Distribution Adaptation (https://openreview.net/pdf?id=Wl2optQcng)
Questions for authors
  1. What will be the performance of CFL and IFCA if they continue training for 10 additional rounds with full participation after 200 rounds?
  2. During the performance comparison, are all methods trained for 10 additional rounds with full participation after the initial 200 rounds?
  3. At the end, how do the client clusters generated by the proposed framework differ from those generated by CFL or IFCA after 200 rounds? How do the clusters evolve during training?
  4. How will the algorithm behave if we train the proposed framework from scratch?
2 comments
AuthorsOpenReview ↗

Baseline Comparisons and Training Protocol

Thank you for the questions. We address Q1–Q4 and weaknesses W1–W3.

1. New CFL/IFCA comparison (Q1)

Yes. We ran a new experiment comparing FedBHG with continued CFL or IFCA training. The original local-all and joint-all controls froze upstream groups, so they did not answer this question.

We use five paired seeds in four selected settings. Each pair shares a regenerated round-200 checkpoint and optimizer states, then trains through round 210 (Table 3, Appendix G.2).

SettingContinued CFL/IFCA accuracyFedBHG accuracyLoss reduction
Fashion S3/CFL85.136%88.215%0.0606
EMNIST S3/IFCA68.806%70.789%0.0547
Shakespeare/CFL42.205%42.210%0.0024
Shakespeare/IFCA43.229%43.385%0.0027

FedBHG improves both accuracy and held-out loss in three of four settings. Paired intervals favor FedBHG except on Shakespeare/CFL, where both include zero. Accuracy is sample weighted, loss is client uniform, and intervals reflect five seeds on fixed populations.

2. Training budgets (Q2)

Yes, the methods have matched checkpoints, ten continuation rounds, and ten local epochs per client. No, their learning rates and total computation are not matched. All clients participate and complete one local epoch per round. Continued CFL/IFCA keeps its upstream learning rate, sample-count gradient aggregation, and splitting or reassignment rules. FedBHG fixes client assignments and gradient-aggregation coefficients and uses one-tenth of that rate. Vision training uses SGD at 0.01 versus 0.001; Shakespeare uses Adam at 0.001 versus 0.0001.

The experiment compares full training procedures. FedBHG additionally collects gradient summaries and fits variances, while native IFCA evaluates reassignments. Equal epochs do not imply equal computation.

A separate four-arm experiment controls the training convention. It compares full-cluster versus FedBHG-selected sharing, each with sample-count or inverse-total-variance gradient coefficients. All branches share the reduced learning rate, restored optimizer states, and client shuffles (Section 5.3, Appendix G.2).

3. Client assignments (Q3)

FedBHG assigns clients once at round 200. Within each existing CFL/IFCA cluster, clients with estimated score below 1/21/2 train independently. The others share one model with fixed estimated inverse-total-variance coefficients for combining gradients. These client-to-model assignments and aggregation coefficients stay fixed for ten rounds, while model parameters keep updating. There is no rerouting across or within upstream clusters. Appendix G.2 records groups at rounds 200, 201, 205, and 210.

4. Upstream clustering and initialization (Q4)

We do not report a controlled experiment that applies FedBHG alone from random initialization. Our method is designed to refine an existing CFL or IFCA grouping at a trained checkpoint. The Bayesian hierarchy models gradients around a common population mean within each candidate group. In heterogeneous data, all clients need not belong to one such group. CFL or IFCA first identifies groups; FedBHG then decides which clients in each group should continue locally or share a model.

Starting from random initialization would require routing before upstream groups and checkpoint gradients were established. Section 4 clarifies this design scope.

5. Writing and Revisions (W1–W3)

We added Table 3 and the continued CFL/IFCA comparison to Section 5.2, the selection-versus-weighting study to Section 5.3, and explicit optimizer, budget, and assignment details to Section 4 and Appendix G.2.

Experiments use 100 clients, full participation, three datasets, varied data quantities and corruption, and four fresh vision partitions. Client-count and partial-participation studies and benchmarks against the five suggested methods remain outside the evaluation.

Thank you for providing the additional experiments and further clarifications in response to my questions and concerns. While I appreciate the efforts, my concerns regarding the listed weaknesses remain. Therefore, I would like to keep my score unchanged.

Reviewer_rCT7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FedBHG, a method determining whether clients should continue training locally or jointly from a shared checkpoint. Under a Gaussian hierarchical model, FedBHG compares each client's sampling variance with the between-client variance to guide the routing decision. Specifically, clients with relatively reliable local gradient estimates remain local, while clients with higher sampling uncertainty continue joint training. The paper further analyzes the resulting routing decisions from the perspective of coalition stability and evaluates the proposed method on both synthetic and neural-network experiments.

Critical strengths

The paper introduces an interesting perspective on PFL. Starting from a shared checkpoint, clients can adaptively choose between local and shared continuation according to the uncertainty of their own gradient estimates and the degree of gradient heterogeneity between clients.

Critical weaknesses

The main concern is the gap between the theoretical analysis and the practical implementation. The theoretical analysis models gradients using isotropic scalar covariances. However, the covariance structure of gradients in modern neural networks can be highly anisotropic and correlated across parameter dimensions. Therefore, it remains unclear to what extent the theoretical routing guarantees derived under the scalar covariance model can explain the empirical improvements observed with neural networks.

Questions for authors
  1. Could the authors further clarify how the theoretical results are expected to translate to the practical setting?

  2. Since the routing decision depends on estimated variance components, estimation errors may cause some clients to be incorrectly routed. Have the authors considered mechanisms to solve this problem?

2 comments
AuthorsOpenReview ↗

Theory and Experimental Evidence

We thank the reviewer for questioning the theory-to-training connection. We label the concern W1 and the questions Q1 and Q2.

1. Bayesian decisions and trained-model results (W1, Q1)

The theory derives which clients should train locally and which should share a gradient estimate. With known variances under the Gaussian hierarchy, Theorem 3.2 assigns clients whose sampling variance is below between-client variance to local estimates and the others to a shared estimate with inverse-total-variance coefficients. Under the specified disclosure rule, no coalition can lower every member's posterior estimation risk while strictly improving one member. The theorem justifies these assignments and coefficients under its assumptions.

Proposition 4.1 connects this decision to learning loss under shared quadratic curvature. The same local-or-shared comparison determines expected loss for the corresponding Newton updates. FedBHG estimates variances at a checkpoint and continues with SGD or Adam. Gradient-shape checks on fresh client partitions reject the spherical working model, so our claims for SGD and Adam rely on measured accuracy and loss.

New direct comparisons show predictive gains from FedBHG's checkpoint procedure. Starting from matched round-200 checkpoints, it improves accuracy and held-out loss over continued CFL/IFCA training on Fashion/CFL, EMNIST/IFCA and Shakespeare/IFCA. Both paired intervals include zero on Shakespeare/CFL (Table 3, Appendix G.2). CFL/IFCA retains its learning rate and clustering rules. FedBHG fixes client assignments and aggregation coefficients and uses one-tenth of that rate. The comparison tests the full procedures. A separate four-arm test holds training settings fixed and measures client selection and gradient weighting separately (Section 5.3). The no-deviation theorem concerns posterior risk under its stated model.

Theorem 3.4 also covers anisotropic covariance models when all directions favor the same choice. This includes Σ=σ2K\Sigma=\sigma^2K and Ni=viKN_i=v_iK for any positive-definite KK. Theorem 3.5 shows that conflicting directions can make every partition unstable. These results describe when the grouping theorem extends beyond spherical covariances.

2. Routing errors and decision reliability (Q2)

Existing synthetic experiments show how incorrect client assignments affect the no-deviation outcome. Among six-client trials, 93.0% of fitted groupings with correct assignments have no blocking coalition. The rate falls to 14.5% with one incorrect assignment and below 1% with two or more. Each trial checks all 63 nonempty coalitions. Incorrect assignments are associated with far fewer no-deviation outcomes.

Estimated scores are particularly sensitive near the threshold of 1/21/2. All oracle assignments remain unchanged if max⁡i∣B^i−Bi∣<min⁡i∣Bi−1/2∣\max_i|\widehat B_i-B_i|<\min_i|B_i-1/2|. A six-client example shows a fitted score crossing the threshold even as gradient dimension grows.

Bootstrap refits identify uncertain decisions in the network experiments. Average assignment-flip rates across four fresh client partitions are 1.39 to 3.47%, but the most sensitive client in each setting changes assignment in 42.8 to 51.4% of refits. Bootstrap refits measure repeatability, whereas synthetic trials allow comparisons against known oracle assignments. The refits identify clients with uncertain assignments near the threshold, even when the average decision is stable across resamples.

Thank you for the detailed response and the additional analyses. Nevertheless, my main concern is only partially resolved. As the response acknowledges, the theoretical connection to the practical setting using SGD or Adam remains limited. I encourage the authors to reconsider how the relationship between the theoretical analysis and the empirical results is framed in the paper, and to clearly distinguish the settings covered by the theoretical guarantees from those supported primarily by empirical evidence. Given this remaining concern, I will maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.