- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposed FedSAM, a method for model heterogeneous FL, based on Bayesian hyerarchical framework. The idea is to alternate a local beta process to locally adapt a neural network to each client data and a global beta process to shape a global model, aggragating parameter via standard FedAvg.
Critical strengths
The main strength of the paper is that the proposed approach is principled and rooted in Bayesian inference: it treats client structure as a latent variable, inferred jointly with the weights, and shares structural statistics across clients through a hierarchical prior. In addition, the communication overhead is limited, since the approach transmits only two additional scalars per layer. Finally, the results show consistent improvements over the baselines across the image datasets, and roughly aligned with other SOTA on text datasets
Critical weaknesses
The novelty and contribution of the paper, in particular how the proposal differs from the existing literature, is not so clear. The local model follows the beta-process depth/width adaptation of the cited KC et al. (2021) and Thapa & Li (2024), and the hierarchical extension directly applies the hierarchical beta process of Thibaux & Jordan (2007). The paper briefly discusses these works, and the positioning relative to them seems to be mainly at the level of the setting: prior structure-adaptation methods consider a single network, and prior Bayesian personalized FL methods assume homogeneous structures. It does not discuss what is technically new in the model or in the inference scheme. As presented, the contribution appears to be a combination of existing components in the model-heterogeneous FL setting.
Questions for authors
-
Source of the gain of FedSAM: Table 4 shows that the variant of FedSAM without the global beta process already outperforms all baselines. Moreover, FedSAM averages predictions over 10 structural samples at inference (Appendix C.3), whereas the baselines use a single forward pass. This seems to suggests that much of performance gain come from stochastic regularization and ensembling rather than from the hierarchical sharing of structure. Could the authors discuss on this and, for instance, report results with a single structural sample at inference, and for the same backbone without structural variables, to isolate the contribution of the hierarchical prior?
-
Number of seeds: How many random seeds were used for the reported results? Several differences, particularly on the text datasets, are within one standard error.
-
On personalized baselines. FedSAM’s local models keep client-specific structural variables and are evaluated on local test sets, which resemble a form of personalization that the considered baselines lack. Could the authors discuss this point and possibly include a personalized or Bayesian personalized FL baselines (e.g., already cited Zhang et al., 2022; Thapa & Li, 2025)?
3 comments
Reply to authors
Thank you for the additional experiments and clarifications. Most of my initial concerns have been addressed, particularly regarding the novelty of the proposed approach and the contribution of the different prior components. Regarding the distinction between structural personalization and personalized FL, I believe this aspect could benefit from further clarification.
Overall, the rebuttal has addressed most of my doubts, and I will consider increasing my score.
We thank Reviewer o8fw for the further comments and for raising the score.
As Reviewer o8fw requested, we further clarify the distinction between personalized FL and our framework below:
-
Personalized FL methods (Zhang et al., 2022; Thapa & Li, 2025) use the same neural network structure for all clients and only personalize the neural network weights: each client learns its own client-specific weights to fit its local data distribution. Since every client trains an identical network structure, these methods are not directly applicable when clients have to train different-sized neural network structures due to their devices’ unique computing and memory constraints.
-
In contrast, our framework automatically personalizes network structures for the clients with global weights shared among them. Each client infers its own network structure within its computational capacity budget (Eq. 11). The server aggregates the local network structures through the global beta process, and yields a generalized global model (Eq. 12). Evaluated on the union of all clients' test sets, this global model outperforms all baselines in Table 1 by at least 5.1 percentage points on the image datasets.
We will clarify these in the camera-ready version.
Response to Reviewer o8fw
We thank Reviewer o8fw for the constructive feedback.
“The novelty and contribution of the paper, in particular how the proposal differs from the existing literature, is not so clear.”
Our novelty and contribution lie in the aspects as follows:
The hierarchical beta process in Thibaux & Jordan (2007) is implemented with Monte Carlo sampling. It is computationally expensive and not scalable to high-dimensional models such as deep learning. In contrast, we develop a novel structured variational inference scheme for the computation of the joint inference of the global and local client-specific network structures for model-heterogeneous FL. The scalable and efficient inference procedure accommodates heterogeneous client model capacities through client-level structure adaptation (Eq. 11) and server-level structural information aggregation (Eq. 12).
We further provide theoretical analysis of statistical and algorithmic convergence of our modeling and inference methods. Theorem 4.1 characterizes the asymptotic stochastic convergence of the local activation probabilities towards global mean during sampling. Based on Theorem 4.1, Theorem 4.2 establishes an algorithmic convergence bound per mask sample from the local activation probabilities.
We will clarify the novelty of our method in camera-ready version.
Q1: As Reviewer o8fw suggested, to better understand the source of performance gain, we further investigate our method’s performance without the hyper-pior or the local-priors in model-heterogeneous FL. We summarize the results in the table below for greater clarity.
Average test accuracy (%) on CIFAR10 dataset under high data heterogeneity (α = 0.3). “gBP” stands for global beta process prior and “lBP” is for local beta process prior. With only lBP, each client infers its structure independently, so no global structure exists to construct a global model (“-”).
In the setting without either global prior or local prior, our method degenerates to dropout. The results highlight the complementary benefits of local structural adaptation and the hierarchical prior. Local beta-Bernoulli adaptation improves average local accuracy over standard dropout (r=0.5) by 3.9 percentage points, while incorporating the global beta process yields an additional 3.7-point gain.
Q2: We use 3 random seeds for all reported results and report the mean accuracy and standard error.
On the text datasets in Table 2, our method’s superior performance is statistically significant (with 3 seeds, we achieve p≈0.07–0.08) on both text datasets under high data heterogeneity (α = 0.3), outperforming the strongest baselines by 1.6 and 0.9 percentage points in average local test accuracy, respectively. Under low data heterogeneity (α = 10.0), although our method achieves the second-best performance, the accuracy difference is not significant between our method and the best-performing baselines. We will clarify the points in the camera-ready version.
Q3: Personalized FL (Zhang et al., 2022; Thapa & Li, 2025) assumes homogeneous local model structures for all the clients. They primarily address statistical data heterogeneity by personalizing model weights. Personalized FL with homogeneous local network structures is a less challenging task.
In contrast, our proposed framework as structural personalization aligns with model-heterogeneous FL, in the sense that it primarily addresses system and hardware heterogeneity, where participating clients have different computational resources and cannot train identical network structures.
The two categories of FL methods focus on addressing different challenges. That said, integrating model heterogeneity with parameter personalization represents a key avenue for our future research.