acceptodds
Under review as a conference paper at ICLR 2027

FedSAM: Hierarchical Structure Adaptation For Model-Heterogeneous Federated Learning

Abstract

In real-world Federated Learning (FL) settings, clients often differ in both computational resources and local data characteristics. Model-heterogeneous FL addresses this issue by assigning clients sub-models of different-sized network structures partitioned from a single large neural network. However, existing methods typically rely on heuristic partitioning criteria and do not allow clients to customize their sub-models to local data's unique statistical properties. In this paper, we propose FedSAM, a novel hierarchical Bayesian framework that enables probabilistic inference on client-specific network structures while sharing collaborative knowledge across the local models in the hierarchy. Specifically, in the framework we adapt a local model's neural network structure by modeling network depth and width as beta and conjugate Bernoulli processes. We further specify a global-level beta process as a hyper-prior to aggregate and share local models' structural information at central server. We develop a structured variational inference scheme for efficient computation. Extensive empirical studies demonstrate that by adapting local models' structures to non-IID data and sharing the structural information, our framework yields state-of-the-art performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Model-Heterogeneous Federated Learning, Hierarchical Bayesian Inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_o8fwOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed FedSAM, a method for model heterogeneous FL, based on Bayesian hyerarchical framework. The idea is to alternate a local beta process to locally adapt a neural network to each client data and a global beta process to shape a global model, aggragating parameter via standard FedAvg.

Critical strengths

The main strength of the paper is that the proposed approach is principled and rooted in Bayesian inference: it treats client structure as a latent variable, inferred jointly with the weights, and shares structural statistics across clients through a hierarchical prior. In addition, the communication overhead is limited, since the approach transmits only two additional scalars per layer. Finally, the results show consistent improvements over the baselines across the image datasets, and roughly aligned with other SOTA on text datasets

Critical weaknesses

The novelty and contribution of the paper, in particular how the proposal differs from the existing literature, is not so clear. The local model follows the beta-process depth/width adaptation of the cited KC et al. (2021) and Thapa & Li (2024), and the hierarchical extension directly applies the hierarchical beta process of Thibaux & Jordan (2007). The paper briefly discusses these works, and the positioning relative to them seems to be mainly at the level of the setting: prior structure-adaptation methods consider a single network, and prior Bayesian personalized FL methods assume homogeneous structures. It does not discuss what is technically new in the model or in the inference scheme. As presented, the contribution appears to be a combination of existing components in the model-heterogeneous FL setting.

Questions for authors
  1. Source of the gain of FedSAM: Table 4 shows that the variant of FedSAM without the global beta process already outperforms all baselines. Moreover, FedSAM averages predictions over 10 structural samples at inference (Appendix C.3), whereas the baselines use a single forward pass. This seems to suggests that much of performance gain come from stochastic regularization and ensembling rather than from the hierarchical sharing of structure. Could the authors discuss on this and, for instance, report results with a single structural sample at inference, and for the same backbone without structural variables, to isolate the contribution of the hierarchical prior?

  2. Number of seeds: How many random seeds were used for the reported results? Several differences, particularly on the text datasets, are within one standard error.

  3. On personalized baselines. FedSAM’s local models keep client-specific structural variables and are evaluated on local test sets, which resemble a form of personalization that the considered baselines lack. Could the authors discuss this point and possibly include a personalized or Bayesian personalized FL baselines (e.g., already cited Zhang et al., 2022; Thapa & Li, 2025)?

3 comments
AuthorsOpenReview ↗

Response to Reviewer o8fw

We thank Reviewer o8fw for the constructive feedback.

“The novelty and contribution of the paper, in particular how the proposal differs from the existing literature, is not so clear.”

Our novelty and contribution lie in the aspects as follows:

  • The hierarchical beta process in Thibaux & Jordan (2007) is implemented with Monte Carlo sampling. It is computationally expensive and not scalable to high-dimensional models such as deep learning. In contrast, we develop a novel structured variational inference scheme for the computation of the joint inference of the global and local client-specific network structures for model-heterogeneous FL. The scalable and efficient inference procedure accommodates heterogeneous client model capacities through client-level structure adaptation (Eq. 11) and server-level structural information aggregation (Eq. 12).

  • We further provide theoretical analysis of statistical and algorithmic convergence of our modeling and inference methods. Theorem 4.1 characterizes the asymptotic stochastic convergence of the local activation probabilities towards global mean during sampling. Based on Theorem 4.1, Theorem 4.2 establishes an algorithmic convergence bound per mask sample from the local activation probabilities.

We will clarify the novelty of our method in camera-ready version.

Q1: As Reviewer o8fw suggested, to better understand the source of performance gain, we further investigate our method’s performance without the hyper-pior or the local-priors in model-heterogeneous FL. We summarize the results in the table below for greater clarity.

Average test accuracy (%) on CIFAR10 dataset under high data heterogeneity (α = 0.3). “gBP” stands for global beta process prior and “lBP” is for local beta process prior. With only lBP, each client infers its structure independently, so no global structure exists to construct a global model (“-”).

gBPlBPLocal Model AverageGlobal Model
✓\checkmark✓\checkmark77.5±0.576.3±0.6
✓\checkmark73.8±1.9-
69.9±0.770.8±1.1

In the setting without either global prior or local prior, our method degenerates to dropout. The results highlight the complementary benefits of local structural adaptation and the hierarchical prior. Local beta-Bernoulli adaptation improves average local accuracy over standard dropout (r=0.5) by 3.9 percentage points, while incorporating the global beta process yields an additional 3.7-point gain.

Q2: We use 3 random seeds for all reported results and report the mean accuracy and standard error.

On the text datasets in Table 2, our method’s superior performance is statistically significant (with 3 seeds, we achieve p≈0.07–0.08p ≈ 0.07–0.08) on both text datasets under high data heterogeneity (α = 0.3), outperforming the strongest baselines by 1.6 and 0.9 percentage points in average local test accuracy, respectively. Under low data heterogeneity (α = 10.0), although our method achieves the second-best performance, the accuracy difference is not significant between our method and the best-performing baselines. We will clarify the points in the camera-ready version.

Q3: Personalized FL (Zhang et al., 2022; Thapa & Li, 2025) assumes homogeneous local model structures for all the clients. They primarily address statistical data heterogeneity by personalizing model weights. Personalized FL with homogeneous local network structures is a less challenging task.

In contrast, our proposed framework as structural personalization aligns with model-heterogeneous FL, in the sense that it primarily addresses system and hardware heterogeneity, where participating clients have different computational resources and cannot train identical network structures.

The two categories of FL methods focus on addressing different challenges. That said, integrating model heterogeneity with parameter personalization represents a key avenue for our future research.

Reviewer_o8fwOpenReview ↗

Reply to authors

Thank you for the additional experiments and clarifications. Most of my initial concerns have been addressed, particularly regarding the novelty of the proposed approach and the contribution of the different prior components. Regarding the distinction between structural personalization and personalized FL, I believe this aspect could benefit from further clarification.

Overall, the rebuttal has addressed most of my doubts, and I will consider increasing my score.

We thank Reviewer o8fw for the further comments and for raising the score.

As Reviewer o8fw requested, we further clarify the distinction between personalized FL and our framework below:

  • Personalized FL methods (Zhang et al., 2022; Thapa & Li, 2025) use the same neural network structure for all clients and only personalize the neural network weights: each client learns its own client-specific weights to fit its local data distribution. Since every client trains an identical network structure, these methods are not directly applicable when clients have to train different-sized neural network structures due to their devices’ unique computing and memory constraints.

  • In contrast, our framework automatically personalizes network structures for the clients with global weights shared among them. Each client infers its own network structure within its computational capacity budget (Eq. 11). The server aggregates the local network structures through the global beta process, and yields a generalized global model (Eq. 12). Evaluated on the union of all clients' test sets, this global model outperforms all baselines in Table 1 by at least 5.1 percentage points on the image datasets.

We will clarify these in the camera-ready version.

Reviewer_y4WhOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a method to incorporate the role of the local client data in model-heterogenous Federated Learning. It use a hierarchical beta process prior at the global server, which shares prior knowledge about which components are useful, and the local beta-bernoulli process adapt those components to the local client's data. This presents an alternative way to the existing approaches of prediefined subnetwork selection. It conducts experiments on five datasets, CIFAR10, CIFAR100, Tiny-ImageNet, AGI-News and IMDB, and report considerable test accuracy gains and faster convergence, mostly in image datasets.

Critical strengths
  1. Sharing structural information and adapting network to client's data using hierarchical beta-bernoulli process is a principled alternative to the existing predefined submodel selection works in the model-heterogenous federated learning.
  2. Results on Image datasets consistently beat baselines by a good margin across all settings.
  3. Convergence is faster in both CIFAR10 and CIFAR100 across different heterogeneity levels(Figure 2, Figure 5 and Figure 6).
  4. The plot with inferred layer-wise activation probabilities (Figure 3) qualitatively depicts the method and highlights the difference between the earlier layers and the deeper layers, as well as the closeness of the global model and the local models' average despite variability in the client models.
Critical weaknesses
  1. FedRolex and FIARSE baseline comparison for text datasets is missing.
  2. Results indicate that there is marginal or no improvement across text datasets.
  3. Convergence plots are only available for 2 image datasets(CIFAR10, CIFAR100).
  4. The paper mentions "infinite depth networks" as one of the main building blocks of the core methodology but the practical implementation uses pre-specified finite truncation levels, similar to other works.
  5. The choice of fixed and pre-specified local model capacity ratios should be justified.
  6. It is unclear what explains the gains that the paper achieves. The work should do some ablations, or isolate factors highlighting how and why the gains occur.
  7. The paper does not report memory and/or FLOPs requirements, and only reports "test accuracy vs communication rounds" and "time required to reach target accuracy". It is worthy enough to check if the gains sustain under these matching constraints as well.
Questions for authors
  1. Can the authors clarify the key methodological novelty of FedSAM over prior Bayesian approaches to neural network structure adaptation, beyond introducing hierarchical structural sharing in federated learning? Also mention how the work relates to FedMA and to Thapa & Li(ICML 2024), and state the novelty over the latter specifically.
  2. The work is restricted to 5 datasets, and could be substantially improved if the method is demonstrated to work on foundation models, multi-modal FL, or real-world client-heterogenous medical imaging datasets, etc. (Suggestion, will not affect the score)
4 comments
AuthorsOpenReview ↗

Response to Reviewer y4Wh (1/2)

We thank Reviewer y4Wh for the constructive feedback.

W1: As Reviewer y4Wh requested, we have conducted additional experiments comparing our FedSAM with FedRolex and FIARSE on AGNews and IMDB under both high and low data heterogeneity. The average local model and global model test accuracy (%) are reported below.

DatasetαFedRolex (Local Model)FedRolex (Global Model)FIARSE (Local Model)FIARSE (Global Model)Ours (Local Model)Ours (Global Model)
AGNews0.388.4±0.187.7±0.388.6±0.187.7±0.690.7±0.488.6±0.2
AGNews10.089.1±0.388.8±0.488.8±0.289.1±0.289.1±0.289.1±0.3
IMDB0.381.9±0.681.7±0.481.4±1.581.1±0.983.8±0.381.7±1.1
IMDB10.083.4±0.282.4±0.382.1±0.182.9±0.283.2±0.483.2±0.4

Our proposed framework achieves superior local model accuracy under high data heterogeneity (α=0.3), outperforming the stronger of these two baselines by 2.1 and 1.9 percentage points on AGNews and IMDB, respectively. Under low data heterogeneity (α=10.0), FedSAM remains competitive. For global model accuracy, our proposed framework achieves the best or tied-best among the three methods in this table across all four settings. We will add the results to the camera-ready version.

W2: As shown in Table 2, our method’s superior performance is statistically significant (i.e., with 3 seeds, we achieved p≈0.07–0.08p ≈ 0.07–0.08) on both text datasets under high data heterogeneity (α = 0.3), outperforming the strongest baselines by 1.6 and 0.9 percentage points in average local test accuracy, respectively. Under low data heterogeneity (α = 10.0), although our method achieves the second-best performance, the accuracy difference is not significant between our method and the best-performing baselines.

W3: We additionally present the convergence behavior on Tiny-ImageNet in Figure 8 of the revised manuscript (Appendix). The results demonstrate that our framework consistently achieves rapid convergence under different levels of data heterogeneity.

W4: The infinite-depth network structures are induced by the stochastic process as a prior of the true model, for which exact inference is computationally intractable, as stated in Line 201-202. Therefore, we employ a variational distribution with a finite truncation to approximate the posterior over the infinite dimensional structure space, as described in Section 3.4. The truncation level pre-specified according to each client's computational capacity serves only as an upper bound, while the effective network structure is inferred within the assigned capacity budget.

W5: We choose the capacity ratio in reference to what is established in the baselines. For example, HeteroFL chooses 1, 1/2, 1/4, 1/8, 1/16, ScaleFL chooses 1, 3/4, 1/2, 1/4, and Fiarse chooses 1, 1/4, 1/16, 1/64. For a rigorous comparison, we implement these methods following a similar capacity ratio scheme. The capacity ratios are sufficiently heterogeneous to demonstrate and test the performance of model-heterogeneous FL.

W6: We provided ablation study in Section 5.1 Line 473-479. Specifically, Table 4 shows that in comparison with independently adapting local network structures, our framework benefits from the global beta process sharing structural information and regularizing local models’ learning process.

To better understand the source of performance gain, we further investigate our method’s performance without the hyper-pior or the local-priors in model-heterogeneous FL. We summarize the results in the table below for greater clarity.

Average test accuracy (%) on CIFAR10 dataset under high data heterogeneity (α = 0.3). “gBP” stands for global beta process prior and “lBP” is for local beta process prior. With only lBP, each client infers its structure independently, so no global structure exists to construct a global model (“-”).

gBPlBPLocal Model AverageGlobal Model
✓\checkmark✓\checkmark77.5±0.576.3±0.6
✓\checkmark73.8±1.9-
69.9±0.770.8±1.1

In the setting without either global prior or local prior, our method degenerates to dropout. The results highlight the complementary benefits of local structural adaptation and the hyper-prior for global sharing. Local structural adaptation improves average local accuracy over standard dropout (r=0.5) by 3.9 percentage points, while incorporating the global beta process yields an additional 3.7-point gain, demonstrating the effectiveness of cross-client structural information sharing.

AuthorsOpenReview ↗

Response to Reviewer y4Wh (2/2)

W7: in federated learning, accuracy vs communication rounds and wall-clock time to reach the best performance (reported in Figure 2 and Table 3) are the major efficiency metrics researchers take into consideration.

With that said, as Reviewer y4Wh requested, we provided total computation (PFLOPs) required to reach target global model test accuracy on CIFAR-100 under different data heterogeneity levels below:

αTarget Accuracy (%)HeteroFLFedRolexDepthFLScaleFLFIARSEOurs
0.338.047.0521.4514.2932.5016.2917.14
40.0—33.5124.29—27.1620.00
42.0—47.5248.58—43.4525.72
44.0—————34.29
50.0—————101.16
10.048.020.1414.778.5817.6310.8811.44
50.0—21.4811.4451.2016.3111.44
52.0—32.2217.17—24.4714.31
54.0—53.1640.06——17.17
60.0—————40.06

And peak training memory (MB) across different client model capacity levels on CIFAR-100.

MethodT1T2T3T4
HeteroFL283389551682
FedRolex283387551682
DepthFL290420549682
ScaleFL259410548682
FIARSE1588171518421971
Ours422603790976

As shown in the tables above, our method requires 25.72 PFLOPs to reach 42% accuracy under α = 0.3, compared with 43.45–48.58 PFLOPs for the baselines that reach this target. Under α = 10.0, FedSAM requires 17.17 PFLOPs to reach 54% accuracy, compared with 40.06–53.16 PFLOPs for the corresponding baselines.

Regarding memory requirements, our method incurs additional overhead due to structural inference but maintains substantially lower peak training memory than FIARSE across all model capacity levels.

Since our method uses more memory at each capacity level, a fairer comparison is our model against the baselines' next larger model, which has similar memory. On CIFAR-100 (α = 0.3, Table 1), our method remains more accurate in this comparison: our T1 model (422 MB) reaches 48.5% vs. 45.9% for the best baseline T2 model (387–420 MB), our T2 model (603 MB) reaches 51.1% vs. 46.1% for baseline T3 (548–551 MB), and our T3 model (790 MB) reaches 52.7% vs. 44.1% for baseline T4 (682 MB). Baseline memory figures exclude FIARSE, which uses more memory than our method at every level.

Q1: Our key methodological novelty lie in the following three aspects:

  • We extend beta-Bernoulli structure inference for a single neural network (Thapa & Li ICML 2024) to joint inference for multiple local models’ structures by integrating them through a hyper-prior at global level. This extension is non-trivial, as it enables collaborative structural information sharing across heterogeneous client models while preserving client-specific network adaptation.

  • We develop an efficient structured variational inference scheme for the computation of the joint inference of the global and local client-specific network structures in model-heterogeneous FL. The inference procedure accommodates heterogeneous client model capacities through client-level structure adaptation (Eq. 11) and server-level structural information aggregation (Eq. 12).

  • We further provide theoretical analysis of statistical and algorithmic convergence of our modeling and inference methods. Theorem 4.1 characterizes how the global structural prior balances shared structural information and client-specific adaptation, and Theorem 4.2 establishes an algorithmic convergence bound for our method under partial client participation.

Comparison with Thapa & Li (ICML 2024): The beta-Bernoulli structure inference in (Thapa & Li, 2024) is only for a single neural network in continual learning. In contrast, we extend Bayesian structure adaptation to joint inference for multiple local models’ structures in a hierarchical framework for model-heterogeneous FL. This extension requires jointly inferring multiple client-specific network structures while enabling structural information to be shared across heterogeneous models.

Comparison with FedMA: FedMA is based on homogeneous local model structures. It is not a model-heterogeneous FL method. It constructs a global model by matching and averaging neurons across identical network structures,rather than inferring structure. In contrast, our model-heterogeneous FL framework enables clients to adapt their network structures to local data while sharing structural information across clients in a hierarchy.

We will clarify the novelty of our method in camera-ready version.

Q2: Although we reported the performance of our method with transformer backbone models on text data in Table 2, scaling our framework to even larger-scale foundation models is a natural step of our future research. We agree with Reviewer y4Wh that extending our method to multi-modal FL and real-world client-heterogeneous medical imaging datasets is an interesting avenue for us to explore in the future.

Reviewer_y4WhOpenReview ↗

Response to Authors

Most of my primary concerns are addressed by the authors in their reply. Thankyou for the detailed response with additional experiments and clarifications.

The authors say "We develop an efficient structured variational inference scheme for the computation of the joint inference of the global and local client-specific network structures in model-heterogeneous FL.", and just as a matter of curiosity and to be inline with state-of-the-art research, can the authors mention existing works which do the similar thing, or do they claim to be the first one to pursue this.

Meanwhile, I would prefer to raise my score . Good Luck.

We thank Reviewer y4Wh for the further comments and positive assessment of our work.

To the best of our knowledge, our work is the first to perform joint variational inference over global and client-specific network structures in model-heterogeneous federated learning (FL).

State-of-the-art model-heterogeneous FL methods (e.g., Ilhan et al., 2023; Wu et al., 2024; Zhang et al., 2025; Hu et al., 2026) cited in our paper do not infer network structures from clients' local data. Instead, they obtain heterogeneous sub-models for the clients by partitioning a global model using predefined heuristics or parameter-importance estimates.

In contrast, our joint variational inference scheme operates at two levels. At the client level, each client adapts its own network structure to its local data characteristics within its computational capacity (Eq. 11). At the server level, the hyper-prior aggregates these local structures into a global network structure (Eq. 12), sharing common and complementary structural information among the clients.

Reviewer_gj6hOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes FedSAM, which learns a subnetwork for each client within a depth limit set by its resources. A hierarchical Bayesian prior shares the structural information across the clients, while allowing different local activation probabilities. The proposed method learns these distributions through the variational inference technique, and aggregates the weights over compatible layers. Experiments have been done on the image and text classification under heterogeneous data and model capacities.

Critical strengths
  1. The idea of sharing a prior over the network structure is a reasonable way to let the clients learn from one another without requiring them to use the same subnetwork.

  2. The experiments have covered several image and text classification tasks. In addition, the ablation of the global prior shows where the structure sharing helps, and the results on the computational complexity make its additional training cost visible.

Critical weaknesses
  1. The proposed method uses the well-established Bayesian structure-learning tools, and the convergence analysis follows FIARSE. It is thus unclear what the main technical advance is in adapting these ideas to federated learning? Could the authors analytically explain how the hierarchical structure sharing affects the mask-induced gradient discrepancy?

  2. Two recent referecnes, FedEL (2025) and FedLAGC (2026), are discussed but not included in the experiment comparison. Both of them address the problem of sub-model construction for the heterogeneous devices. A comparison with these two methods seems necessary to validate what the proposed structure sharing additionally introduces.

  3. Table 5 shows a substantial sensitivity to the global prior. The average local accuracy drops from 77.5% to 60.7% when its parameters change from (10, 1) to (10, 10). How can one choose these parameters for a new task?

Questions for authors

Please see the weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer gj6h

We thank Reviewer gj6h for the constructive feedback.

W1: Our main technical advances lie in three aspects as follows:

  • The existing Bayesian structure-learning tools are only for a single neural network. In contrast, we extend it to joint inference for multiple local models’ network structures by introducing a hyper-prior at global level.

    Our hierarchical structure inference operates at two levels. At the client level, each client adapts its own network structure to its local data characteristics within its computational capacity. At the server level, the hyper-prior aggregates these local structures into a global network structure, sharing common and complementary structural information among the clients.

    Our ablation study in Table 4 demonstrates the advantage of sharing structural information via hyper-prior over independent structure adaptation for every single network.

  • We develop an efficient structured variational inference scheme for the computation of the joint inference for the global and local client-specific network structures in model-heterogeneous FL.

    The inference procedure accommodates heterogeneous client model capacities through client-level structure adaptation (Eq. 11) and server-level structural information aggregation (Eq. 12).

  • We further provide theoretical analysis of statistical and algorithmic convergence of our modeling and inference methods.

    Theorem 4.1 characterizes the asymptotic stochastic convergence of the local activation probabilities. Tied to Theorem 4.1, Theorem 4.2 establishes an algorithmic convergence bound per mask sampled from the local activation probabilities. Although we adopt similar assumptions made in FIARSE, we bound our local masked objectives within the stochastic sampling process.

We will clarify the novel contributions in the camera-ready version.

".. How the hierarchical structure sharing affects the mask-induced gradient discrepancy?"

Given structural masks Mc(k)M_c^{(k)}, sampled from local layer-wise activation probabilities which are governed by both the global structural prior and client-specific estimates in the hierarchy (Theorem 4.1), the mask-induced gradient discrepancy in Eqn.(31) is assumed to be bounded for the weights activated by the masks through element-wise multiplication, as in Wc⊙Mc(k)W_c \odot M_c^{(k)}, in order to derive the optimization convergence within the asymptotic sampling process.

W2: As Reviewer gj6h requested, we have conducted additional experiments comparing our proposed framework with FedEL (2025) and FedLAGC (2026) on image classification tasks under both high and low levels of data heterogeneity. The global model test accuracy (%) is reported below. Both baselines use the same backbone and the same client capacity split as in our paper.

DatasetαFedELFedLAGCOurs
CIFAR100.373.5±0.771.9±1.276.3±0.6
CIFAR1010.076.5±0.976.2±0.278.4±0.7
CIFAR1000.344.9±0.545.2±0.550.3±0.5
CIFAR10010.057.3±0.357.9±0.361.7±0.4
Tiny-ImageNet0.330.4±0.830.6±1.133.8±0.3
Tiny-ImageNet10.034.2±0.236.0±0.437.5±0.1

Compared with the stronger of the two baselines in each setting, our method achieves improvements of 2.8 and 1.9 percentage points on CIFAR10, 5.1 and 3.8 points on CIFAR100, and 3.2 and 1.5 points on Tiny-ImageNet, under high and low data heterogeneity, respectively. We will include these results in the camera-ready version.

W3: As described in Appendix C.3, we tuned the global prior only once, using CIFAR-10 under high data heterogeneity, and applied the same setting across all experiments. This indicates that a single setting transfers well to new tasks. The global prior parameters encode an inductive bias over local network structures. Specifically, (10,1) favors deeper network structures, whereas (10,10) favors relatively shallower and wider structures. Setting a prior favoring deeper network structures would be beneficial in general.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.