acceptodds
Under review as a conference paper at ICLR 2027

Phase-Free Bidirectional Structural Adaptation in Bayesian Neural Networks

Abstract

Biological neural systems continuously adapt their structure through the formation and elimination of neural connections, whereas artificial neural networks typically retain a fixed architecture throughout training. Amongst existing structural adaptation approaches, most rely on pruning rather than growth, with the combination of the two being the exception, typically relying on pre-defined adaptation phases. We propose, to the best of our knowledge, the first phase-free bidirectional structural adaptation algorithm that leverages parameter statistics in a Bayesian neural network framework to control optional growth or pruning steps through a single unified criterion. Concretely, candidates for growth and pruning are proposed based on weight-level uncertainty and signal-to-noise-ratio, while candidate selection follows a common penalised evidence lower bound that explicitly accounts for model complexity. This enables network capacity to continuously increase, decrease, or remain unchanged throughout training. We evaluate the proposed framework on fully connected and convolutional architectures using Fashion-MNIST and CIFAR-10. Across our experiments, the method achieves favourable performance–complexity trade-offs compared with static architectures and dynamic baselines that rely on predefined growth and pruning schedules or separate structural criteria.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Bayesian neural networks, Neural Network Pruning, Neural Network Growth, Neuroplasticity

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_fSG5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a framework for adapting the width of Bayesian neural networks during training. At each epoch, posterior variance guides a growth proposal, posterior signal-to-noise ratio guides a pruning proposal, and the unchanged architecture forms a third candidate. After brief candidate training, a validation-based, complexity-penalized variational objective selects the next model. Experiments on Fashion-MNIST and CIFAR-10 compare accuracy and Brier score against parameter count using shallow fully connected and convolutional networks. Additional analyses examine pruning alone, retraining the discovered architectures, growth-location selection, growth magnitude, and a class-distribution shift.

Critical strengths

The idea is somewhat interesting. Growth and pruning proposals are evaluated against continued training on the current architecture rather than being applied automatically. This might be a useful design choice, as it permits capacity changes in either direction while explicitly accounting for parameter count and allowing the model to retain its structure.

Critical weaknesses
  1. The dataset is too small. The experiment was only validated on small-scale datasets such as CIFAR and MNIST, which have already been extensively studied, and most weights could easily achieve higher accuracy, such as over 95%. This paper did not produce competitive results. Also, no large-scale dataset is tested, such as ImageNet. I feel it is somewhat pointless to develop new architectures on small datasets, given the current evolution of LLMs.

  2. The bidirectional adaptation is not convincing. The method combines posterior-based proposals with a complexity-penalized variational selection objective, but the justification for the growth statistic is weak. After within-layer median/MAD normalization, Equation (2) primarily measures the relative shape of the variance distribution rather than its absolute uncertainty level. It is unclear why the selected layer should be the one that needs additional capacity.

  3. I am a little bit confused about what happens during candidate training. Not so clear what is the roles of M and M1. Also, which parameters are frozen during isolated growth steps, and whether all their candidates receive the same total number of updates?

Questions for authors

Please see weaknesses above. Along those, which growth rule is implemented? The lowest normalized uncertainty shown in Figure 1, or the highest score specified in Equation (2)?

1 comment
AuthorsOpenReview ↗

Response to the Review

We thank the reviewer for their time and for recognizing the value of comparing structural changes with continued training of the existing architecture. We address the specific questions about the growth statistic and candidate training below.

1. Experimental scope. We disagree that the development of LLMs makes structural adaptation in smaller Bayesian networks “pointless.” Learning when to add or remove capacity remains a research question.

We compare methods within the same Bayesian architecture specification and report accuracy against parameter count. Accuracies above 95% from other architectures and training procedures do not establish whether our structural procedure helps or harms the networks we study.

2. Meaning of the growth statistic. The reviewer correctly identifies that the normalized statistic does not measure absolute variance. In Eq. (2), we use the upper tail of the within-layer normalized variance distribution. We make this choice to avoid selecting a layer simply because its posterior variances have a larger overall scale.

We use this statistic to propose where to add capacity. We do not claim that it proves the layer needs more units. After training the candidates, we compare growth with pruning and retention under the same validation objective; we can reject the growth proposal.

In Fig. 3(b), we compare this proposal rule with random selection of the growth layer, while keeping the other components unchanged. We observe a better mean accuracy–parameter-count trade-off for the proposed rule in the tested CNN setting. This provides empirical support for the heuristic in that setting, not a general guarantee that normalized variance identifies insufficient capacity.

3. Candidate training: M and M1. We train each candidate for M warm-start optimization steps. For the first M1 steps of the growth candidate, we update only parameters associated with the new unit, including its incoming and outgoing connections, and freeze the existing parameters. For the remaining M − M1 steps, we update the full model. We update all parameters of the retain and prune candidates throughout their M steps.

M1 is therefore part of M, not an additional training budget. In Table 2, we use (M, M1) = (64, 16) for Fashion-MNIST FNN, (64, 32) for CIFAR-10 FNN and (32, 16) for CIFAR-10 CNN. We use the same training batches across candidates and evaluate them on the same held-out validation batches. We give the three candidates equal numbers of optimization steps.

4. Highest versus lowest. We select the layer with the highest normalized score, as specified by argmax in Eq. (2). Figure 1 incorrectly says “lowest”. We will update it accordingly.

Reviewer_QQZ1OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a method for dynamically adjusting the width of Bayesian neural networks during training, letting the network grow, shrink, or stay the same at every epoch. Its novelty is described as being due to its ‘phase free’ growth and pruning, unlike alternative approaches which often follow a schedule for growth or pruning phases. Units’ posterior weight uncertainties and signal-to-noise ratios are used as the criterion for weighing where to prune/grow. A grown, pruned, or retained network structure is chosen based upon the building of three corresponding candidate models after each epoch: one that adds a unit, one that removes a unit(s), and one that keeps the network as is. Each candidate is thereafter trained on shared mini-batches and a winner is chosen based upon the best score on a KL-weighted ELBO plus an explicit per-parameter (architecture size) penalty. On two-layer fully connected networks (Fashion-MNIST, CIFAR-10) and a small CNN (CIFAR-10), the authors report slightly better accuracy vs parameter-count trade-offs than static networks and Bayesian adaptations of two grow-and-prune (phase based) methods.

Critical strengths
  1. The use of one common score for all structural decisions is a neat and ideal approach. Even though different heuristics are used to produce the candidates, every candidate is judged on the same scale. Furthermore the λ term gives users one interpretable knob for the size-versus-performance trade-off, rather than separate growth and pruning thresholds.

  2. The “retain” candidate is really important, it allows the network to only change if its growth/prune actually beats the current model. This helps to avoid unnecessary modification of the model.

Critical weaknesses
  1. Although this method is described as phase-free, the appendix seems to show that it goes through a serious pruning phase before a steady phase. It also seems that your results show that the method can prune well but there is no clear evidence that it can grow well (i.e. starting with an undersized model). Thus, the evidence so far seems to suggest that this method is primarily a pruning method
  2. The empirical evidence is very limited. The CIFAR-10 models have accuracy around 44% for the MLP and 72% for the CNN and this is far below the level to which they are regularly trained in machine learning paper and benchmarks. Improvements are a few tenths of a percent and its not clear that these methods really are improving the models much (error bars would be appreciated and NeST even appears to win in some small network cases). Thus, the real efficacy of this method of as a structure search is not demonstrated.
Questions for authors
  • If applied to an extra small model, does growth of the models nodes dominate and raise the model size? If yes, does it converge to a similar accuracy (or model size) as one arrives at from a model which is too large/wide being automatically pruned?

  • In Figure 1, 2a says that the "lowest normalized uncertainty" layer grows a unit, but the text says that it is the layer with largest (L251, argmax) normalized uncertainty grows a unit. Would be good to clarify this as these are contradictory.

  • It seems that your heuristics for choosing a layer for node growth and node pruning could easily choose the same layer, is that right? Can you indicate how often the same layer is selected for growth and pruning?

  • The method currently has a limited set of baseline comparisons - could it be compared to a more recent attempt such as that by Erica et al. 2026 (https://arxiv.org/abs/2501.15889)?

1 comment
AuthorsOpenReview ↗

Response to the Review

We thank the reviewer for their time and for identifying the value of the common selection score and the retain candidate. The questions about small initial networks and Adaptive Width Networks help us state the contribution more precisely.

1. Phase-free adaptation and growth. By “phase-free,” we mean that we allow growth, pruning and retention at every structural decision. We do not prescribe when the network must grow or shrink. An early sequence of pruning decisions does not contradict this definition.

We deliberately start from overparameterized networks, as explained in Section 4. We also remove a fraction of units when pruning but add only one unit when growing. These choices explain why compression dominates the trajectories. Figures 8–10 nevertheless record growth interleaved with pruning and retention. In Fig. 3(a), we compare the full method with Prune-Only and observe a small additional mean benefit from allowing growth.

Growth from a substantially undersized network remains a limitation. In preliminary trials, we did not reliably grow such networks to a size with competitive performance. We cannot claim that small and large initializations converge to similar architectures or accuracies. Our evidence concerns bidirectional refinement from initially overparameterized networks.

2. Accuracy and baselines. Comparing our absolute accuracies with results from different architectures and training procedures does not isolate the effect of structural adaptation. Within each setting, we use the same Bayesian architecture specification across methods and compare accuracy as a function of parameter count.

We explicitly state in Section 5 that NeST remains competitive, particularly at lower parameter counts. We do not claim that our method wins at every size. We repeat configurations five times, but the main plots omit dispersion.

3. Growth direction. We select the layer with the highest score, as specified by argmax in Eq. (2). Figure 1 incorrectly says “lowest” we will adjust it to highest.

4. Growth and pruning in the same layer. We select a layer for the growth proposal, but we rank units globally for the pruning proposal. We may therefore propose adding capacity to a layer while also identifying low-SNR units in that layer for removal. We evaluate these as alternative architectures and select one action using the common validation score.

5. Adaptive Width Networks. Errica et al. is relevant related work. They learn width parameters jointly with network weights through backpropagation and rescale neuron activations according to an importance ordering. We instead construct discrete growth, pruning and retention candidates from posterior statistics, train them briefly, and compare them using a penalized validation score. Our contribution is about the proposal and selection procedure.

Reviewer_Y4LTOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This research paper tries to solve the elasticity of a neural network in contrast to fixed networks, thereby reducing computational costs by pruning or growing the weights or parameters based on a novel Bayesian approach.

Critical strengths

The paper tries to solve the elasticity of a neural network in contrast to fixed networks, thereby reducing computational costs by pruning or growing depending on the signal to noise ratio. The paper claims to introduce a novel Bayesian bidirectional approach which is quite commendable.

Critical weaknesses

The manuscript presents a new, potentially useful method for proposing structured growth, pruning, or retention in Bayesian neural networks. Few concerns hinder the interpretation and the novel idea as proposed in the paper is that the score used in the experiments is vague at mini-batch scale, while the appendix’s evidence-bound derivation does not justify the KL weighting used in those experiments. These points affect the interpretation and reproducibility of the unified loss criterion.

The empirical comparisons are promising as descriptive results, but the strength of the superiority and efficiency claims should be calibrated to the reported metrics and uncertainty.

The paper’s principal claim of a Bayesian, common evidence-based criterion depends on what objective is actually optimized and compared. In Eq 1 in Section 3.1 without an architecture prior, the joint-posterior interpretation is somewhat incomplete.

In the experiments use β = 0.002 or β = 0.01 (Table 2). As the draft itself notes below Eq. (1), values other than 1 give a KL- weighted variational objective, not the usual evidence lower bound. Thus the empirical score is not the ELBO as derived in Appendix A, which changes the narrative of the study.

In section 3.3, what if training and validation batches are not shared? By sharing the training and validation batches the model is trained with an inductive bias. Although the rationale is sound, the authors are encouraged to be transparent about reporting the results when the training and validation batches are kept separate.

In the entire architecture setup the principle guide is Equation 1, however the architecture prior is needed when talking about joint-posterior interpretation. One can only find it in the Appendix.

In the architecture setup the normalization and activation should be experimented on. Are the architectures dependent on the activation function? Further study is needed justifying the choice of activation functions.

About the priors in the architecture; we find mention of the same in page 7, however experimentation on other priors (than the standard normal prior) needs to be studied for robust analysis.

Whilst the mean of the evaluated configurations are reported in the graph, the standard deviation or confidence intervals are missing.

In Table 2 of Appendix the choice of β should have been justified. OR at the very least various choices of β should have been reported for better understanding the model behavior.

The Distribution Shift study is a a bit of gray area, because in the datasets studied in the experiments does not mention how the distribution shifts gets achieved. All the datasets are fairly standard and have equal distribution in the target classes in the train, val and test partitions. So the claims demonstrated in the Brier score complexity in Appendix C poses serious questions.

One of my major concerns is that the authors do not evaluate their method on ImageNet. Evaluations on Fashion-MNIST and CIFAR-10 alone are insufficient to establish the effectiveness and scalability of the proposed approach. The authors should include experiments on ImageNet to enable a more meaningful comparison with existing approaches and demonstrate the effectiveness of their method on a larger, more challenging dataset.

Questions for authors

How do you conduct Phase 1 and Phase 2 training?

Why just Image Classification tasks, why not Language training tasks or audio processing?

1 comment
AuthorsOpenReview ↗

Response to Review

We thank the reviewer for their time and detailed comments on the objective and experimental protocol, and for recognizing the potential of Bayesian bidirectional adaptation.

1. Objective and architecture prior. We define the architecture prior in Appendix A.2 as p(A) ∝ exp[−λN(A)]. With a point-mass approximation over architectures, the architecture-level KL contributes λN(A), up to an additive constant. The prior is therefore present in the derivation, although we should introduce it alongside Eq. (1).

We also state immediately below Eq. (1) that β = 1 gives the usual ELBO, whereas other values give a KL-weighted variational objective. Appendix A derives the ordinary bound before introducing β. We should therefore describe the experimental objective consistently as a complexity-penalized, KL-weighted variational score. We cannot interpret it as the ordinary ELBO for arbitrary β. This distinction does not change how we compare growth, pruning and retention: we evaluate all three with the same score.

In Eq. (5), we average that score over K validation mini-batches, using the same β and λ for every candidate. We should specify the mini-batch likelihood normalization and corresponding KL scaling more precisely.

2. Training and validation batches. We share training batches across the three candidates and, separately, share validation batches across them. We optimize parameters on the training batches and evaluate candidates on held-out validation batches. “Shared” does not mean that we use the same examples for training and validation. The reviewer’s proposed separation is already part of the procedure.

3. Distribution shift and the two training phases. We describe the shift in Section 5, p. 8. In Phase 1, approximately 90% of training examples come from Trouser, Sandal, Sneaker, Bag and Ankle boot, and 10% from the remaining five classes. In Phase 2, we reverse these proportions. We then use the validation-selected Phase-1 weight posterior as the Phase-2 weight prior.

After Phase 2, we evaluate on the full balanced Fashion-MNIST test set. In Appendix E, we also evaluate on a test distribution with the Phase-1 class proportions. The fact that the original dataset is balanced does not preclude this constructed shift. These phases describe changes in the data distribution; we do not prescribe separate growth and pruning phases.

4. Evidence and scope. We repeat each configuration five times, as stated in Section 4. We report means in the main performance figures. We measure parameter count, so the results support claims about parameter efficiency, not measured training-time savings.

We use the same activation, normalization and weight-prior choices across methods within each setting.

We evaluate shallow Bayesian networks on Fashion-MNIST and CIFAR-10. ImageNet, language and audio experiments are an intersting application for future work. However we propose a new training algorithm, which is independent of the underlying task.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.