- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a framework for adapting the width of Bayesian neural networks during training. At each epoch, posterior variance guides a growth proposal, posterior signal-to-noise ratio guides a pruning proposal, and the unchanged architecture forms a third candidate. After brief candidate training, a validation-based, complexity-penalized variational objective selects the next model. Experiments on Fashion-MNIST and CIFAR-10 compare accuracy and Brier score against parameter count using shallow fully connected and convolutional networks. Additional analyses examine pruning alone, retraining the discovered architectures, growth-location selection, growth magnitude, and a class-distribution shift.
Critical strengths
The idea is somewhat interesting. Growth and pruning proposals are evaluated against continued training on the current architecture rather than being applied automatically. This might be a useful design choice, as it permits capacity changes in either direction while explicitly accounting for parameter count and allowing the model to retain its structure.
Critical weaknesses
-
The dataset is too small. The experiment was only validated on small-scale datasets such as CIFAR and MNIST, which have already been extensively studied, and most weights could easily achieve higher accuracy, such as over 95%. This paper did not produce competitive results. Also, no large-scale dataset is tested, such as ImageNet. I feel it is somewhat pointless to develop new architectures on small datasets, given the current evolution of LLMs.
-
The bidirectional adaptation is not convincing. The method combines posterior-based proposals with a complexity-penalized variational selection objective, but the justification for the growth statistic is weak. After within-layer median/MAD normalization, Equation (2) primarily measures the relative shape of the variance distribution rather than its absolute uncertainty level. It is unclear why the selected layer should be the one that needs additional capacity.
-
I am a little bit confused about what happens during candidate training. Not so clear what is the roles of M and M1. Also, which parameters are frozen during isolated growth steps, and whether all their candidates receive the same total number of updates?
Questions for authors
Please see weaknesses above. Along those, which growth rule is implemented? The lowest normalized uncertainty shown in Figure 1, or the highest score specified in Equation (2)?
Response to the Review
We thank the reviewer for their time and for recognizing the value of comparing structural changes with continued training of the existing architecture. We address the specific questions about the growth statistic and candidate training below.
1. Experimental scope. We disagree that the development of LLMs makes structural adaptation in smaller Bayesian networks “pointless.” Learning when to add or remove capacity remains a research question.
We compare methods within the same Bayesian architecture specification and report accuracy against parameter count. Accuracies above 95% from other architectures and training procedures do not establish whether our structural procedure helps or harms the networks we study.
2. Meaning of the growth statistic. The reviewer correctly identifies that the normalized statistic does not measure absolute variance. In Eq. (2), we use the upper tail of the within-layer normalized variance distribution. We make this choice to avoid selecting a layer simply because its posterior variances have a larger overall scale.
We use this statistic to propose where to add capacity. We do not claim that it proves the layer needs more units. After training the candidates, we compare growth with pruning and retention under the same validation objective; we can reject the growth proposal.
In Fig. 3(b), we compare this proposal rule with random selection of the growth layer, while keeping the other components unchanged. We observe a better mean accuracy–parameter-count trade-off for the proposed rule in the tested CNN setting. This provides empirical support for the heuristic in that setting, not a general guarantee that normalized variance identifies insufficient capacity.
3. Candidate training: M and M1. We train each candidate for M warm-start optimization steps. For the first M1 steps of the growth candidate, we update only parameters associated with the new unit, including its incoming and outgoing connections, and freeze the existing parameters. For the remaining M − M1 steps, we update the full model. We update all parameters of the retain and prune candidates throughout their M steps.
M1 is therefore part of M, not an additional training budget. In Table 2, we use (M, M1) = (64, 16) for Fashion-MNIST FNN, (64, 32) for CIFAR-10 FNN and (32, 16) for CIFAR-10 CNN. We use the same training batches across candidates and evaluate them on the same held-out validation batches. We give the three candidates equal numbers of optimization steps.
4. Highest versus lowest. We select the layer with the highest normalized score, as specified by argmax in Eq. (2). Figure 1 incorrectly says “lowest”. We will update it accordingly.