- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces the Self-Organizing Artificial Perceptron (SOAP), a variational framework for jointly learning the width and depth of an MLP during training. SOAP treats the network as unbounded in both width and depth, and uses learnable variational parameters—an exponential importance distribution for neuron ordering and width truncation, and a truncated, discretized centered Skew Normal for depth—to trigger discrete structural additions and removals via quantile thresholds. The model uses a multi-readout architecture and weights local predictions by the depth posterior. The claimed contributions are the variational formulation for joint unbounded width–depth learning, a concrete training procedure driven by backpropagation through continuous structural parameters, and an empirical stability analysis of the learned topologies across initializations, regularization, and batch sizes.
Critical strengths
Strength 1: The joint width–depth formulation is a conceptually meaningful extension of adaptive-width methods. Prior unbounded-network work has largely adapted width while keeping depth fixed, or has relied on external constructive/pruning heuristics. SOAP’s formulation makes both dimensions learnable within a single ELBO-driven training loop, and the multi-readout construction gives a principled way to combine predictions from different active depths. This is the paper’s most consequential contribution and is worth discussing even if the empirical scope is limited.
Strength 2: The stability analysis provides useful qualitative insight into joint structural adaptation. The experiments on SpiralHard examine how initial width, initial depth, regularization strength, and batch size affect converged topologies. The observation that SOAP can adapt primarily along width in some regimes and along depth in others, and that depth convergence is relatively stable across initializations, is a useful empirical characterization that goes beyond reporting a single accuracy number.
Critical weaknesses
Weakness 1: The empirical evaluation is too narrow to support the paper’s broad claims about joint architectural learning and its competitiveness. The experiments are limited to six tabular classification datasets, small MLPs, and only two baselines: a grid-searched fixed MLP and AWN, which adapts width only. There is no comparison with NAS, constructive methods, pruning/distillation, or other adaptive-depth/width approaches discussed in related work. The paper explicitly states that its goal is not SOTA, but the central claim is that joint width–depth learning is a meaningful and stable regime; demonstrating this only on small tabular tasks against two baselines is insufficient to establish that the framework is broadly useful or that joint adaptation is preferable to simpler alternatives. This is a substantive rejection concern because the methodological contribution and the empirical evidence are mismatched in scope.
Weakness 2: The method is complex and relies on several specific distributional and implementation choices whose necessity and robustness are not adequately established. SOAP combines an exponential width-importance distribution, a truncated centered Skew Normal depth posterior, quantile-based truncation, multi-readout weighting, and discrete structural updates with retention/reactivation policies. The paper acknowledges that some choices lack solid theoretical motivation, and the stability analysis is concentrated on one synthetic dataset. In addition, the multi-readout inference evaluates all active readouts, and the paper leaves pruning/renormalization to future work, so the computational cost of the proposed architecture is not fully characterized. The capacity comparisons are also confounded by model selection favoring unregularized configurations, which the authors themselves note. Together, these issues make it difficult to determine whether the reported behavior is due to the core variational mechanism or to a combination of empirical tuning and implementation details. For a methods paper, this weakens confidence in the generality and practical value of the proposed framework.
Questions for authors
-
Why are the empirical comparisons limited to a fixed MLP and AWN? Can you provide comparisons against at least one NAS method, one constructive/growing method, and one pruning-based adaptive method under the same protocol? Without such comparisons, it is difficult to assess whether joint width–depth learning offers advantages over existing architectural adaptation strategies.
-
The paper reports that model selection often favors unregularized configurations, leading to larger SOAP networks than AWN. How do the accuracy and topology results change under a fixed parameter budget or a fixed computational budget? This directly affects whether SOAP’s capacity allocation is meaningful rather than a consequence of allowing larger networks.
-
The stability analysis is primarily on SpiralHard. How stable are the depth and width convergence results on the real-world datasets and on tasks with different input dimensionality or class imbalance? Can you report topology variance across seeds and model-selection runs for all datasets?
-
The depth posterior uses a centered Skew Normal and the width importance uses an exponential distribution. How sensitive are the learned topologies and predictive performance to these distributional choices? For example, what happens with a Poisson or Folded Normal depth posterior under the same centered parameterization, or with a different width-importance family?
-
The multi-readout inference evaluates all active readouts, including those with negligible posterior mass. What is the actual inference cost relative to a fixed MLP of comparable accuracy, and how much would pruning low-mass readouts change both cost and predictive performance?
2 comments
Dear Reviewer MudK,
We wish to thank you again for the opportunity to strengthen our work. We have done our best to address the questions raised with additional empirical evidence, in particular:
- we provide new experiments in Table 1 and in the appendix
- The Pareto frontier plots in Appendix M show that capacity allocation of configurations is indeed meaningful in SOAP, and the frontier can be navigated according to a desired computational budget
- Appendix F shows topological stability on real world datasets as well
- we clarified why the centered parametrization applies to the skew normal distribution and how the distribution solves previous problems
- the appendix now provides ablation studies on the multi-readout structure including the resulting savings in time and parameters
As the rebuttal period draws to a close, we would greatly appreciate any feedback from the Reviewer, and we hope that our responses have sufficiently addressed their concerns to warrant a reconsideration of the score.
Kind regards, the authors
Authors’ Response to Reviewer MudK
We sincerely thank the Reviewer for recognizing the meaningful contribution of our method and the qualitative insights we provided. In what follows we have done our best to address the concerns of the Reviewer.
W1 & Q1: We took action to address the concern of the Reviewer. We have extended the empirical comparison to 4 new tabular datasets and 1 machine translation dataset. These are now reported in Table 1. Our claims seem to still stand.
In addition, compression experiments and comparison with pruning baselines as in Errica et al. 2026 are now reported in Appendix G, showing that SOAP is easy to compress and the compression works generally better and only once on par with magnitude-based pruning.
As discussed in Section 2, and as was the case for Nazaret and Blei and Errica et al., the comparison is not against NAS or other constructive approaches for one reason, which is that those frameworks are complementary to this line of approaches. The main methodological advantage of SOAP is the backpropagation-driven, unbounded topological adaptations, a combination of features that could only be found in UDN and AWN so far. Therefore, the advantage is not a mere matter of empirical results, but it has profound consequences on how we will design and learn neural networks in the future.
Nevertheless, we agreed with the Reviewer that the scope of the empirical evaluation was too narrow, and we hope to have amended that with the new experiments of Table 1 and the additional analyses in the Appendix, which are highlighted in blue text.
We thank the Reviewer for the strong encouragement to strengthen the paper in this direction, which we believe has substantially improved the empirical support of our claims.
W2: While we believe SOAP is simple to implement in practice (and many higher-level hyper-parameters do not play a role in topological convergence, as shown in Appendices F and H), the added complexity of SOAP can be seen the price to pay for the advantages discussed above, namely automatizing model design. The distributional choices were already present in AWN and UDN, and our centered Skew Normal proposal theoretically improved on limitations of previous depth distributions.
Our additional analyses now address the Reviewer’s points: i) we expanded the stability analyses to multiple real world datasets in Appendix F; ii) we investigated multi-readout effect in Appendix G; iii) we compared against pruning baselines in Appendix G; iv) the computational costs are now described in Appendix L; v) finally, we remove the confounding factor of the model selection analysis by investigating Pareto frontiers of model selection in Appendix M, showing there exist much more efficient configurations at negligible cost that an expert could have chosen instead of relying on an automatic model selection procedure based on validation accuracy. Taken together, we hope this would increase the Reviewer’s confidence in the practical value of our proposed approach.
Once more, we thank the Reviewer for pointing out specific aspects to address. We are very glad to have been given the change to work on this and update the paper accordingly. We feel the paper is now much better supported in terms of empirical evidence.
Q2: We show a nice Pareto frontier between accuracy and topology complexity in Appendix M, as mentioned earlier, which we can consider as imposing constraints on the parameter budget. SOAP finds architectures with negligible performance differences but much reduced parameter counts.
Q3: Appendix F now reports the stability analysis for all real-world datasets as well, showing general topological convergence also in this case with the exception of a few runs.
Q4: We are not aware of a way to apply the centered parametrization to the Folded Normal, whereas the Poisson does not need it since it is parametrized by a single value.
Errica et al. 2026 already provide an analysis for the width distribution, which we did not duplicate, showing how the exponential implements a good trade-off between performance and learned neurons. For the depth, the Poisson suffers from equidispersion, meaning the variance is tied to the mean and can generate large depths, and the Folded Normal collapses when the mean goes to 0, generating larger depths by flattening the distribution, while the SN distribution we propose solves both issues (please refer to Appendix D.4 for a discussion).
Q5: This analysis is now provided in Appendix G.1 for MLPs and G.1 for Transformers, in particular Figures 17, 18, 21 and 22.
We thank the Reviewer for the constructive criticism and we hope that our new experiments give a better feeling that this method is robust enough to be applied in practice. We would greatly appreciate it if the Reviewer could consider revising their score in light of our efforts.