acceptodds
Under review as a conference paper at ICLR 2027

Self-Organizing Artificial Perceptron: Backpropagating through the Width and Depth of Neural Networks

Abstract

In machine learning, the hypotheses' space of neural networks is encoded by their architecture, and it cannot change during training or inference in response to the task. As discrete architectural changes are not differentiable, neural networks are essentially unable to self-organize their knowledge and computational structure. This work introduces the Self-Organizing Artificial Perceptron (SOAP), a multi-layer perceptron (MLP) that can jointly learn its architecture, namely depth, width, and the associated parameters via standard backpropagation. Crucially, we do not impose upper bounds on depth or width, meaning the network is formally infinite but practically finite, in the style of Bayesian models. The underlying idea is to make the addition and removal of new layers and neurons compatible with the gradient signal. We test the joint learning of width and depth on 6 tabular datasets, showing SOAP maintains or exceeds the performance of MLPs. In addition, we analyze the behavior of depth and width in response to different batch sizes and starting conditions. Our contribution paves the way for new investigations about a new class of model whose inductive bias adapts to the task's complexity.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-organizing neural networks, adaptive width, adaptive depth, variational inference, multi-layer perceptron

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_MudKOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces the Self-Organizing Artificial Perceptron (SOAP), a variational framework for jointly learning the width and depth of an MLP during training. SOAP treats the network as unbounded in both width and depth, and uses learnable variational parameters—an exponential importance distribution for neuron ordering and width truncation, and a truncated, discretized centered Skew Normal for depth—to trigger discrete structural additions and removals via quantile thresholds. The model uses a multi-readout architecture and weights local predictions by the depth posterior. The claimed contributions are the variational formulation for joint unbounded width–depth learning, a concrete training procedure driven by backpropagation through continuous structural parameters, and an empirical stability analysis of the learned topologies across initializations, regularization, and batch sizes.

Critical strengths

Strength 1: The joint width–depth formulation is a conceptually meaningful extension of adaptive-width methods. Prior unbounded-network work has largely adapted width while keeping depth fixed, or has relied on external constructive/pruning heuristics. SOAP’s formulation makes both dimensions learnable within a single ELBO-driven training loop, and the multi-readout construction gives a principled way to combine predictions from different active depths. This is the paper’s most consequential contribution and is worth discussing even if the empirical scope is limited.

Strength 2: The stability analysis provides useful qualitative insight into joint structural adaptation. The experiments on SpiralHard examine how initial width, initial depth, regularization strength, and batch size affect converged topologies. The observation that SOAP can adapt primarily along width in some regimes and along depth in others, and that depth convergence is relatively stable across initializations, is a useful empirical characterization that goes beyond reporting a single accuracy number.

Critical weaknesses

Weakness 1: The empirical evaluation is too narrow to support the paper’s broad claims about joint architectural learning and its competitiveness. The experiments are limited to six tabular classification datasets, small MLPs, and only two baselines: a grid-searched fixed MLP and AWN, which adapts width only. There is no comparison with NAS, constructive methods, pruning/distillation, or other adaptive-depth/width approaches discussed in related work. The paper explicitly states that its goal is not SOTA, but the central claim is that joint width–depth learning is a meaningful and stable regime; demonstrating this only on small tabular tasks against two baselines is insufficient to establish that the framework is broadly useful or that joint adaptation is preferable to simpler alternatives. This is a substantive rejection concern because the methodological contribution and the empirical evidence are mismatched in scope.

Weakness 2: The method is complex and relies on several specific distributional and implementation choices whose necessity and robustness are not adequately established. SOAP combines an exponential width-importance distribution, a truncated centered Skew Normal depth posterior, quantile-based truncation, multi-readout weighting, and discrete structural updates with retention/reactivation policies. The paper acknowledges that some choices lack solid theoretical motivation, and the stability analysis is concentrated on one synthetic dataset. In addition, the multi-readout inference evaluates all active readouts, and the paper leaves pruning/renormalization to future work, so the computational cost of the proposed architecture is not fully characterized. The capacity comparisons are also confounded by model selection favoring unregularized configurations, which the authors themselves note. Together, these issues make it difficult to determine whether the reported behavior is due to the core variational mechanism or to a combination of empirical tuning and implementation details. For a methods paper, this weakens confidence in the generality and practical value of the proposed framework.

Questions for authors
  1. Why are the empirical comparisons limited to a fixed MLP and AWN? Can you provide comparisons against at least one NAS method, one constructive/growing method, and one pruning-based adaptive method under the same protocol? Without such comparisons, it is difficult to assess whether joint width–depth learning offers advantages over existing architectural adaptation strategies.

  2. The paper reports that model selection often favors unregularized configurations, leading to larger SOAP networks than AWN. How do the accuracy and topology results change under a fixed parameter budget or a fixed computational budget? This directly affects whether SOAP’s capacity allocation is meaningful rather than a consequence of allowing larger networks.

  3. The stability analysis is primarily on SpiralHard. How stable are the depth and width convergence results on the real-world datasets and on tasks with different input dimensionality or class imbalance? Can you report topology variance across seeds and model-selection runs for all datasets?

  4. The depth posterior uses a centered Skew Normal and the width importance uses an exponential distribution. How sensitive are the learned topologies and predictive performance to these distributional choices? For example, what happens with a Poisson or Folded Normal depth posterior under the same centered parameterization, or with a different width-importance family?

  5. The multi-readout inference evaluates all active readouts, including those with negligible posterior mass. What is the actual inference cost relative to a fixed MLP of comparable accuracy, and how much would pruning low-mass readouts change both cost and predictive performance?

2 comments
AuthorsOpenReview ↗

Authors’ Response to Reviewer MudK

We sincerely thank the Reviewer for recognizing the meaningful contribution of our method and the qualitative insights we provided. In what follows we have done our best to address the concerns of the Reviewer.

W1 & Q1: We took action to address the concern of the Reviewer. We have extended the empirical comparison to 4 new tabular datasets and 1 machine translation dataset. These are now reported in Table 1. Our claims seem to still stand.

In addition, compression experiments and comparison with pruning baselines as in Errica et al. 2026 are now reported in Appendix G, showing that SOAP is easy to compress and the compression works generally better and only once on par with magnitude-based pruning.

As discussed in Section 2, and as was the case for Nazaret and Blei and Errica et al., the comparison is not against NAS or other constructive approaches for one reason, which is that those frameworks are complementary to this line of approaches. The main methodological advantage of SOAP is the backpropagation-driven, unbounded topological adaptations, a combination of features that could only be found in UDN and AWN so far. Therefore, the advantage is not a mere matter of empirical results, but it has profound consequences on how we will design and learn neural networks in the future.

Nevertheless, we agreed with the Reviewer that the scope of the empirical evaluation was too narrow, and we hope to have amended that with the new experiments of Table 1 and the additional analyses in the Appendix, which are highlighted in blue text.

We thank the Reviewer for the strong encouragement to strengthen the paper in this direction, which we believe has substantially improved the empirical support of our claims.

W2: While we believe SOAP is simple to implement in practice (and many higher-level hyper-parameters do not play a role in topological convergence, as shown in Appendices F and H), the added complexity of SOAP can be seen the price to pay for the advantages discussed above, namely automatizing model design. The distributional choices were already present in AWN and UDN, and our centered Skew Normal proposal theoretically improved on limitations of previous depth distributions.

Our additional analyses now address the Reviewer’s points: i) we expanded the stability analyses to multiple real world datasets in Appendix F; ii) we investigated multi-readout effect in Appendix G; iii) we compared against pruning baselines in Appendix G; iv) the computational costs are now described in Appendix L; v) finally, we remove the confounding factor of the model selection analysis by investigating Pareto frontiers of model selection in Appendix M, showing there exist much more efficient configurations at negligible cost that an expert could have chosen instead of relying on an automatic model selection procedure based on validation accuracy. Taken together, we hope this would increase the Reviewer’s confidence in the practical value of our proposed approach.

Once more, we thank the Reviewer for pointing out specific aspects to address. We are very glad to have been given the change to work on this and update the paper accordingly. We feel the paper is now much better supported in terms of empirical evidence.

Q2: We show a nice Pareto frontier between accuracy and topology complexity in Appendix M, as mentioned earlier, which we can consider as imposing constraints on the parameter budget. SOAP finds architectures with negligible performance differences but much reduced parameter counts.

Q3: Appendix F now reports the stability analysis for all real-world datasets as well, showing general topological convergence also in this case with the exception of a few runs.

Q4: We are not aware of a way to apply the centered parametrization to the Folded Normal, whereas the Poisson does not need it since it is parametrized by a single value.

Errica et al. 2026 already provide an analysis for the width distribution, which we did not duplicate, showing how the exponential implements a good trade-off between performance and learned neurons. For the depth, the Poisson suffers from equidispersion, meaning the variance is tied to the mean and can generate large depths, and the Folded Normal collapses when the mean goes to 0, generating larger depths by flattening the distribution, while the SN distribution we propose solves both issues (please refer to Appendix D.4 for a discussion).

Q5: This analysis is now provided in Appendix G.1 for MLPs and G.1 for Transformers, in particular Figures 17, 18, 21 and 22.


We thank the Reviewer for the constructive criticism and we hope that our new experiments give a better feeling that this method is robust enough to be applied in practice. We would greatly appreciate it if the Reviewer could consider revising their score in light of our efforts.

Dear Reviewer MudK,

We wish to thank you again for the opportunity to strengthen our work. We have done our best to address the questions raised with additional empirical evidence, in particular:

  • we provide new experiments in Table 1 and in the appendix
  • The Pareto frontier plots in Appendix M show that capacity allocation of configurations is indeed meaningful in SOAP, and the frontier can be navigated according to a desired computational budget
  • Appendix F shows topological stability on real world datasets as well
  • we clarified why the centered parametrization applies to the skew normal distribution and how the distribution solves previous problems
  • the appendix now provides ablation studies on the multi-readout structure including the resulting savings in time and parameters

As the rebuttal period draws to a close, we would greatly appreciate any feedback from the Reviewer, and we hope that our responses have sufficiently addressed their concerns to warrant a reconsideration of the score.

Kind regards, the authors

Reviewer_srDLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes the Self-Organizing Artificial Perceptron (SOAP), a neural network framework that can jointly adapt its width and depth during training through gradient-based optimization. By introducing a variational formulation with learnable distributions governing network width and depth, the authors enable the addition and removal of neurons and layers in response to the training objective.

The authors evaluate SOAP on six binary classification datasets and compare its predictive performance and learned architectures with fixed multilayer perceptrons and Adaptive Width Networks (AWN). They also investigate the convergence of the learned architectures under different initial conditions and hyperparameter settings.

The reviewer finds the proposed framework interesting, particularly its ability to incorporate both width and depth adaptation into the backpropagation-based learning process. The reviewer also appreciates that the authors do not aim to achieve state-of-the-art predictive performance, but rather to demonstrate the feasibility of joint architectural adaptation.

However, several questions remain concerning the advantages of the proposed approach over existing self-organizing networks, the efficiency of the learned architectures, their sensitivity to initial conditions, and the generality of the experimental findings.

Critical strengths
  1. The proposed mechanism for jointly adapting network width and depth through gradient-based optimization is interesting and technically appealing. In particular, the use of a variational formulation to make architectural adaptation compatible with standard backpropagation is a meaningful methodological contribution.

  2. The proposed framework does not require the final network width or depth to be specified in advance. This provides a flexible approach to network construction in which the architecture can evolve during training.

  3. The authors provide experiments investigating the convergence of the learned network structures under different initial conditions and hyperparameter settings. These analyses are useful for understanding the behavior of the proposed architectural adaptation mechanism.

Critical weaknesses

The reviewer finds the proposed approach interesting and does not identify a fundamental flaw in its formulation. However, several concerns should be addressed to establish its advantages and practical applicability more convincingly.

  1. Advantages over existing self-organizing networks

The authors should clarify the advantages of SOAP over existing self-organizing and constructive neural network approaches.

For example, Self-Organizing Incremental Neural Networks (SOINN; Shen and Hasegawa, 2006) can incrementally add and remove neurons in response to the characteristics of the input data. Although SOINN and SOAP address different learning problems and employ different learning mechanisms, a discussion of their respective approaches to architectural adaptation would help clarify the distinctive contribution of SOAP.

In particular, the reviewer would appreciate a clearer explanation of the practical advantages of gradient-driven architectural adaptation, beyond the ability to modify network structure during training.

  1. Efficiency of the discovered network architectures

The comparison with AWN is not entirely convincing in terms of network efficiency.

As shown in Table 2, SOAP tends to generate substantially larger networks than AWN on most benchmark datasets, despite achieving comparable predictive accuracy in several cases.

The authors acknowledge that the larger architectures are partly attributable to the model-selection procedure and the absence of weight regularization in some selected configurations. Nevertheless, this raises an important question concerning the efficiency of the proposed self-organization mechanism.

The reviewer believes that the efficiency of the discovered architectures remains important, especially against other networks with similar performance.

A more systematic comparison of network complexity, including the total number of trainable parameters and computational cost, would strengthen the evaluation. Comparisons under similar accuracy and regularization conditions would be particularly informative.

  1. Sensitivity to initial structures and potential overfitting

The authors investigate the sensitivity of the learned architectures to initial conditions in Figures 3 and 4. However, these results also indicate that the final network structures can depend on the initial width, depth, and regularization settings.

The reviewer would appreciate further discussion of how strongly the discovered architectures are determined by task complexity rather than initial conditions and hyperparameter choices.

Furthermore, given that SOAP sometimes produces considerably larger networks than the baselines, the potential effects of excessive network growth on generalization should be investigated. In particular, it would be useful to examine whether the learned network complexity remains appropriate when the amount of available training data is limited.

  1. Limited scope of benchmark problems

The experimental evaluation is restricted to six binary classification datasets, three of which are synthetic.

Although these experiments demonstrate the feasibility of the proposed method, they provide limited evidence of its scalability and general applicability to more complex learning problems.

The reviewer suggests including at least one more challenging benchmark, such as a multiclass classification problem with higher-dimensional inputs, and reporting how the learned architecture and computational requirements change as problem complexity increases.

Questions for authors
  1. Advantages of gradient-driven self-organization

What are the principal practical advantages of SOAP over earlier self-organizing or constructive neural networks, beyond the ability to adapt width and depth through gradient-based optimization?

In particular, does the proposed mechanism offer advantages in terms of architectural efficiency, convergence, stability, or generalization?

  1. Network efficiency

Table 2 indicates that SOAP often produces substantially larger networks than AWN. How should this result be interpreted in the context of self-organizing network architectures?

Can the authors provide a comparison of total trainable parameters, training and inference costs, or accuracy–complexity trade-offs under comparable regularization conditions?

  1. Dependence on initial architecture

Figures 3 and 4 show that initial width, depth, and regularization can influence the final network architecture.

To what extent does SOAP discover a network structure determined by the learning task, rather than by the initial configuration and hyperparameter settings?

  1. Generalization and overfitting

How does SOAP behave when the training dataset is relatively small? Does the proposed architectural adaptation mechanism effectively control unnecessary network growth and the associated risk of overfitting?

  1. Scalability to more complex problems

Can the authors provide additional evidence or discussion regarding the scalability of SOAP to more complex problems, particularly multiclass classification and high-dimensional datasets?

How do the computational cost and learned network complexity change as the difficulty of the learning task increases?

  1. Architecture selection for the fixed MLP baseline

In Section 4, the authors state that the width and depth of the fixed MLP baselines were selected through grid search, with the results taken from Errica et al. (2026).

Could the authors clarify the exact grid-search procedure used to determine these architectures? In particular, what ranges of widths and depths were considered, what criteria were used for selecting the final architectures, and how did the search budget compare with that used for SOAP?

Since the efficiency of the learned architectures is an important aspect of this study, clarifying the baseline architecture-selection procedure would help establish the fairness of the comparisons reported in Tables 1 and 2.

5 comments
AuthorsOpenReview ↗

Authors’ Response to Reviewer srDL (Part 1)

We deeply thank the Reviewer for the detailed review, which highlights both the strengths of our approach correctly and gives us an opportunity to strengthen the work by addressing the Reviewer’s concerns. We have done our best to provide an answer to all points raised.

W1 & Q1: We thank the Reviewer for bringing SOINN to our attention, which is now covered (both the 2006 and 2010 versions) in Section 2. To the best of our understanding, SOINN focuses on criteria to incrementally learn under non-stationary data and learns a potentially unbounded number of nodes to tackle this scenario.

We agree that architectural adaptation is not unique to SOAP as we discussed in Section 2. We believe that the advantage of relying on gradient-based structural adaptation, in the long run, will be a significant reduction of the effort needed by practitioners into designing ad-hoc procedures for topological adaptation and reducing the time spent designing a suitable model, including those that in the past dealt with topological adaptations by human-defined criteria. We thank the Reviewer for encouraging us to reflect more on this point; we have revised our Conclusions accordingly.

Our Figures 3 and 4, and now Appendix F, show that, as it happened with prior AWN work, SOAP enjoys convergence to similar topologies on multiple datasets, especially under a mild weight regularization. As discussed in prior work, weight regularization helps encourage an ordering of neurons, thus favoring stable convergence.

In terms of generalization, SOAP enjoys equal or better performance compared to baselines, as further confirmed by the 5 new and more challenging experiments of Table 1.

As regards architectural efficiency, we show that there exist model selection configurations of SOAP that are extremely more efficient at negligible performance loss than the ones selected automatically using validation accuracy only (Appendix M). In addition, SOAP is prone to strong compression (Appendix G) and tends to adapt well to the difficulty of the task (Appendix H) as well as to the amount of training data (Appendix J).

We hope our analyses will be sufficient to convince the Reviewer of SOAP’s practical and attractive advantages compared to manually searching for architectures.

W2 & Q2: We understand the concern of the Reviewer. Appendix M now shows that SOAP inductive biases are not necessarily the reason for the larger architectures observed in Table 2. Indeed, we observed how in many cases there exist model selection configurations on the Pareto frontier of accuracy vs parameters that we could have selected at negligible performance loss. This means that Table 2 results are a consequence of an automatic model selection mechanism more than some effect of SOAP’s design choices.

To satisfy the Reviewer’s requests, we included:

  • analysis of trainable parameters and computational costs (Appendices G for compression and accuracy-complexity trade-offs, Appendix L for time comparisons, and Appendix M for Pareto frontier analyses)
  • additional hyper-parameter comparisons in Appendix F

W3 & Q3: Appendix H now investigates more in depth the relation between learned architectures and task complexity, showing clear trends. Appendix F now shows that convergence regardless of initial conditions is achieved on most real world datasets as well. The concern of the Reviewer about learning larger networks is addressed, as mentioned earlier, mainly in Appendix M, where we show how automatic model selection process selected architectures larger than necessary. Finally, our Appendix J now investigate the limited data regime, showing SOAP shrinks when the amount of data is reduced. Therefore, SOAP does not seem to produce unnecessarily large architectures in the end. There is a tendency to grow in width and depth as more data is used, but at some point the model’s width stabilizes. This is consistent with the exponential width distribution playing the role of the “simplicity bias” as discussed in [1]

[1] Wilson, Andrew Gordon. "Deep learning is not so mysterious or different." ICML 2025.

W4 & Q5: We thank the Reviewer for suggesting this Tables 1 and 2 have been extended with 4 additional tabular datasets with different characteristics as well as a Transformer experiment on the Multi30K translation task (Vaswani et al., 2017). These datasets are multiclass (up to 100) and include a much larger number of features compared to before (up to 2000). However, it is hard to quantify the difficulty of these datasets since they are not synthetic; what we observe is that SOAP remains competitive with the baselines. Dataset complexity might be measured in terms of learned SOAP architectures, but because of the discussions above about architectural efficiency and how the model selection process affects the final result, we prefer not making any claim about it at this stage.

AuthorsOpenReview ↗

Authors’ Response to Reviewer srDL (Part 2)

Q6: The fixed MLP baselines in AWN’s paper considered [1,2,4] possible layers and [8,16,24,128, 256]. AWN and SOAP initial width and depth choices are within these ranges, although both show that solving a task might require values for these architectural choices beyond a manually defined grid. Validation accuracy was the sole criteria to select the best model, which was then retrained 10 times as we do in our work.

The MLP grid budget consisted of 32 configurations, compared to the 27 configurations tried for AWN and 24 for SOAP on the real world datasets (AWN reported results had a budget of 180 configurations). Also, we realized that Table 4 reports only the 288 configurations we used for the synthetic datasets in the preliminary explorations, but we forgot to specify the restricted grid we used for the real-world datasets. We have refined Section E.2 accordingly.


We hope to have satisfactorily answered the Reviewer points and we stand by should further clarifications be necessary. We would appreciate it if the Reviewer could consider rewarding our efforts by revising the scores the way the Reviewer sees most fit.

Reviewer_srDLOpenReview ↗

Responses to authors' revision

I would like to thank the authors for their diligent revisions of this paper. The additional datasets in the experiments strengthen the technical reliability of this paper. The authors have addressed almost all of my previous concerns, so I am increasing my rating from 2 to 3.

However, in pg. 8, "Structural Trends", the authors claimed that the proposed SOAP tends to allocate its capacity in response to the increase in task's complexity. I think that this statement is unjustified, as in their response (W4 & B5), the authors mentioned that "it is hard to quantify the difficulty of the datasets". The complexity of a dataset is not necessarily directly connected to the number of readout neurons or the size of the data, but to how well the data distribution matches the class distribution for a classification task. As long as the authors do not mathematically define a complexity measure, this claim is an overstatement.

If any, please provide a negative example, i.e., a fixed MLP converges much better than SOAP (this is not a condition for acceptance).

Reviewer_5VL5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces the Self-Organizing Artificial Perceptron (SOAP), a multilayer perceptron that jointly learns network width, depth, and model parameters through gradient-based optimization. The method employs probabilistic distributions to control neuron allocation and network depth, using quantile-based truncation to obtain finite architectures from formally unbounded networks. A variational objective integrates predictive learning with regularization over weights and architectural variables. Experiments on six classification datasets demonstrate competitive predictive performance against fixed MLPs and adaptive-width networks (AWN), while additional analyses investigate architectural convergence under different initializations, regularization settings, and batch sizes.

Critical strengths
  1. The paper presents a unified probabilistic framework for jointly adapting network width and depth through gradient-based learning. Combining these two architectural dimensions provides an interesting extension of existing adaptive neural network methods.
  2. The proposed formulation connects continuous optimization with discrete architectural changes through learnable distributions and quantile-based truncation, providing a coherent mechanism for dynamically expanding and shrinking neural networks.
  3. The experiments demonstrate competitive predictive performance across six classification datasets. The additional analyses of architectural convergence under different initializations and regularization settings provide useful insights into the stability of joint structural adaptation.
Critical weaknesses

The necessity and practical benefits of the proposed variational framework remain insufficiently justified. While SOAP provides a unified approach to jointly learning network width and depth, its objective introduces multiple latent distributions and regularization terms without clearly demonstrating why such complexity is necessary. Moreover, the learned architectures are often substantially larger than those of the baselines despite achieving comparable predictive performance, raising questions about the efficiency and practical advantages of joint architectural adaptation.

Questions for authors
  1. Could the authors clarify whether the full variational objective is necessary for joint architectural adaptation, potentially through comparisons with simpler cross-entropy-based objectives?
  2. Could the authors provide additional ablations isolating width and depth adaptation, together with comparisons of parameter counts and computational costs, to better demonstrate the benefits of their joint optimization?
3 comments
AuthorsOpenReview ↗

Authors’ Response to Reviewer 5VL5

We wish to thank the Reviewer for providing constructive feedback as well as recognizing the merits of our approach. We provide an answer to the Reviewer’s concerns below.

Justification of variational framework: SOAP’s probabilistic framework shows that our design choices do not come out of nowhere, rather they are the result of a variational approximation of a probabilistic model with explicit assumptions. Please note that, in practice, this does not add significant complexity to the model, as implementing the model is rather simple. Of course, some degree of additional complexity is required to unlock the joint learning of width and depth via backpropagation, which can have huge potential in automating model design of neural networks. We have clarified this point in the revised Section 3.2.

Learned architectures are larger than baselines: thank you for raising this point, which encouraged us to dive deeper into our results. Part of the reason why this happens, as we mentioned in lines 426-429, is that our automatic model selection code was driven by validation accuracy only, affecting the results of Table 2. However, our new Pareto frontier plots in Appendix M clearly show that, for many datasets, a practitioner could have chosen configurations with negligible validation accuracy different at a fraction of the current parameters, for instance 10-30% for pol and Helena datasets. We hope that this result shows that learning larger architectures is not an issue of SOAP per se, but rather an effect of the model selection process built around it. In the future, it might be a good idea to start evaluating Pareto frontiers across performance and size as part of a model selection strategy.

Finally, we would like to stress that the efficiency argument lies not only in the final architecture learned, which can nevertheless be significantly compressed as we now show in Appendix G, but also in the reduced amount of model selection time spent trying different architectures (discussed in Appendix L).

Q1: Following on from our answer above, it is not strictly necessary that the full variational objective is used. This is the case, for instance, in the new Transformer experiment in Tables 1 and 2, where we simply replaced the feedforward MLPs with a SOAP variant without touching the the original Transformer model. We thank the Reviewer for stressing this: we have clarified this aspect in Section 3.3.

Q2: The requested width and depth ablations, comparisons of model selection time as in AWN paper, and the parameter counts are now provided in Appendix I and Appendix L. We ran an ad-hoc model selection on Volkert to thoroughly investigate the effect of the ablation, showing the advantage of automating both width and depth learning, and we provide time comparisons and theoretical speedups showing how SOAP is slightly more expensive than AWN but allows to reduce the time spent running realistic grids with many configurations.


We thank the Reviewer for the opportunity to clarify the methodological, rather than practical, need for a principled probabilistic framework and for suggesting the additional ablations that further strengthened the paper. We sincerely hope the Reviewer appreciates our efforts to address the raised points and will consider adjusting the scores as a result.

Thank you for engaging so thoughtfully with my review. I appreciate the time and effort you put into responding to my questions and running additional experiments during the discussion. The new width-only and depth-only ablations on Volkert directly address my question about the value of joint adaptation. The Pareto analysis and the expanded reporting on parameter counts and computational costs also give a much clearer picture of the practical trade-offs. The revision is stronger as a result of these additions, and the new evidence addresses my main concerns. I am raising my overall rating from 2 to 3. Thank you again for the constructive exchange.

We sincerely thank the Reviewer for engaging with the rebuttal and for raising the score as a result of our efforts!

Reviewer_7aN1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Self‑Organizing Artificial Perceptron (SOAP), a variational MLP framework that jointly learns both network depth and width purely via back‑propagation, without pre‑fixed architectural upper bounds. Built upon prior infinite‑network variational inference work, SOAP models theoretically unbounded layers and neurons with learnable variational distributions: a centered‑parameterized truncated skew‑normal distribution controls network depth, and exponential distributions govern per‑layer neuron width. Discrete network resizing, which adds or delets layers and neurons, is triggered by quantile‑based truncation of these continuous variational posterior distributions. The objective optimizes an ELBO combining predictive loss, weight priors, and complexity‑regularizing KL‑divergence terms. A first‑order Taylor approximation makes intractable infinite‑dimensional expectations computable. The authors theoretically diagnose optimization failure modes of the previously‑used folded‑normal distribution and demonstrate that their centered skew‑normal parameterization avoids vanishing‑gradient collapse at degenerate locations. Empirical evaluations run on six tabular binary‑classification datasets, i.e., 3 synthetic, 3 real‑world. SOAP is compared against fixed‑grid‑searched static MLP and AWN. It matches or surpasses baseline classification accuracy. Extensive stability ablation studies investigate how final learned topology depends on initialization, batch‑size, weight‑prior strength, and readout configuration. Experiments show the model can converge to task‑appropriate depth‑width configurations regardless of many starting‑topology choices.

Critical strengths
  1. Unlike existing methods that adapt only width or only depth, SOAP enables simultaneous differentiable learning of both network depth and width within a single variational‑inference framework.

  2. The paper identifies a critical gradient‑vanishing pathology for folded‑normal depth distributions and provides mathematical proof.

  3. The paper derives the full ELBO objective for unbounded‑width‑and‑depth networks, clearly separates continuous variational‑parameter optimization from discrete physical network‑resizing steps.

Critical weaknesses
  1. All evaluations are restricted to low‑dimensional tabular binary‑classification tasks. There are no experiments on images, text or deep non‑MLP architectures such as CNN or transformer. Therefore, the experimental scope is limited.

  2. The gradients only update variational parameters, not directly the discrete topology itself. Actual addition/deletion of layers and neurons are out‑of‑graph discrete operations.

  3. There are so many hyper-parameters, such as quantile thresholds, distribution hyper‑parameters, weight‑prior scales, initial variational rates require careful tuning.

Questions for authors
  1. All experiments are limited to tabular binary‑classification MLPs. Could you extend the experiments to convolutional or transformer‑based models?

  2. Could you add hyper‑parameter sensitivity ablation to quantify how much final depth and width vary across reasonable hyper‑parameter ranges?

  3. How would SOAP behave under distribution‑shift or continual‑learning conditions? Would it reliably grow or shrink depth or width to adapt to new task complexity?

2 comments
AuthorsOpenReview ↗

Authors’ Response to Reviewer 7aN1

We sincerely thank the Reviewer for the positive assessment of our work and the constructive criticism. Below, we did our best to address the Reviewer’s minor concerns.

W1 & Q1: We agree with the Reviewer and we extended the evaluation to 4 additional tabular classification datasets, with input dimensionality ranging from 27 to 2000, size ranging from 10.000 to 581.000 samples, and 5 to 100 output classes. The details are now in Section E.1, and we updated Tables 1 and 2 accordingly.

In addition, we extended the evaluation to a Transformer as in the AWN paper, using the same experimental setup. We found out that AWN had two evaluation issues with this particular experiment: i) Adam’s weight decay was unintentionally applied to the width parameters as well, causing the collapse of all widths; and ii) the cross entropy loss did not mask out the padding tokens, which account for approx. 80% of the tokens. Moreover, the learning rate was too high and with a valid cross entropy loss, the model failed to learn the correct output distribution. We reduced it with no further modifications to its schedule. We have therefore performed a whole new comparison between a fixed model, SOAP, and AWN on the Multi30k dataset. The new experiments seem to confirm the trends observed with the original datasets, thus strengthening our claims.

W2: We understand the Reviewer’s concern, but please note that it is mathematically impossible to differentiate against a discrete architectural change; what the Reviewer sees as a limitation, we do instead see as the main conceptual advancement brought forward by SOAP. Indeed, discrete architectural changes are modeled via continuous proxies by UDN, AWN, and ultimately SOAP. This is the only way, to the best of our knowledge, to inform discrete architectural changes via backpropagation, and we have extensively shown, as prior work did, that the inductive biases introduced in the model function as one would intuitively expect, for instance by adapting the topology to the difficulty of a task.

W3: We would like to stress that the quantile threshold is not a hyper-parameter to be generally tuned; higher results are always better, but cost additional capacity that does not significantly impact the output. Empirically, earlier UDN and AWN works found that quantile thresholds such as 0.9 and 0.95 are an excellent compromise, so we recommend fixing these values. Secondly, the weight prior scale is not an additional hyper-parameter, as it is the classical weight decay of neural networks. Finally, our sensitivity analyses (Figure 3, for instance) show that there is convergence to similar widths and/or depths regardless of the starting points on multiple synthetic and real world datasets.

We agree that the distributions have to be chosen somehow; for width, the exponential has been shown to be a good compromise for the width (please refer to Table 2 in AWN’s paper), while we propose an improved depth distribution that solves issues with prior depth distributions. These choices are nonetheless higher-level than having to select fixed widths and depths from a grid, and Bayesian theory tells us these higher-level choices affect the end result less [1].

[1] Bernardo, J. M. and Smith, A. F. Bayesian theory, volume 405. John Wiley & Sons, 2009.

Q2: Thank you for the suggestion: Appendix F now provides more hyper-parameter analyses for the real-world datasets as well. On real world datasets, we still observe a general convergence trend to similar topologies with the exception of a few runs across all datasets.

Please also note that Appendix F already provided an in-depth topological stability analysis by running more than 1000 different configurations on SpiralHard, trying to quantify the true impact of each hyper-parameter. Unfortunately, due to space reasons, we could not include it in the main paper.

Q3: This is a really interesting question, thank you for posing it. We investigate the topological behavior of SOAP under distribution shift on the three synthetic datasets, for which task difficulty is clearly ordered. We ran two experiments, one where task difficulty increases and one in which it decreases, training for a fixed number of epochs on each dataset. These results are now in Appendix K in the revised paper, and show that the network grows or shrink dynamically depending on the difficulty of the new data distribution to model. We hope the Reviewer appreciates the analysis and we thank them again for proposing it.


We wish to thank the Reviewer again for the suggestions for improvement. We believe the paper’s robustness has improved substantially and we would appreciate it if the Reviewer could consider revising the scores accordingly.

Dear Reviewer 7aN1,

We wish to thank you again for providing valuable suggestions for improvement of our work. We have worked hard over the past week to address the concerns raised, and as a result we believe the strength of our claims has increased significantly, in particular regarding:

  • the breadth of the empirical evaluations (Table 1 + the new appendix sections)
  • the extended ablation studies on hyper-parameters demonstrating convergence to similar topologies on multiple real world datasets regardless of the initial choice of hyper-parameters
  • ⁠SOAP's behavior under distribution-shift and its ability to adapt the model's topology accordingly

As the rebuttal period draws to a close, we would greatly appreciate any feedback from the Reviewer, and we hope that our responses have sufficiently addressed their concerns to warrant a reconsideration of the score.

Kind regards, the authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

We thank the Reviewers for the constructive feedback. Revised Paper is uploaded.

We wish to thank all Reviewers for the in-depth and constructive feedback. We have a feeling the review process served its purpose and the paper's empirical support has been much strengthened.

We have uploaded a revised PDF of our manuscript with the changes written in blue.

Best regards, the authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.