Self-Organizing Artificial Perceptron: Backpropagating through the Width and Depth of Neural Networks

In machine learning, the hypotheses' space of neural networks is encoded by their architecture, and it cannot change during training or inference in response to the task. As discrete architectural changes are not differentiable, neural networks are essentially unable to self-organize their knowledge and computational structure. This work introduces the Self-Organizing Artificial Perceptron (SOAP), a multi-layer perceptron (MLP) that can jointly learn its architecture, namely depth, width, and the associated parameters via standard backpropagation. Crucially, we do not impose upper bounds on depth or width, meaning the network is formally infinite but practically finite, in the style of Bayesian models. The underlying idea is to make the addition and removal of new layers and neurons compatible with the gradient signal. We test the joint learning of width and depth on 6 tabular datasets, showing SOAP maintains or exceeds the performance of MLPs. In addition, we analyze the behavior of depth and width in response to different batch sizes and starting conditions. Our contribution paves the way for new investigations about a new class of model whose inductive bias adapts to the task's complexity.