acceptodds
Under review as a conference paper at ICLR 2027

Von Neumann Networks

Abstract

In the mid-twentieth century, mathematician and polymath John von Neumann created a computational system on an array of cells as a simple model of the human brain, where each cell had one of a finite set of roles or states that he predicted would be modelled by a diffusion process. In this work, we show that such a system, when developed in a modern deep learning setting, enables the construction of an artificial neuron having specialized roles that can be learnt. We refer to this neuron as the Von Neumann neuron, and the resulting neural network from such neurons result in a self-engineered design whose architecture is only dependent on the structure and locations of its inputs and outputs on this cellular array. The mathematical framework for these Von Neumann Networks (VNNs) is also constructed and shows that they are based on the extension of neural operators and the learning of Green's functions with convolutions on a cellular topology having a diffusion signature. We also prove that these VNNs are part of a more general computational system called Cellular Machine that are computationally universal. Initial experiments show that VNN based multi-layered perceptrons outperform their equivalent deep learning variant on basic tasks, while being more parameter efficient and are capable of learning new types of tasks. This includes the ability to solve for and construct an extension of the Von Neumann (hardware) architecture common to all modern computers to cells and suggests new opportunities that could be explored.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
artificial neural networks, self-replicating machines, architecture learning, cellular automata, representation learning, architecture search, deep learning, neurocell

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Cf1wOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Von Neumann Networks (VNNs), which arrange computational cells on a grid. Each cell has an activation value and cell-specific learned weights, biases, and a gate. Learned convolution kernels pass information between cells, while the gate controls the balance between linear processing and nonlinear activation.

The authors evaluate VNNs on MNIST and CIFAR-10 classification, arithmetic and logic tasks, recovery through retraining after simulated cell damage, and prediction of Game of Life transitions. They also propose a mathematical interpretation using Green's functions and claim that VNNs are Turing-complete.

Critical strengths
  1. VNNs combine communication between grid cells with a learned choice between linear and nonlinear processing at each cell. This is an interesting idea of learning multiple roles or jobs for the grid cells.

  2. The VNNs use fewer parameters than the selected image-classification MLPs while reaching similar or better accuracy.

Critical weaknesses
  1. Missing comparisons with closely related approaches. For MNIST and CIFAR-10, I would expect a comparison with a CNN, since VNNs already use convolutional connections. A neural cellular automaton would also be a particularly relevant baseline for the full VNN. Comparing them would help test whether this cell-specific processing provides an advantage over a shared update rule. An ablation with a fixed cell gate would also be interesting. Without these comparisons, it is difficult to tell whether the reported gains come from the proposed gates, the convolutional connections, or repeated computation.

  2. The choice of baseline configurations is insufficiently justified. It is unclear how their architectures and parameter budgets were selected, or whether they received a comparable amount of hyperparameter tuning. For example, the ALU MLP has 2.0K parameters compared with 16.3K for the full VNN, whereas the CIFAR-10 MLP has 11.1M compared with 1.2M. A more systematic comparison across model sizes would help establish whether the reported gains reflect the proposed architecture or the particular configurations chosen. Furthermore, the paper does not specify how many independent runs or random seeds were used for each task and model, or whether the reported results represent single runs, averages, or the best-performing runs.

Questions for authors

Q1. How were the baseline architectures and parameter budgets selected? Were the baselines and VNNs given comparable hyperparameter-tuning budgets? Why is the ALU MLP much smaller than the full VNN, while the CIFAR-10 MLP is much larger?

Q2. How many independent training runs and random seeds were used for each task and model? Do the reported values represent single runs, averages, or the best-performing runs? Please provide the mean and standard deviation across runs and explain how the reported runs or checkpoints were selected.

Q3. How does VNN compare with a CNN or NCA on MNIST and CIFAR-10.

Q4. How does performance change when the cell gates are fixed during training rather than learned, while the other model parameters remain trainable? This would help assess whether learning the cell roles contributes to the reported gains.

2 comments
AuthorsOpenReview ↗

Author Response 1/2 - Comparisons and Gate Ablation

We thank the reviewer for their comments, especially with insights with respect to the cell gates that prompted a detail study into them.

Missing Comparisons and Gate Ablation

We have added comparisons to CNNs, Diff Logic and Neural CA for MNIST/CIFAR10 experiments (see update Table 1 in revision) and most importantly added VNN-CNN model that builds upon a 3D VNN model to compete with CNNs in image classification (see Appendix G in the revision. These details can be also found in the Global Response 2/4.

We appreciate the suggestion to do a ablation study on the cell gates, i.e. the Codd states. This has been added to Appendix H in the revision and details are mentioned in Global Response 4/4. Given the ALU model, where the VNN provided the most benefit as shown in Table 1, we run three neuron designs (with ss, s=1s=1 always, and deep learning) with a fixed training recipe to learn the ALU model on the same scale and topology. The main result of this study is summarized in the Global Response 4/4, but most clearly shown in Figure 13 of the revised manuscript. The Von Neumann Neuron with the Codd state provided better convergence and fit to the ALU task relatively independently of the number of Green's functions showing their value to spatial connectivity.

We also provide the following table of the analysis of connectivity metrics for the ALU model from the Codd state study for it.

M = 4M = 8M = 16M = 32M = 16, adopted
Committed (%)17.940.161.565.495.5
Wires (%)15.237.858.864.094.5
Neurons (%)2.62.32.61.41.0
Wire components17112250473
Largest wire component holds (%)37.757.780.388.499.9
Committed-mask IoU / chance0.218 / 0.0980.333 / 0.2480.522 / 0.4430.548 / 0.4820.929 / 0.913
Cohen's κ0.2190.1690.1870.1650.185
Non-wire IoU / chance0.788 / 0.7350.527 / 0.4500.370 / 0.2580.312 / 0.2150.182 / 0.028
Neuron IoU / chance0.099 / 0.0130.084 / 0.0120.089 / 0.0130.052 / 0.0060.109 / 0.005

The Table shows evidence that the VNN model reliably organizes itself into a connected wire fabric with a sparse scattering of neurons. The wires consolidate as MM and training length grow, from over a hundred fragments to a single fabric in terms of spatial connectivity.

This ablation also helped us refine our contributions with a new figure 2 in the revised manuscript attempts to show that VNNs use two mechanisms to learn spatial neuron connectivity:

  • An extension of neural operators to cellular arrays with learnable Green's functions that allows for neural networks with a cellular topology having deep learning as a special case.
  • A novel neuron design with a Codd state ss per cell that fundamentally enables learning of cell neighborhoods across the array directly with gradient descent. We provide a neuron design to synergize with the Green's functions to learn connectivity at the local neighborhood level to route signals or decisions through cells.

The learnt Codd-state model is worth most where directions MM / Green's functions are scarce. Extra directions give the state-free array more ways to carry a signal across the grid and so stand in, in part, for the transmitting mode the state provides.

AuthorsOpenReview ↗

Author Response 2/2 - Baselines and Questions

Baseline Configurations/Justifications

We have now added all experiments across multiple seeds and details can be found in the Global Response 2/4. The revised Table 1 in the revision reports mean ± s.d. over three seeds and the Codd states study reports over 6 seeds.

In terms of model tuning, the VNN and MLPs configurations were settled over a few exploratory runs per task and then fixed by hand; the MLP baselines were built on the same inputs and outputs and run once at the recipe now stated per task. The goal of the model configurations were to compare between models that are of similar, yet minimal size that accomplishes the task. Since the VNNs are small compact spatial structures, the comparison was designed on 'how would would similar deep learning models (size matched) perform?' The MNIST and CIFAR10 experiments however all use standard pre-processing and training recipes for that dataset that VNNs are forced to follow, such as the DAWNBench ResNet baseline which is already tuned with super-convergence on the CIFAR10 task (35 Epochs with AdamW on a Cosine LR schedule). The CIFAR-10 MLP's 11.1 M parameters has been confirmed and its architecture (a residual MLP on the same seven-plane slab as the VNN) is now stated. This model was the best performing MLP model found and tuned by hand to complete the CIFAR10 task. The relative poor performance with respect to the CNN is expected because of the lack of color mixing and translational invariance.

Questions

How were the baseline architectures and parameter budgets selected? Were the baselines and VNNs given comparable hyperparameter-tuning budgets? Why is the ALU MLP much smaller than the full VNN, while the CIFAR-10 MLP is much larger?

As mentioned above, goal of the model configurations were to compare between models that are of similar, yet minimal size that accomplishes the task. The full VNN model appears larger in parameter size because it includes cells (hence parameters) of the padding areas of the grid, whose cells effectively don't take part in the gradient backprop. Recent developments of an optimized Neurocell framework that implements VNNs is able to forecast the computable region per epoch to remove these, but are difficult to estimate as they depend on the initial configuration of the inputs/outputs on the grid.

How many independent training runs and random seeds were used for each task and model? Do the reported values represent single runs, averages, or the best-performing runs? Please provide the mean and standard deviation across runs and explain how the reported runs or checkpoints were selected.

We have now updated the Tables to use multiple seeds, see previous comment and the Global Responses 2/4 and 4/4.

How does VNN compare with a CNN or NCA on MNIST and CIFAR-10.

We have added a VNN-CNN, ResNets and a NCA results to Table 1, see Global Responses 2/4 and Appendix G for details. The VNN-CNN is able to classify images at the same accuracy as the highly optimized DAWNBench ResNet 18 with 8.5x less parameters.

How does performance change when the cell gates are fixed during training rather than learned, while the other model parameters remain trainable? This would help assess whether learning the cell roles contributes to the reported gains.

We thank the review for this suggestion. The response above described the exact benefits of the Codd state in a study now added to the Appendix H of the revision and Global Response 4/4.

Reviewer_JD5BOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a novel computational model, VNN (Von Neumann Networks), composed of Von Neumann neurons, which is proved to be computationally universal. This network is able to learn its own architectures and connectivity. On the one hand, its neuronal interaction is simulated with extensions to neural operators on a cellular topology. On the other hand, Green’s functions are introduced in creating the network on cellular arrays. The experimental results also demonstrate the parameter efficiency of the model and its capability to learn new tasks.

Critical strengths
  1. The presentation is of very high quality. The background and insights (e.g., cellular automata) around such a novel computational framework are well-organized and comprehensive. The same applies to the following methodology and experimental validation.
  2. Overall, the paper is novel and technically sound. The idea of establishing a cellular model from the concept of the Von Neumann neuron and ending up with a modernized model that can learn different shapes and tasks is quite refreshing. From the basic computational unit design and construction of the entire VNN to the proof of Turing completeness, the paper is also theoretically rigorous and solid.
  3. The network is demonstrated to be parameter-efficient and flexible, as indicated by Tab. 1.
Critical weaknesses
  1. The implementation and architectural forms of VNN in Sec. 3.2 are hard to comprehend. Although in Fig. 2(c), a cellular topology is provided, it’s still hard to get the point of how propagation works exactly (e.g., a clear visualization of the computational flow like Fig. 6, which is not referred to in this part, may be beneficial).
  2. The topology utilized in this paper is limited to a rather small scale, and the evaluation is conducted only on simple tasks (solving binary arithmetic, MNIST & CIFAR-10 recognition).
Questions for authors
  1. I’m curious about whether such a network could be regarded as a computational model for the discovery of the mechanism behind self-evolving organization in neural systems (e.g., Locally Adaptive Cellular Automata [1]). Could the authors provide more insights about it?
  2. How could the adaptation to other forms of topology be realized, and what may be the main benefits of adopting the Von Neumann neuron as the basic computing unit (i.e., modern DL architectures like Transformers and CNNs; a demo would be nice but is not required)? Could the authors provide further elaboration?

Reference: [1] Khajehabdollahi, Sina, et al. "Locally adaptive cellular automata for goal-oriented self-organization." Artificial Life Conference Proceedings 35. Vol. 2023. No. 1. One Rogers Street, Cambridge, MA 02142-1209, USA journals-info@ mit. edu: MIT Press, 2023.

1 comment
AuthorsOpenReview ↗

Author Response

We thank the reviewer for this helpful suggestions and creative insights to the possible benefits of VNNs beyond what has been shown in the paper.

Visualization of the computational flow

We agree that the propagation of a mixture of Green's functions is hard to follow. The revision now features an improved novelty figure (new Figure 2(d) in the revised manuscript) that also shows the (four) propagation steps of a VNN on a 2D array and defines the hyperplane (a line in this case) to the output. Hopefully this makes the VNN computational flow more visible and clearer.

Evaluation Scale

The submitted models were small so that their layouts can be easily read and understood. The revision adds a VNN CNN-class instance of the same neuron, a three-stage pyramid (40×40, 20×20, 10×10 cells; 64/128/256 propagation directions, see Appendix G), which reaches the accuracy of a highly tuned ResNet-style comparator under the shared 35-epoch recipe at 9.5× fewer parameters, at 7.7× of its per-step cost. Larger datasets are future work.

Questions

I’m curious about whether such a network could be regarded as a computational model for the discovery of the mechanism behind self-evolving organization in neural systems (e.g., Locally Adaptive Cellular Automata [1]). Could the authors provide more insights about it?

We thank the reviewer for pointing out this work and making the astute observation. The locally adaptive CAs (or ACAs) appear to make the update rules themselves dynamic and localized, where the "learning" is entirely local, self-organizing, and dynamical. We believe that the VNNs are capable of reducing to these ACAs when using the 3x3 Green's functions and the appropriate neural operator, a diffusion operator in the case of the Ising model, i.e. the Green's functions are the CA's rules. Alternatively, as the reviewer suggests, we can also discover the Green's functions (and thus the operators) by learning the Green's functions of the Ising model or another model through training. It is our belief that because the Neurocell framework supports both forward and back propagation, the corresponding learnt operators could be used to run partial differential operators or rule based systems like GoL and ACAs forward and backwards to discover new patterns or rules. We have updated the conclusion to mention this discovery aspect and added a citation of the ACA work.

How could the adaptation to other forms of topology be realized, and what may be the main benefits of adopting the Von Neumann neuron as the basic computing unit (i.e., modern DL architectures like Transformers and CNNs; a demo would be nice but is not required)? Could the authors provide further elaboration?

We thanks the reviewer for the question as other topologies are most certainly possible but with dramatically increased computational and conceptual complexity. The presented Neurocell framework relies on convolution and the theory of neural operators. We believe that a graph structure is the ultimate form of such a network just like our Human brains. The deep learning equivalent on such a graph network would be using graph convolution with the Laplace-Beltrami operator, such as those on triangulated meshes. Then the VNN would be learning the operator based on the tasks being conducted and generalizing beyond this special case. The cellular approach as presented in this work however is still the most practical approach for the same reasons that John von Neumann constructed his self replication theory, i.e. it simultaneously provides a simple mathematical framework that efficient to implement. The approach on other topologies would definitely be a fruitful area of future research though.

Reviewer_VUZ9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Von Neumann Networks (VNNs), a cellular-array neural framework in which state-based neurons and learned convolutional Green's functions jointly determine computation and connectivity. The paper presents a formalism based on neural operators, claims computational universality, and reports experiments on MNIST, CIFAR-10, XOR, an arithmetic logic unit, and a hardware-like cellular design. The central idea is creative, but the current evidence does not yet separate the contribution from existing cellular neural networks, neural cellular automata, neural operators, dynamic routing, and neural architecture search.

Critical strengths
  1. The cellular view of a learnable network is conceptually interesting. A single state variable per cell combined with spatial signal propagation gives an intuitive mechanism for emergent sparse computation.
  2. The paper attempts to connect the proposed model to several formal viewpoints, including neural operators, cellular machines, and Turing-machine constructions. The appendix and supplied code appear to aim at reproducibility.
  3. The experiments show that VNNs can learn nontrivial spatial layouts and can reduce parameter counts on selected toy tasks. The full 2D ALU example and the learned tape-like XOR construction are useful demonstrations of the intended behavior.
Critical weaknesses
  1. The novelty boundary is not sufficiently established. The proposed mechanism is closely related to cellular neural networks, neural cellular automata, convolutional recurrent systems, neural operators, differentiable routing, and architecture search. The paper needs controlled comparisons that implement these alternatives under the same grid, compute budget, initialization, and training protocol.
  2. The universality claims are stronger than the experiments support. A mapping from VNNs to a universal system or an equivalence to Game of Life does not by itself show that gradient training can reliably discover useful universal computations. The proofs should state the exact assumptions, allowed states, boundary conditions, precision model, and whether the construction is constructive and trainable.
  3. The empirical tasks are mostly small and highly structured. On MNIST, VNN accuracy is similar to an MLP while inference and training are slower; on CIFAR-10, the result is far below standard convolutional baselines. The paper should include larger-scale tasks, multiple random seeds, variance, and comparisons against CNNs or modern spatial recurrent baselines.
  4. The claim that the network self-engineers its architecture is difficult to evaluate from the current tables. Please define an objective measure of learned sparsity, connectivity, effective depth, repairability, and architecture stability across seeds. Visual examples alone cannot show that the architecture is useful or reproducible.
  5. The states are continuous rather than the discrete roles emphasized by the von Neumann analogy, and the required kernels can be large (the paper notes sizes greater than 13). The computational and memory costs of these kernels should be reported, along with ablations for kernel size, dimensionality, and state quantization.
Questions for authors
  1. What is the strongest controlled baseline among neural cellular automata, cellular neural networks, neural operators, or dynamic routing when matched for grid size and compute?
  2. Please state the exact assumptions and constructive steps in the universality proof, including precision, boundary conditions, and discrete versus continuous states.
  3. Can the authors report mean and standard deviation over seeds for all main tasks and quantify learned sparsity, effective depth, and connectivity stability?
  4. How do kernel size, grid dimension, and state quantization affect accuracy, memory, and runtime?
  5. What evidence shows that the learned architectures transfer across tasks or remain repairable after perturbing cells?
3 comments
AuthorsOpenReview ↗

Response 1/3 - Novelty

We thank the reviewer for the detailed feedback enabling us to improve the quality and strength of the contribution of our work. We provide a section by section response of the comments provided.

Novelty

We thank the reviewer for pointing out concerns on the novelty and we have taken the opportunity to improve the introduction with a new figure, help distinguish between our contribution with more evidence for the novel mechanisms in our work and sharpen the narrative around the main advantages of VNNs with respect to architecture learning.

Clarity of Contribution

The new figure 2 in the revised manuscript attempts to show that VNNs use a two mechanisms to learn neuron connectivity:

  • An extension of neural operators to cellular arrays with learnable Green's functions that allows for neural networks with a cellular topology having deep learning as a special case.
  • A novel neuron design with a Codd state ss per cell that fundamentally enables learning of cell neighborhoods across the array directly with gradient descent. We provide a neuron design to synergize with the Green's functions to learn connectivity at the local neighborhood level to route signals or decisions through cells.

Neural operators have not been applied to cellular arrays and learning Green's functions over these arrays applies naturally to the spatial nature of this topology. Thus, we can propagate signals between cells as neurons (of any design) using them to control spatial connectivity. Convolution becomes a efficient computational tool to do this. The result is a CeNN-like framework, but for the modern deep learning era, where we can apply stochastic gradient descent directly on the arrays and dropping the reliance on hand crafted neighborhood templates.

The figure therefore shows two other closely related methods: Nerual CAs and Differential Logic Gates Networks. Neural CAs use a shared MLP to learn the (immediate, small) usually 3×33\times 3 neighborhoods to produce patterns and classification is done through 'self-classification' as only neighborhood rules are learnt. Differential Logic Gates Networks use convolutions to learn logic gates as layers of a network.

Additional Evidence

Given the two novel points above, we added an additional ablation to study the role of Codd states in the VNN models with details found in Appendix H. Given the ALU model, where the VNN provided the most benefit as shown in Table 1, we run three neuron designs (with ss, s=1s=1 always, and deep learning) with a fixed training recipe to learn the ALU model on the same scale and topology. The hypothesis is that the learnt Codd states allow neurons the flexibility to individually act as a wire or as a non-linear switch (i.e. traditional neuron) and that flexibility enables more beneficial spatial connections and improved convergence for physical networks like VNNs especially when propagation directions are few.

The main result of this study is summarized in the Global Response 4/4, but most clearly shown in Figure 13 of the revised manuscript. The Von Neumann Neuron with the Codd state provided better convergence and fit to the ALU task relatively independently of the number of Green's functions showing their value to spatial connectivity.

Architecture Learning Narrative

We now agree in hindsight that the term Architecture Learning is too broad for the mechanisms being presented in this work and thanks the reviewer for highlighting it. The Codd state study shows that the two mechanisms of learning cellular Green's functions and Codd states lead to spatial connectivity learning instead and we have amended our use of "architecture learning" to "spatial connectivity" where appropriate in the revision.

AuthorsOpenReview ↗

Response 2/3 - Universality, Baselines and Connectivity

Universality

We have attempted to improve the assumptions, boundary conditions and constructivity in the Definitions, Proposition 1 and 2 in the revised manuscript. More details are provided in the Global Response 3/4.

Proposition 1 has been expanded to mention that the array supports multiple boundary conditions including sufficient zero padding and circular with a suitable window. We assume every trained VNN in this work is a finite array and is run for a fixed number of steps. We also explicitly point out the Church-Turing thesis for the mapping between models and state the mappings between model components (like tape of the TM etc.) more clearly.

Proposition 2 has been improved by provided a construction by exhaustive example concerning all possible combinations of a 3×33\times 3 to show that a network can have basic properties that will result in the GoL exactly. Details are in Appendix E of the revision, but its essentially possible to setup a VNN with exactly the right form to provide the GoL rules and thus show universality through the GoL Turing completeness.

Baselines and Tasks

We have added a VNN-CNN that is compared with CNNs and Diff Logic works in a new Table 1 with multiple seeds with variance that is competitive. The revision features a new Appendix G for the VNN-CNN that was developed after submission. This new section provides an ablation of how a 3D VNN hyperplane model was modified to create a specialized network like a CNN through make it equivariant. We would also like to note that works on new networks, such as Neural CAs, Diff Logic gates and KANs (Liu et al., 2024), it is standard practice to demonstrate performance on datasets such as MNIST and CIFAR10, since there are the first works of their kind.

Architecture Learning

We now agree in hindsight that the term Architecture Learning is too broad for the two mechanisms of learning cellular Green's functions and Codd states that lead to spatial connectivity learning and we have amended our use of "architecture learning" to "spatial connectivity" where appropriate in the revision.

We understand that the evidence for connectivity is somewhat similar and therefore provide the following table of the analysis of metrics for the ALU model from the Codd state study for it.

M = 4M = 8M = 16M = 32M = 16, adopted
Committed (%)17.940.161.565.495.5
Wires (%)15.237.858.864.094.5
Neurons (%)2.62.32.61.41.0
Wire components17112250473
Largest wire component holds (%)37.757.780.388.499.9
Committed-mask IoU / chance0.218 / 0.0980.333 / 0.2480.522 / 0.4430.548 / 0.4820.929 / 0.913
Cohen's κ0.2190.1690.1870.1650.185
Non-wire IoU / chance0.788 / 0.7350.527 / 0.4500.370 / 0.2580.312 / 0.2150.182 / 0.028
Neuron IoU / chance0.099 / 0.0130.084 / 0.0120.089 / 0.0130.052 / 0.0060.109 / 0.005

The Table shows evidence that the VNN model reliably organizes itself into a connected wire fabric with a sparse scattering of neurons. The wires consolidate as MM and training length grow, from over a hundred fragments to a single fabric in terms of spatial connectivity. The neurons never consolidate; they stay in 10–50 small groups.

AuthorsOpenReview ↗

Response 3/3 - Costs and Questions

Continuous vs Discrete, Computational and Memory Costs

The Codd states by formulation are indeed continuous, however John von Neumann predicted that non-linearity can result in discretization. The connectivity ablations of the previous comment and circuit like models resulting in fixed circuit-like models suggest that long and correct training procedures could produce discrete states naturally. Quantized weights and exact back-propagation could also be avenues for future exploration.

In terms of computational and memory costs, our recent VNN-CNN model for image classification show that VNNs could become competitive with current models with additional development of optimizations. Note that the VNN-CNN convolutions are implemented in the spectral domain where computational costs of different kernel sizes are the same. The table below shows the comparison of models and their compute costs:

ALU array VNNVNN-CNNCNN comparator
Task8-bit ALUCIFAR-10CIFAR-10
Cells62 × 62 array40² / 20² / 10² planes, 112 planes deep—
Propagation steps T43 × 3, and a readout step8 basic blocks
Directions M1664 / 128 / 256—
Kernel support23 × 1339² / 19² / 9² in plane, 21 depth taps3 × 3
Parameters16,316563,6595,362,250
of which kernels4,784 (29 %)201,600 (36 %)nearly all
Kernel memory (32-bit)19 KiB0.77 MiB20.5 MiB
Batch B1289696
Peak GPU memory0.22 GiB7.9 GiB0.66 GiB
Time per step2.9 ms334–345 ms43–45 ms
Throughput (samples/s)44,4002802,200
Full training run13 min (300 epochs)1.9 h (35 epochs)0.26 h (35 epochs)
Time per step, spatialO(B T M N K)O(B T M P N K)O(B C² N K) per layer
Time per step, spectralO(B T M N log N)O(B T M P N log N)—
Activation memoryO(B T M N)O(B T M P N)O(B C N) per layer
Parameters3N + M Ksee App. KO(C² K) per layer

B is the batch, T the propagation steps, M the directions, N the cells in the array or plane, K the cells in the kernel support, P the live planes per step (11 in the VNN-CNN) and C the comparator's channels. The VNN-CNN model is trained in the spectral form, whose cost does not depend on the kernel support.

For the compute costs with respect to MM, the table below shows the costs for the ALU model that were part of he Codd state study.

MParametersTime per stepThroughput (samples/s)Peak GPU memory
412,7281.15 ms111,6000.099 GiB
813,9241.72 ms74,5000.144 GiB
1616,3162.89 ms44,4000.219 GiB
3221,1005.76 ms22,2000.385 GiB

On a final note, ResNet and deep learning MLPs have seen decades of optimization. This is the first instance of VNNs and deep learning on cells that involve hand written code from the authors. AI assisted code re-writes have already been released open-source that show promising 3-6x speed up. Future work could see further kernel and Codd state optimizations that should close the gap.

Questions

What is the strongest controlled baseline ... matched for grid size and compute?

We have implemented a Neural CA to match the grid and compute of the ALU model. This is provided in Table 1 and does not succeed in converging. CA are incapable of learning as a network, CeNN doesn't support deep learning and we reported Diff Logic Gates results in the tables for CIFAR10 and MNIST. We provide an extension of neural operators to cellular arrays.

Please state the exact assumptions and constructive steps in the universality proof, including precision, boundary conditions, and discrete versus continuous states.

This has been amended into Proposition 1 and details are provided in the Universality response above.

Can the authors report mean and standard deviation over seeds for all main tasks and quantify learned sparsity, effective depth, and connectivity stability?

Table 1 and other ablations now have multiple seeds. Table above shows the connectivity/stability that point towards evidence of spatial connectivity learning.

How do kernel size, grid dimension, and state quantization affect accuracy, memory, and runtime?

We have provided a MM ablation and model based costs in the response above. We will note that optimized VNN implementations, starting with the VNN-CNN implement their computations largely in the spectral domain with the costs with respect to the kernel is fixed. Future work could see further kernel and Codd state optimizations that should close the gap.

What evidence shows that the learned architectures transfer across tasks or remain repairable after perturbing cells?

This is a great question that we hope to explore in future work.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Global Response (1/4) - Summary & Novelty

We thank the reviewers for their time and feedback. We have prepared the following global response followed by individual reviewer responses separately.

Summary

The following improvements have been made with details below and revised manuscript:

  • Sharpened novelty and added a novelty figure in the revision to pinpoint the significant differences between the proposed VNNs to Neural CA and Differential Logic approaches, as well as additional notes below with respect to other approaches and additional baselines in the main table (see below).
  • Added an equivariant VNN model that behaves like a VNN-CNN with significant parameter efficiency and achieves comparable ResNet performance on CIFAR10, while learning some its major architectural components. It out-performs other baselines added to the comparisons table. A new appendix highlights this model in more detail (see Appendix G).
  • Updated Table 1 of quantitative results with additional baseline comparisons of Neural CA and Diff Logic approaches, as well as the VNN-CNN above and 3-seed results where possible. The advantages of the proposed VNNs remain intact despite the additions and seeds.
  • Improved clarity around computational universality within current three proofs and added an additional content to the definition and proof to ensure assumptions and conditions are better explained and utilize the Church-Turing thesis more directly. The Church-Turing thesis points to the equivalence between computational models as a standard mechanism to prove universality.
  • Added study of Codd states ss to highlight its key role in VNNs. Refined claims around "learning their own architecture" to "learning their own connectivity". Refined contribution text to be more granular around separating DL neurons on cells, DL cellular neurons with Codd states to form Von Neumann neurons and their connectivity via Green's functions leading to VNNs.

Novelty and Baselines

To sharpen the novelty boundary as the reviewer VUZ9 requested, we have provided a few additional changes:

  • Added Figure 2 to the revised manuscript that highlights the differences between Neural CA and the most recent Differential Logic Gates works.
  • Added controlled experiments for the Neural CA work on the ALU task to table. NCAs were never demonstrated on CIFAR10 as far as the author is aware and we have reported their MNIST results from the literature, which relate to correctly generating MNIST digit pixels not classifying images because NCAs are not directly neural networks but pattern creators using learnt neighborhoods.
  • We have reported the results of the Different Logic Gates works in the literature for the CIFAR10 task as training times are quite large.
  • We have adjusted the contribution sentences in the introduction to clarify VNN novelty better.

We would additionally like to point out that Appendix B is devoted to ensure related works are highlighted and contributions of the work clearly differentiated from the different literatures of CAs, Neural CA, Differential Logic Gates, CeNNs and architecture search. We also point out how Neural Operators through Green's functions and their theory is used to construct VNN theory as a modern approach to CeNNs that have no deep learning equivalent formulation as far as the authors are aware. We have directly explained these points in more detail in the individual reviewer response below.

AuthorsOpenReview ↗

Global Response (2/4) - VNN-CNN & Table Updates (3-seeds and baselines)

VNN-CNN and CNN Comparisons

To address VNN scalability and CNN comparisons concerns from reviewers JD5B and Cf1w, we have added a VNN-CNN by creating a VNN with equivariant properties, added a comparison to two ResNet CNN models and added their CIFAR10 performance to Table 1 across 3-seeds.

This was achieved by noticing that the 3D VNN already learns convolutional kernels in the form of Green's functions, but the input and output locations are fixed. Since CNNs are translationally invariant, in our VNN-CNN we make the output as planes that are global average pooled (GAP) across 10 planes, one for each class. To make the 3D VNN more efficient and mimic enlarging receptive fields, we alter its cellular design to mimic a three level pyramid by learning a strided Green's function to reduce the array size by a factor of 2 twice. We also found that this learnt functions can be replaced by a mean 2x2 pooling is more parameter and computationally efficient.

Upon submission, work commenced on creating this VNN-CNN model in co-operation with AI models to add appropriate specialized model features using the VNN 3D model as a base. The resulting model was then rebuilt using another independent AI model to maintain the pure VNN aspect of the model and adding model features as an ablative study into the design of the VNN-CNN. These methods are documented in Appendix G and will be released open source with these responses. We have updated the AI Use Statement of the paper appropriately to match.

An interesting feature of this model is how the model doesn't explicitly need residual connections, i.e. there are no residual blocks/connections within this VNN-CNN, these are achieved naturally along the propagation direction of the network via the Green's function's extent in that direction. Other important features of the VNN-CNN that were required to reach parity with the ResNet were the need for batchnorm per plane, explicit weight decay and initial color channel mixing VNN. The details of this VNN-CNN are presented in a dedicated new Appendix G and the updated Table 1 below.

Updated Table and Multiple Seeds

Updated Table 1 with new comparison baselines and multiple seeds is provided below.

TaskModel#ParamsAcc. (%)SeedsTrain (s)Throughput (samples/s)
XOR (8-bit)MLP (128/128/128)36.2 K100.0 ± 0.033279,010
Linear regression1361.11——
Polynomial regression7.8 K66.31——
Random forest6.1 M36.31——
VNN3.3 K99.9 ± 0.21010324,780
ALU (8-bit)MLP (24/20/16/12/10)2.0 K67.3 ± 1.1335358,070
Neural CA, matched16.7 K0.712,771443
VNN (1D hyperplanes)7.9 K68.3 ± 0.9372528,250
VNN 2D full16.3 K99.3 ± 0.833,8535,320
CIFAR-10MLP (residual, 7-plane slab) †11.1 M62.41——
ResNet-20272.5 K90.1 ± 0.231,6661,050
ResNet DAWNBench5.36 M94.2 ± 0.139081,927
LogicTreeNet-S (Petersen et al., 2024)0.40 M gates60.4———
LogicTreeNet-G (Petersen et al., 2024)61.0 M gates86.3———
VNN 3D841.7 K71.91——
VNN 3D full1.2 M77.71——
VNN-CNN (App. G)563.7 K94.3 ± 0.337,010250

Important points of note regarding the results shown:

  • The VNN models and their relative performance against all baselines remains unchanged to the original Table 1
  • The Neural CA is no known CIFAR10 model and Diff Logic work also is significantly below current CNN performance levels unlike the VNN-CNN
AuthorsOpenReview ↗

Global Response (3/4) - Universality

Computational Universality

We have improved the definitions, theorems and proves related to computational universality as requested by reviewer VUZ9, especially with respect to the GoL equivalence.

There are a total of three proofs provided for universality with two of them involving a mapping from VNNs to other computational models such as the GoL and deep learning. This mapping approach is a standard because we utilize the Church-Turing thesis, a brief description of which has been added to the Computation appendix. This thesis enables a computational model to adapt the universality of another model if the corresponding model can be constructed within it. The third proof is now constructive and found in the main body of the paper as Proposition 1.

Firstly, we have improved the definition and proofs to list the assumptions, boundary conditions and whether the construction is constructive within Proposition 1. Secondly, to improve the equivalence to the GoL in Proposition 2, we added a theoretic construction of GoL within cells via encoding all 512 3x3 possible neighborhoods that could contain the GoL rules.

For the cellular machine, the construction establishes that the update of Eqn (4), with a specific choice of parameters from a specific hand-set, computes Life exactly:

  • At any generation, in every field, the next value of a cell depends only on its 3×3 neighbourhood and all neighbourhoods 29=5122^9=512 give the right answer.
  • Counting the concrete neighborhoods for being alive gives 56 ways to choose three of the eight neighbors, plus 28 ways to choose two neighbors with a live centre: 84 patterns, no two of which can occur at once.
  • Let every cell be a neuron (S≡1S \equiv 1) with bias −2 and ReLU. Each 3×3 kernel has weight +1 on its three pattern cells and −1 on the remaining neighbors. On a 0/1 field: an exact match sums to 3, so the cell emits ReLU(3 − 2) = 1; a missing cell gives at most 2, so ReLU(0) = 0; an extra live neighbor hits a −1 weight and pushes it lower still. So each kernel fires 1 on its own pattern and 0 on everything else.
  • One kernel with a single center tap of weight 252 (3 × 84), same bias and ReLU: 252 × (1/84) − 2 = 1, and 252 × 0 − 2 is negative, so 0. Two steps are one generation.
  • For any number of generations, the outputs are exactly 0 or 1, so each generation starts from clean binary values and the argument repeats by induction and does not require any training.
  • Hence VNNs are Turing complete because such a construction is possible. Life simulates a universal Turing machine, so a cellular machine that contains Life is universal.

The trained VNN GoL model it got the center cell right on 361 of the 512 neighborhoods and predicts 47.5 % of grids exactly on its own training distribution. So the experiment shows that gradient descent finds an approximation of the rule on those patterns and a longer, more concerted training recipe should find this theoretical construction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.