- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces Von Neumann Networks (VNNs), which arrange computational cells on a grid. Each cell has an activation value and cell-specific learned weights, biases, and a gate. Learned convolution kernels pass information between cells, while the gate controls the balance between linear processing and nonlinear activation.
The authors evaluate VNNs on MNIST and CIFAR-10 classification, arithmetic and logic tasks, recovery through retraining after simulated cell damage, and prediction of Game of Life transitions. They also propose a mathematical interpretation using Green's functions and claim that VNNs are Turing-complete.
Critical strengths
-
VNNs combine communication between grid cells with a learned choice between linear and nonlinear processing at each cell. This is an interesting idea of learning multiple roles or jobs for the grid cells.
-
The VNNs use fewer parameters than the selected image-classification MLPs while reaching similar or better accuracy.
Critical weaknesses
-
Missing comparisons with closely related approaches. For MNIST and CIFAR-10, I would expect a comparison with a CNN, since VNNs already use convolutional connections. A neural cellular automaton would also be a particularly relevant baseline for the full VNN. Comparing them would help test whether this cell-specific processing provides an advantage over a shared update rule. An ablation with a fixed cell gate would also be interesting. Without these comparisons, it is difficult to tell whether the reported gains come from the proposed gates, the convolutional connections, or repeated computation.
-
The choice of baseline configurations is insufficiently justified. It is unclear how their architectures and parameter budgets were selected, or whether they received a comparable amount of hyperparameter tuning. For example, the ALU MLP has 2.0K parameters compared with 16.3K for the full VNN, whereas the CIFAR-10 MLP has 11.1M compared with 1.2M. A more systematic comparison across model sizes would help establish whether the reported gains reflect the proposed architecture or the particular configurations chosen. Furthermore, the paper does not specify how many independent runs or random seeds were used for each task and model, or whether the reported results represent single runs, averages, or the best-performing runs.
Questions for authors
Q1. How were the baseline architectures and parameter budgets selected? Were the baselines and VNNs given comparable hyperparameter-tuning budgets? Why is the ALU MLP much smaller than the full VNN, while the CIFAR-10 MLP is much larger?
Q2. How many independent training runs and random seeds were used for each task and model? Do the reported values represent single runs, averages, or the best-performing runs? Please provide the mean and standard deviation across runs and explain how the reported runs or checkpoints were selected.
Q3. How does VNN compare with a CNN or NCA on MNIST and CIFAR-10.
Q4. How does performance change when the cell gates are fixed during training rather than learned, while the other model parameters remain trainable? This would help assess whether learning the cell roles contributes to the reported gains.
2 comments
Author Response 2/2 - Baselines and Questions
Baseline Configurations/Justifications
We have now added all experiments across multiple seeds and details can be found in the Global Response 2/4. The revised Table 1 in the revision reports mean ± s.d. over three seeds and the Codd states study reports over 6 seeds.
In terms of model tuning, the VNN and MLPs configurations were settled over a few exploratory runs per task and then fixed by hand; the MLP baselines were built on the same inputs and outputs and run once at the recipe now stated per task. The goal of the model configurations were to compare between models that are of similar, yet minimal size that accomplishes the task. Since the VNNs are small compact spatial structures, the comparison was designed on 'how would would similar deep learning models (size matched) perform?' The MNIST and CIFAR10 experiments however all use standard pre-processing and training recipes for that dataset that VNNs are forced to follow, such as the DAWNBench ResNet baseline which is already tuned with super-convergence on the CIFAR10 task (35 Epochs with AdamW on a Cosine LR schedule). The CIFAR-10 MLP's 11.1 M parameters has been confirmed and its architecture (a residual MLP on the same seven-plane slab as the VNN) is now stated. This model was the best performing MLP model found and tuned by hand to complete the CIFAR10 task. The relative poor performance with respect to the CNN is expected because of the lack of color mixing and translational invariance.
Questions
How were the baseline architectures and parameter budgets selected? Were the baselines and VNNs given comparable hyperparameter-tuning budgets? Why is the ALU MLP much smaller than the full VNN, while the CIFAR-10 MLP is much larger?
As mentioned above, goal of the model configurations were to compare between models that are of similar, yet minimal size that accomplishes the task. The full VNN model appears larger in parameter size because it includes cells (hence parameters) of the padding areas of the grid, whose cells effectively don't take part in the gradient backprop. Recent developments of an optimized Neurocell framework that implements VNNs is able to forecast the computable region per epoch to remove these, but are difficult to estimate as they depend on the initial configuration of the inputs/outputs on the grid.
How many independent training runs and random seeds were used for each task and model? Do the reported values represent single runs, averages, or the best-performing runs? Please provide the mean and standard deviation across runs and explain how the reported runs or checkpoints were selected.
We have now updated the Tables to use multiple seeds, see previous comment and the Global Responses 2/4 and 4/4.
How does VNN compare with a CNN or NCA on MNIST and CIFAR-10.
We have added a VNN-CNN, ResNets and a NCA results to Table 1, see Global Responses 2/4 and Appendix G for details. The VNN-CNN is able to classify images at the same accuracy as the highly optimized DAWNBench ResNet 18 with 8.5x less parameters.
How does performance change when the cell gates are fixed during training rather than learned, while the other model parameters remain trainable? This would help assess whether learning the cell roles contributes to the reported gains.
We thank the review for this suggestion. The response above described the exact benefits of the Codd state in a study now added to the Appendix H of the revision and Global Response 4/4.
Author Response 1/2 - Comparisons and Gate Ablation
We thank the reviewer for their comments, especially with insights with respect to the cell gates that prompted a detail study into them.
Missing Comparisons and Gate Ablation
We have added comparisons to CNNs, Diff Logic and Neural CA for MNIST/CIFAR10 experiments (see update Table 1 in revision) and most importantly added VNN-CNN model that builds upon a 3D VNN model to compete with CNNs in image classification (see Appendix G in the revision. These details can be also found in the Global Response 2/4.
We appreciate the suggestion to do a ablation study on the cell gates, i.e. the Codd states. This has been added to Appendix H in the revision and details are mentioned in Global Response 4/4. Given the ALU model, where the VNN provided the most benefit as shown in Table 1, we run three neuron designs (with s, s=1 always, and deep learning) with a fixed training recipe to learn the ALU model on the same scale and topology. The main result of this study is summarized in the Global Response 4/4, but most clearly shown in Figure 13 of the revised manuscript. The Von Neumann Neuron with the Codd state provided better convergence and fit to the ALU task relatively independently of the number of Green's functions showing their value to spatial connectivity.
We also provide the following table of the analysis of connectivity metrics for the ALU model from the Codd state study for it.
The Table shows evidence that the VNN model reliably organizes itself into a connected wire fabric with a sparse scattering of neurons. The wires consolidate as M and training length grow, from over a hundred fragments to a single fabric in terms of spatial connectivity.
This ablation also helped us refine our contributions with a new figure 2 in the revised manuscript attempts to show that VNNs use two mechanisms to learn spatial neuron connectivity:
The learnt Codd-state model is worth most where directions M / Green's functions are scarce. Extra directions give the state-free array more ways to carry a signal across the grid and so stand in, in part, for the transmitting mode the state provides.