- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces LeWi (Learned Wiring), a neural network architecture that jointly learns connection weights and neuron positions in a low-dimensional space. Connections between neurons are determined by their distances, allowing the network to learn sparse, modular computational structures through gradient descent. The authors introduce CSFRBench, a benchmark of synthetic functions with hidden compositional structures, and show that LeWi achieves better sample efficiency than dense MLPs and several sparsity-based baselines, reducing the mean learning-curve area by 27% across 32 randomly generated computation graphs. They also evaluate LeWi on 17 world-model prediction tasks, outperforming dense MLPs on 16 of them. Through ablation studies, they demonstrate that learning connectivity is more beneficial than imposing fixed random sparsity, while additional analyses show that the learned representations and connections align with the underlying computational structure. However, LeWi is less effective on functions involving many interacting variables and introduces significant computational overhead.
Critical strengths
- The paper addresses an interesting problem by proposing a novel approach to neural network training that is intuitive and well-motivated.
- The paper is well-written, clearly structured, and easy to follow.
- The paper provides a comprehensive set of experiments, with strong and promising empirical results.
Critical weaknesses
-
The paper evaluates the proposed method primarily on synthetic tasks and relatively old simulation benchmarks. There is no evaluation on real-world datasets, and even standard benchmarks such as MNIST and CIFAR-10 are not considered. This makes it difficult to assess the practical applicability and generalizability of the proposed approach beyond the specific settings studied in the paper.
-
The experiments are limited to relatively small MLP architectures. However, modern deep learning relies on a much broader range of architectures, such as CNNs, Transformers, and GNNs. I believe that demonstrating the effectiveness of the proposed approach on architectures beyond small MLPs is important for assessing the broader significance and potential impact of the proposed paradigm.
Questions for authors
-
Have the authors considered evaluating LeWi on standard classification datasets such as MNIST or CIFAR-10? This would help assess whether the benefits extend beyond synthetic and simulation-based prediction tasks.
-
Can the proposed approach be extended to other architectures, such as CNNs, Transformers, or GNNs? What are the main technical challenges in doing so?
-
Have the authors considered first pretraining a dense MLP and then learning the neuron embeddings while keeping the pretrained weights fixed or fine-tuning them? I believe this is an interesting experiment.
5 comments
[W2, Q2] Extensions beyond MLPs
We thank the reviewer for highlighting the importance of evaluating LeWi beyond MLPs. We use a common MLP backbone to compare methods for learning connectivity. This choice is consistent with theoretical work on learning compositionally sparse functions using feedforward networks [9–11], which establishes the theoretical sample-efficiency gains available from such functions and directly motivates our work. MLPs are also widely used as the primary architecture in related research: Neural Sculpting [12] studies hierarchical modularity through pruning, and Boopathy et al. [13] investigate sample complexity in fully connected modular networks.
We agree that extending LeWi to broader architectures is an important direction, and we are actively investigating it. However, a systematic extension to Transformers, CNNs, and GNNs is beyond the scope of the present study. Nevertheless, our preliminary Transformer experiments provide encouraging results, and we outline possible CNN and GNN adaptations below.
Transformers. We investigated applying LeWi’s distance-based connectivity masks to the weight matrices that project token representations into queries, keys, and values, and transform attention outputs (QKVO), or to the two linear layers of each feed-forward network (FFN), or both.
We evaluated these adaptations using a three-block Transformer (width 64, four attention heads) in which each input coordinate is represented as a token. The experiments cover three CSFRBench settings (depth-4 tree, sampled depth-4 DAGs, and Fourier tree; nine instances total), with 64,000 training steps. To assess the contribution of learned connectivity, we also compare LeWi-QKVO against a fixed-random-mask control with the same initial number of active connections and effective weight norm in each projection. The control trains its weights but keeps its connectivity fixed.
| Adaptation | Comparison | Relative AULC (95% CI) |
|---|---|---|
| LeWi-FFN | Dense | +12.1% [+7.7%, +16.5%] |
| LeWi-QKVO+FFN | Dense | −7.8% [−14.3%, −1.3%] |
| LeWi-QKVO | Dense | −21.6% [−29.1%, −14.2%] |
| LeWi-QKVO | Matched random QKVO | −4.2% [−7.6%, −0.9%] |
Negative differences indicate lower AULC.
Applying LeWi to QKVO reduces AULC by 21.6% relative to Dense, whereas applying it only to FFNs increases AULC by 12.1%. This might be because each token represents one input coordinate, so interactions between input coordinates occur through attention, whereas FFNs transform each token independently. LeWi-QKVO also achieves 4.2% lower AULC than the matched random control, providing preliminary evidence of a benefit from learned connectivity beyond initial sparsity and weight scale.
CNNs. Following work on spatially organized convolutional units [14, 15], LeWi could assign an embedding to each feature channel and learn connectivity between input and output channels.
GNNs. Message-passing GNNs use shared neural networks to compute messages and update node features [16]. LeWi could be applied within these functions to learn feature-level connectivity without modifying the graph's adjacency.
We will include the preliminary Transformer results in the revised appendix and discuss these architectural extensions in the Discussion.
[Q3] Learning LeWi after Dense pretraining
We thank the reviewer for this suggestion. Since LeWi's neuron embeddings and weights are closely coupled, it is not immediately clear how best to adapt a pretrained Dense network. We investigate one approach using weight-informed initialization to preserve pretrained predictions, followed by either optimizing embeddings alone or jointly fine-tuning embeddings and weights.
Setup. We conduct experiments on CSFRBench depth-4 trees and reuse DAGs (16 instances each), with . We pretrain a Dense MLP for 32,000 steps, introduce LeWi-style connectivity through weight-informed initialization, and train for another 32,000 steps under two settings: (1) optimizing neuron embeddings while freezing pretrained weights, and (2) jointly fine-tuning embeddings and weights. We compare against continued Dense training with the same optimizer restart and LeWi trained from scratch for 64,000 steps. Training schedules are matched, with hyperparameters selected by validation error.
Weight-informed initialization. Directly applying LeWi with randomly initialized embeddings closes approximately 40% of the pretrained connections and substantially disrupts predictions. To enable a meaningful post-training comparison, we initialize LeWi's geometry from the pretrained network:
- Score connections using Wanda-style importance [17], , where contains the sending neuron’s activations over the training samples.
- Select the highest-scoring positive-importance connections, up to 60% per layer.
- Fit neuron embeddings using 2,000 Adam steps and a squared-hinge objective that encourages selected connections to lie within LeWi's connection radius and other connections outside it.
- Rescale the retained weights to compensate for LeWi's soft gates.
This conversion preserves validation NMSE to within 5% of the pretrained network in 87/96 conditions (32 target instances × 3 training-set sizes).
| Training strategy | Depth-4 tree | Reuse DAG |
|---|---|---|
| Pretrained Dense (32k steps) | 0.763 | 0.394 |
| Continued Dense (64k) | 0.752 | 0.379 |
| Dense → LeWi-style wiring, weights frozen (64k) | 0.755 | 0.378 |
| Dense → LeWi-style wiring, joint fine-tuning (64k) | 0.721 | 0.354 |
| LeWi from scratch (64k) | 0.648 | 0.256 |
Results. Joint fine-tuning improves AULC over continued Dense training by 4.1% on depth-4 trees and 6.6% on reuse DAGs, outperforming it on 15/16 and 16/16 instances, respectively. In contrast, optimizing embeddings alone with pretrained weights frozen yields no significant improvement (+0.4% and −0.2%). Joint fine-tuning also outperforms frozen-weight adaptation by 4.5% and 6.4%. However, LeWi trained from scratch remains substantially better, with joint post-training adaptation yielding 11.3% and 38.0% higher AULC, respectively.
These results show that adapting a pretrained Dense network using LeWi-style connectivity can improve prediction when weights and embeddings are jointly optimized, but learning computational structure from the beginning is more effective, potentially because it allows more time for the coupled optimization of connectivity and local functions.
We will include the experimental protocol and complete results in the revised appendix.
[Additional clarification] Computational overhead
We thank the reviewer for noting the computational overhead of learning connectivity. LeWi’s additional training cost comes from computing neuron distances, gates, masks, and optimizing the neuron embeddings. We have accelerated training by batching independent networks and using fused C++ kernels for LeWi, while also optimizing Dense through batching and compilation. Under these matched CPU measurements, LeWi’s end-to-end runtime per network is 1.32x and 1.01x that of Dense on the 32x4 and 64x8 MLPs, respectively (Appendix D.7, Table 6).
Our implementation still uses dense operations during training, so exploiting the learned sparsity to further reduce computation remains a direction for future work. At inference, the learned gates and masks can be folded into fixed weights, eliminating the need to compute neuron distances, gates, or masks during prediction.
[W1, Q1] Evaluation beyond synthetic and simulation-based tasks
We thank the reviewer for raising this concern. Our world-model tasks come from stable-worldmodel [1], a 2026 evaluation platform built on established benchmarks that remain widely used in recent world-model research, including TD-MPC2 [2], DreamerV3 [3], DINO-WM [4], and LeWorldModel [5]. We focus on world-model prediction tasks because such dynamics functions can exhibit local dependencies among state and action variables, as studied in factored MDPs [6] and object-centric world models [7]. This provides a natural setting to test the practical applicability of LeWi’s inductive bias towards compositional sparsity. We will clarify this motivation and the benchmarks' relevance in the paper.
To assess the generalizability of LeWi beyond these benchmarks, we conducted additional sample-efficiency experiments on the OpenML-CTR23 regression datasets [8], a curated suite of 35 tabular regression tasks. LeWi substantially improves on Dense and achieves competitive average performance with the strongest baselines in this evaluation.
Experimental setup. Since the datasets within OpenML-CTR23 come in different sizes, to measure sample efficiency across sample sizes, we use 19 datasets large enough to support training sizes of {256,512,1024,2048,4096} samples, together with separate validation sets one-quarter as large. We follow each dataset’s 10 predefined train/test splits, using the same examples for every method. Training sets are nested, so each larger set contains the smaller ones. LeWi and the MLP baselines use 256x4 MLPs (four hidden layers of width 256). We compare against Dense, L0, L1, L2, RigL, and XGBoost. For each dataset, split, and training size, every method receives 32 Sobol-sampled hyperparameter configurations. The selected configuration is then retrained with three random seeds.
Results. LeWi reduces AULC relative to Dense by 33.4% on average, with lower AULC on 18 of 19 tasks.
Mean per-task relative AULC change, with 95% (t)-intervals over tasks (negative values favor LeWi).
LeWi is competitive with the stronger baselines on average: mean differences range from −6.0% to +3.9%, with all confidence intervals including zero. The wide intervals reflect substantial variation across this diverse benchmark, motivating a closer examination of individual tasks.
Task-level results. We hypothesize that LeWi benefits from compositional sparsity, where prediction can be built from functions of small groups of variables. Exploratory analyses of four tasks where LeWi performs well reveal dependencies consistent with this hypothesis:
These cases suggest that LeWi’s gains align with localized interactions among relevant variables. We will include these results and experimental protocols in the revised paper.