acceptodds
Under review as a conference paper at ICLR 2027

Learning Computational Structure for Sample-Efficient Prediction

Abstract

Statistical learning in high dimensions suffers from the curse of dimensionality. Fortunately, many functions of practical interest are compositionally sparse, meaning they admit a composition of local functions, each depending on only a few variables, allowing sample complexity to depend on local arity rather than ambient dimension. Yet realizing this benefit requires exploiting hidden compositional structure, which theory does not establish for practical gradient-based training. We study this gap with CSFRBench, a controlled benchmark that hides the target's compositional computation during training but retains it for diagnosis, and find that dense networks and sparsity-based baselines leave much of that benefit unrealized. Inspired by the spatial organization of biological neural circuits, we introduce _Learned Wiring_ (LeWi): each neuron has a learned low-dimensional embedding, only nearby neurons connect, and embeddings and weights are trained jointly. LeWi gains most on hierarchical targets and those with intermediate reuse. Across 32 additional targets generated by independently sampled computation graphs, LeWi is the most sample-efficient method not given the target graph, with 27% lower mean learning-curve area than dense networks. Gains are smaller for flat low-arity compositions, while dense networks perform better on targets with a single high-arity function. LeWi's representations and connectivity align with the target computation. On 17 world-model prediction tasks spanning control, navigation, and manipulation, LeWi beats the dense baseline on 16 and wins 99/119 baseline comparisons. These results show that jointly learning connectivity and weights can exploit computational structure to improve sample efficiency.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
compositional sparsity, learned connectivity, sample efficiency, neuron embeddings, sparse neural networks, computational structure, world models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JXMnOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces LeWi (Learned Wiring), a neural network architecture that jointly learns connection weights and neuron positions in a low-dimensional space. Connections between neurons are determined by their distances, allowing the network to learn sparse, modular computational structures through gradient descent. The authors introduce CSFRBench, a benchmark of synthetic functions with hidden compositional structures, and show that LeWi achieves better sample efficiency than dense MLPs and several sparsity-based baselines, reducing the mean learning-curve area by 27% across 32 randomly generated computation graphs. They also evaluate LeWi on 17 world-model prediction tasks, outperforming dense MLPs on 16 of them. Through ablation studies, they demonstrate that learning connectivity is more beneficial than imposing fixed random sparsity, while additional analyses show that the learned representations and connections align with the underlying computational structure. However, LeWi is less effective on functions involving many interacting variables and introduces significant computational overhead.

Critical strengths
  1. The paper addresses an interesting problem by proposing a novel approach to neural network training that is intuitive and well-motivated.
  2. The paper is well-written, clearly structured, and easy to follow.
  3. The paper provides a comprehensive set of experiments, with strong and promising empirical results.
Critical weaknesses
  1. The paper evaluates the proposed method primarily on synthetic tasks and relatively old simulation benchmarks. There is no evaluation on real-world datasets, and even standard benchmarks such as MNIST and CIFAR-10 are not considered. This makes it difficult to assess the practical applicability and generalizability of the proposed approach beyond the specific settings studied in the paper.

  2. The experiments are limited to relatively small MLP architectures. However, modern deep learning relies on a much broader range of architectures, such as CNNs, Transformers, and GNNs. I believe that demonstrating the effectiveness of the proposed approach on architectures beyond small MLPs is important for assessing the broader significance and potential impact of the proposed paradigm.

Questions for authors
  1. Have the authors considered evaluating LeWi on standard classification datasets such as MNIST or CIFAR-10? This would help assess whether the benefits extend beyond synthetic and simulation-based prediction tasks.

  2. Can the proposed approach be extended to other architectures, such as CNNs, Transformers, or GNNs? What are the main technical challenges in doing so?

  3. Have the authors considered first pretraining a dense MLP and then learning the neuron embeddings while keeping the pretrained weights fixed or fine-tuning them? I believe this is an interesting experiment.

5 comments

[W1, Q1] Evaluation beyond synthetic and simulation-based tasks

We thank the reviewer for raising this concern. Our world-model tasks come from stable-worldmodel [1], a 2026 evaluation platform built on established benchmarks that remain widely used in recent world-model research, including TD-MPC2 [2], DreamerV3 [3], DINO-WM [4], and LeWorldModel [5]. We focus on world-model prediction tasks because such dynamics functions can exhibit local dependencies among state and action variables, as studied in factored MDPs [6] and object-centric world models [7]. This provides a natural setting to test the practical applicability of LeWi’s inductive bias towards compositional sparsity. We will clarify this motivation and the benchmarks' relevance in the paper.

To assess the generalizability of LeWi beyond these benchmarks, we conducted additional sample-efficiency experiments on the OpenML-CTR23 regression datasets [8], a curated suite of 35 tabular regression tasks. LeWi substantially improves on Dense and achieves competitive average performance with the strongest baselines in this evaluation.

Experimental setup. Since the datasets within OpenML-CTR23 come in different sizes, to measure sample efficiency across sample sizes, we use 19 datasets large enough to support training sizes of {256,512,1024,2048,4096}\{256, 512, 1024, 2048, 4096\} samples, together with separate validation sets one-quarter as large. We follow each dataset’s 10 predefined train/test splits, using the same examples for every method. Training sets are nested, so each larger set contains the smaller ones. LeWi and the MLP baselines use 256x4 MLPs (four hidden layers of width 256). We compare against Dense, L0L_0, L1L_1, L2L_2, RigL, and XGBoost. For each dataset, split, and training size, every method receives 32 Sobol-sampled hyperparameter configurations. The selected configuration is then retrained with three random seeds.

Results. LeWi reduces AULC relative to Dense by 33.4% on average, with lower AULC on 18 of 19 tasks.

BaselineLeWi’s relative AULC change (95% CI)
Dense−33.4% [−49.9%, −16.8%]
L0L_0−0.4% [−30.1%, +29.3%]
L1L_1+3.9% [−40.5%, +48.2%]
L2L_2−6.0% [−48.0%, +35.9%]
RigL−4.7% [−14.9%, +5.5%]
XGBoost−4.7% [−35.4%, +26.1%]

Mean per-task relative AULC change, with 95% (t)-intervals over tasks (negative values favor LeWi).

LeWi is competitive with the stronger baselines on average: mean differences range from −6.0% to +3.9%, with all confidence intervals including zero. The wide intervals reflect substantial variation across this diverse benchmark, motivating a closer examination of individual tasks.

Task-level results. We hypothesize that LeWi benefits from compositional sparsity, where prediction can be built from functions of small groups of variables. Exploratory analyses of four tasks where LeWi performs well reveal dependencies consistent with this hypothesis:

  • Grid stability. LeWi outperforms all baselines. The strongest estimated interactions pair reaction time and price elasticity within each grid node, and LeWi’s learned wiring preferentially connects these pairs.
  • Airfoil self-noise. LeWi outperforms all baselines except L2L_2, where there is no clear difference. Much of the benefit from modeling interactions comes from a single coupling between frequency and boundary-layer displacement thickness.
  • Auction verification. LeWi outperforms all MLP baselines, while XGBoost performs better. The estimated dependencies center on a product indicator that changes how capacity and price affect verification time.
  • Naval propulsion. LeWi outperforms Dense, L0L_0, L1L_1, and XGBoost, with no clear difference from L2L_2 or RigL. Predicting compressor degradation is well captured by pairwise interactions among sensor readings.

These cases suggest that LeWi’s gains align with localized interactions among relevant variables. We will include these results and experimental protocols in the revised paper.

[W2, Q2] Extensions beyond MLPs

We thank the reviewer for highlighting the importance of evaluating LeWi beyond MLPs. We use a common MLP backbone to compare methods for learning connectivity. This choice is consistent with theoretical work on learning compositionally sparse functions using feedforward networks [9–11], which establishes the theoretical sample-efficiency gains available from such functions and directly motivates our work. MLPs are also widely used as the primary architecture in related research: Neural Sculpting [12] studies hierarchical modularity through pruning, and Boopathy et al. [13] investigate sample complexity in fully connected modular networks.

We agree that extending LeWi to broader architectures is an important direction, and we are actively investigating it. However, a systematic extension to Transformers, CNNs, and GNNs is beyond the scope of the present study. Nevertheless, our preliminary Transformer experiments provide encouraging results, and we outline possible CNN and GNN adaptations below.

Transformers. We investigated applying LeWi’s distance-based connectivity masks to the weight matrices that project token representations into queries, keys, and values, and transform attention outputs (QKVO), or to the two linear layers of each feed-forward network (FFN), or both.

We evaluated these adaptations using a three-block Transformer (width 64, four attention heads) in which each input coordinate is represented as a token. The experiments cover three CSFRBench settings (depth-4 tree, sampled depth-4 DAGs, and Fourier tree; nine instances total), with 64,000 training steps. To assess the contribution of learned connectivity, we also compare LeWi-QKVO against a fixed-random-mask control with the same initial number of active connections and effective weight norm in each projection. The control trains its weights but keeps its connectivity fixed.

AdaptationComparisonRelative AULC (95% CI)
LeWi-FFNDense+12.1% [+7.7%, +16.5%]
LeWi-QKVO+FFNDense−7.8% [−14.3%, −1.3%]
LeWi-QKVODense−21.6% [−29.1%, −14.2%]
LeWi-QKVOMatched random QKVO−4.2% [−7.6%, −0.9%]

Negative differences indicate lower AULC.

Applying LeWi to QKVO reduces AULC by 21.6% relative to Dense, whereas applying it only to FFNs increases AULC by 12.1%. This might be because each token represents one input coordinate, so interactions between input coordinates occur through attention, whereas FFNs transform each token independently. LeWi-QKVO also achieves 4.2% lower AULC than the matched random control, providing preliminary evidence of a benefit from learned connectivity beyond initial sparsity and weight scale.

CNNs. Following work on spatially organized convolutional units [14, 15], LeWi could assign an embedding to each feature channel and learn connectivity between input and output channels.

GNNs. Message-passing GNNs use shared neural networks to compute messages and update node features [16]. LeWi could be applied within these functions to learn feature-level connectivity without modifying the graph's adjacency.

We will include the preliminary Transformer results in the revised appendix and discuss these architectural extensions in the Discussion.

[Q3] Learning LeWi after Dense pretraining

We thank the reviewer for this suggestion. Since LeWi's neuron embeddings and weights are closely coupled, it is not immediately clear how best to adapt a pretrained Dense network. We investigate one approach using weight-informed initialization to preserve pretrained predictions, followed by either optimizing embeddings alone or jointly fine-tuning embeddings and weights.

Setup. We conduct experiments on CSFRBench depth-4 trees and reuse DAGs (16 instances each), with n∈{512,2048,8192}n\in\{512,2048,8192\}. We pretrain a Dense MLP for 32,000 steps, introduce LeWi-style connectivity through weight-informed initialization, and train for another 32,000 steps under two settings: (1) optimizing neuron embeddings while freezing pretrained weights, and (2) jointly fine-tuning embeddings and weights. We compare against continued Dense training with the same optimizer restart and LeWi trained from scratch for 64,000 steps. Training schedules are matched, with hyperparameters selected by validation error.

Weight-informed initialization. Directly applying LeWi with randomly initialized embeddings closes approximately 40% of the pretrained connections and substantially disrupts predictions. To enable a meaningful post-training comparison, we initialize LeWi's geometry from the pretrained network:

  1. Score connections using Wanda-style importance [17], Sij=∣Wij∣∣aj∣2S_{ij}=|W_{ij}||a_j|_2, where aja_j contains the sending neuron’s activations over the training samples.
  2. Select the highest-scoring positive-importance connections, up to 60% per layer.
  3. Fit neuron embeddings using 2,000 Adam steps and a squared-hinge objective that encourages selected connections to lie within LeWi's connection radius and other connections outside it.
  4. Rescale the retained weights to compensate for LeWi's soft gates.

This conversion preserves validation NMSE to within 5% of the pretrained network in 87/96 conditions (32 target instances × 3 training-set sizes).

Training strategyDepth-4 treeReuse DAG
Pretrained Dense (32k steps)0.7630.394
Continued Dense (64k)0.7520.379
Dense → LeWi-style wiring, weights frozen (64k)0.7550.378
Dense → LeWi-style wiring, joint fine-tuning (64k)0.7210.354
LeWi from scratch (64k)0.6480.256

Results. Joint fine-tuning improves AULC over continued Dense training by 4.1% on depth-4 trees and 6.6% on reuse DAGs, outperforming it on 15/16 and 16/16 instances, respectively. In contrast, optimizing embeddings alone with pretrained weights frozen yields no significant improvement (+0.4% and −0.2%). Joint fine-tuning also outperforms frozen-weight adaptation by 4.5% and 6.4%. However, LeWi trained from scratch remains substantially better, with joint post-training adaptation yielding 11.3% and 38.0% higher AULC, respectively.

These results show that adapting a pretrained Dense network using LeWi-style connectivity can improve prediction when weights and embeddings are jointly optimized, but learning computational structure from the beginning is more effective, potentially because it allows more time for the coupled optimization of connectivity and local functions.

We will include the experimental protocol and complete results in the revised appendix.

[Additional clarification] Computational overhead

We thank the reviewer for noting the computational overhead of learning connectivity. LeWi’s additional training cost comes from computing neuron distances, gates, masks, and optimizing the neuron embeddings. We have accelerated training by batching independent networks and using fused C++ kernels for LeWi, while also optimizing Dense through batching and compilation. Under these matched CPU measurements, LeWi’s end-to-end runtime per network is 1.32x and 1.01x that of Dense on the 32x4 and 64x8 MLPs, respectively (Appendix D.7, Table 6).

Our implementation still uses dense operations during training, so exploiting the learned sparsity to further reduce computation remains a direction for future work. At inference, the learned gates and masks can be folded into fixed weights, eliminating the need to compute neuron distances, gates, or masks during prediction.

Reviewer_Q3wUOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether gradient-based training can exploit hidden compositional sparsity (targets built from low-arity local functions on a sparse DAG) to improve sample efficiency. It makes two contributions. (i) CSFRBench, a synthetic regression benchmark (d = 8 active inputs, arity 2; one depth-5 extension with d = 16). Its targets are built from fixed and sampled computation graphs with additive–multiplicative or random-Fourier local functions. It also includes graph-informed masked-MLP references (GraphOracle(ℓ)) and diagnostics: linear probes, Boundless-DAS interventions, and module-graph edge F1. (ii) LeWi, which gives each neuron a learned r = 3 embedding. Adjacent-layer neurons connect only within a fixed radius R, through a sigmoid gate times a hard mask, and embeddings and weights are trained jointly. Across 16–32 target instances per setting, LeWi has the lowest mean AULC among methods without graph access on the low-arity settings. Its headline gains over Dense are 20% (depth-4 tree), 34% (reuse DAG) and 27% (32 sampled DAGs). It loses to Dense on the high-arity control and shows better probe and intervention alignment with target intermediates. On 17 stable-worldmodel state-prediction tasks (256×4 MLP, 16-step rollouts), it wins 99 of 119 task–baseline comparisons.

Critical strengths
  1. Very careful experimental protocol. The authors froze the search spaces on development targets before sampling the evaluation targets, used 16–32 independent target instances, reported paired per-instance confidence intervals with an explicit "interval contains zero ⇒ inconclusive" rule, and audited hyperparameter boundaries (App. C.5, C.12, C.13). This makes the CSFRBench comparisons more credible than is typical for architecture papers, and it is the main reason I weigh the results seriously.

  2. Predictive gains are linked to mechanism, not just reported. Graph-oracle references quantify how much benefit target connectivity can offer, and LeWi closes about 51% of the Dense→GraphOracle(all) gap on the depth-4 tree. Probes and DAS interventions show the target hierarchy in LeWi's representations (Fig. 5), and the ablations show that a random sparse mask and frozen embeddings are both worse (Fig. 6c). Together these support the claim that the gain comes from learned connectivity rather than generic sparsity.

Critical weaknesses

Neither weakness undermines the main contribution; both are fixable in revision.

  1. Framing of effect sizes (presentation, not substance). The abstract and §6.2 quote gains relative to Dense, which is often the least stable comparator (Fig. 6a notes divergent Dense runs; the depth-5 Dense AULC is 1.185 ± 0.572). Against the strongest learned baseline the gains are smaller but still consistent: depth-4 tree 0.508 vs 0.569 (L0), about 11%; sampled DAGs 0.197 vs 0.241 (RigL), about 18%; reuse DAG 0.179 vs 0.211 (BIMT), about 15% (Tables 3/4). The "32% lower AULC than Dense on the depth-5 tree" (§6.2) has a paired interval of [−0.950, +0.203] (Table 4), which by the authors' own rule (§6.1, App. C.13) is inconclusive. I suggest reporting gains against the best baseline in the abstract and introduction, and restating the depth-5 result as inconclusive. This would make the claims more accurate without weakening the conclusion that LeWi improves on strong sparsity baselines.

  2. Scale of the evidence. The CSFRBench experiments are deliberately controlled and small (8 active inputs, 16 for depth-5; arity-2 local functions; 32×4 or 64×8 MLPs). This is appropriate for a first study, but it leaves open how the advantage evolves as the problem grows. A modest experiment scaling d with arity fixed would show whether LeWi's advantage grows as the theory suggests. For the world-model study, it would help to note that the CIs are over training seeds on a fixed dataset, to report effect sizes alongside win counts (several wins are small, e.g. 2–10% NMSE on Cheetah, Walker, Quadruped and Hopper, Fig. 22), and to mention that the 119 comparisons are not corrected for multiplicity.

Questions for authors
  1. [Clarification] Hyperparameter budget: LeWi searches only the learning rate (r, R, σ_e are fixed constants chosen on development targets), while ℓ1, ℓ2, L0, RigL, BIMT and Fiedler search a 2-D space with the same 32 Sobol configurations. Is the per-dimension search density comparable? How do the baselines change with a 64-configuration budget, or with LeWi's geometric constants added to its search?
  2. [Clarification] Model selection uses a fixed 4,096-example labelled validation set for every n, including n = 512 (App. C.8). How do conclusions change with a validation set that scales with n (e.g. 20% of n)? This bears directly on the sample-efficiency claim, and methods with more tunable hyperparameters may benefit differently.
  3. [Clarification] The learned-neuron-positions baseline (Erb et al., 2025) is far worse than Dense everywhere (e.g. 0.861 vs 0.633 on the depth-4 tree; 0.233 vs 0.018 on flat modules, Table 3). Is this a faithful reimplementation, and what explains it failing so badly even on flat modules? As the closest prior method, it matters that it is well tuned.
  4. [Clarification] Please restate the depth-5 Dense comparison in light of the paired interval containing zero, and report gains relative to the best non-oracle baseline in the abstract.
  5. [Suggestion] A scaling experiment in d (e.g. d ∈ {8, 16, 32, 64} active inputs, arity fixed) showing whether LeWi's gap to Dense and to GraphOracle(all) grows or shrinks.
  6. [Clarification] On the world-model tasks, are the CIs over training seeds only, with the dataset fixed? Would resampling trajectories change the win/tie/loss counts?
  7. [Clarification] Section 2 contrasts LeWi with Lin et al. (2026), who infer dependencies from data to construct a fixed sparse architecture before training. Since both aim to recover target computational structure, how do the connectivity patterns LeWi learns compare with those obtained by such a data-driven construction on CSFRBench targets? A comparison of predictive results, or at least of how well each recovers the target graph (e.g. module-graph edge F1 against the ground truth, which the benchmark already provides), would clarify whether learning wiring during training offers advantages over inferring it beforehand.
4 comments

[W1] Effect-size reporting

We thank the reviewer for this helpful suggestion. We agree that comparisons against the strongest non-oracle baselines provide a more informative characterization of LeWi's improvements. We will revise the abstract, introduction, and Section 6.2 accordingly, and clarify the statistically inconclusive depth-5 comparison against Dense, as discussed further in our responses to [Q1] and [Q4] below.

[W2, Q5] Scaling to larger compositional targets

We thank the reviewer for suggesting this scaling experiment. We extended CSFRBench’s balanced tree targets to d=32d = 32 active inputs, keeping local arity fixed at two and increasing network capacity and training budgets with target graph size. We evaluated 16 independently sampled target functions at training-set sizes n∈{217,219,221}n\in\{2^{17}, 2^{19}, 2^{21}\}, using 1,024,000 training steps for each method. The resulting mean AULCs are:

LeWiDenseL0L_0RigLGraphOracle(all)
0.7850.8990.8960.8960.224

LeWi achieves lower AULC than Dense by 0.115 (paired 95% CI [0.092, 0.137]), and then both L0L_0 and RigL by 0.111, outperforming all three on 16/16 targets. These results demonstrate that LeWi’s sample efficiency advantage extends to larger compositional targets under the evaluated training budget. Relative to GraphOracle(all), LeWi closes approximately 17% of the gap from Dense, indicating that substantial benefits remain available from exploiting hidden computational structure. The magnitude of this advantage may also depend on the training budget. We will include the complete scaling results and experimental protocols in the revised appendix.

[Q1] Baseline tuning fairness

We thank the reviewer for this question. To assess the effect of additional tuning, we increased the search budget from 32 to 64 configurations for the strongest non-oracle baseline on each of three central CSFRBench settings, retaining LeWi’s original 32 configurations. The additional trials extend the original Sobol sequences, with unchanged search ranges and model-selection procedures.

SettingBaselineLeWi AULC (32 trials)Baseline AULC (32 → 64)LeWi advantage (32 → 64)
Depth-4 treeL0L_00.5080.569 → 0.56310.7% → 9.8%
Reuse DAGBIMT0.1790.211 → 0.20015.1% → 10.8%
Sampled DAGsRigL0.1970.241 → 0.23118.3% → 14.9%

The additional tuning improves all three baselines, but LeWi retains lower AULC, with paired 95% confidence intervals excluding zero in all three settings. We will include the results in the revised appendix.

[Q2] Validation data budget

We thank the reviewer for pointing out the importance of accounting for validation labels. We repeated hyperparameter and checkpoint selection using nval=⌈0.2n⌉n_{\mathrm{val}}=\lceil0.2n\rceil validation examples, rather than the fixed 4,096 examples used in the submission.

We evaluated the depth-4 tree (16 instances), reuse DAG (16 instances), and sampled depth-4 DAGs (32 instances), using three independently selected validation subsets per condition. The training runs, hyperparameter candidates, and test sets remain unchanged.

SettingLeWi − strongest baseline, original20% validation
Depth-4 tree (L0)−0.061−0.068
Reuse DAG (BIMT)−0.032−0.034
Sampled DAGs (RigL)−0.044−0.044

All differences retain paired 95% confidence intervals excluding zero. The changes in LeWi's advantage between validation protocols are not statistically conclusive. These results show that LeWi's relative sample-efficiency advantage persists when validation data scale with the training-set size. We will report learning curves against both training-set size and total labeled-data budget in the revised appendix.

[Q3] Neuron-position baseline

We thank the reviewer for highlighting this comparison. We double-checked our implementation against Erb et al. (2025) and their released code, including forward computations, gradients, parameter counts, and training behavior. Our original implementation follows their inverse-distance weight parameterization but uses fixed interlayer distances and the benchmark's standard linear readout. Erb et al. also consider learned interlayer distances, which remove the upper bound on effective weight magnitudes imposed by fixed unit spacing.

To investigate the poor performance, we measured training error and evaluated additional configurations. The original neuron-position model exhibits substantial underfitting: its training/test AULCs are 0.198/0.233 on flat modules and 0.794/0.861 on depth-4 trees, compared with Dense training AULCs of 0.013 and 0.385, respectively.

On flat modules, learning interlayer distances reduces the neuron-position model's test AULC from 0.233 to 0.169 (paired difference −0.064, 95% CI [−0.076, −0.051]), although it remains above Dense (0.018) and LeWi (0.008). Erb et al. similarly report improved MNIST accuracy with learned interlayer distances (90.18% to 92.17%), while conventional MLPs still perform better (94.29%–97.45%). Their experiments use single-hidden-layer networks for image classification, whereas ours evaluate deeper networks on compositional function regression. On depth-4 trees, none of the tested variants provides a comparable improvement.

We will clarify the original baseline configuration and include the implementation audit, training-error diagnostics, and learned-gap results in the revised appendix.

[Q4] Depth-5 comparison and abstract reporting

We thank the reviewer for these suggestions. On the depth-5 tree, LeWi achieves mean AULC 0.812 versus Dense's 1.185, but the paired difference of -0.374 has a 95% confidence interval of [-0.950,+0.203]. We agree that this comparison is statistically inconclusive and will revise Section 6.2 accordingly. The wide interval partly reflects an extreme Dense run with test NMSE 52.6 despite validation NMSE near 1. Importantly, LeWi still outperforms the strongest non-oracle baseline, L0L_0 (AULC 0.882), with a paired difference of −0.070 (95% CI [−0.089, −0.052]). We will also revise the abstract to report the improvement against the strongest non-oracle baseline.

[W2, Q6] World-model uncertainty and effect sizes

We thank the reviewer for highlighting these statistical considerations. The confidence intervals in our original submission reflect 16 training seeds on a fixed trajectory dataset per task. We have now assessed variability across training datasets on all 17 tasks.

Effect sizes and multiple comparisons. Applying the Holm–Bonferroni correction across all 119 task–baseline comparisons changes the original 99 wins, 6 ties, and 14 losses to 93 wins, 13 ties, and 13 losses, preserving all 16/17 wins against Dense. Among the original wins, the median reduction in geometric-mean rollout NMSE is 38%, though 19 improvements are below 10% (e.g., Walker 2.1%, Quadruped 5.1%, and Cheetah 7.8% against Dense).

Dataset variability. We additionally retrained LeWi and Dense on all 17 tasks using five independently sampled training datasets and four paired training seeds per dataset (20 runs per method and training-set size), with architectures, training budgets, hyperparameters, and test episodes unchanged. The outcome against Dense is unchanged on all 17 tasks: 16 wins, 0 ties, and 1 loss. On each of the 16 winning tasks, LeWi has lower seed-averaged AULC on all five new training datasets.

For Walker (control), Two-room (navigation), and Manipulator (manipulation), the LeWi–Dense differences in AULC of log-rollout NMSE (negative favors LeWi) are:

TaskOriginal training set, 16 seeds (95% CI)Resampled datasets (95% CI)
Walker−0.009 [−0.011, −0.007]−0.009 [−0.011, −0.006]
Two-room−0.306 [−0.371, −0.241]−0.421 [−0.518, −0.324]
Manipulator+0.062 [+0.026, +0.098]+0.071 [+0.009, +0.134]

For the resampled experiments, we average the paired differences over four seeds within each dataset, then compute 95% confidence intervals across the five dataset means. For these three tasks, the conclusions also remain unchanged when evaluated on newly generated test episodes.

We will include the corrected comparisons, full effect-size distribution, and dataset-resampling analysis in the revised appendix.

[Q7] Comparison with Lin et al.

We thank the reviewer for suggesting this comparison. We evaluated the published HNN architecture and an adaptation of its inferred connectivity to our MLP, measuring predictive sample efficiency and module-graph alignment on five CSFRBench settings comprising 96 targets.

1. Comparison with the published HNN architecture. We implemented HNN [1] using its published architecture and released dependency-inference code. HNN organizes input dependencies into subsets, assigns one neuron per subset, and fixes connectivity before weight training. We report the stronger HNN (m-s), which uses target-conditioned input correlations. As an implementation check, its mean rank across the 35 OpenML-CTR23 [2] tasks closely matches Lin et al. ’s reported result under their training protocol (3.74 versus 3.77 among seven methods).

CSFRBench settingLeWi AULC ↓HNN (m-s) AULC ↓LeWi edge F1 ↑HNN (m-s) edge F1 ↑
Depth-4 tree0.5080.7000.430.28
Reuse DAG0.1790.3850.620.33
Sampled depth-4 DAGs0.1970.3100.470.38
Flat modules0.00760.02680.850.68
High-arity node0.0850.1640.400.33

AULC measures the area under the test-NMSE learning curve, with lower being better. Edge F1 uses the same activation-probe alignment and graph-scoring procedure, with LeWi’s learned neuron clusters and HNN’s predefined subset neurons as modules. F1 scores are averaged over seven training-set sizes.

LeWi achieves lower AULC on all 96 targets and higher graph alignment on the four compositionally sparse settings. The edge-F1 difference on the high-arity control is inconclusive.

HNN differs from our MLP in its per-subset capacity, normalization, and readouts. To further investigate whether learning wiring during training offers advantages over inferring it beforehand, we additionally compare its inferred connectivity within the same MLP backbone.

2. HNN-derived masks on the same MLP. Similar to how we construct connectivity masks for GraphOracle references, here we construct a fixed mask from Lin et al. ‘s inferred structure and apply it to the same 32x4 MLP backbone, called IFN-full-MLP:

  1. Estimate the hierarchy. Retain every inferred pair, triple, and quadruple of input features from Lin et al. ‘s MFCF procedure.
  2. Convert inferred subsets into neuron blocks. Pair blocks read their two input variables; triple blocks read their constituent pair blocks; quadruple blocks read their constituent triple blocks.
  3. Allocate layers and neurons using GraphOracle-style masking. Layers 1–3 compute pairs, triples, and quadruples; layer 4 refines each quadruple block locally. Each computed block receives at least one neuron, with remaining neurons distributed evenly within the width of 32. If a pair or triple does not feed into a larger subset, its output is passed through the remaining layers to reach the final readout.

We then train the MLP with the fixed mask, without HNN’s intermediate-layer readouts.

We evaluate seven sample sizes with three optimization seeds and 32 random hyperparameter configurations per condition, using separate development targets to set search ranges and validation error for model selection.

CSFRBench settingLeWi AULC ↓IFN-full-MLP AULC ↓LeWi edge F1 ↑IFN-full-MLP edge F1 ↑
Depth-4 tree0.50670.76790.600.46
Reuse DAG0.18360.44720.760.58
Sampled depth-4 DAGs0.19840.35820.580.54
Flat modules0.00840.03290.920.77
High-arity node0.08470.32890.370.43

Results average three seeds per target. F1 is evaluated at n=215n=2^{15}, using IFN-full-MLP’s predefined blocks as modules and the same alignment procedure.

LeWi again achieves lower AULC on all 96 targets. All five predictive differences remain significant after Holm correction over the ten planned predictive and structural comparisons. F1 gains remain significant on depth-4 trees, reuse DAGs, and flat modules; the other two structural differences are inconclusive. Permuting the inferred masks’ input assignments worsens prediction on every setting, confirming that the inferred dependencies are informative.

Discussion. These results are consistent with a benefit from coupling connectivity learning with local-function learning: LeWi can adjust which intermediate computations interact as their functions develop. The fixed hierarchy commits to a decomposition before training. In the MLP adaptation, it also allocates few neurons per subset and restricts nonlinear interactions to at most four inputs, contributing to underfitting. Within this backbone, LeWi achieves better prediction alongside stronger module-graph alignment on several settings.

We will include the complete comparisons and controls in the revised appendix.

AuthorsOpenReview ↗

References for our responses

[1] Lin, H., Briola, A., Wang, Y., & Aste, T. (2026). Compositional Sparsity as an Inductive Bias for Neural Architecture Design. arXiv preprint arXiv:2605.14764.

[2] Fischer, S. F., Feurer, M., & Bischl, B. (2023). OpenML-CTR23–a curated tabular regression benchmarking suite. In AutoML Conference 2023 (Workshop).

Reviewer_hGQgOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Learned Wiring (LeWi), a biologically inspired method that learns spatial embeddings for neurons connections to nearby neighbors.

Critical strengths

On 17 world-model prediction tasks spanning control, navigation, and manipulation, LeWi beats the dense baseline on 16 and wins 99/119 baseline comparisons. These results show that jointly learning connectivity and weights can exploit computational structure to improve sample efficiency.

Critical weaknesses

While the motivation of exploiting compositional sparsity is sound and the geometric approach is conceptually interesting, the technical novelty is limited, and the empirical evaluation relies heavily on favorable synthetic settings while hiding critical failure modes.

Questions for authors

1.Limited Technical Novelty: The core mechanism of LeWi—using a binary mask for connectivity and propagating gradients through a soft gate (Eq. 2 and Eq. 3)—is essentially a variant of the Straight-Through Estimator (STE). These techniques have been extensively studied in dynamic sparse training (DST) and network pruning (e.g., Bengio et al., 2013;). The authors need to explicitly clarify the fundamental algorithmic difference between LeWi and existing dynamic sparse training methods. 2.Weak Failure Analysis: LeWi exhibits worse sample efficiency than the dense baseline at small sample sizes, and only barely catches up with massive amounts of data in Figure 4. The authors claim LeWi learns compositional structure, but it clearly fails when the target function requires global interactions.  3.Over-engineering: Many technical details, mathematical derivations, and data generation processes are relegated to the Appendix. The main text lacks sufficient detail for a reader to reproduce or fully understand the method. For instance, the derivation of Eq. (3), which is the heart of the "learned wiring" mechanism, is just hand-waved in the main text and outsourced to the appendix.

1 comment

[W1, Q1] Technical novelty: distinction from STE and dynamic sparse training

We thank the reviewer for highlighting the connection between LeWi and existing dynamic sparse training (DST) methods. While LeWi also uses binary connectivity masks, its gradient computation does not use a straight-through estimator (STE). During backpropagation, the binary masks are held fixed, without surrogate gradients propagating through them. Gradients propagate only through the sigmoid gates of active connections, while inactive connections receive no direct embedding gradient.

More importantly, LeWi’s algorithmic novelty lies in its shared geometric parameterization of connectivity. DST methods such as SET [1] and RigL [2] update each individual connection through weight-based pruning and random or gradient-based regrowth, respectively. LeWi instead determines active connections from distances between learned low-dimensional neuron embeddings. Updating one embedding simultaneously changes the scores of all its incident connections, coupling connectivity decisions through shared geometry. This induces a low-rank connection-score matrix (Appendix B.2), which we hypothesize provides a useful inductive bias for compositional sparsity.

We have made the gradient computation and its distinction from STE more explicit in Section 4.2 and added the derivation to Appendix B.

[1] Mocanu, D. C., Mocanu, E., Stone, P., Nguyen, P. H., Gibescu, M., & Liotta, A. (2018). Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 9(1), 2383.

[2] Evci, U., Gale, T., Menick, J., Castro, P. S., & Elsen, E. (2020, November). Rigging the lottery: Making all tickets winners. In International conference on machine learning (pp. 2943-2952). PMLR.

[W2] Scope of the empirical evaluation

We thank the reviewer for raising the question of generalization beyond synthetic settings. CSFRBench is controlled by design: its known computational graphs allow us to isolate how compositional structure affects sample efficiency and assess whether learned wiring recovers that structure. We also include a high-arity negative control without a low-arity decomposition, where Dense outperforms LeWi.

Our original submission also includes 17 world-model prediction tasks from stable-worldmodel (SWM), spanning navigation, control, and manipulation. LeWi outperforms Dense on 16 of 17 tasks. These experiments evaluate sample efficiency on established simulation benchmarks whose dynamics are not constructed to have a prescribed compositional structure.

As discussed in the response to reviewer JXMn’s Q1 below, we have further added sample-efficiency experiments on 19 OpenML-CTR23 tabular regression datasets. LeWi reduces AULC relative to Dense by 33.4% on average, with lower AULC on 18 of 19 tasks, while remaining competitive on average with the stronger baselines. We will include these additional results and experimental protocols in the revised paper.

[W3, Q2] Clarification of Figure 4 and high-arity limitations

We thank the reviewer for raising the concerns about LeWi’s performance relative to Dense at small sample sizes (Figure 4) and on targets requiring global interactions, and welcome the opportunity to clarify our results. First, Figure 4 shows that LeWi improves sample efficiency on compositionally sparse targets, including at small training-set sizes. Relative to Dense, LeWi reduces mean AULC by 20% on the depth-4 tree, 34% on the reuse DAG, and 27% across 32 independently sampled depth-4 DAGs. These gains are visible at small sample sizes, rather than LeWi merely catching up at large sample sizes.

Second, we deliberately include the “High-arity node” setting as a control and explicitly discuss it in the Abstract, Introduction, and Section 6.2 of our original submission. These targets consist of a single eight-input nonlinear function without a low-arity decomposition. Dense outperforms LeWi on this control (mean AULC 0.066 vs 0.085), consistent with LeWi’s inductive bias being less useful when there is no low-arity structure to exploit. This result illustrates the boundary of LeWi’s benefits, which are strongest on sparse compositions rather than high-arity interactions.

[Q3] Clarification of Eq. (3) and reproducibility details

We thank the reviewer for pointing out that the presentation of Eq. (3) could be clearer. While Eq. (3) and its gradient interpretation are already provided in the main text, we agree that a more explicit derivation would help readers understand the mechanism. As noted in our response to [W1, Q1] above, we have added the full derivation to Appendix B and clarified the gradient computation in Section 4.2.

We will also add more references to the appendix, where detailed experimental protocols, hyperparameters, and data generation procedures are provided to ensure reproducibility while respecting the main-text page limit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.