- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how neural networks allocate computation across different paths during training. The authors derive theoretical results on routing dynamics, stability, and the activation of dormant subnetworks, supported by experiments on linear and ReLU networks.
Critical strengths
The paper addresses an interesting theoretical question and provides a unified framework for understanding routing dynamics. The mathematical analysis appears to offer useful insights into path specialization.
Critical weaknesses
The work appears theoretically interesting, but I am uncertain about its broader implications. My assessment is tentative given my limited confidence in evaluating the technical proofs.
- Most theoretical results rely on restrictive assumptions, particularly positive scalar networks. Their applicability to practical neural networks remains unclear.
- The main stability results are asymptotic, while practical training operates under finite budgets. The paper provides limited evidence connecting these theoretical predictions to typical training dynamics.
- The nonlinear experiments are mostly illustrative and do not directly validate the proposed theoretical mechanisms.
Questions for authors
Please see weakness
2 comments
Reviewer 3, Response 2 of 2
Predicting the amount of collapse and the selected subnetwork. Linearizing the routing flow at the symmetric state, with no simulation and no fitted constant, predicts the share at the fit with correlation 0.95 across 241 graphs at an initial weight of (mean error 3 percentage points), and predicts a route lying in the selected subnetwork in a median of 88% of starts, against 11% by chance (Appendix O.3).
Typical nonlinear training. The ordering by coherence depth carries over to signed, matrix, tanh and ReLU networks from the standard initialization (Appendix Q). In the new Figure 4, every ReLU seed from the standard initialization settles on one candidate subnetwork while most of the error remains, then fits on it; the retained candidate carries 98–100% of the output at the fit.
Finite budgets and weight decay. Without weight decay the routing state freezes at the fit, before collapse is exact. With weight decay , equal-depth positive scalar routing follows the same flow under the clock , which keeps advancing after the fit, and generically approaches a stable product state at finite capacity whenever its limit is nonzero (Theorem 6, Appendix P). In the census, 89–100% of runs with weight decay end on layerwise-complete subnetworks in every range of coherence depth.
Nonlinear evidence: Weakness 3
The nonlinear experiments are mostly illustrative and do not directly validate the proposed theoretical mechanisms.
We agree that the original nonlinear experiments were illustrations. We have replaced or supplemented them with experiments that test specific predictions of the theory, and we report every run.
Figure 4. The single ReLU run is replaced by every seed: eight ReLU networks from the standard LeCun initialization on a census graph with four layerwise-complete candidate subnetworks, beside eight positive scalar networks on the same graph. The theory predicts that routing selects one candidate early and that learning then continues on it. Every ReLU seed shows this: routing settles at 35–85% of the time to fit, while 53–98% of the squared error remains, and the retained candidate (different in different seeds) carries 98–100% of the output at the fit, while the total squared weight grows by only 4–5% (Appendix K).
Predictions tested across graphs (Appendix Q). On 117 graphs in signed, matrix, tanh and ReLU networks, the amount of selection at the fit is ordered by coherence depth in every class; long training with weight decay ends on layerwise-complete supports, the form of the scalar stable states, in 90–100% of runs on graphs with ; and with several outputs, supports become layerwise complete only per singular mode in linear and tanh networks and per input item in ReLU networks, consistent with each such component being routed separately.
Dormant recruitment on real data (Appendix S, pilot). On CIFAR-10, a dormant branch whose last layers start at a small scale is recruited after a delay that grows with and with , consistent with the scalar activation-order calculation, and only while the active network cannot fit the data. In four-stream CNNs started at half the He scale, parallel streams concentrate 82% of each class's output on one stream, against 59–60% when the streams mix.
Figure 6B. The ReLU rescaling comparison now uses 24 paired seeds evaluated at a common fitting progress (see our response to Reviewer 1, Question 2).
These experiments extend the empirical evidence beyond the scalar setting, although the scalar theorems do not prove the corresponding claims for nonlinear networks.
Reviewer 3, Response 1 of 2
Your comments are quoted below and keep their references to the submitted paper; references in our responses point to the revised paper.
Scope: Weakness 1
Thank you. We have added a systematic test of the theory's predictions in signed, matrix and nonlinear networks, and in residual architectures with paths of mixed depth.
Signed, matrix and nonlinear networks (Appendix Q). We trained signed scalar, linear matrix, tanh and ReLU networks on 117 graphs of depth up to 10, from the standard LeCun initialization and six smaller scales. First, how much a network selects at the fit is largely ordered by coherence depth in every class: at the standard initialization, graphs with λ≥7 carry a median of 81% (matrix), 82% (tanh) and 97% (ReLU) of their output on the eventually selected subnetwork. Second, after long training with weight decay, the thresholded supports have the layerwise-complete form of the scalar stable states in 90–100% of runs on graphs with λ≥5, with exceptions only where the subnetwork must contain a block of two active vertices on each side of a layer (Appendix Q.2). Third, with several outputs (the hierarchical task of Shi et al., 2022), supports become layerwise complete only when measured per singular mode in linear and tanh networks, and per input item in ReLU networks (Appendix Q.3).
Mixed depth (Appendix R). In ResNet-like architectures, all 304 positive scalar runs end numerically consistent with balanced product states on layerwise-complete subnetworks whose paths have one depth. Small changes to the architecture, or to where the initialization is concentrated, change which subnetwork is selected, and ReLU networks with matrix vertices make the same choices (see our responses to Reviewer 2).
We present these results as existence demonstrations, since a theory for networks whose activation patterns change during training remains open, but they show the ordering by coherence depth and the layerwise-complete endpoints of the scalar theory in all of these network classes.
Finite training: Weakness 2
We have now added an extensive finite-training analysis, including a large empirical suite and several analytic results (Appendix O, a new passage in Section 5 and a new Figure 4; see also our response to Reviewer 1). Ordinary initializations can produce substantial collapse, especially in graphs with long route memory, and coherence depth predicts where. The analysis also predicts when to expect collapse, how much occurs at the fit, and which subnetwork is selected.
How much collapse at an ordinary fit. We measure the share of the output carried at the fit by the subnetwork that routing eventually selects. In a census of 244 random four-stream graphs at an initial weight of 1/3, comparable to standard schemes, that subnetwork carries at least half of the output in 70% of graphs with λ≥7 and 28% with 5≤λ<7, but in only one of 168 graphs with λ<5; at 0.1, the first two numbers rise to 91% and 77%. Graphs with less collapse still move toward the eventual subnetwork, since the share rises monotonically along every run.
When to expect it. For a balanced positive scalar network of depth d with initial output f0, the per-layer weight growth at the fit is at most f0−1/d, about a factor of 2 at an initial weight of 1/3 (Proposition 6). Near the symmetric state an unstable routing mode grows like Gλi−2, so this modest budget produces substantial selection only when route memory is long. For parallel routes the result is exact (Proposition 8): with a 10% capacity gap, the losing route falls below 1% of the output from initial weights of 0.44 at depth 8, but only below 0.012 at depth 3, and at depth 2 the shares never change.