acceptodds
Under review as a conference paper at ICLR 2027

The Dynamics of Routing in Deep Linear Networks

Abstract

Over the course of learning, computation in biological and artificial neural networks is distributed across the underlying network in complex ways, reflecting the interaction of network architecture and the dynamics of learning, but we have limited theoretical understanding of this process. Here, we develop a mathematical formalism to study the routing dynamics exhibited by gradient descent learning in linear networks defined over a directed acyclic graph. We show that the conserved quantities satisfied by gradient descent allow learning to be decomposed into an overall capacity variable and a routing variable living in a finite polytope, measuring the relative distribution of weights across edges in the network. In balanced networks of uniform depth, in the positive scalar setting, we show that learning separates cleanly into scalar growth and an autonomous dynamical flow on the routing polytope, which is controlled by the distribution of computation across paths in the network. We prove that the stable fixed points of these projective dynamics correspond to chain-equivalent subgraphs, allowing us to predict when a network will collapse to a smaller subgraph in the large-target limit; we also show evidence for this effect in non-linear DAG networks. When depth is mixed or conserved charges are non-zero, the geometry yields transient depth races and complex routing movement across the graph, before ultimately converging to the previous homogeneous case in the large-target limit. We also show that the local dynamics at interior fixed points are governed by a kind of path-memory, while the local behavior at boundary fixed points allows for spontaneous circuit formation. Finally, we show that richness in linear DAG networks is governed by an electrical formalism. We demonstrate that several of these phenomena generalize in some form beyond our exact setting, to matrix and non-linear networks. Our results provide an analytical bridge between structural graph topology, conserved quantities, and emergent routing behaviors in the deep linear setting, which we hope can eventually help support further understanding of complex dynamics in more general networks.

Primary area:
learning theory
Keywords:
Deep Linear Networks, Graphs, Conserved Quantities, Neural Network Theory, Routing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8gB9OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how neural networks allocate computation across different paths during training. The authors derive theoretical results on routing dynamics, stability, and the activation of dormant subnetworks, supported by experiments on linear and ReLU networks.

Critical strengths

The paper addresses an interesting theoretical question and provides a unified framework for understanding routing dynamics. The mathematical analysis appears to offer useful insights into path specialization.

Critical weaknesses

The work appears theoretically interesting, but I am uncertain about its broader implications. My assessment is tentative given my limited confidence in evaluating the technical proofs.

  1. Most theoretical results rely on restrictive assumptions, particularly positive scalar networks. Their applicability to practical neural networks remains unclear.
  2. The main stability results are asymptotic, while practical training operates under finite budgets. The paper provides limited evidence connecting these theoretical predictions to typical training dynamics.
  3. The nonlinear experiments are mostly illustrative and do not directly validate the proposed theoretical mechanisms.
Questions for authors

Please see weakness

2 comments
AuthorsOpenReview ↗

Reviewer 3, Response 1 of 2

Your comments are quoted below and keep their references to the submitted paper; references in our responses point to the revised paper.

The work appears theoretically interesting, but I am uncertain about its broader implications. My assessment is tentative given my limited confidence in evaluating the technical proofs.

Scope: Weakness 1

Most theoretical results rely on restrictive assumptions, particularly positive scalar networks. Their applicability to practical neural networks remains unclear.

Thank you. We have added a systematic test of the theory's predictions in signed, matrix and nonlinear networks, and in residual architectures with paths of mixed depth.

Signed, matrix and nonlinear networks (Appendix Q). We trained signed scalar, linear matrix, tanh and ReLU networks on 117 graphs of depth up to 10, from the standard LeCun initialization and six smaller scales. First, how much a network selects at the fit is largely ordered by coherence depth in every class: at the standard initialization, graphs with λ≥7\lambda\ge7 carry a median of 81% (matrix), 82% (tanh) and 97% (ReLU) of their output on the eventually selected subnetwork. Second, after long training with weight decay, the thresholded supports have the layerwise-complete form of the scalar stable states in 90–100% of runs on graphs with λ≥5\lambda\ge5, with exceptions only where the subnetwork must contain a block of two active vertices on each side of a layer (Appendix Q.2). Third, with several outputs (the hierarchical task of Shi et al., 2022), supports become layerwise complete only when measured per singular mode in linear and tanh networks, and per input item in ReLU networks (Appendix Q.3).

Mixed depth (Appendix R). In ResNet-like architectures, all 304 positive scalar runs end numerically consistent with balanced product states on layerwise-complete subnetworks whose paths have one depth. Small changes to the architecture, or to where the initialization is concentrated, change which subnetwork is selected, and ReLU networks with matrix vertices make the same choices (see our responses to Reviewer 2).

We present these results as existence demonstrations, since a theory for networks whose activation patterns change during training remains open, but they show the ordering by coherence depth and the layerwise-complete endpoints of the scalar theory in all of these network classes.

Finite training: Weakness 2

The main stability results are asymptotic, while practical training operates under finite budgets. The paper provides limited evidence connecting these theoretical predictions to typical training dynamics.

We have now added an extensive finite-training analysis, including a large empirical suite and several analytic results (Appendix O, a new passage in Section 5 and a new Figure 4; see also our response to Reviewer 1). Ordinary initializations can produce substantial collapse, especially in graphs with long route memory, and coherence depth predicts where. The analysis also predicts when to expect collapse, how much occurs at the fit, and which subnetwork is selected.

How much collapse at an ordinary fit. We measure the share of the output carried at the fit by the subnetwork that routing eventually selects. In a census of 244 random four-stream graphs at an initial weight of 1/31/3, comparable to standard schemes, that subnetwork carries at least half of the output in 70% of graphs with λ≥7\lambda\ge7 and 28% with 5≤λ<75\le\lambda\lt 7, but in only one of 168 graphs with λ<5\lambda\lt 5; at 0.1, the first two numbers rise to 91% and 77%. Graphs with less collapse still move toward the eventual subnetwork, since the share rises monotonically along every run.

When to expect it. For a balanced positive scalar network of depth dd with initial output f0f_0, the per-layer weight growth at the fit is at most f0−1/df_0^{-1/d}, about a factor of 2 at an initial weight of 1/31/3 (Proposition 6). Near the symmetric state an unstable routing mode grows like Gλi−2G^{\lambda_i-2}, so this modest budget produces substantial selection only when route memory is long. For parallel routes the result is exact (Proposition 8): with a 10% capacity gap, the losing route falls below 1% of the output from initial weights of 0.44 at depth 8, but only below 0.012 at depth 3, and at depth 2 the shares never change.

AuthorsOpenReview ↗

Reviewer 3, Response 2 of 2

Predicting the amount of collapse and the selected subnetwork. Linearizing the routing flow at the symmetric state, with no simulation and no fitted constant, predicts the share at the fit with correlation 0.95 across 241 graphs at an initial weight of 1/31/3 (mean error 3 percentage points), and predicts a route lying in the selected subnetwork in a median of 88% of starts, against 11% by chance (Appendix O.3).

Typical nonlinear training. The ordering by coherence depth carries over to signed, matrix, tanh and ReLU networks from the standard initialization (Appendix Q). In the new Figure 4, every ReLU seed from the standard initialization settles on one candidate subnetwork while most of the error remains, then fits on it; the retained candidate carries 98–100% of the output at the fit.

Finite budgets and weight decay. Without weight decay the routing state freezes at the fit, before collapse is exact. With weight decay ν\nu, equal-depth positive scalar routing follows the same flow under the clock τ=log⁡(Q/Q0)+2νt\tau=\log(Q/Q_0)+2\nu t, which keeps advancing after the fit, and generically approaches a stable product state at finite capacity whenever its limit is nonzero (Theorem 6, Appendix P). In the census, 89–100% of runs with weight decay end on layerwise-complete subnetworks in every range of coherence depth.

Nonlinear evidence: Weakness 3

The nonlinear experiments are mostly illustrative and do not directly validate the proposed theoretical mechanisms.

We agree that the original nonlinear experiments were illustrations. We have replaced or supplemented them with experiments that test specific predictions of the theory, and we report every run.

Figure 4. The single ReLU run is replaced by every seed: eight ReLU networks from the standard LeCun initialization on a census graph with four layerwise-complete candidate subnetworks, beside eight positive scalar networks on the same graph. The theory predicts that routing selects one candidate early and that learning then continues on it. Every ReLU seed shows this: routing settles at 35–85% of the time to fit, while 53–98% of the squared error remains, and the retained candidate (different in different seeds) carries 98–100% of the output at the fit, while the total squared weight grows by only 4–5% (Appendix K).

Predictions tested across graphs (Appendix Q). On 117 graphs in signed, matrix, tanh and ReLU networks, the amount of selection at the fit is ordered by coherence depth in every class; long training with weight decay ends on layerwise-complete supports, the form of the scalar stable states, in 90–100% of runs on graphs with λ≥5\lambda\ge5; and with several outputs, supports become layerwise complete only per singular mode in linear and tanh networks and per input item in ReLU networks, consistent with each such component being routed separately.

Dormant recruitment on real data (Appendix S, pilot). On CIFAR-10, a dormant branch whose last kk layers start at a small scale δ\delta is recruited after a delay that grows with kk and with 1/δ1/\delta, consistent with the scalar activation-order calculation, and only while the active network cannot fit the data. In four-stream CNNs started at half the He scale, parallel streams concentrate 82% of each class's output on one stream, against 59–60% when the streams mix.

Figure 6B. The ReLU rescaling comparison now uses 24 paired seeds evaluated at a common fitting progress (see our response to Reviewer 1, Question 2).

These experiments extend the empirical evidence beyond the scalar setting, although the scalar theorems do not prove the corresponding claims for nonlinear networks.

Reviewer_EBVvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This article studies how architecture and initialization affect routing of computation in linear deep neural networks on directed acyclic graphs (DAGs). The author points out that hidden base transformations can result in conservative matrix imbalances, thereby limiting capacity allocation during the learning process. For positive scalar networks, this paper derives an accurate capacity flow equation and proves that in a balanced network with equal path depth, a stable extreme routing state is equivalent to a chain structure. In addition, this paper explores the relationship between routing instability and path memory, and presents preliminary experimental phenomena of matrix and ReLU networks.

Critical strengths
  1. This paper provides a clear theoretical perspective to explain how capacity evolves with gradient flow and mathematically separates global growth from routing dynamics.

  2. Linking routing selection with network initialization and conservation laws provides an interesting physical perspective for understanding the evolution of network structure.

Critical weaknesses
  1. The core theoretical results of the paper, such as the precise capacity-flow equation, are based on the simplified assumption of "positive scalar networks". Modern deep learning architectures not only rely on matrix multiplication, but also heavily on nonlinear activation functions such as ReLU. Although the author attempted to extend some of the conclusions to mixed depth and networks with ReLU in Sec. 6 and experiments, these extensions lack theoretical guarantees and mostly remain at the empirical observation level. The extent to which the dynamics of scalar networks can generalize to high-dimensional parameter spaces remains a major question.

  2. In my understanding, many key derivations, such as stability analysis in Secs. 4 and 5, require the network to be "balanced" and "have a consistent path depth". In practical complex DAG structures such as ResNet or Transformer computation streams, these conditions are hard to satisfy. Thus, this assumption is difficult to match the actual situation.

  3. The experiment is not very convincing because it mainly focuses on small topological graphs (e.g., Y-networks, Wheatstone bridges, small Crown graphs). Lack of evaluation on datasets with practical significance such as CIFAR/ImageNet or modern standard network architectures. Thus, can "the recruitment of dormant computations" phenomena will occur during real-world, large-scale, high-dimensional non-linear optimization?

Questions for authors
  1. The conserved quantities in linear networks and ReLU networks and their limitations on learning dynamics have been widely studied (e.g. Du et al., 2018; Kunin et al., 2021; Marcotte et al., 2023). Although this paper introduces the concept of capacities, it is largely a restatement of existing symmetry and conservation laws in mathematical essence. Can the author explain the differences from existing works?

  2. May I ask if the theoretical framework of this article will completely fail with the introduction of weight decay?

  3. Could you provide a concrete example on a standard deep learning task demonstrating how the theory could guide us in designing better network architectures?

5 comments
AuthorsOpenReview ↗

Reviewer 2 post 1 of 4

Your comments are quoted below and keep their references to the submitted paper; references in our responses point to the revised paper.

Nonlinear scope: Weakness 1

The core theoretical results of the paper, such as the precise capacity-flow equation, are based on the simplified assumption of "positive scalar networks". Modern deep learning architectures not only rely on matrix multiplication, but also heavily on nonlinear activation functions such as ReLU. Although the author attempted to extend some of the conclusions to mixed depth and networks with ReLU in Sec. 6 and experiments, these extensions lack theoretical guarantees and mostly remain at the empirical observation level. The extent to which the dynamics of scalar networks can generalize to high-dimensional parameter spaces remains a major question.

Thank you. We have added a systematic test of the theory's predictions outside the positive scalar setting. We note in addition that the point of restricting to a more narrow setting is our strong theoretical results, which would not be possible in a more complex, realistic network setting; as with most theory papers, these results are meant to serve as a groundwork for later empirical understanding of larger networks. We view these additional experiments as significant evidence that our results can serve as a starting point for understanding collapse in more complex setting.

Matrix, signed and nonlinear networks (Appendix Q). We trained signed scalar, linear matrix, ReLU and tanh networks on 117 graphs of depth up to 10, from the standard LeCun initialization and six smaller scales, six starts each (15 long-decay runs that lost their fit are identified in Appendix Q.2 and excluded from the completeness summaries). In every class, how much a network selects at the fit is largely ordered by coherence depth, as in the scalar theory: at the standard initialization, graphs with λ≥7\lambda\ge7 carry a median of 81% (matrix), 82% (tanh) and 97% (ReLU) of their output on the eventually selected subnetwork. Furthermore, after long training with weight decay, the thresholded supports have the layerwise-complete form of the scalar stable states in 95–100% of runs on graphs with λ≥5\lambda\ge5 for matrix, ReLU and tanh networks, and 90% for signed scalar ones; the exceptions all occur where the subnetwork must contain a block of two active vertices on each side of a layer. With several outputs (the hierarchical task of Shi et al., 2022), the experiments favor a modewise description for linear networks and an itemwise one for ReLU networks; the exact modewise reduction requires simultaneous alignment with the target under identity inputs (Appendix Q.3–Q.4). The new Figure 4 shows selection in ReLU networks from the standard initialization, with every seed, beside positive scalar networks on the same graph.

Mixed depth. We also added ResNet-like architectures with paths of mixed depth (Appendix R; see also Weakness 2). At the final simulated clock, all 304 positive scalar runs are numerically consistent with balanced product states on layerwise-complete supports whose paths have one depth. The general mixed-depth theorem gives convergence to a balanced single-depth fixed point; stability needs the qualifications discussed below. Small changes to the architecture (grouping the residual branches, or adding crossing shortcuts to one block) or to where the initialization is concentrated change which subnetwork is selected. Linear and ReLU networks with matrix vertices make the same choices, partly at the standard initialization and largely at a third of it.

A theory for networks whose activation patterns change during training remains open.

AuthorsOpenReview ↗

Reviewer 2 post 2 of 4

Balance and mixed depth: Weakness 2

In my understanding, many key derivations, such as stability analysis in Secs. 4 and 5, require the network to be "balanced" and "have a consistent path depth". In practical complex DAG structures such as ResNet or Transformer computation streams, these conditions are hard to satisfy. Thus, this assumption is difficult to match the actual situation

Balance and equal depth are not restrictions on the applicability of our large-capacity fixed-point analysis: they are the only kinds of limiting fixed points that the routing state of a general imbalanced, mixed-depth positive scalar network can converge to. This is the content of our Section 6 and Appendix H (Theorem 2), show that the routing state of every such network converges to a balanced fixed point on an equal-depth subnetwork as capacity tends to infinity, Therefore our fixed-point analysis applies to the limiting states of general imbalanced, mixed-depth positive scalar networks. We have also added demonstrations in mixed-depth architectures, including ResNet-like graphs, in Appendix R.

The one remaining qualification is a full proof of generic convergence to stable fixed points. We prove this for equal-depth networks with arbitrary initial imbalance (Appendix L, Theorem 3) and for mixed-depth networks whose limiting support has maximal depth (Appendix M, Theorem 4); in the general mixed-depth case it remains a conjecture, and Appendix N gives a conditional result under the stated gradient-finiteness conjecture. In principle, then, a positive-measure set of mixed-depth initializations could converge to a balanced equal-depth subnetwork that is not layerwise complete, but we have not found any such example in our experiments.

Small graphs and real data: Weakness 3

The experiment is not very convincing because it mainly focuses on small topological graphs (e.g., Y-networks, Wheatstone bridges, small Crown graphs). Lack of evaluation on datasets with practical significance such as CIFAR/ImageNet or modern standard network architectures. Thus, can "the recruitment of dormant computations" phenomena will occur during real-world, large-scale, high-dimensional non-linear optimization?

Larger graphs. We now test the theory on hundreds of graphs much larger than the examples in the main text. The census of Appendix O has 244 random four-stream graphs of depths 4 to 10, with 16 to 124 edges and up to about 110,000 input–output paths. Appendix Q trains signed scalar, matrix, tanh and ReLU networks on 117 graphs from the same family (depths 4 to 10), with eight units at every vertex of the matrix and nonlinear networks (hidden layers of 32 units), and Appendix R adds ResNet-like networks with paths of mixed depth. On these graphs, how much a network selects at the fit is ordered by coherence depth in every class; after long training with weight decay, the selected supports are layerwise complete, the form of the theory's stable states, in 90–100% of runs on graphs with λ≥5\lambda\ge5; and in the positive scalar census the initial state predicts the selected subnetwork in a median of 88% of starts, against 11% by chance. Many of the paper's figures still use small graphs such as the Y network, the Wheatstone bridge and crown graphs, because on a small graph every candidate subnetwork and its dynamics can be inspected directly.

Recruitment of dormant computations on CIFAR-10 (Appendix S, pilot). A narrow ReLU MLP is trained together with a wider parallel branch whose last kk layers start at δ\delta times the He scale. For k=1k=1, the branch carries a quarter of the output within 50 steps; for k=2k=2, this takes 250, 1,850 and 3,200 steps at δ=10−1,10−2,10−3\delta=10^{-1},10^{-2},10^{-3}, roughly logarithmic in 1/δ1/\delta; for k=3k=3 the dependence is steeper. This is qualitatively consistent with the scalar activation-order calculation for approximately constant positive drive (in the neural experiment the drive evolves). The branch is not recruited when the active network is wide enough to fit the data. In four-stream CNNs on CIFAR-10 started at half the He scale, parallel streams put 82% of each class's output on the leading stream, against 59–60% when the streams mix; from the He scale, the architectures do not differ (55–60%). These are single-seed pilots on small networks, showing behavior consistent with the proposed mechanisms rather than a large-scale validation.

AuthorsOpenReview ↗

Reviewer 2 post 3 of 4

Prior work: Question 1

The conserved quantities in linear networks and ReLU networks and their limitations on learning dynamics have been widely studied (e.g. Du et al., 2018; Kunin et al., 2021; Marcotte et al., 2023). Although this paper introduces the concept of capacities, it is largely a restatement of existing symmetry and conservation laws in mathematical essence. Can the author explain the differences from existing works?

Note: below is a general description of our new results and how they are distinct from prior work. If you were interested specifically in the comparison between our methodological contributions and prior conserved quantity work, see "Previous work on symmetries and conservation laws" below. Thanks!

The conserved quantities themselves are established background, not a claimed contribution of this paper, and we will make this distinction explicit earlier in the manuscript. Our contribution is a theory of routing and subnetwork specialization in positive scalar DAG networks, together with a new capacity/routing polyhedron framework that enables this theory. Our results characterize which subnetworks can persist under learning, which routing configurations are unstable and lead to specialization, and how dormant subnetworks can be recruited.

Conservation laws constrain learning trajectories to invariant sets, but do not by themselves determine how learning proceeds within these constraints, which subnetworks are stable, or how computation becomes concentrated in particular routes. Our work addresses these questions for general DAG architectures. Our classification of stable routing states generalizes previous results on specialization in parallel pathways (Shi et al., 2022) to arbitrary DAGs, allowing us to study specialization in large network architectures with complex connections and shared edges among paths. Beyond classifying stable routing states, our results connect routing instability to coherence depth, characterize the recruitment of dormant subnetworks, and establish connections to rich and lazy learning.

Our principal methodological contribution is the capacity/routing polyhedron formalism and its associated dynamical equations, which enable these results. Using squared edge weights and conserved imbalances, we construct the capacity/routing polyhedra as natural spaces for studying routing dynamics. In particular, their boundary faces organize subnetwork specialization and allow us to analyze the limiting routing configurations as edges become inactive. We derive simple expressions for the dynamics on these spaces, allowing us to study both competition among routes within an active subnetwork and the growth of additional edges from its boundary. Thus, the conserved quantities provide the starting point for our analysis, rather than its conclusion: the new geometry and dynamics allow us to answer questions about routing and subnetwork selection that the conservation laws alone do not resolve.

To clarify the relationship with prior work, it is useful to distinguish our two categories of contributions:

(1) Previous work on routing and specialization. The most directly related comparisons for our dynamical results are Shi et al. (2022), Jarvis et al. (2025), Saxe et al. (2022), Zhang et al. (2023), and Veit et al. (2016). Our work extends previous specialization results to general DAG architectures and provides new results on the classification of stable routing states, the graph structure governing instability through coherence depth, and the recruitment of dormant subnetworks. We will expand the related-work discussion to make the relationship between these results and prior specialization theories more explicit.

(2) Previous work on symmetries and conservation laws. To our knowledge, none of these works develops the capacity/routing polyhedron framework or uses its boundary geometry and associated dynamics to analyze routing stability and subnetwork selection. Two particularly close recent works help illustrate the distinction:

  • Nurisso et al. (2026) also study conservation laws on general DAG networks, expressing them as linear constraints on squared edge weights, as we do. Their focus, however, is on connectivity, singularities, and regularization approaches to pruning. There are natural mathematical connections between their pruning-related singularities and the boundary configurations studied in our framework, as well as between pruning and our results on network collapse. However, they do not develop our capacity/routing polyhedron framework or use its geometry and dynamics to study routing stability and specialization. Our results and approaches are complementary.
  • Marcotte et al. (2026) investigate when training dynamics on conserved-quantity level sets can be expressed in reduced path-product coordinates, a distinct question from the routing dynamics and stability problems studied here.
Reviewer_PdA5OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how gradient flow allocates computation among paths in linear networks represented by directed acyclic graphs. Conserved matrix imbalances motivate squared-weight capacity coordinates; for positive scalar networks, these yield an exact capacity-flow equation. Under balance and equal path depth, the dynamics separate into a scale variable and autonomous routing. The main theoretical contributions characterize stable routing states as product flows on layerwise-complete supports, which become chains after twin aggregation; relate local routing instability to path covariance and a coherence-depth threshold; derive an exact conductance-depth expression for instantaneous kernel evolution; and characterize recruitment of dormant edges through activation order and a coactivation spectrum. The appendices extend the convergence analysis to unequal depths and nonzero charge. Scalar simulations and selected matrix/ReLU teacher-student experiments illustrate routing specialization, staged recruitment, and initialization-dependent kernel change.

Critical strengths
  1. A substantive characterization of routing beyond disjoint parallel pathways. The capacity reduction and the path-covariance stability analysis handle branching, merging, and shared edges. Most notably, Theorem 1 establishes a necessary and sufficient characterization of stable normalized states, including boundary states and continuous fixed-point families, rather than merely exhibiting specialization in examples. The connection between dependence across layers, instability, and chain-equivalent product flows provides a useful structural explanation of how architecture constrains the computations retained by learning (Sections 4-5; Appendices F-G).

  2. Several learning phenomena are connected through explicit, checkable identities. The distinction between coherence depth, which controls routing instability, and conductance depth, which controls kernel evolution, is informative. The dormant-edge expansion further explains why the number of simultaneously required new edges matters more than total path length. The analysis addresses boundary feasibility, distinguishes convergence from stability, and extends the asymptotic result to mixed depths and nonzero imbalance (Sections 6-8; Appendices H-J). These qualifications make the theoretical contribution more convincing than a purely geometric analogy.

Critical weaknesses
  1. The most decisive structural result is asymptotic, while the finite-training implication remains qualitative. Chain-equivalent stable states concern normalized routing as capacity tends to infinity. The paper correctly notes that fitting a finite target may stop much earlier, but does not give a criterion or rate relating the initial allocation, target scale, and accessible capacity growth to proximity to the limiting support. Figure 12 deliberately chooses a large target after the routing trajectory, and the ReLU example uses the equivalent of very small initialization (Appendix K, p. 26). Thus the theory identifies possible endpoints more directly than it predicts which finite training problems will exhibit substantial route selection. This limits the immediate explanatory reach of the result, although it does not invalidate the theorem.

  2. The experiments outside the exact scalar regime provide limited evidence for a general mechanism. Figure 3 supplies a useful six-seed architecture comparison, but Figure 4 is a selected run from eight seeds and two target amplitudes, while the ReLU panels of Figure 6 average only two seeds. Moreover, the latter use fixed update budgets and initialization-dependent learning rates, with different amounts of capacity growth (Appendix K, pp. 26-27). These comparisons do not isolate conductance depth from differences in training progress, and the paper does not establish whether route stabilization is distinct from small-initialization alignment in the selected example. The authors are appropriately candid that these are illustrations; consequently, I place substantially more weight on the scalar theory than on the proposed extension to practical nonlinear networks.

Questions for authors
  1. Clarification of finite-scale relevance: Can the existing scalar trajectories be used to quantify how close routing is to its limiting support at a fixed relative fitting error, across initial scales or finite target sizes? Even a restricted case or a bound for parallel routes would help determine whether the asymptotic characterization predicts ordinary fitting trajectories or principally a large-growth regime.

  2. Clarification of the nonlinear comparisons: For the Figure 6 runs already performed, what are the final losses and relative predictor changes for each rescaling? Does the ordering of kernel change persist when evaluated at a common fitting progress, rather than at the fixed update budget? This would affect how much evidence these panels provide for an extension of the scalar richness identity.

  3. Clarification of robustness: How often did the two-stage behavior and chain-equivalent retained support of Figure 4 occur across the eight seeds and two target amplitudes considered? Reporting the existing selection pool would clarify whether the chosen example is representative or an existence demonstration.

3 comments
AuthorsOpenReview ↗

Reviewer 1 post 1 of 3

Your comments are quoted below and keep their references to the submitted paper; references in our responses point to the revised paper.

Finite scale: Weakness 1 and Question 1

The most decisive structural result is asymptotic, while the finite-training implication remains qualitative. Chain-equivalent stable states concern normalized routing as capacity tends to infinity. The paper correctly notes that fitting a finite target may stop much earlier, but does not give a criterion or rate relating the initial allocation, target scale, and accessible capacity growth to proximity to the limiting support. Figure 12 deliberately chooses a large target after the routing trajectory, and the ReLU example uses the equivalent of very small initialization (Appendix K, p. 26). Thus the theory identifies possible endpoints more directly than it predicts which finite training problems will exhibit substantial route selection. This limits the immediate explanatory reach of the result, although it does not invalidate the theorem.

Clarification of finite-scale relevance: Can the existing scalar trajectories be used to quantify how close routing is to its limiting support at a fixed relative fitting error, across initial scales or finite target sizes? Even a restricted case or a bound for parallel routes would help determine whether the asymptotic characterization predicts ordinary fitting trajectories or principally a large-growth regime.

Thank you. In response to this question, we have added an extensive appendix of empirical results, and several analytical results: Appendix O on finite-scale collapse, a new Figure 4, and a new passage in Section 5. Our finding is that ordinary initializations can produce substantial collapse, especially in graphs with long route memory.

How close routing is at an ordinary fit. We measure closeness as the share of the output carried by the subnetwork that routing eventually selects. In a census of 244 random four-stream graphs of depths 4 to 10 (16 starts each), at an initial weight of 1/31/3 (about the mean weight size of standard schemes for these layers, 0.27–0.46), that subnetwork carries at least half of the output at the fit in 70% of graphs with λ≥7\lambda\ge7 and 28% with 5≤λ<75\le\lambda\lt 7, against one of 168 graphs with λ<5\lambda\lt 5; at an initial weight of 0.1, these become 91% and 77%. Graphs with less collapse still move toward that subnetwork: at 1/31/3, those with 5≤λ<75\le\lambda\lt 7 that stay below half have more than doubled its share. Across initial scales, the share is nondecreasing in 6,657 of 6,748 runs (Appendix O.1). Because our graphs were chosen to span coherence depth, these results show where collapse is substantial, not how common it is in practical architectures.

When to expect collapse. Initial scale and target size enter through the accessible weight growth. For a balanced positive scalar network of depth dd with unit target, initial output f0f_0 and fitting checkpoint ffitf_{\mathrm{fit}}, Gfit≤(ffit/f0)1/d≤f0−1/dG_{\mathrm{fit}}\le(f_{\mathrm{fit}}/f_0)^{1/d}\le f_0^{-1/d} (Proposition 6); in the imbalanced census, growth is typically within about 15% of this budget, about a factor of 2 at initial weight 1/31/3. Near the symmetric state an unstable routing mode grows like Gλi−2G^{\lambda_i-2}, so long route memory can amplify a small initial difference within this budget. For two balanced parallel routes of depth d≥3d\ge3 this is exact: gradient flow conserves rp1−d/2−rq1−d/2r_p^{1-d/2}-r_q^{1-d/2}, so with losing-route initial weight ω\omega and relative capacity advantage δ>0\delta\gt 0 for the winner, the loser's share at output ffitf_{\mathrm{fit}} is at most B/ffitB/f_{\mathrm{fit}}, where B=ωd[1−(1+δ)−(d/2−1)]−d/(d−2),B=\omega^{d}\bigl[1-(1+\delta)^{-(d/2-1)}\bigr]^{-d/(d-2)}, asymptotically tight for small weights (Proposition 8). At unit interpolation, a 10% capacity gap leaves the loser below 1% of the output from initial weights of 0.44, 0.30, 0.10 and 0.012 at depths 8, 6, 4 and 3, and never at depth 2.

Quantifying collapse. For general graphs we linearize the routing flow at the symmetric state and evolve each mode of the initial state at its rate λi/2−1\lambda_i/2-1 for the growth the initialization allows. Across 241 graphs, the predicted share at the fit has correlation 0.95 with the measured share at an initial weight of 1/31/3 (mean error 3 percentage points), 0.87 at 1/51/5 and 0.77 at 1/101/10 (Figure 18); on census graph 160 (depth 9, λ=7\lambda=7), 89% is measured against 90% predicted. At much smaller initial weights, nonlinear coupling between modes decides close competitions and the estimate loses accuracy (Appendix O.3).

AuthorsOpenReview ↗

Reviewer 1 post 2 of 3

Which subnetwork. The same linearization, evolved until its fastest mode is of order one, predicts a route that lies in the selected subnetwork in a median of 88% of starts per graph, against 11% by chance (Appendix O.3 and Figure 19).

At a fixed relative error, and with weight decay. The census is recorded at ffit=0.99f_{\mathrm{fit}}=0.99, a 1% relative output error; at relative error ε\varepsilon the growth bound uses ffit=1−εf_{\mathrm{fit}}=1-\varepsilon. Without weight decay, every edge retains at least G−2G^{-2} of its initial normalized capacity at the fit, so exact pruning has not occurred (Proposition 7). With weight decay ν\nu, equal-depth positive scalar routing follows the same flow under the clock τ=log⁡(Q/Q0)+2νt\tau=\log(Q/Q_0)+2\nu t, which diverges at any nonzero limit; Theorem 6 (Appendix P) proves generic convergence to a stable product state with excluded edges tending to zero, and gives a sufficient condition excluding the zero limit. In the census, 100%, 97% and 89% of runs with decay pass the 0.1% layerwise-completeness test for λ≥5\lambda\ge5, 3≤λ<53\le\lambda\lt 5 and 2<λ<32\lt \lambda\lt 3.

Figure 12 and the ReLU example. You are right that Figure 12 uses a large target and that the original ReLU example is equivalent to a very small initialization. Figure 12's caption now says so and points to Appendix O. The ReLU example is replaced by the new Figure 4, with all eight seeds per condition at ordinary scalar and standard ReLU initialization (see Weakness 2).

Experiments: Weakness 2 and Questions 2 and 3

The experiments outside the exact scalar regime provide limited evidence for a general mechanism. Figure 3 supplies a useful six-seed architecture comparison, but Figure 4 is a selected run from eight seeds and two target amplitudes, while the ReLU panels of Figure 6 average only two seeds. Moreover, the latter use fixed update budgets and initialization-dependent learning rates, with different amounts of capacity growth (Appendix K, pp. 26–27). These comparisons do not isolate conductance depth from differences in training progress, and the paper does not establish whether route stabilization is distinct from small-initialization alignment in the selected example. The authors are appropriately candid that these are illustrations; consequently, I place substantially more weight on the scalar theory than on the proposed extension to practical nonlinear networks.

Clarification of the nonlinear comparisons: For the Figure 6 runs already performed, what are the final losses and relative predictor changes for each rescaling? Does the ordering of kernel change persist when evaluated at a common fitting progress, rather than at the fixed update budget? This would affect how much evidence these panels provide for an extension of the scalar richness identity.

Clarification of robustness: How often did the two-stage behavior and chain-equivalent retained support of Figure 4 occur across the eight seeds and two target amplitudes considered? Reporting the existing selection pool would clarify whether the chosen example is representative or an existence demonstration.

We have replaced all our non-robust figures with robust versions, as described below.

New Figure 4. The new figure keeps the original panels but shows every seed: eight positive scalar networks at an initial weight of 1/31/3 and eight ReLU networks from the standard LeCun initialization, on a census graph with coherence depth 7.1 and four layerwise-complete candidate subnetworks. Every ReLU seed shows the two-stage behavior: routing settles on one candidate at 35–85% of the time to fit (median 74%), while 53–98% of the squared error remains, and the network then fits on that candidate, which carries 98–100% of the output at the fit. In every seed of both classes the retained subnetwork is one of the four candidates, and different seeds retain different ones (Appendix K).

Relation to small-initialization alignment. Positive scalar edges have no internal direction to align, so their support changes are described entirely by routing; the product-state structure is related to established matrix-alignment results (Section 9). In the ReLU row of Figure 4, the total squared weight grows by only 4–5% from the standard scale, yet the retained candidate carries 98–100% of the output at the fit. Across the 116 suite graphs with λ>2\lambda\gt 2, those with λ≥7\lambda\ge7 carry a median of 81% (matrix) and 97% (ReLU) on their eventual subnetwork at the fit from standard initialization, after per-layer output growth of only 1.4–1.9 (Appendix Q). These observations establish finite-scale selection, though they do not by themselves separate routing from alignment mechanisms in ReLU networks.

AuthorsOpenReview ↗

Reviewer 1 post 3 of 3

Chain-equivalent retained support, at the population level. This test is to verify that the limiting supports obey our theorem. After long training with weight decay, thresholded supports are layerwise complete in 99% (matrix), 100% (tanh), 95% (ReLU) and 90% (signed scalar) of runs on graphs with λ≥5\lambda\ge5. Failures concentrate, mostly in signed scalar and ReLU networks, in graphs whose supports contain a block of at least two active vertices on each side of a layer (Appendix Q.2 discusses conflicting signs as an unproved explanation). On the 70 graphs whose subnetworks never contain such a block, every reported run of every class passes (Table 1).

Figure 6 (Question 2)

We replaced the two-seed, fixed-budget ReLU comparison with 24 paired seeds, a common learning rate, and evaluation at the same 95% reduction in training loss. Within each seed and depth, the rescalings preserve the initial predictor and use the same data and teacher. We now show input as well as output interventions, with seed means and bootstrap intervals.

All 312 primary runs and 208 smaller-step and width controls reach this checkpoint. At depths four and six, enlargement produces greater mean kernel growth than shrinkage at both positions, with a position-dependent magnitude; at depth two, where both rescalings leave one small layer, neither changes the kernel much. These results persist with smaller steps, wider networks, and evaluation at matched predictor displacement. Appendix K reports checkpoint losses and relative predictor changes for each rescaling, with paired uncertainty estimates.

We also removed the depth-drift panel and added the exact scalar counterpart. The revised Figure 6B (Section 7) therefore supports a specific rescaling effect at matched fitting progress, consistent with our scalar formula, without claiming a conductance-depth identity for ReLU networks.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

References in our responses point to the revised paper.

We thank all of the reviewers for their thoughtful feedback, questions, and comments on our paper. In response to the reviews, we have incorporated a number of significant additions and modifications into our new draft, which we summarize below.

Finite-fitting analysis, experiments and theory. To address whether our theory matters for ordinary training, and how its effects scale with small initialization, we added a large suite of experiments that measure collapse at the point where the target is fit, as a function of initialization scale, on 244 randomly generated four-stream graphs of depths 4 to 10, with 16 starts each. The main finding is that ordinary initializations can produce substantial collapse, especially in graphs with long route memory, while graphs with small coherence depth need smaller initializations to show collapse; in general, coherence depth is a strong first-order predictor of where collapse occurs. At an initial weight of 1/31/3, comparable to standard schemes, the subnetwork that routing eventually selects carries at least half of the output at the fit in 70% of graphs with coherence depth λ≥7\lambda\ge7 (91% at an initial weight of 0.1), and graphs with less collapse still show movement toward that subnetwork. We also add an exact solution for parallel paths which bounds these effects, and a linear estimate built from coherence depth and the other unstable modes of the amount of collapse at the fit. The full results are in Section 5 and Appendix O. It would be an interesting question for future work to understand the additional graph structure that controls the details of collapse speed beyond the initial spectrum.

Scope. To test how far our theory extends beyond positive scalar networks, we ran the same analyses as above for signed scalar, matrix, tanh and ReLU networks, with standard (LeCun) initializations at varying magnitude (Appendix Q). Collapse is again largely ordered by coherence depth, and is substantial in larger coherence depth sectors. We show via long training with weight decay that we obtain layerwise-complete thresholded supports in most runs. A systematic theory of stable routing in these settings is beyond the scope of this paper, so we present these results as existence demonstrations, which we view as significant evidence that our results can serve as a starting point for understanding collapse in more complex settings.

We also include demonstrations of collapse in mixed-depth, ResNet-like networks (positive scalar, linear and ReLU; Appendix R), which were missing in our original work.

Experiment robustness. We have replaced Figure 4 with all eight seeds in each of the positive scalar and ReLU conditions from our finite-scale experimental suite. We have also redone Figure 6B with 24 paired seeds, a common learning rate, and evaluation at the same fitting progress (Section 7 and Appendix K).

Weight decay. Appendix P describes how our results change with L2L^2 regularization, with supplemental Figures 20 and 21 showing the finite-time effects. Our formalism incorporates weight decay directly. For equal-depth positive scalar networks, routing follows the same trajectories under a new clock; at a nonzero limit that clock diverges, so routing generically approaches a stable product state at finite capacity and excluded edges tend to zero in weight. Theorem 6 gives a sufficient condition excluding convergence to the zero network.

Dormant nucleation in more complex networks. Appendix S adds a pilot on CIFAR-10 images: a ReLU MLP trained together with a wider parallel branch whose last kk layers start at a small scale δ\delta. Recruitment is prompt for k=1k=1, has a roughly logarithmic dependence on seed scale for k=2k=2, and a steeper dependence for k=3k=3. This pattern is consistent with the scalar activation-order calculation under approximately constant positive drive; the neural experiments have an evolving drive and one seed per setting.

New preliminary network design application. We have included a preliminary network design application in a multi-modal network setting. Please see the response to Reviewer 2 or the new Appendix T for more information.

We have also made two new revisions not explicitly requested by the reviewers:

  • Apps. L–N establish convergence for all strictly positive equal-depth scalar initializations, balanced or unbalanced, and generic stability in this setting and for maximal-depth selected supports; we hadn't included clear versions of these in the original text. They also establish generic stable convergence in the general case conditional on the semialgebraic form of Kurdyka’s gradient-finiteness conjecture, which we hadn't addressed in the original text.
  • Section 8 and App. J now formulate nucleation along moving boundary trajectories, distinguishing instantaneous growth from accumulated recruitment, extending the theory's reach.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.