acceptodds
Under review as a conference paper at ICLR 2027

WIDTH AND DEPTH LIMITS COMMUTE DURING TRAINING IN LINEAR RESIDUAL NETWORKS

Abstract

Recent work showed that width and depth limits in residual networks with branches scaled with commute at *initialization* in the sense that no matter how those limits are taken (e.g. width first, then depth, or depth first, then width, or with a fixed width/depth ratio, etc), the network output converges to the same limit at initialization. In this paper, we extend this result to commutativity *during training* for linear residual networks: we show that under the same branch scaling, for any fixed number of gradient steps with gradient clipping, the network output converges in as , implying commutativity of the width and depth limits during training. We further show that such result does not hold without gradient clipping, and a weaker result can be obtained in that case under the weaker convergence in probability.

Primary area:
learning theory
Keywords:
muP, deep learning theory, scaling limits, training dynamics

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_HUDKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proves commutativity of infinite width and infinite depth limits for linear ResNets when their residual branches are scaled as L−1/2L^{-1/2}, where LL is depth, after any fixed finite number of training steps. Previous works either proved commutativity only at initialization, or scaled residual branches as L−1L^{-1}, which gave a more "degenerate" deterministic limit, akin to LLN (L−1L^{-1}) vs CLT (L−1/2L^{-1/2}). The main theorems are: (1) commutativity in probability for conventional GD, (2) non-commutativity in L2L_2 for conventional GD, (3) commutativity in L2L_2 for clipped GD. The commutativity is proven in two steps. First, for fixed width nn, each forward pass is compared to the corresponding SDE, and a uniform in nn O(L−1/2)O(L^{-1/2}) error bound is established. Second, for the resulting finite-nn SDE, some of the variables are substituted with their infinite nn limits, and the corresponding error is proven to be O(n−1/2)O(n^{-1/2}). The setup asssumes square loss and a single training sample; this is standard for Tensor Programs-style analysis. The paper includes toy experiments on synthetic Gaussian and HIGGS datasets which certify correctness of the established rigorous results.

Critical strengths
  1. The paper is well-organized, relatively easy to read, and gives a clear overview of the main result.
Critical weaknesses
  1. [1] already derives a finite-width/depth error estimate of order O(n−1/2)+O(L−1)O(n^{-1/2}) + O(L^{-1}) (Appendix I, Equation 116), and its predicted L−1L^{-1} depth rate is consistent with the empirical results of the present paper. Although the analysis in [1] is non-rigorous and does not establish the same theorem for clipped GD, it substantially anticipates both the commutativity phenomenon and the convergence-rate behavior studied here. Thus, the main novelty of the present work appears to be the rigorous (L2L^2) convergence guarantee under gradient clipping, rather than the observation of commutativity during training or the predicted scaling rates. I believe this distinction should be emphasized more clearly in the introduction and related-work discussion.
  2. [2] appears to be a very related work, as relevant as [1], but it is not mentioned. Also, [1] is mentioned only briefly as "heuristic analysis", without mentioning the (tighter and empirically consistent) rates they estimate. To my opinion, the paper needs a more detailed related work section, comparing claimed rates, residual branch scalings, and level of rigor.

I will raise my score if both of the issues get addressed during the rebuttal phase

[1] Blake Bordelon, Lorenzo Noci, Mufan Bill Li, Boris Hanin, and Cengiz Pehlevan. Depthwise hyperparameter transfer in residual networks: Dynamics and scaling limit, ICLR'2024.

[2] Blake Bordelon and Cengiz Pehlevan. Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer, ICML'2025.

Questions for authors
  1. Am I right that if one wanted to prove mere commutativity (without rates), they would only prove fn,L(k),τ→Fn(k),τf^{(k),\tau}_{n,L} \to F^{(k),\tau}_n uniformly in nn, while the second (large nn) limit is not needed? Indeed, if depth convergence is uniform in nn then, by Moore-Osgood, for the joint limit to exist, is suffices to have an infinite-width limit for any LL. But the latter appears obtainable by standard Tensor Programs machinery.
  2. Could authors please comment on the proof chains in Section 4.2? I am not sure if I understood in which order arrows are established.
  3. How fast does the constant CTC_T grow with TT?
  4. Do you have a guess for why the exponent in LL becomes loose?
  5. Only two datasets are used: synthetic Gaussian and HIGGS. Are more traditional MNIST and CIFAR really more computationally prohibitive compared to HIGGS? At first sight, this is large depth that is the main bottleneck, not the input size.
  6. It would be nice to see plots, akin to that of Figure 3, in Appendix G, particularly, for experiments with activation functions. How does the error behave with nn and LL?
6 comments

Comment 1: Prior work and Questions 1–3

We thank the reviewer for the constructive feedback. We revised the Introduction (p. 1) and Related Work (p. 2) accordingly.

Relation to [1], [2], and [5]. Our L−1/2L ^{ -1/2 } branch scaling matches [1] for the linear activation and [2] when its branch constant is one: because WℓW _{ \ell } has entry variance 1/n1/n, L−1/2Wℓ=Gℓ/nLL ^{ -1/2 } W _{ \ell } = G _{ \ell } / \sqrt{ nL }. Bordelon et al. [1] use dynamical mean-field theory to study the large-width and large-depth limits of training and predict N−1/2N ^{ -1/2 } finite-width fluctuations and an L−1L ^{ -1 } finite-depth error. Our Figure 3 empirically matches these rates. Bordelon and Pehlevan [2] analyze linear ResNets in a proportional large-data, large-width regime and derive an infinite-depth limit.

These works motivate a joint-limit prediction at the level of physics rigor, but separate width and depth asymptotics do not prove joint convergence along arbitrary paths. The O(n−1/2)O(n ^{ -1/2 }) and O(L−1)O(L ^{ -1 }) rates appear in [1] and [5], while [5] establishes commutativity with these rates only at initialization (Corollary 2). Our contribution is a rigorous training-time theorem for the linear ResNet considered here. For clipped GD and every fixed training horizon, ∥fn,L(k),τ−f∞(k),τ∥L2≤CT(n−1/2+L−1/2)\| f _{ n,L } ^{ (k), \tau } - f _{ \infty } ^{ (k), \tau } \| _{ L ^ 2 } \le C _{ T } (n ^{ -1/2 } + L ^{ -1/2 }) along every joint growth path. We also prove joint convergence in probability for ordinary GD and show that the depth-first second moment can diverge at fixed finite width. Our L−1/2L ^{ -1/2 } term is a width-uniform upper bound and does not contradict the sharper infinite-width prediction in [1] or the initialization result in [5]. We added this comparison to the Introduction and Related Work.

Q1. Is the large-width comparison needed for qualitative commutativity? No. Given a width-uniform depth limit and width convergence at each fixed depth, Moore--Osgood yields qualitative commutativity. Our continuous-depth large-width comparison provides the quantitative joint rate O(n−1/2+L−1/2)O(n ^{ -1/2 } + L ^{ -1/2 }) along arbitrary growth paths.

Q2. What is the order of the arrows in Section 4.2? Let Si+1:jn,L:=Aj(0)⋯Ai+1(0)S_{i+1:j}^{n,L}:=A_j^{(0)}\cdots A_{i+1}^{(0)} be the discrete initialization propagator, and let Φti,tj\Phi_{t_i,t_j} be the solution operator of the corresponding matrix-Brownian SDE from depth tit_i to tjt_j. The base comparison is max⁡0≤i≤j≤L∥Si+1:jn,L−Φti,tj∥L2(op)=O(L−1/2)\max_{0\le i\le j\le L}\|S_{i+1:j}^{n,L}-\Phi_{t_i,t_j}\|_{L^2(\mathrm{op})}=O(L^{-1/2}), uniformly in width.

The first diagram compares the finite-depth network in the top row with its finite-width continuous-depth counterpart in the bottom row. At training step kk, the forward and backward states determine the clipped coefficient γ(k)\gamma^{(k)}. This coefficient determines the updated drift V(k+1)V^{(k+1)}, which then determines the forward and backward states at step k+1k+1. The vertical comparisons show that the O(L−1/2)O(L^{-1/2}) rate is preserved through this induction.

The second diagram compares the finite-width continuous-depth states with surrogate states whose scalar coefficients and empirical inner products are replaced by their infinite-width limits. An O(n−1/2)O(n^{-1/2}) state comparison at step kk first controls γ(k)\gamma^{(k)} and then the states at step k+1k+1. This gives the O(n−1/2)O(n^{-1/2}) bound for term (I) of the output decomposition. Conditional Gaussian concentration and normalized-trace convergence control term (II). We added this order explicitly before the diagrams in Section 4.2 (pp. 7--8).

Q3. How does CTC_T grow? For θ=(q0,η,τ,∣y⋆∣)\theta=(q_0,\eta,\tau,|y_\star|), CT≤exp⁡{Kθ(T+1)4}C_T\le\exp\{K_\theta(T+1)^4\}, with KθK_\theta independent of T,n,LT,n,L. Tracking constants gives exp⁡{Oθ((T+1)2)}\exp\{O_\theta((T+1)^2)\} from the depth comparison and exp⁡{Oθ((T+1)4)}\exp\{O_\theta((T+1)^4)\} from the width comparison. We added the calculation to Appendix H (pp. 46--47).

Comment 2: Questions 4–6 and references

Q4. Why might the L−1/2L^{-1/2} depth bound be loose? The proof begins with the standard strong Euler--Maruyama rate O(L−1/2)O(L^{-1/2}), and the training induction propagates this rate to the output bound. Establishing the empirical L−1L^{-1} rate would require new proof machinery beyond the present strong discretization estimates. Our theorem therefore states the rigorous L−1/2L^{-1/2} upper bound, while Figure 3 reports the faster empirical rate, consistent with the infinite-width prediction in [1]. We added this discussion to Appendix G.3 (p. 45).

Q5. Why not MNIST or CIFAR? We added both datasets in Section 5.1, Figure 2 (pp. 8--9). Using the same fully connected residual architecture, we evaluate five paths n=Ln=L, n=2Ln=2L, L=2nL=2n, n=10Ln=10L, and n=20Ln=20L. For each dataset, we train a ten-class classifier on 1,024 examples by full-batch GD without clipping for ten steps using cross-entropy loss. The independent reference uses (n,L)=(3072,320)(n,L)=(3072,320), and the logit-vector distance is evaluated on 1,024 fixed test examples. The RMS distance from the reference decreases with joint scale on both datasets, and the five paths approach one another as the common scale grows. These experiments provide evidence that the phenomenon is not restricted to the synthetic and HIGGS tasks.

Q6. Can Appendix G show nonlinear error curves? We added Figure 3-style sweeps through 50 GD steps in Appendix I.4, Figure 9 (p. 52). At step 50, the width/depth exponents are 0.45/0.790.45/0.79 (ReLU), 0.48/0.800.48/0.80 (tanh), 0.50/1.000.50/1.00 (GELU), 0.49/0.960.49/0.96 (SiLU), and 0.51/0.630.51/0.63 (centered sigmoid). Raw sigmoid has a fixed 1/21/2 offset that is propagated through every block and can mask depth-dependent differences [4], motivating ϕ(z)=σ(z)−1/2\phi(z)=\sigma(z)-1/2. ReLU, tanh, GELU, SiLU, and centered sigmoid show decreasing width and depth errors over the fitted regimes. Centering removes the fixed mean while leaving the sigmoid derivative unchanged. These empirical results support width--depth commutativity beyond the linear setting and show that the same width and depth convergence trends persist across the tested nonlinear activations.

We thank the reviewer again for the constructive comments and hope that they will adjust the scores accordingly.

[1] Bordelon et al., Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit, ICLR 2024.

[2] Bordelon and Pehlevan, Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer, ICML 2025.

[3] Kloeden and Platen, Numerical Solution of Stochastic Differential Equations, Springer, 1992.

[4] Glorot and Bengio, Understanding the Difficulty of Training Deep Feedforward Neural Networks, AISTATS 2010.

[5] Hayou, Commutative Width and Depth Scaling in Deep Neural Networks, JMLR 2024.

Thank you again for your detailed review. In our response and revised manuscript, we clarified the relation to prior width–depth scaling results, expanded the proof explanation and the dependence on the training horizon, discussed the finite-depth rate, and added new experiments on MNIST, CIFAR-10, and several nonlinear activations.

As the discussion period closes on November 18, we want to follow up whether our responses and revisions have addressed your concerns. We would be happy to provide further details. Thank you for your time!

Reviewer_GZjmOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies width–depth limits during training in linear residual networks. It shows that ordinary gradient descent admits a joint limit in probability but can fail to converge in L2L^2. With gradient clipping, it establishes L2L^2 convergence with explicit width and depth rates.

Critical strengths
  • The paper shows that convergence in probability survives ordinary GD while L2L^2 convergence can fail, and that clipping restores the stronger guarantee. This clarifies which scaling guarantees survive training and is the main reason I lean toward acceptance.

  • The quantitative error bound allows width and depth to grow at arbitrary relative rates, making the result useful beyond a particular scaling path. The paper is also clearly written, and the proof overview makes the main ideas easy to follow.

Critical weaknesses
  • Model assumptions: The analysis is limited to linear networks, with only the residual layers trained. Although the experiments include ReLU and other nonlinear activations, it remains unclear whether the distinction between convergence in probability and in L2L^2, and the role of clipping, carry over to these networks.
  • Experiments: The experiments do not specify whether gradient clipping is used or give its threshold. Clarifying this would help connect the experiments to the theoretical results.
Questions for authors

1.Were the experiments run with gradient clipping? If so, what threshold and clipping rule were used?

2.How sensitive are the fitted convergence rates to the size and random seeds of the networks used to construct the empirical reference?

3.On page 18, the bound for term (b) uses a supremum over both endpoints of the flow. How does this follow from Eq. (19), which controls each fixed starting point?

2 comments

Q1. Were the experiments run with gradient clipping? If so, what threshold and clipping rule were used?

We did not use gradient clipping in any experiment. Appendix D proves joint convergence in probability for ordinary GD along arbitrary width-depth growth paths at every fixed training horizon, sufficing to observe commutativity empirically. Clipping is introduced only for the stronger L2L ^ 2 result in Theorem 4.1. We clarified this in Sec. 5 (p. 8).

Q2. How sensitive are the fitted convergence rates to the size and random seeds of the networks used to construct the empirical reference?

For Fig. 3, the original reference averages 25 trajectories at m=3072m = 3072 from five width-depth paths and seeds 0-4. We reconstructed it with seeds 5-9 and compared m=2048m = 2048 and m=1024m = 1024, keeping sweep data and fitting scales fixed. Each entry is (αn,αL)(\alpha_{n}, \alpha_{L}).

Step km=3072m = 3072, seeds 0-4m=3072m = 3072, seeds 5-9m=2048m = 2048, seeds 0-4m=1024m = 1024, seeds 0-4
1(0.427, 0.738)(0.444, 0.661)(0.405, 0.761)(0.362, 0.818)
5(0.425, 0.899)(0.463, 0.863)(0.399, 0.934)(0.352, 0.930)
10(0.447, 1.019)(0.473, 1.031)(0.433, 1.047)(0.326, 1.111)
15(0.535, 0.960)(0.523, 1.052)(0.539, 1.006)(0.404, 1.084)

Changing reference seeds or reducing mm to 2048 changes fitted rates slightly. Finite-size effects become noticeable at m=1024m = 1024. We added these comparisons to Appendix I.1, Table 2 (p. 49).

Q3. On page 18, the bound for term (b) uses a supremum over both endpoints of the flow. How does this follow from Eq. (19), which controls each fixed starting point?

Thanks for catching this. Eq. (19) gives an LpL ^ p bound on A(s):=sup⁡t∈[s,1]∣Φs,t∣opA(s) := \sup _{ t \in [s, 1] } \\| \Phi _{ s, t } \\| _{ \mathrm{op} } uniformly for each fixed ss, but it doesn't justify moving the supremum inside the norm. For s∈Iℓs \in I _{ \ell }, set ΔV(s):=Vn,L(k)(tℓ)−Vn(k)(s)\Delta V(s) := V _{ n, L } ^{ (k) }(t _{ \ell }) - V _{ n } ^{ (k) }(s), and let MY:=max⁡m≤L∣Yn,L(k),τ(tm−1)∣nM _{ Y } := \max _{ m \le L } \\| Y _{ n, L } ^{ (k), \tau }(t _{ m - 1 }) \\| _{ n }. For each jj, since ⋃ℓ=1jIℓ=[0,tj]\bigcup _{ \ell = 1 } ^ j I _{ \ell } = [0, t _{ j }], the triangle inequality and ∣ΔV(s)∣op≤∣ΔV(s)∣F\\| \Delta V(s) \\| _{ \mathrm{op} } \le \\| \Delta V(s) \\| _{ \mathrm{F} } give

R _{ j } &:= \left\| \sum _{ \ell = 1 } ^ j \int _{ I _{ \ell } } \mathrm{(b)}(s) \, ds \right\| _{ n } \\ &\le M _{ Y } \sum _{ \ell = 1 } ^ j \int _{ I _{ \ell } } \| \Phi _{ s, t _{ j } } \| _{ \mathrm{op} } \| \Delta V(s) \| _{ \mathrm{F} } \, ds \\ &\le M _{ Y } \int _{ 0 } ^ { t _{ j } } A(s) \| \Delta V(s) \| _{ \mathrm{F} } \, ds \le M _{ Y } \left( \int _{ 0 } ^ 1 A(s) ^ 2 \, ds \right) ^ { 1/2 } \| \Delta V \| _{ L _{ t } ^ 2 (\mathrm{F}) }. \end{aligned}$$ The right-hand side is independent of $j$, so the inequality holds after taking $\max _{ j \le L }$. Minkowski's inequality and Eq. (19) then give, for the moment order $p$ used in the proof, $$\left\\| \left( \int _{ 0 } ^ 1 A(s) ^ 2 \\, ds \right) ^ { 1/2 } \right\\| _{ L ^ p } \le \left( \int _{ 0 } ^ 1 \\| A(s) \\| _{ L ^ p } ^ 2 \\, ds \right) ^ { 1/2 } \le C _{ \mathrm{base} }(p).$$ Set $p = 8r$ above. Proposition F.1 and the induction hypothesis (29) control the state and drift-difference factors, respectively. Hölder's inequality on these factors yields the required $O(L ^{ -1/2 })$ bound for term (b). We replaced the unsupported two-endpoint supremum bound with this fixed-start integral estimate in App. B.2 of the revised manuscript (p. 20). **Model assumptions (Critical Weaknesses):** Extending the theory to nonlinear residual blocks is a substantial open problem. In the linear model, the Jacobian of a residual block with respect to its hidden state is $I + L ^{ -1/2 } W _{ \ell }$, which does not depend on the current hidden state. This yields a closed matrix-flow representation of the forward and backward dynamics and finite-rank parameter updates. These structures are essential to our width-uniform strong Euler comparison and the Brownian-flow and Volterra estimates. For a nonlinear block $h \mapsto h + L ^{ -1/2 } W _{ \ell } \phi(h)$, the Jacobian contains $\operatorname{Diag}(\phi'(h))$ and therefore depends on the evolving hidden trajectory. This couples the forward states, backward flows, and parameter updates across layers and training steps. The linear representation and resulting estimates do not transfer directly, so a new proof strategy is required. App. I.4 provides empirical evidence with ReLU, tanh, GELU, SiLU, and centered sigmoid, but we do not claim theoretical convergence in probability or in $L ^ 2$ for these nonlinear networks. We added the theoretical discussion to App. G.2, “Nonlinear residual blocks” (p. 45), and the experiments to App. I.4, Fig. 9 (p. 52). **We thank the reviewer again for the constructive comments and hope that they will adjust the scores accordingly.**

Thank you for the response. The clarification about clipping and the additional reference-sensitivity checks address my experimental concerns. The revised estimate for term (b) also resolves the proof issue I raised. I maintain my score of 3.

Reviewer_GpG2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the problem of commutativity between width and depth in limits during training for linear residual networks. It is proved that they commute in probability after one gradient step; however, it diverges in L2L_2 depth-wise under gradient descent. The main result shows that, by clipping the gradients, the width and depth limits commute during training and there is an upper bound given by CT(n−1/2+L−1/2)C_T(n^{-1/2}+L^{-1/2}) where CTC_T is a finite constant and n,Ln,L are width and depth. In particular, the n−1/2n^{-1/2} term is sharp and cannot be improved. The main difficulty of the commutativity in L2L_2 under the gradient descent scenario is also discussed. It is due to the fact that after the gradient step, the residual blocks are no longer independent so the proof technique of independent-increment Brownian coupling cannot be applied. Experimental results show the depth rate of L−1/2L^{-1/2} is likely not tight given the faster rate observed empirically.

Critical strengths

This paper proves an interesting property that the depth and width limits commute in L2L_2 during training for linear residual networks when gradient clipping is applied. The difficulty and the proof techniques are well presented and motivated.

Critical weaknesses

For the main result (Theorem 4.1), it is not clear whether for a fixed TT and τ\tau one can achieve arbitrary estimation error. The result can be more insightful if the target y∗y_* is the limit or the estimation error is involved. The most obvious weakness of this work is probably its practical impact. The main result shows that the depth and width limits commute for a very specific network type, the family of linear residual networks, under a quite restrictive setting (gradient clipping). It is not clear how to use this result to design a better architecture or improve training performance.

Questions for authors
  1. Given that the clipping radius τ\tau is independent of TT and it doesn't decrease during training, does the converged output still predict the target accurately?
  2. Is it possible to extend the main result (Theorem 4.1) to nonlinear residual networks?
  3. In Theorem 4.1, is it possible to replace f∞(k),τf_{\infty}^{(k),\tau} with y∗y_* and relax the upper bound to include the estimation error?
  4. Does the main result motivate a better design of residual networks? Are there any practical implications from the main result?
5 comments

Thank you for the constructive comments. We revised the manuscript to separate the finite-size guarantee from optimization of the limiting dynamics and to clarify the practical scope.

Q1. Does a fixed clipping radius prevent accurate prediction of the target? Theorem 4.1 holds for every fixed finite training horizon TT and controls every iterate k≤Tk\leq T. For each such kk, the target error satisfies

∣fn,L(k),τ−y⋆∣L2≤CT(n−1/2+L−1/2)+∣f∞(k),τ−y⋆∣.|f_{n,L}^{(k),\tau}-y_{\star}|_{L^2} \le C_T(n^{-1/2}+L^{-1/2})+\lvert f_{\infty}^{(k),\tau}-y_{\star}\rvert.

The second term is the optimization error after kk steps. Whether the limiting training dynamics converges to y⋆y_\star is a separate optimization question and is not the main focus of this paper, which studies commutativity of the width and depth limits at each fixed finite training horizon. Clipping is used only to obtain the stronger L2L^2 commutativity result. We do not expect clipping to be a bottleneck for optimization convergence. For squared loss, ∇L=(f−y⋆)∇f\nabla\mathcal L=(f-y_\star)\nabla f, so as the prediction approaches the target, the loss gradient becomes smaller than the fixed clipping threshold. Clipping is then inactive and the update is identical to ordinary GD, so it does not induce a nonzero prediction-error floor. In addition, Appendix D proves joint width--depth convergence in probability for ordinary GD without clipping. We added the target-error decomposition and the discussion of the optimization term to Appendix G.1, “Optimization error” (p. 45).

Q2. Can Theorem 4.1 be extended to nonlinear residual networks? This is still an open problem and would require new proof machinery. The key use of linearity is that the Jacobian of a residual block with respect to its hidden state is I+L−1/2WℓI+L^{-1/2}W_\ell. It depends on the weight matrix but not on the current hidden state. This gives a closed matrix-flow representation of the forward and backward dynamics and finite-rank parameter updates, which underlie our width-uniform strong Euler comparison and the Brownian and Volterra estimates. For a nonlinear block h↦h+L−1/2Wℓϕ(h)h\mapsto h+L^{-1/2}W_\ell\phi(h), the Jacobian becomes I+L−1/2WℓDiag⁡(ϕ′(h))I+L^{-1/2}W_\ell\operatorname{Diag}(\phi'(h)). The propagation operator therefore depends on the evolving trajectory of hidden states, coupling the forward states, backward flows, and parameter updates across layers and training steps. The linear matrix-flow representation and the resulting estimates do not transfer directly to this setting. Appendix I.4 reports experiments with ReLU, tanh, GELU, SiLU, and centered sigmoid. For every tested activation, the outputs along different joint width--depth growth paths converge as the common scale increases, providing empirical evidence for commutativity beyond the linear setting. We do not make a theoretical convergence claim for these nonlinear networks. We added the theoretical discussion to Appendix G.2, “Nonlinear residual blocks” (p. 45), and the experiments to Appendix I.4, Figure 9 (p. 52).

Q3. Can the theorem include estimation error? The displayed bound in Q1 gives the requested decomposition. Its first term is the finite-width/depth approximation error, and the second is the optimization error of the deterministic limiting dynamics. Although every block is linear, training is nonconvex because the output depends jointly on all residual matrices:

f(W1,…,WL)=u⊤∏ℓ=1L(I+L−1/2Wℓ)v.f(W_1,\ldots,W_L)=u^\top\prod_{\ell=1}^L(I+L^{-1/2}W_\ell)v.

It is linear in each WℓW_\ell separately but not jointly convex. A finite-step optimization rate would require extra conditions, such as a uniform lower bound on ∥∇f∞(k),τ∥2\|\nabla f_\infty^{(k),\tau}\|^2, smoothness, and a compatible learning rate. These are not automatic here. We therefore leave ∣f∞(k),τ−y⋆∣|f_\infty^{(k),\tau}-y_\star| explicit and added the explanation to Appendix G.1 (p. 45).

Q4. Practical impact? Figure 3 empirically suggests

E(n,L)≈Cwn−1/2+CdL−1.E(n,L)\approx C_{\mathrm w}n^{-1/2}+C_{\mathrm d}L^{-1}.

Balancing the terms gives n≍L2n\asymp L^2. Representative open-weight models follow this order-one n/L2n/L^2 pattern:

ModelnnLLn/L2n/L^2
Mistral 7B4,096324.00
OLMo 2 13B5,120403.20
Qwen2.5 32B5,120641.25
Llama 3 70B8,192801.28
Llama 3.1 405B16,3841261.03

These aspect ratios were likely tuned to identify effective width--depth combinations and are consistent with our empirical finding: multiplying depth by aa suggests multiplying width by approximately a2a^2 to keep the two finite-size errors comparable. We added this implication and the larger comparison table to Section 5.2, Table 1 (p. 10).

We thank the reviewer again for the constructive comments and hope that they will adjust the scores accordingly.

Thank you again for your constructive review. In our response and revised manuscript, we addressed each of your four points by clarifying the role of gradient clipping and the optimization error, adding nonlinear experiments, and discussing the practical width–depth scaling implication.

As the discussion period closes on November 18, we want to ask whether our responses and revisions have addressed your concerns. We would be happy to clarify any remaining points or provide further details. Thank you for your time!

As the discussion period comes to an end, we hope to make the best use of the remaining time for further discussion.

We have carefully addressed all the comments and concerns raised in your review and have revised the manuscript accordingly.

We would be grateful if you could let us know whether our responses and revisions address your concerns. We are happy to provide any additional clarification that may be helpful. Thank you again for your time and consideration.

Comments on the paper

1 comment

We sincerely thank all reviewers and the Area Chair for their constructive and insightful feedback. We have conducted additional experiments, strengthened the theoretical results, and revised the manuscript to address the main questions raised during the discussion. The key responses and revisions are summarized below.

  • Strengthened the theoretical results and clarified the role of clipping. We prove convergence in probability under ordinary gradient descent for every fixed finite number of training steps. We extend the main commutativity result to fixed finite training sets and finite-dimensional outputs. We also give an explicit bound for how the convergence constant grows with the number of training steps. To clarify the role of clipping, we separate the finite-width and finite-depth approximation error from the optimization error of the limiting dynamics.

    Manuscript revisions: The main results appear in Sections 3 and 4. The additional proofs and extensions appear in the theoretical appendices. The discussion of clipping and optimization error appears in the Discussion appendix.

  • Expanded the experiments and convergence-rate analysis. We add MNIST and CIFAR-10 experiments along multiple joint width--depth growth paths. We report additional experiments with nonlinear residual blocks using ReLU, tanh, GELU, SiLU, sigmoid, and centered sigmoid activations. We also provide more complete width and depth convergence results throughout training. We analyze the sensitivity of the fitted rates to the reference-model scale and initialization seeds.

    Manuscript revisions: The MNIST and CIFAR-10 results appear in Section 5.1 and Figure 2. The width and depth convergence-rate results appear in Section 5.2 and Figure 3. The nonlinear-activation and reference-sensitivity results appear in the experimental appendices.

  • Clarified the relationship with related work. We revise the discussion of Bordelon et al. (2023) and Bordelon and Pehlevan (2025). The revised discussion clarifies how their analyses of width--depth scaling and training dynamics relate to the rigorous commutativity results established in our work.

    Manuscript revisions: The corresponding text appears in the second paragraph of the Introduction and the final paragraph of Related Work.

Overall, the revised manuscript strengthens the theoretical results, expands the empirical evidence, and more clearly presents the contribution and scope of the work.

As the discussion period closes on November 18, we would be grateful to hear from the remaining reviewers on whether our responses and revisions address their concerns. We are happy to provide any additional clarification that may be helpful. Thank you again for your time and careful consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.