- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proves commutativity of infinite width and infinite depth limits for linear ResNets when their residual branches are scaled as , where is depth, after any fixed finite number of training steps. Previous works either proved commutativity only at initialization, or scaled residual branches as , which gave a more "degenerate" deterministic limit, akin to LLN () vs CLT (). The main theorems are: (1) commutativity in probability for conventional GD, (2) non-commutativity in for conventional GD, (3) commutativity in for clipped GD. The commutativity is proven in two steps. First, for fixed width , each forward pass is compared to the corresponding SDE, and a uniform in error bound is established. Second, for the resulting finite- SDE, some of the variables are substituted with their infinite limits, and the corresponding error is proven to be . The setup asssumes square loss and a single training sample; this is standard for Tensor Programs-style analysis. The paper includes toy experiments on synthetic Gaussian and HIGGS datasets which certify correctness of the established rigorous results.
Critical strengths
- The paper is well-organized, relatively easy to read, and gives a clear overview of the main result.
Critical weaknesses
- [1] already derives a finite-width/depth error estimate of order (Appendix I, Equation 116), and its predicted depth rate is consistent with the empirical results of the present paper. Although the analysis in [1] is non-rigorous and does not establish the same theorem for clipped GD, it substantially anticipates both the commutativity phenomenon and the convergence-rate behavior studied here. Thus, the main novelty of the present work appears to be the rigorous () convergence guarantee under gradient clipping, rather than the observation of commutativity during training or the predicted scaling rates. I believe this distinction should be emphasized more clearly in the introduction and related-work discussion.
- [2] appears to be a very related work, as relevant as [1], but it is not mentioned. Also, [1] is mentioned only briefly as "heuristic analysis", without mentioning the (tighter and empirically consistent) rates they estimate. To my opinion, the paper needs a more detailed related work section, comparing claimed rates, residual branch scalings, and level of rigor.
I will raise my score if both of the issues get addressed during the rebuttal phase
[1] Blake Bordelon, Lorenzo Noci, Mufan Bill Li, Boris Hanin, and Cengiz Pehlevan. Depthwise hyperparameter transfer in residual networks: Dynamics and scaling limit, ICLR'2024.
[2] Blake Bordelon and Cengiz Pehlevan. Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer, ICML'2025.
Questions for authors
- Am I right that if one wanted to prove mere commutativity (without rates), they would only prove uniformly in , while the second (large ) limit is not needed? Indeed, if depth convergence is uniform in then, by Moore-Osgood, for the joint limit to exist, is suffices to have an infinite-width limit for any . But the latter appears obtainable by standard Tensor Programs machinery.
- Could authors please comment on the proof chains in Section 4.2? I am not sure if I understood in which order arrows are established.
- How fast does the constant grow with ?
- Do you have a guess for why the exponent in becomes loose?
- Only two datasets are used: synthetic Gaussian and HIGGS. Are more traditional MNIST and CIFAR really more computationally prohibitive compared to HIGGS? At first sight, this is large depth that is the main bottleneck, not the input size.
- It would be nice to see plots, akin to that of Figure 3, in Appendix G, particularly, for experiments with activation functions. How does the error behave with and ?
6 comments
Comment 2: Questions 4–6 and references
Q4. Why might the depth bound be loose? The proof begins with the standard strong Euler--Maruyama rate , and the training induction propagates this rate to the output bound. Establishing the empirical rate would require new proof machinery beyond the present strong discretization estimates. Our theorem therefore states the rigorous upper bound, while Figure 3 reports the faster empirical rate, consistent with the infinite-width prediction in [1]. We added this discussion to Appendix G.3 (p. 45).
Q5. Why not MNIST or CIFAR? We added both datasets in Section 5.1, Figure 2 (pp. 8--9). Using the same fully connected residual architecture, we evaluate five paths , , , , and . For each dataset, we train a ten-class classifier on 1,024 examples by full-batch GD without clipping for ten steps using cross-entropy loss. The independent reference uses , and the logit-vector distance is evaluated on 1,024 fixed test examples. The RMS distance from the reference decreases with joint scale on both datasets, and the five paths approach one another as the common scale grows. These experiments provide evidence that the phenomenon is not restricted to the synthetic and HIGGS tasks.
Q6. Can Appendix G show nonlinear error curves? We added Figure 3-style sweeps through 50 GD steps in Appendix I.4, Figure 9 (p. 52). At step 50, the width/depth exponents are (ReLU), (tanh), (GELU), (SiLU), and (centered sigmoid). Raw sigmoid has a fixed offset that is propagated through every block and can mask depth-dependent differences [4], motivating . ReLU, tanh, GELU, SiLU, and centered sigmoid show decreasing width and depth errors over the fitted regimes. Centering removes the fixed mean while leaving the sigmoid derivative unchanged. These empirical results support width--depth commutativity beyond the linear setting and show that the same width and depth convergence trends persist across the tested nonlinear activations.
We thank the reviewer again for the constructive comments and hope that they will adjust the scores accordingly.
[1] Bordelon et al., Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit, ICLR 2024.
[2] Bordelon and Pehlevan, Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer, ICML 2025.
[3] Kloeden and Platen, Numerical Solution of Stochastic Differential Equations, Springer, 1992.
[4] Glorot and Bengio, Understanding the Difficulty of Training Deep Feedforward Neural Networks, AISTATS 2010.
[5] Hayou, Commutative Width and Depth Scaling in Deep Neural Networks, JMLR 2024.
Thank you again for your detailed review. In our response and revised manuscript, we clarified the relation to prior width–depth scaling results, expanded the proof explanation and the dependence on the training horizon, discussed the finite-depth rate, and added new experiments on MNIST, CIFAR-10, and several nonlinear activations.
As the discussion period closes on November 18, we want to follow up whether our responses and revisions have addressed your concerns. We would be happy to provide further details. Thank you for your time!
Comment 1: Prior work and Questions 1–3
We thank the reviewer for the constructive feedback. We revised the Introduction (p. 1) and Related Work (p. 2) accordingly.
Relation to [1], [2], and [5]. Our L−1/2 branch scaling matches [1] for the linear activation and [2] when its branch constant is one: because Wℓ has entry variance 1/n, L−1/2Wℓ=Gℓ/nL. Bordelon et al. [1] use dynamical mean-field theory to study the large-width and large-depth limits of training and predict N−1/2 finite-width fluctuations and an L−1 finite-depth error. Our Figure 3 empirically matches these rates. Bordelon and Pehlevan [2] analyze linear ResNets in a proportional large-data, large-width regime and derive an infinite-depth limit.
These works motivate a joint-limit prediction at the level of physics rigor, but separate width and depth asymptotics do not prove joint convergence along arbitrary paths. The O(n−1/2) and O(L−1) rates appear in [1] and [5], while [5] establishes commutativity with these rates only at initialization (Corollary 2). Our contribution is a rigorous training-time theorem for the linear ResNet considered here. For clipped GD and every fixed training horizon, ∥fn,L(k),τ−f∞(k),τ∥L2≤CT(n−1/2+L−1/2) along every joint growth path. We also prove joint convergence in probability for ordinary GD and show that the depth-first second moment can diverge at fixed finite width. Our L−1/2 term is a width-uniform upper bound and does not contradict the sharper infinite-width prediction in [1] or the initialization result in [5]. We added this comparison to the Introduction and Related Work.
Q1. Is the large-width comparison needed for qualitative commutativity? No. Given a width-uniform depth limit and width convergence at each fixed depth, Moore--Osgood yields qualitative commutativity. Our continuous-depth large-width comparison provides the quantitative joint rate O(n−1/2+L−1/2) along arbitrary growth paths.
Q2. What is the order of the arrows in Section 4.2? Let Si+1:jn,L:=Aj(0)⋯Ai+1(0) be the discrete initialization propagator, and let Φti,tj be the solution operator of the corresponding matrix-Brownian SDE from depth ti to tj. The base comparison is max0≤i≤j≤L∥Si+1:jn,L−Φti,tj∥L2(op)=O(L−1/2), uniformly in width.
The first diagram compares the finite-depth network in the top row with its finite-width continuous-depth counterpart in the bottom row. At training step k, the forward and backward states determine the clipped coefficient γ(k). This coefficient determines the updated drift V(k+1), which then determines the forward and backward states at step k+1. The vertical comparisons show that the O(L−1/2) rate is preserved through this induction.
The second diagram compares the finite-width continuous-depth states with surrogate states whose scalar coefficients and empirical inner products are replaced by their infinite-width limits. An O(n−1/2) state comparison at step k first controls γ(k) and then the states at step k+1. This gives the O(n−1/2) bound for term (I) of the output decomposition. Conditional Gaussian concentration and normalized-trace convergence control term (II). We added this order explicitly before the diagrams in Section 4.2 (pp. 7--8).
Q3. How does CT grow? For θ=(q0,η,τ,∣y⋆∣), CT≤exp{Kθ(T+1)4}, with Kθ independent of T,n,L. Tracking constants gives exp{Oθ((T+1)2)} from the depth comparison and exp{Oθ((T+1)4)} from the width comparison. We added the calculation to Appendix H (pp. 46--47).