- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper examined upscaling via depth using concat vs stacked and analyzed it from a discretization of a continuous ODE point of view. Unlike some prior works, it was found that concat was less stable than in-place stacking as the number of layers expanded. The authors proved that this is due to the fact that the stacked version corresponds to a stable discretization, where the forward and adjoint solutions converges as .
Critical strengths
- The authors were able to show, with some amounts of analytical rigor, a phenomenon found via empirical testing: in-place stacking worked better as for deeper networks, and might potentially lead to training larger/deeper networks more efficiently. The analysis was relatively straightforward and classical, but that was enough to match the scenario from practice.
Critical weaknesses
-
Concern over essentially two experiments: the GPT2/Qwen experiments themselves feel limited. It seems that the theory suggest that upscaling to much larger scales can work, so why not do it instead of just the 2x. DeepNet (a paper you guys already cite/use) manage to scale quite large. Similarly, Figure 4 and 5's validation losses are quite high to me, suggesting that additional comparisons/work is needed for useable upscaling.
-
Originality: while the theory ostensibly is for transformers, nothing really restricts it to that architecture. The paper mentions in the conclusion about multigrid ideas, but fail to mention several papers which are quite relevant and have overlapping results. For example a.(5) and (6) of "Stateful ODE-nets using basis function expansions" is essentially lines 182 and (3) of the present paper b. (2.3) of "Globally Convergent Multilevel Training of Deep Residual Networks" is the same as (1) with minor details that doesn't play too much into the results. c. \S 2.3 of "Layer-Parallel Training of Deep Residual Neural Networks" is the same adjoint training. I think additional great emphasis have to be made in the present manuscript of what is sort of known/established versus what is actually novel.
Questions for authors
-
There are some bounds that are proven with , can one show on toy problems that this is true in empirical fashion? E.g. make plot with L on the axis and show the decay up to a point.
-
Figures 1/2: both use Qwen on the x-axis. isn't it better to just use "layer" count, or some sort of normalized? The way that figure is constructed highly suggests that the x-axis is a continuous spectrum.
-
Isn't prop 1 a tautology by def of Lipschitz?
-
Why is it that scaling in lines ~140 is not introduced for (4)?
-
Oddly enough, I found the counter-example in the appendix to be very interesting. Can you expand on this and make it into a general statement?
5 comments
W2. Relation to prior work and theoretical contributions
Thank you for pointing us to these related works. Your feedback is particularly helpful for clarifying how our perspective differs from prior work and for stating our contributions precisely.
-
Techniques used in our work. Piecewise-constant and linear interpolation are established techniques, including those used in [2,3]. As stated in our Contributions paragraph, our contribution is not to propose these methods, but to prove the consistency of simple-continuation and scaled stacking for serial Transformers.
-
Discrepancies in our infinite-depth formulation. We do not adopt the common "formulate in continuous time and then discretize" perspective. The optimal value of discrete problem does not generally converge to its continuous counterpart (see Example 2). The relation between general ResNet and its continuous limit is yet unclear [8]. Therefore, without sufficient regularities, we consider to prolong the discretely trained solution to continuous time. Despite similar in structure, it differs from multigrid methods as our starting point.
-
Serial Transformer dynamics. Some of our results are not limited to Transformers. However, a serial Transformer block differs from the single residual in a standard ResNet block. For the continuous composite dynamics
a serial Transformer corresponds to a forward-forward splitting discretization, whereas forward Euler gives a parallel Transformer. To study depth up-scaling of serial Transformers from an infinite-depth perspective, we first construct a weak continuation of the trained discrete controls for the -dependent controlled ODE , and then refine its grid using forward Euler. Appendix B.4 also compares the serial and parallel Transformer formulations.
-
Consistent grid refinement across depth-dependent ODEs. Standard multigrid alternates between coarse and fine grids to solve a fixed controlled ODE problem [1,2]. For serial Transformers, however, successive growth stages instead move between different controlled ODEs. For example, the -fold grid refinement of generally differs from -step forward Euler applied to the up-scaled serial dynamics . For scaled stacking, this field is in our analysis.
-
Contribution to LLM pretraining. Our contribution is a depth-dependent understanding of model growth. A strategy unfavored by ablations previously on shallower models may become reliable as depth increases. Our theory predicts increasing consistency of scaled stacking with depth, and our experiments validate this prediction.
Q1. Empirical error scaling
We have completed the requested experiments examining the error as a function of and added the results in Figure 10, where the zeroth- and first-order consistencies are observed as prediction.
Q2. Layer counts in Figures 1 and 2
Thank you for the suggestion. The checkpoint labels identify the complete Qwen3 family, including models that share the same depth. We agree that the layer counts should be explicit when interpreting these plots. We will emphasize this point.
Q3. The role of Proposition 1
Yes. Proposition 1 follows directly from Lipschitz continuity. This basic concept for ResNets motivates the analogous scaling assumption for the attention and FFN residuals in a serial Transformer.
Q4. Explicit coefficients and residual scaling
Cohen et al. [4] study architectures with trainable residual coefficients, DeepNet [5] uses fixed scaling of the skip connection and weight initialization, and ReZero [6] introduces trainable residual gates. Such depth-dependent behavior can arise without an additional trainable coefficient: Chang et al. [7] report an approximately inverse-depth trend in the average residual norm of ResNets without an explicit residual multiplier. For comparison, we retain the standard serial Transformer blocks without adding these architectural modifications.
Q5. A sufficient condition for terminal-state consistency
Thank you for suggesting this generalization. Under Assumptions 1 and 2, a sufficient condition for terminal-state consistency is
Stability relative to simple-continuation, together with its Euler error, gives . We provide the detailed proof in Appendix A.2.
[1] Alena Kopaničáková and Rolf Krause. Globally Convergent Multilevel Training of Deep Residual Networks. SIAM Journal on Scientific Computing, 45(3):S254–S280, 2023.
[2] Stefanie Günther, Lars Ruthotto, Jacob B. Schroder, Eric C. Cyr, and Nicolas R. Gauger. Layer-Parallel Training of Deep Residual Neural Networks. SIAM Journal on Mathematics of Data Science, 2(1):1–23, 2020.
[3] Alejandro Queiruga, N. Benjamin Erichson, Liam Hodgkinson, and Michael W. Mahoney. Stateful ODE-Nets using Basis Function Expansions. NeurIPS, 2021.
[4] Alain-Sam Cohen, Rama Cont, Alain Rossier, and Renyuan Xu. Scaling Properties of Deep Residual Networks. ICML, 2021.
[5] Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. DeepNet: Scaling Transformers to 1,000 Layers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(10):6761–6774, 2024.
[6] Thomas Bachlechner, Bodhisattwa Prasad Majumder, Henry Mao, Gary Cottrell, and Julian McAuley. ReZero is all you need: Fast convergence at large depth. UAI, 2021.
[7] Bo Chang, Lili Meng, Eldad Haber, Frederick Tung, and David Begert. Multi-level Residual Networks from Dynamical Systems View. ICLR, 2018.
[8] Raphael Barboni, Gabriel Peyre, and Francois-Xavier Vialard. Understanding the training of infinitely deep and wide ResNets with conditional optimal transport. Communications on Pure and Applied Mathematics, 78(11):2149–2205.
Thank you for your valuable insights and constructive comments.
W1. Experimental scope and validation loss
Larger growth factors. Our experiments have included immediate post-growth evaluation of Qwen3-14B at M=3 and M=4 (Appendix B.5.3), as well as GPT-2 continual training at M=4 (Figures 19 and 21). For Qwen3-14B, scaled stacking outperforms direct stacking on 0.3429 of 14 metrics at each growth factor. For GPT-2, optimizer-state inheritance improves continual training (Figure 19), and stacking achieves lower loss and higher token accuracy than concatenation (Figure 21). These results support our conclusions at growth factors up to M=4.
Additional architectures. We add immediate post-growth evaluations at growth factor M=2 for:
We observe immediate performance preservation across all three model families, including those with nonsmooth routing or alternating attention patterns.