acceptodds
Under review as a conference paper at ICLR 2027

Transformer Depth Up-Scaling: An Infinite-Depth Perspective

Abstract

Progressive training iteratively up-scales small checkpoints to reduce the cost of training large language models. We observe that, as base-model depth increases, layer concatenation leads to greater performance degradation immediately after up-scaling, whereas in-place stacking becomes more stable. Inspired by this depth-dependence trend, we consider an infinite-depth formulation of Transformers, treating depth up-scaling as continuation of parameters and study the consistency problems with a controlled ODE. A criterion on this ODE's terminal-state ensures the consistent next-token predictions between the base and the up-scaled models. For base models of depth , piecewise-constant simple-continuation yields an terminal-state error and thus asymptotically preserves immediate performance. As a concrete continuation scheme, scaled stacking approximates the Euler refinement of simple-continuation, with the same error order uniformly over growth-factors. Moreover, the ODE's adjoint gives -consistent directional derivatives, providing a criterion for preserving descent directions. Benchmarkings on Qwen3 family and progressive trainings on GPT-2 provide supports for these consistency results.

Primary area:
foundation or frontier models, including LLMs
Keywords:
deep learning optimization, model growth, Transformer up-scaling, neural ODEs, large language models, looped Transformer

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_2wRTOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper examined upscaling via depth using concat vs stacked and analyzed it from a discretization of a continuous ODE point of view. Unlike some prior works, it was found that concat was less stable than in-place stacking as the number of layers expanded. The authors proved that this is due to the fact that the stacked version corresponds to a stable discretization, where the forward and adjoint solutions converges as L→∞L \to \infty.

Critical strengths
  1. The authors were able to show, with some amounts of analytical rigor, a phenomenon found via empirical testing: in-place stacking worked better as for deeper networks, and might potentially lead to training larger/deeper networks more efficiently. The analysis was relatively straightforward and classical, but that was enough to match the scenario from practice.
Critical weaknesses
  1. Concern over essentially two experiments: the GPT2/Qwen experiments themselves feel limited. It seems that the theory suggest that upscaling to much larger scales can work, so why not do it instead of just the 2x. DeepNet (a paper you guys already cite/use) manage to scale quite large. Similarly, Figure 4 and 5's validation losses are quite high to me, suggesting that additional comparisons/work is needed for useable upscaling.

  2. Originality: while the theory ostensibly is for transformers, nothing really restricts it to that architecture. The paper mentions in the conclusion about multigrid ideas, but fail to mention several papers which are quite relevant and have overlapping results. For example a.(5) and (6) of "Stateful ODE-nets using basis function expansions" is essentially lines 182 and (3) of the present paper b. (2.3) of "Globally Convergent Multilevel Training of Deep Residual Networks" is the same as (1) with minor details that doesn't play too much into the results. c. \S 2.3 of "Layer-Parallel Training of Deep Residual Neural Networks" is the same adjoint training. I think additional great emphasis have to be made in the present manuscript of what is sort of known/established versus what is actually novel.

Questions for authors
  1. There are some bounds that are proven with O(L−1)O(L^{-1}), can one show on toy problems that this is true in empirical fashion? E.g. make plot with L on the axis and show the decay up to a point.

  2. Figures 1/2: both use Qwen on the x-axis. isn't it better to just use "layer" count, or some sort of normalized? The way that figure is constructed highly suggests that the x-axis is a continuous spectrum.

  3. Isn't prop 1 a tautology by def of Lipschitz?

  4. Why is it that scaling in lines ~140 is not introduced for (4)?

  5. Oddly enough, I found the counter-example in the appendix to be very interesting. Can you expand on this and make it into a general statement?

5 comments

Thank you for your valuable insights and constructive comments.

W1. Experimental scope and validation loss

  1. Larger growth factors. Our experiments have included immediate post-growth evaluation of Qwen3-14B at M=3M=3 and M=4M=4 (Appendix B.5.3), as well as GPT-2 continual training at M=4M=4 (Figures 19 and 21). For Qwen3-14B, scaled stacking outperforms direct stacking on 0.3429 of 14 metrics at each growth factor. For GPT-2, optimizer-state inheritance improves continual training (Figure 19), and stacking achieves lower loss and higher token accuracy than concatenation (Figure 21). These results support our conclusions at growth factors up to M=4M=4.

  2. Additional architectures. We add immediate post-growth evaluations at growth factor M=2M=2 for:

  • gpt-oss-20b: This MoE model has nonsmooth routing. It alternates local sliding-window and global attention, making residual mappings do not have the same functional form across layers.
  • Gemma 2-9B: This model uses both pre- and post-normalization. It also alternates local sliding-window and global attention. Its residual mappings likewise differ in functional form across layers.
  • Llama 3-8B: This model satisfies the architectural conditions of our analysis.

We observe immediate performance preservation across all three model families, including those with nonsmooth routing or alternating attention patterns.

Gemma2-9BGPT-OSS-20BLlama3-8B
ARC-C ↑16.590.5154.33
ARC-E ↑57.9688.5675.41
HellaSwag ↑17.87100.8382.06
MMLU ↑3.2957.9153.55
GPQA ↑8.630.010.02
IFEval ↑78.0391.81160.94
Lambada (ACC) ↑20.5852.2669.17
MGSM ↑4.9973.7244.02
GLUE ↑11.8157.0813.78
TruthfulQA (ACC) ↑0.0147.560.0
TruthfulQA (GEN) ↑75.22391.9387.18
Lambada (PPL) ↓225.04182.814.06
Pile 10K (PPL) ↓30.853.142.42
WikiText-2 (PPL) ↓10.952.01.94
  1. Validation loss at larger model scales. Figures 4 and 5 report results for 25.2M models, whose validation losses are constrained by parameter scale. However, the larger 100.8M and 0.9B models attain lower validation losses (Figures 20 and 6). Stacking also achieves lower loss than concatenation at both scales.

W2. Relation to prior work and theoretical contributions

Thank you for pointing us to these related works. Your feedback is particularly helpful for clarifying how our perspective differs from prior work and for stating our contributions precisely.

  1. Techniques used in our work. Piecewise-constant and linear interpolation are established techniques, including those used in [2,3]. As stated in our Contributions paragraph, our contribution is not to propose these methods, but to prove the consistency of simple-continuation and scaled stacking for serial Transformers.

  2. Discrepancies in our infinite-depth formulation. We do not adopt the common "formulate in continuous time and then discretize" perspective. The optimal value of discrete problem does not generally converge to its continuous counterpart (see Example 2). The relation between general ResNet and its continuous limit is yet unclear [8]. Therefore, without sufficient regularities, we consider to prolong the discretely trained solution to continuous time. Despite similar in structure, it differs from multigrid methods as our starting point.

  3. Serial Transformer dynamics. Some of our results are not limited to Transformers. However, a serial Transformer block differs from the single residual in a standard ResNet block. For the continuous composite dynamics

    dxdt=A~(x)+F~(x),\frac{\mathrm{d}\mathbf{x}}{\mathrm{d}t}=\tilde{\boldsymbol{A}}(\mathbf{x})+\tilde{\boldsymbol{F}}(\mathbf{x}),

    a serial Transformer corresponds to a forward-forward splitting discretization, whereas forward Euler gives a parallel Transformer. To study depth up-scaling of serial Transformers from an infinite-depth perspective, we first construct a weak continuation of the trained discrete controls for the LL-dependent controlled ODE dx/dt=fL\mathrm{d}\mathbf{x}/\mathrm{d}t=f_L, and then refine its grid using forward Euler. Appendix B.4 also compares the serial and parallel Transformer formulations.

  4. Consistent grid refinement across depth-dependent ODEs. Standard multigrid alternates between coarse and fine grids to solve a fixed controlled ODE problem [1,2]. For serial Transformers, however, successive growth stages instead move between different controlled ODEs. For example, the MM-fold grid refinement of fLf_L generally differs from MLML-step forward Euler applied to the up-scaled serial dynamics fMLf_{ML}. For scaled stacking, this field is gL,Mg_{L,M} in our analysis.

  5. Contribution to LLM pretraining. Our contribution is a depth-dependent understanding of model growth. A strategy unfavored by ablations previously on shallower models may become reliable as depth increases. Our theory predicts increasing consistency of scaled stacking with depth, and our experiments validate this prediction.

Q1. Empirical error scaling

We have completed the requested experiments examining the error as a function of LL and added the results in Figure 10, where the zeroth- and first-order consistencies are observed as prediction.

Q2. Layer counts in Figures 1 and 2

Thank you for the suggestion. The checkpoint labels identify the complete Qwen3 family, including models that share the same depth. We agree that the layer counts should be explicit when interpreting these plots. We will emphasize this point.

Q3. The role of Proposition 1

Yes. Proposition 1 follows directly from Lipschitz continuity. This basic concept for ResNets motivates the analogous scaling assumption for the attention and FFN residuals in a serial Transformer.

Q4. Explicit coefficients and residual scaling

Cohen et al. [4] study architectures with trainable residual coefficients, DeepNet [5] uses fixed scaling of the skip connection and weight initialization, and ReZero [6] introduces trainable residual gates. Such depth-dependent behavior can arise without an additional trainable coefficient: Chang et al. [7] report an approximately inverse-depth trend in the average residual norm of ResNets without an explicit residual multiplier. For comparison, we retain the standard serial Transformer blocks without adding these architectural modifications.

Q5. A sufficient condition for terminal-state consistency

Thank you for suggesting this generalization. Under Assumptions 1 and 2, a sufficient condition for terminal-state consistency is

∫01sup⁡x∈K∥fL(x,θL,∗(t))−fL(x,θsimpL,∗(t))∥dt→0,\int_0^1\sup_{x\in K}\left\|f_L\bigl(x,\boldsymbol{\theta}^{L,\ast}(t)\bigr)-f_L\bigl(x,\boldsymbol{\theta}_{\mathrm{simp}}^{L,\ast}(t)\bigr)\right\|\mathrm{d}t\to 0,

Stability relative to simple-continuation, together with its Euler error, gives ∥xL,∗(1)−xLL,∗∥→0\|\mathbf{x}^{L,\ast}(1)-x_L^{L,\ast}\|\to 0. We provide the detailed proof in Appendix A.2.

[1] Alena Kopaničáková and Rolf Krause. Globally Convergent Multilevel Training of Deep Residual Networks. SIAM Journal on Scientific Computing, 45(3):S254–S280, 2023.

[2] Stefanie Günther, Lars Ruthotto, Jacob B. Schroder, Eric C. Cyr, and Nicolas R. Gauger. Layer-Parallel Training of Deep Residual Neural Networks. SIAM Journal on Mathematics of Data Science, 2(1):1–23, 2020.

[3] Alejandro Queiruga, N. Benjamin Erichson, Liam Hodgkinson, and Michael W. Mahoney. Stateful ODE-Nets using Basis Function Expansions. NeurIPS, 2021.

[4] Alain-Sam Cohen, Rama Cont, Alain Rossier, and Renyuan Xu. Scaling Properties of Deep Residual Networks. ICML, 2021.

[5] Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. DeepNet: Scaling Transformers to 1,000 Layers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(10):6761–6774, 2024.

[6] Thomas Bachlechner, Bodhisattwa Prasad Majumder, Henry Mao, Gary Cottrell, and Julian McAuley. ReZero is all you need: Fast convergence at large depth. UAI, 2021.

[7] Bo Chang, Lili Meng, Eldad Haber, Frederick Tung, and David Begert. Multi-level Residual Networks from Dynamical Systems View. ICLR, 2018.

[8] Raphael Barboni, Gabriel Peyre, and Francois-Xavier Vialard. Understanding the training of infinitely deep and wide ResNets with conditional optimal transport. Communications on Pure and Applied Mathematics, 78(11):2149–2205.

Reviewer_QrRTOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores the depth expansion of Transformer models from an "infinite-depth perspective," aiming to preserve original model performance and optimization characteristics when increasing the depth of pre-trained language models. The authors model depth expansion as a process of parameter continuousization and grid refinement within the framework of controlled ordinary differential equations (ODEs). The study establishes theoretical guarantees for "zero-order consistency" (maintaining identical predictions for the next token) and "first-order consistency" (preserving the gradient and descent direction of the loss function). Building on these theoretical foundations, the paper proposes a "scaled stacking" method—which involves replicating layer structures in situ while scaling the residual outputs—combined with an optimizer-state inheritance mechanism to facilitate continued training. Experiments conducted on the Qwen3 model series and through progressive training based on GPT-2 demonstrate that this approach outperforms traditional stacking or concatenation methods, particularly in its ability to maintain model performance immediately following depth expansion.

Critical strengths
  1. Novel and interesting research problem. This paper explores an important and relatively under-researched issue: how to maintain consistency in model functionality and optimization when scaling Transformer models to greater depths. This question is particularly significant for the efficient scaling and progressive training of large language models.

  2. Detailed and comprehensive theoretical analysis. Its mathematical formulation is rigorous, and its theoretical results are supported by detailed proofs. The analysis of forward and backward consistency provides valuable insights into the behavior of deep, extended Transformer models.

  3. Clear writing and presentation. The paper is rigorously structured and clearly written. The research motivation, theoretical framework, and experimental methodology are presented in a well-organized manner, making the primary contributions easy to understand.

Critical weaknesses
  1. Limited evaluation across model families. Current evaluations of pre-trained language models are primarily limited to the Qwen3 series. Although the authors consider various model scales and include a Qwen3 MoE checkpoint, these experiments are insufficient to demonstrate the generalizability of the proposed method across different model architectures and training regimes. Evaluating other modern model families—such as GPT-OSS and Gemma 4—would be highly valuable, as it would help confirm that the observed "depth-scaling behavior" and "performance retention" characteristics are not unique to the Qwen3 models.

  2. Insufficient analysis of computational costs and practical efficiency. Although efficient model scaling is a key motivation, the paper does not systematically compare the computational cost of incremental depth expansion with training from scratch to reach comparable performance. This comparison should account for base-checkpoint training, continued training after expansion, and the added inference cost. Reporting total training FLOPs, GPU hours, or wall-clock time at matched performance is necessary to substantiate the claimed efficiency gains.

  3. Insufficient evidence that the additional depth can be effectively utilized during subsequent training. Preserving the model’s original functionality after depth expansion does not establish that the added layers improve representational capacity or performance. Although continual training experiments on GPT-2 outperform other scaling strategies, the paper lacks comparisons with both continued training of the original model and training an equally deep model from scratch under matched data and computational budgets. Without these baselines, it remains unclear whether depth expansion provides substantive benefits beyond preserving functionality at increased computational cost. In addition, the authors should provide training and validation loss curves—under strictly matched training conditions—for the original model, the depth-extended model, and a model of equivalent depth trained from scratch, while also comparing their performance on downstream tasks.

  4. Limited and inconsistent performance improvements across downstream tasks. Depth-expanded models often perform comparably to or worse than the original models, with only minor differences among expansion strategies. The results primarily demonstrate performance preservation rather than improved capabilities, and do not establish whether depth-expanded models can surpass their original counterparts after sufficient continued training. The paper should distinguish functional preservation from performance improvement and provide stronger evidence that depth expansion yields gains commensurate with its additional training and inference costs.

  5. Limited relevance of the continual training experiments to modern large language models. The experiments use GPT-2 models scaled up to approximately 0.9B parameters, which differ substantially from modern LLMs in architecture, scale, and training configuration. It therefore remains unclear whether the observed optimization gains and training stability generalize to depth expansion of modern pre-trained LLMs. Experiments on modern model families are needed to substantiate this practical applicability.

Questions for authors
  1. Under matched data and computational budgets, does the depth-expanded model outperform continued training of the original model? Please provide training and validation loss curves and downstream results to establish whether the added layers provide substantive benefits.

  2. How does the depth-expanded model compare with an equally deep model trained from scratch using the same data and total computational budget, including base-checkpoint training? Please report final performance, training FLOPs, and wall-clock time to reach a target performance level.

  3. Does depth expansion yield consistent downstream performance gains over the original model after sufficient continued training? Given the limited and sometimes negative improvements reported, could the authors clarify when these gains justify the additional training and inference costs?

  4. Could the authors conduct continual pre-training experiments on modern model families, such as GPT-OSS and Gemma 4, to assess whether the theoretical predictions and optimization benefits generalize beyond GPT-2?

5 comments

Thank you for your valuable insights and constructive comments on the practical benefits and applicability of depth up-scaling.

W1. Generalization across model families

Concern: Whether the observed preservation after depth up-scaling generalizes beyond Qwen3 to other architectures and model families.

  • Theoretical scope. As specified in the Background section, our framework considers pre-norm, decoder-only Transformers with smooth residual mappings and a same residual functional form across layers. Dense Qwen3 and GPT-2 satisfy these architectural conditions. Our Qwen3 MoE experiments also exhibit immediate performance preservation, despite the nonsmoothness introduced by top-kk routing.

  • Additional model families. We add immediate post-growth evaluations at growth factor M=2M=2 for:

    • gpt-oss-20b: This MoE model has nonsmooth routing. It alternates local sliding-window and global attention, making residual mappings do not have the same functional form across layers.
    • Gemma 2-9B: This model uses both pre- and post-normalization. It also alternates local sliding-window and global attention. Its residual mappings likewise differ in functional form across layers.
    • Llama 3-8B: This model satisfies the architectural conditions of our analysis.

We observe immediate performance preservation across all three models, including those with nonsmooth routing or alternating attention patterns. Gemma 4 was not evaluated because its multimodal architecture departs substantially from the text-only setting assumed in our framework.

Gemma2-9BGPT-OSS-20BLlama3-8B
ARC-C ↑16.590.5154.33
ARC-E ↑57.9688.5675.41
HellaSwag ↑17.87100.8382.06
MMLU ↑3.2957.9153.55
GPQA ↑8.630.010.02
IFEval ↑78.0391.81160.94
Lambada (ACC) ↑20.5852.2669.17
MGSM ↑4.9973.7244.02
GLUE ↑11.8157.0813.78
TruthfulQA (ACC) ↑0.0147.560.0
TruthfulQA (GEN) ↑75.22391.9387.18
Lambada (PPL) ↓225.04182.814.06
Pile 10K (PPL) ↓30.853.142.42
WikiText-2 (PPL) ↓10.952.01.94

W2–W5. Effectiveness of depth up-scaling

Concern: Whether depth up-scaling improves training efficiency and downstream performance relative to other growth methods and training from scratch, and whether these benefits persist in modern language models.

Progressive training through depth replication has already been extensively validated in prior work by Du et al. [1] and Meituan LongCat Team [2]. Our theoretical study builds on this established empirical foundation. For a target model of prescribed width and depth, Du et al. compare four growth operators applied along the width or depth with training from scratch. They assess the FLOPs required to reach a matched loss and downstream performance at matched total compute, including base-model training. We summarize their results as follows:

  • Training efficiency (W2). For a 1.1B Llama-like target, depth replication achieves a reported 49.1% speedup over training from scratch at matched loss and outperforms the other evaluated growth methods.
  • Progressive training of a prescribed target model (W3). As in Du et al. [1], our continual-training study concerns progressive training of a target model with prescribed width and depth. The smaller model is an intermediate checkpoint in this process. This differs from further expanding a model that has already completed extensive pre-training and post-training. Du et al. show that concatenation attains lower loss than training the same target architecture from scratch at matched tokens. Within the same model family and data distribution, empirical scaling laws also suggest lower loss for the larger target when both models are sufficiently trained. Therefore concatenation also outperforms the original model.
  • Downstream gains (W4). Depth replication also improves downstream performance over training from scratch in their 1.1B, 3B, and 7B comparisons at matched total compute, extending the gains beyond language-modeling loss.
  • Modern architectures and sustained training (W5). The benefits persist at 3B and 7B and in a token-scaling experiment on Llama-like models. Meituan LongCat Team [2] conducts a 560B MoE comparison. These results provide evidence of the validity of concatenation in various architecture, scale, and training configuration.

Du et al. [1] show that concatenation improves training efficiency and downstream performance over training the target model from scratch. LongCat [2] further demonstrates its applicability to large-scale MoE training. Concatenation therefore provides a strong, empirically established baseline for progressive training. Building on this foundation, our study asks whether scaled stacking can further improve the training of deeper target models. We address this question by comparing the two replication strategies under matched training conditions.

Q1–Q4. Evidence provided by prior work

The findings of Du et al. [1] inform our responses to Q1–Q4 as follows:

  • Q1: Comparison with continual training of the original model. Du et al. show that concatenation outperforms training a model of the same width and depth from scratch. Together with the scaling-law argument above, this suggests that concatenation can also outperform continued training of the original model.
  • Q2: Training from scratch and total training FLOPs. At matched loss, Du et al. show that concatenation requires fewer FLOPs than the other evaluated growth strategies and achieves a reported 49.1% speedup over training from scratch.
  • Q3: Downstream gains and their computational cost. Concatenation also performs better on most downstream metrics in Du et al.'s comparisons. We also add downstream task comparisons.
  • Q4: Applicability to modern model architectures. Du et al. [1] demonstrate the effectiveness of concatenation on Llama-like models across different architectural configurations and training settings. LongCat [2] provides further evidence on MoE models.

Additional downstream evaluation of the final 0.9B checkpoints

To complement this prior evidence, we have evaluated the final checkpoints of our 0.9B models trained using scaled stacking and concatenation. Both methods start from exactly the same base-model checkpoint and use identical training data and token counts. They produce the same target architecture, so both the base-model cost and the subsequent training FLOP budgets are matched. The final downstream metrics are listed below.

Stacking (Scaled)Concatenation
ARC-C ↑23.2922.18
ARC-E ↑40.5338.22
HellaSwag ↑31.9729.59
MMLU ↑25.6127.01
Lambada-en (ACC) ↑32.4128.41
Lambada-en (PPL) ↓41.8975.91
WikiText-2 (PPL) ↓35.6744.11

GLUE detailed tasks (all scores multiplied by 100):

Stacking (Scaled)Concatenation
CoLA (MCC) ↑3.992.54
MNLI (ACC) ↑34.2932.69
MNLI-mm (ACC) ↑34.9934.09
MRPC (ACC) ↑57.6038.48
MRPC (F1) ↑69.2731.98
QNLI (ACC) ↑50.5950.69
QQP (ACC) ↑48.3659.33
QQP (F1) ↑43.4325.54
RTE (ACC) ↑54.5154.15
SST-2 (ACC) ↑53.9056.42
WNLI (ACC) ↑54.9356.34

Scaled stacking consistently outperforms concatenation on the evaluated tasks. These gains are obtained at the same training FLOP budget and target model size, and therefore require neither additional parameters nor additional inference layers relative to concatenation.

Q4. Additional Llama continual-training experiment

We have also added a continual-training experiment using the Llama architecture employed by Du et al. [1]. Given the heavy compute needed for pre-training, we will let you know as long as we finish the experiments.

[1] Wenyu Du, Tongxu Luo, Zihan Qiu, Zeyu Huang, Yikang Shen, Reynold Cheng, Yike Guo, and Jie Fu. Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training. NeurIPS, 2024.

[2] Meituan LongCat Team. LongCat-Flash Technical Report. arXiv:2509.01322, 2025.

Reviewer_MkRGOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies Transformer depth expansion through a controlled ODE formulation. It motivates scaled stacking: duplicating layers in place and dividing their attention and FFN residual outputs by the growth factor. Under stability and regularity assumptions, the authors establish (O(1/L)) consistency of terminal hidden states and loss directional derivatives, uniformly over growth factors, supporting prediction preservation and optimizer-state reuse. Experiments evaluate immediate performance on Qwen3 checkpoints and continual training of GPT-2-style models on OpenWebText, showing improved stability relative to concatenation and benefits from inheriting optimizer state.

Critical strengths
  1. Interesting empirical motivation. The observed difference between stacking and concatenation across model depths raises a relevant question for progressive training. It motivates examining when depth expansion preserves a checkpoint’s performance and whether conclusions drawn from shallow models extend to deeper ones.

  2. A clear connection between theory and practice. The controlled ODE formulation connects prediction preservation and loss-derivative consistency to a concrete expansion procedure: scaled stacking with optimizer-state inheritance. The conditional error bounds provide a principled explanation for these choices. The well-explained background and structured presentation help readers follow this connection, while the discussion relates it to looped Transformers and multigrid methods.

Critical weaknesses
  1. The empirical evidence does not sufficiently establish the proposed explanation of depth-dependent stabilization.

The introductory crossover is interesting, but the Qwen3 comparisons vary width and other architectural features alongside depth. The controlled GPT-2-style experiments compare expansion methods at selected configurations without systematically reproducing this crossover while holding other architectural choices fixed. The serial-to-parallel diagnostic provides indirect support for the residual-stability assumptions, but their applicability to trained Transformers remains insufficiently established. Immediate preservation is also evaluated only within the Qwen3 family, and the larger-growth-factor comparison uses a single pretrained checkpoint. These limitations weaken the connection between the observed phenomenon, its theoretical explanation, and its generality.

  1. The additional practical benefit of residual scaling is insufficiently isolated at the larger training scale.

Figure 6 compares scaled stacking against concatenation, changing both layer ordering and residual scaling. It therefore does not establish whether scaling improves upon direct stacking at this scale. This matters because the small-scale AdamW trajectories for direct and scaled stacking nearly overlap, as acknowledged in Appendix B.5.5, while Figure 6 also uses AdamW. Optimizer-state inheritance produces useful empirical gains, but the training trajectories do not directly measure the directional-derivative consistency established by the theory. The experiments show advantages over concatenation and gains from optimizer-state inheritance. However, they do not establish whether residual scaling improves large-scale training beyond direct stacking, and the training trajectories do not directly verify the predicted preservation of loss directional derivatives.

Questions for authors
  1. Contribution of residual scaling. Do you have a direct-stacking baseline for the large-scale experiment in Figure 6? Please clarify how optimizer-state inheritance and rescaling were handled for both reported methods. Given the nearly overlapping small-scale AdamW trajectories in Appendix B.5.5, what evidence supports an additional benefit from residual scaling at the larger scale?

  2. Evidence for the stability assumptions. Appendix B.4 provides a useful serial-to-parallel diagnostic. Beyond this indirect evidence, do you have measurements of residual magnitudes or state sensitivities supporting the assumed depth scaling? Are there controlled comparisons at fixed width that help distinguish depth effects from other differences between Qwen3 checkpoints?

  3. Scope of the optimization guarantee. Section 3.3 states that the descent direction is guaranteed and warmup is unnecessary. Could you clarify how this follows for inherited AdamW updates? Did you conduct an continual training ablation experiment with and without warmup and scaled vs. unscaled stacking?

  4. Architecture details for reproducibility. Please report the hidden width, FFN width, and attention head dimension for each continual-training configuration in Table 3. Do the reported parameter counts include embedding and readout parameters?

Suggestions for further validation: A controlled fixed-width depth sweep would strengthen the proposed explanation of the introductory crossover. Evaluation on another pretrained family and broader testing of larger growth factors would also strengthen generality. These are suggestions for additional validation if relevant results are not already available.

2 comments

Thank you for your valuable insights and constructive suggestions. Your comments help us strengthen the empirical connection to the theory, clarify the role of residual scaling, and improve reproducibility.

W1. Evidence for depth-dependent stabilization

We use the open-weight Qwen3 family to make the post-growth evaluations reproducible. To examine the role of depth under a fixed architecture, we have added controlled GPT-2 experiments in Appendix B.4. These models have depths L∈{16,32,48,64,80}L\in\{16,32,48,64,80\}, a fixed hidden dimension d=128d=128, 8 attention heads, and the same attention and FFN structure.

After scaled stacking with M=2M=2, Figure 10(a) measures the terminal-state discrepancy and Figure 10(b) measures the absolute difference between loss directional derivatives along corresponding parameter directions. Both discrepancies decrease with base-model depth, consistent with the zeroth- and first-order predictions of Theorems 3 and 4.

For the M=4M=4 continual-training experiments, AdamW and Muon use distinct base checkpoints, each pretrained with the corresponding optimizer. Within each optimizer, the compared growth methods start from the same checkpoint. We have also extended the immediate post-growth evaluations to Llama 3-8B, gpt-oss-20b, and Gemma 2-9B, as reported in Appendix B.5.4.

W2. The role of residual scaling and direct evidence of first-order consistency

Our analysis does not establish that scaled stacking must outperform direct stacking in finite-depth training. Both variants approximate the grid refinement associated with simple-continuation. The specific guarantee for scaled stacking is that its terminal-state error remains controlled uniformly over growth factors. Theorem 3 therefore establishes consistency uniformly in MM.

We have also added a direct measurement of directional-derivative consistency in Figure 10(b). We evaluate loss gradients on 10M validation tokens and compare directional derivatives along 1024 sampled directions and their corresponding refinements. The discrepancy decreases with depth, providing direct evidence for the first-order consistency studied in Theorem 4.

Q1. Direct stacking at larger training scales

Within the available compute budget, we concentrated the direct-versus-scaled stacking ablations on smaller models. Figure 17 reports these comparisons for the expansion from 64 to 128 layers. Scaled stacking performs better both under Muon and AdamW. However, direct stacking remains a competitive choice at the modest growth factor M=2M=2 tested here. The theoretical results complement the theoretical distinction: scaled stacking has an error bound uniform in MM, while its practical advantage over direct stacking depends on the finite-depth training setting.

Q2. Direct measurements supporting the stability assumptions

We have added direct measurements of residual magnitudes in Appendix B.4, Figure 9. Using the fixed-width GPT-2 models described above, we measure the norms of the composite Transformer residual T\boldsymbol T, the attention residual A\boldsymbol A, and the FFN residual F\boldsymbol F over 1000 OpenWebText validation sequences. All three residual norms decrease with depth. These measurements provide direct empirical evidence for the residual-magnitude decay underlying Assumption 2, with width and the attention and FFN structure held fixed.

Q3. Descent-direction preservation and warmup

Thank you for prompting this clarification. For AdamW, scaling the first and second moments by 1/M1/M and 1/M21/M^2 gives

=-\frac{\widehat m_k^L}{\sqrt{\widehat s_k^L}+M\epsilon} \approx-\frac{\widehat m_k^L}{\sqrt{\widehat s_k^L}+\epsilon}.$$ Here $\widehat m_k^L$ and $\widehat s_k^L$ are the bias-corrected moments, and the approximation holds when $M\epsilon\ll\sqrt{\widehat s_k^L}$ coordinatewise. The weight-decay term is unchanged for copied parameters, so the inherited AdamW direction is preserved up to this $\epsilon$-dependent perturbation. Theorem 4 then preserves descent when the base-model descent margin exceeds the combined consistency and $\epsilon$-dependent errors. Our added warmup ablation (Figure 22) shows that omitting warmup does not impair continual training with scaled stacking under the tested conditions. **Q4. Architecture details and parameter counts** Our GPT-2 models use eight attention heads. For hidden dimension $d$, the FFN width is $4d$, and the attention head dimension is $d/8$. Bias terms are disabled. Each block contains $4d^2$ attention parameters, $8d^2$ FFN parameters, and $2d$ LayerNorm parameters. Including the final LayerNorm, the parameter count is $$L(4d^2+8d^2+2d)+d =12Ld^2+(2L+1)d.$$ The reported model sizes exclude token embeddings, positional embeddings, and the readout, whose weights are shared with the token embeddings. The architecture details for the configurations in Table 3 are listed below. | Reported model size | Layers $L$ | Hidden dimension $d$ | FFN width | Attention heads | Head dimension | | --- | ---: | ---: | ---: | ---: | ---: | | 12.6M | 64 | 128 | 512 | 8 | 16 | | 50.4M, base model | 64 | 256 | 1024 | 8 | 32 | | 453.2M | 256 | 384 | 1536 | 8 | 48 | | 25.2M | 128 | 128 | 512 | 8 | 16 | | 50.4M, expanded model | 256 | 128 | 512 | 8 | 16 | | 100.8M | 128 | 256 | 1024 | 8 | 32 | | 0.9B | 512 | 384 | 1536 | 8 | 48 |

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.