acceptodds
Under review as a conference paper at ICLR 2027

On the Depth Scaling of Neural Optimizers via L-Smoothness

Abstract

Scaling deep learning models efficiently requires predictable performance transfer across compute regimes. Establishing reliable hyperparameter transfer allows optimal training dynamics and structural choices to be discovered on low-cost proxy models, avoiding catastrophic failures during massive-scale training. Existing depth-transfer analyses are architecture-specific and heuristic: hyperparameter transfer is only justified through toy blocks, and both rely on infinite-depth dynamics, a specific initialization, and feature-learning criteria. Building on -smoothness framework of Xu et al. (2026), we understand hyperparameter transfer as how do the Lipschitz and -smoothness constants of the loss scale with the depth ? We show that inverse-depth residual branch scaling is sufficient to make both constants depth-independent at fixed width, uniformly over the bounded parameter domain and for arbitrary geometries. Because the bounds depend on each residual branch only through its Lipschitz and smoothness constants, the argument applies to general residual networks—including branches of block depth such as transformer and ResNet blocks—without assumptions on initialization or correlations between layer weights. Since the smoothness constant controls the curvature penalty in the descent inequality, our bounds motivate a learning-rate scale without an additional depth-dependent curvature correction for updates bounded in the prescribed norm. This provides a geometric explanation of inverse-depth residual scaling and its application to the operator-norm geometries associated with AdamW, Muon, and MOGA. Learning-rate sweeps on GPT-2 (depths 2–128) and CIFAR-100 ResNets (8–32 residual blocks) confirm that Muon and MOGA exhibit stable learning-rate transfer under these prescriptions, with a common optimal rate across a 64-fold increase in depth.

Primary area:
optimization
Keywords:
Neural Network Optimization, Learning Rate Transfer, Gradient Normalization, L-smoothness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zppHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies depth scaling through the sensitivity and curvature of residual-network losses. For residual branches scaled by beta/K and satisfying depth-uniform regularity assumptions, it proves depth-independent Lipschitz and smoothness bounds in the maximum of prescribed per-block parameter norms. The proof controls both within-block and cross-block second derivatives at finite depth. The authors interpret these bounds as a motivation for transferring learning-rate scales across depths, conditional on comparable dual gradient norms, and evaluate this prescription on residual MLPs, GPT-2-style models and CIFAR-100 ResNets with AdamW, Muon and MOGA.

Critical strengths
  1. The finite-network formulation is clean and does not require random initialization or independence between layers. Its second-order analysis explicitly handles simultaneous updates to all layers: O(1/K) diagonal terms and O(1/K^2) cross-layer terms remain controlled after summation (Appendix B.3). Describing branches through regularity constants makes this an informative and potentially reusable complement to first-order propagation arguments.
  2. The experiments cover a substantial depth range and several optimizer geometries. Figures 2 and 4 support a common low-loss learning-rate region for Muon and MOGA in the tested transformer setting. I also appreciate that Appendix D.2 reports validation results and clearly acknowledges the weaker transfer seen for ResNet validation loss.
Critical weaknesses
  1. The additional theoretical insight relative to closely related frameworks needs to be established more precisely. Section 2.2 appropriately discusses the modular-norm analysis of Large et al. (2024) and existing inverse-depth prescriptions. However, the distinction between an unweighted prescribed block norm and a recursively constructed norm does not yet make clear what additional guarantee becomes available. In particular, it would be helpful to determine whether a suitable equal-mass modular construction gives a norm uniformly equivalent in depth for the present residual network, or identify a concrete assumption or architecture for which it does not. CompleteP already gives inverse-depth residual scaling and no depth correction to AdamW learning rates at fixed width. The present direct proof is useful, but the significance of the extension would be more convincing with a precise comparison of the resulting guarantees and prescriptions.

  2. The experiments demonstrate transfer in the tested setting, but the additional conditions underlying the proposed curvature-based explanation are not measured. Section 3.2 correctly states that comparable dual gradient norms are additionally needed; a depth-independent upper bound on smoothness alone does not imply comparable optimal normalized step sizes. The current experiments do not examine this additional condition, the tightness of the curvature bounds, or the relation between actual optimizer updates and the idealized steepest directions. Moreover, the scaled-versus-unscaled comparison is confined to the residual MLP experiment, while the transformer and ResNet sweeps do not include a matched established depth-scaled prescription. Reporting selected gradient/update/curvature diagnostics and held-out transfer regret for representative shallow/deep pairs would help establish what the framework explains beyond the observed alignment of sampled minima. The six-rate sweeps across seven transformer depths are meaningful evidence of transfer; the remaining issue is the uncertainty and explanatory interpretation of the differences, particularly the validation-optimum shift already acknowledged in Appendix D.2

Questions for authors
  1. Can the unweighted block norm in this work be related explicitly to a modular norm with equal block masses? Which result, assumption or practical prescription differs once depth-independent equivalence factors are accounted for?
  2. For representative existing runs, do the dual gradient norm, actual update norm/alignment and directional curvature remain comparable across depths? Can these measurements explain the different stability behavior of AdamW versus Muon/MOGA in Figures 2 and 4?
  3. Please clarify the mapping from the implemented architectures to the hypotheses. In particular, what activation functions, normalization layers and shortcuts are actually used in the ResNet experiments, and how do they relate to the C2 identity-skip model? How are transformer embeddings, readout and normalization parameters included? It would be helpful to distinguish formally covered configurations from empirical extensions.
  4. What is the validation-loss regret from transferring a shallow model's selected rate to a deep model, relative to tuning at the target depth? A small matched comparison with an established depth-scaled prescription, together with uncertainty near the minima, would help assess whether the proposed explanation accounts for an empirically meaningful difference.
Reviewer_xN7pOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work analyzes hyperparameter transfer with respect to network depth K. It builds on the L-smoothness framework and shows that inverse-depth residual scaling α=Θ(1/K)\alpha = \Theta(1/K) is sufficient to make both the Lipschitz and the smoothness constants of the loss depth-independent at fixed width, which motivates a learning rate without depth correction. Experiments on GPT-2 and ResNets are conducted to verify the prediction.

Critical strengths
  • The analysis is general. Since the bounds only depend on each residual branch through its Lipschitz and smoothness constants, the same argument covers different optimizer geometries (AdamW, Muon, and MOGA).
Critical weaknesses
  • The main result is not new. CompleteP already proposes 1/K residual scaling and shows depth-wise learning rate transfer for transformers with 2–128 layers. Moreover, Depth-μP shows that 1/K scaling has drawbacks: nearby layers become redundant at large depth, and in their experiments, 1/K scaling with a depth-independent Adam learning rate gives worse training loss than Depth-μP and a significant learning rate shift with depth. This work does not discuss these issues. Since the theory only gives upper bounds, which also hold for smaller scalings such as 1/K^2, it cannot tell whether 1/K is a good choice or just a stable one. In addition, I cannot understand why assuming infinite depth and width is problematic here. These limits let prior work characterize feature learning and scale width and depth jointly, while the bounds here only hold at fixed width.

  • Experiments are problematic. The depth range is limited: GPT-2 only goes up to 128 layers, the same range already covered by CompleteP, and ResNets only cover 8–32 residual blocks. In contrast, Depth-μP tests residual networks with up to 2^10 blocks and observes a learning rate shift for 1/K scaling there. Also, the results seem suboptimal. All GPT-2 models are trained for only 1,200 steps (~315M tokens) regardless of depth, so the deeper models are heavily under-trained and the final training loss stays around 4. It is unclear whether the transfer still holds with realistic training lengths. In addition, the loss gap between different depths is small, suggesting that the deeper models gain little from the extra layers. Finally, the GPT-2 and ResNet experiments have no baselines (standard parameterization, Depth-μP, or CompleteP).

Questions for authors
  1. What does this work add beyond CompleteP, which already propose 1/K residual scaling? Why are the infinite-width-then-depth limits used in prior work problematic in practice? Prior work is more flexible in one important respect: it covers joint width–depth scaling, while the bounds here are only shown at fixed width, with constants that may grow with width. Is there a concrete case where Depth-μP or CompleteP fails but the proposed analysis gives the right prediction?
  2. Depth-μP reports a significant learning rate shift with depth for 1/K scaling with a depth-independent Adam learning rate, in residual networks with up to 2^10 blocks. How do the authors reconcile this with their theory, and does the transfer still hold beyond 128 layers to 1024 layers or more ?
  3. Does 1/K scaling reduce the benefit of depth compared with 1/K1/\sqrt{K}? Can the authors run the same learning rate sweeps with 1/K1/\sqrt{K} (Depth-μP) and the standard parameterization, so that the three scalings can be compared in both learning rate transfer and performance as depth increases?
  4. The upper bounds also hold for smaller scalings such as 1/K21/K^2. What in the theory singles out 1/K?
  5. Does the transfer still hold with longer training (e.g., a token budget that grows with model size)?
Reviewer_bsPyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how learning rates tuned on a shallow residual network can transfer to a deeper one. Its main contributions are:

  • Proving that inverse-depth residual branch scaling α= Θ(1/K) yields depth-independent bounds on loss sensitivity and smoothness at fixed width.
  • Developing a framework that extends the depth scaling analysis to a broader range of optimizers including AdamW, Muon, and MOGA, without requiring assumptions on initialization or correlations between layer weights.

They show this applies to general residual networks such as transformers and ResNet blocks, supported by empirical results on GPT-2 (depths 2-128) and ResNets trained on CIFAR-100 (8-32 residual blocks) across AdamW, Muon, and MOGA.

Critical strengths
  • Clear theoretical explanation for scaling residual branches by inverse depth. The paper shows that this scaling keeps loss sensitivity and curvature bounded as networks become deeper, under the stated assumptions. The analysis includes multi-layer residual networks and several optimizer geometries without requiring a particular initialization or infinite width and depth limits. This provides a useful explanation for learning rate scaling across different settings.

  • Promising results across substantially different network depths. Muon and MOGA achieve their best tested learning rates at nearly the same values across GPT-2 models with 2-128 layers, with additional results on residual MLPs and ResNets. This is meaningful as it shows learning rates chosen using inexpensive shallow models can reduce tuning costs for deeper models.

Critical weaknesses
  • Limited experiment results. The experiments do not compare the proposed scaling with existing transfer methods such as Depth µP and CompleteP. This makes it difficult to assess whether the approach improves learning rate transfer over prior methods. Also all reported experiments use limited training budgets; its reliability over longer training remains untested.

  • It remains unclear how much the theory adds to prior work. Earlier studies already use inverse-depth residual scaling and provide depth-independent smoothness bounds. This paper analyzes these ideas across a broader range of optimizer norms, but it should explain more clearly what new guarantees this provides and why they matter. Without it, it is difficult to judge the significance of the theoretical contribution.

Questions for authors
  • What specific guarantee does your analysis provide beyond existing inverse-depth scaling and depth-independent smoothness results?

  • Do the models satisfy the theory's boundedness assumptions throughout training?

  • How does your approach compare with existing methods such as Depth µP and CompleteP? Does transfer remain reliable with longer training steps?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.