- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies depth scaling through the sensitivity and curvature of residual-network losses. For residual branches scaled by beta/K and satisfying depth-uniform regularity assumptions, it proves depth-independent Lipschitz and smoothness bounds in the maximum of prescribed per-block parameter norms. The proof controls both within-block and cross-block second derivatives at finite depth. The authors interpret these bounds as a motivation for transferring learning-rate scales across depths, conditional on comparable dual gradient norms, and evaluate this prescription on residual MLPs, GPT-2-style models and CIFAR-100 ResNets with AdamW, Muon and MOGA.
Critical strengths
- The finite-network formulation is clean and does not require random initialization or independence between layers. Its second-order analysis explicitly handles simultaneous updates to all layers: O(1/K) diagonal terms and O(1/K^2) cross-layer terms remain controlled after summation (Appendix B.3). Describing branches through regularity constants makes this an informative and potentially reusable complement to first-order propagation arguments.
- The experiments cover a substantial depth range and several optimizer geometries. Figures 2 and 4 support a common low-loss learning-rate region for Muon and MOGA in the tested transformer setting. I also appreciate that Appendix D.2 reports validation results and clearly acknowledges the weaker transfer seen for ResNet validation loss.
Critical weaknesses
-
The additional theoretical insight relative to closely related frameworks needs to be established more precisely. Section 2.2 appropriately discusses the modular-norm analysis of Large et al. (2024) and existing inverse-depth prescriptions. However, the distinction between an unweighted prescribed block norm and a recursively constructed norm does not yet make clear what additional guarantee becomes available. In particular, it would be helpful to determine whether a suitable equal-mass modular construction gives a norm uniformly equivalent in depth for the present residual network, or identify a concrete assumption or architecture for which it does not. CompleteP already gives inverse-depth residual scaling and no depth correction to AdamW learning rates at fixed width. The present direct proof is useful, but the significance of the extension would be more convincing with a precise comparison of the resulting guarantees and prescriptions.
-
The experiments demonstrate transfer in the tested setting, but the additional conditions underlying the proposed curvature-based explanation are not measured. Section 3.2 correctly states that comparable dual gradient norms are additionally needed; a depth-independent upper bound on smoothness alone does not imply comparable optimal normalized step sizes. The current experiments do not examine this additional condition, the tightness of the curvature bounds, or the relation between actual optimizer updates and the idealized steepest directions. Moreover, the scaled-versus-unscaled comparison is confined to the residual MLP experiment, while the transformer and ResNet sweeps do not include a matched established depth-scaled prescription. Reporting selected gradient/update/curvature diagnostics and held-out transfer regret for representative shallow/deep pairs would help establish what the framework explains beyond the observed alignment of sampled minima. The six-rate sweeps across seven transformer depths are meaningful evidence of transfer; the remaining issue is the uncertainty and explanatory interpretation of the differences, particularly the validation-optimum shift already acknowledged in Appendix D.2
Questions for authors
- Can the unweighted block norm in this work be related explicitly to a modular norm with equal block masses? Which result, assumption or practical prescription differs once depth-independent equivalence factors are accounted for?
- For representative existing runs, do the dual gradient norm, actual update norm/alignment and directional curvature remain comparable across depths? Can these measurements explain the different stability behavior of AdamW versus Muon/MOGA in Figures 2 and 4?
- Please clarify the mapping from the implemented architectures to the hypotheses. In particular, what activation functions, normalization layers and shortcuts are actually used in the ResNet experiments, and how do they relate to the C2 identity-skip model? How are transformer embeddings, readout and normalization parameters included? It would be helpful to distinguish formally covered configurations from empirical extensions.
- What is the validation-loss regret from transferring a shallow model's selected rate to a deep model, relative to tuning at the target depth? A small matched comparison with an established depth-scaled prescription, together with uncertainty near the minima, would help assess whether the proposed explanation accounts for an empirically meaningful difference.