- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies normalized SGD in a power-law random-feature model to understand how gradient normalization affects convergence, stability, and resource allocation. Using dynamical mean-field theory, it characterizes transient acceleration and three late-time regimes: width-limited, edge-of-stochastic-stability, and deterministic edge-of-stability. It further analyzes how batch size and model width affect effective learning rates and computational efficiency, with numerical validation on synthetic models and linearized ResNets trained on CIFAR-5M.
Critical strengths
-
A coherent theoretical framework. The paper connects normalization acceleration, stability, and scaling within a single tractable model.
-
Explicit, interpretable predictions. The analysis derives a clear transformation of the convergence exponent and identifies conditions governing batch and width efficiency. Distinguishing the stability threshold from the threshold for approximately linear batch scaling is particularly useful.
Critical weaknesses
-
The degenerate model makes the adaptivity lose the stochasticity. In the high-dimensional limit, the gradient norm denominator self-averages, reducing normalization to a deterministic, scalar lr rescaling. It is true that some more practical papers use this setup, but this is a gap for such a purely theoretical paper. The model therefore captures a narrow adaptivity mechanism.
-
Irrelevance to Reality The paper does propose an interesting theoretical model and analysis, but the question is: how is it relevant to reality?
- The paper can only attribute the benefit of methods such as RMSProp, Adam, SOAP, and Muon to learning rate rescaling. While this benefit is especially doubtful under a non-constant learning rate.
- The paper's real-data experiment is too idealistic: linearized ResNet, far from a practical setting. Even under this setup, the experiment cannot cover the EoSS phenomenon as well as the EoSS–EoS transition or the predicted regimes in which increasing batch size or width improves compute efficiency.
-
I suggest that the author compare with the works below. SignGD/SignSGD is a more practical proxy for adaptive methods.
Power-law models with scaling analyses of SignSGD, both have more detailed experimental evidence.- Scaling Laws of SignSGD in Linear Regression: When Does It Outperform SGD?
- Scaling-Law Analysis of SignSGD: From Feature-Space Linear Regression to LLM Pre-training.
Adam (SignGD) dynamical mean-field theory (DMFT):
- Adam Optimizer Dynamics in the High-dimensional Linear Regression: A Dynamical Mean Field Theory Analysis.
-
Less minor, but I find that some asymptotics are weird: The paper fixes while taking , so must scale linearly with . However, for at fixed , the EoSS boundary scales as . Consequently, EoSS disappears in this strict fixed-ratio limit. The paper should clarify the scaling assumptions under which all three regimes remain observable.
Questions for authors
See above.
9 comments
The paper can only attribute the benefit of methods such as RMSProp, Adam, SOAP, and Muon to learning rate rescaling. While this benefit is especially doubtful under a non-constant learning rate.
We thank the reviewer for the interesting question, we believe it has three different aspects:
- Applicability of the theory to practical optimizers:
We performed new experiments with RMSProp, Adam and Muon where we measured the effective learning rate in the same power-law random feature setting as in the paper. We find that both Adam and RMSProp follow the predicted of normSGD in the width-limited phase. We also find RMSProp's is accurately described by that of normSGD in the EoSS phase in our problem, and that Adam's is well approximated by it. We find that in the EoS phase all three optimizers achieve a larger , and that Muon's differs from normSGD throughout. We ablated the Muon experiment and found that momentum fully explains the difference between Muon and normSGD. This is expected since without momentum Muon approximate normSGD in this problem.
To facilitate these results, we give a more general definition of the effective learning rate with the step before the preconditioner (the gradient for RMSProp and the first moment for Adam).
The figure can be found in the anonymized link: https://drive.google.com/file/d/1vB0vf1Gt-BsLt0HRLozRUZDfteYy_yuE/view?usp=sharing.
- How much of the behaviour of different practical optimizers can be attributed to learning rate rescaling:
This is an interesting question. First we note that the manuscript makes no claims on this topic and we think it is outside the scope of the current submission.
Having said that, there's a rich literature on this question and it does indicate the step magnitude/effective learning rate is important. The original paper that introduced grafting (mixing direction from one optimizer with magnitude from a different one) [1] found that magnitude plays a pivotal role in both image classification and NLP (often dictating the best performing optimizer) with evidence for the importance of both the direction and the magnitude of the optimizer. [2] Consider Adam-SGD grafting (their Fig.10) in the large batch regime for LLMs and found the direction to be more important. From a different angle, [3] investigated linear interpolation of model weights and considered Adam-SGD grafting and found that the magnitude, rather than the direction is the deciding factor.
Taken together, we believe these studies make a strong case for the importance of step magnitude / adaptive rescaling, and that its role is distinct from the step direction. In our mind, this makes it a worthy subject of study.
- The role of a learning rate scheduler when combined with an adaptive optimizer:
A learning rate schedule does not contradict an adaptive learning rate, and does not take away from it. For the case of normSGD the effective learning rate simply becomes so the two effects compound. Our DMFT can accommodate that as we already consider a time-varying time integration operator . Additionally, we do not expect a learning rate scheduler to change the existence of the 3 phases or the fundamental acceleration and its breakdown, though it will of course change other aspects of training.
[1] Agarwal N, Anil R, Hazan E, Koren T and Zhang C 2020 Disentangling Adaptive Gradient Methods from Learning Rates Online: http://arxiv.org/abs/2002.11803
[2] Srećković T, Geiping J and Orvieto A 2025 Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling Online: http://arxiv.org/abs/2506.12543
[3] Lucas J, Bae J, Zhang M R, Fort S, Zemel R and Grosse R 2021 Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes Online: http://arxiv.org/abs/2104.11044
The paper's real-data experiment is too idealistic: linearized ResNet, far from a practical setting. Even under this setup, the experiment cannot cover the EoSS phenomenon as well as the EoSS–EoS transition or the predicted regimes in which increasing batch size or width improves compute efficiency.
We thank the reviewer for the comment. We would like to emphasize the experiment corroborates our theory. Our theory predicts that there are no meaningful equal compute speedups in that setting and this is indeed what we find. Per the reviewer's suggestion, we performed new experiments with the same network and data where we train the full feature learning networks and carefully tune the learning rate for every batch size and width, and find the same result: no equal compute speedup in both axes. The results are available at: https://drive.google.com/drive/folders/1IX8JViQkshhfhd-03uBAyk7QTec9kVxa?usp=sharing.
The goal of our experiment it to show test our predictions for a transient accelerated power-law, and its predicted breakdown and degradation to the SGD power-law hold with a realistic but linearized model on a realistic dataset. They also confirm our width-limited stability criterion, which is, to the best of our knowledge, novel.
Additionally, the experiment in Fig. 5 shows our DMFT can track the dynamics in this case. This fact is highly non-trivial, as given the spectrum of the infinite-width kernel and the projection of the target on it, we are able to fully predict the dynamics with zero additional fitted parameters and without ever training the network. To the best of our knowledge, we are the first to demonstrate that the DMFT tracks the dynamics in this setting. Of independent interest, our experiment indicates that linearized networks can be treated as a random projection of the corresponding infinite-width NTK, an idea that had already been put forward by Bordelon et al.
Finally, since this experiment also confirms the power-law assumption we make on the data, it complements and strengthens the experiments on synthetic data which do show this transition.
I suggest that the author compare with the works below. SignGD/SignSGD is a more practical proxy for adaptive methods...
We thank the reviewer for pointing us to those interesting works. We will cite them alongside Xiao et al., 2025; Paquette et al., 2026 when we mention theories that resolve optimizer-dependent preconditioning. Regarding the experimental evidence provided by these papers compared to our own:
- Kim et al. extensively validate scaling exponents in synthetic models, while their Transformer experiments primarily establish qualitative consistency with the theory rather than quantitative predictions of training curves.
- Li et al. demonstrate impressive LLM-scale experiments and successful extrapolation to unseen training conditions, although their quantitative predictions rely on fitting a five-parameter scaling law to observed training losses.
- Adam (SignSGD) DMFT demonstrates quantitative agreement between DMFT and simulations in synthetic linear regression, while its MLP and Transformer experiments provide qualitative evidence from single runs without quantitative theoretical predictions.
In contrast, our experiments systematically validate predictions of full learning dynamics, stability transitions, and batch- and width-dependent efficiency. Crucially, we also quantitatively test these predictions on CIFAR-5M using linearized ResNets with up to 4.1 million parameters, using independently measured NTK spectra and target alignment, without fitting the observed training-loss curves. We believe this provides a distinctive and complementary level of empirical validation connecting analytical theory to neural-network training.
We thank the reviewer for taking the time to review our work and for finding out paper presents a coherent theoretical framework with explicit interpretable prediction. Below, we respond to the weaknesses raised by the reviewer.
We thank the reviewer for raising this point. We would like to clarify that, in our model determining the effective learning rate requires knowing the gradient statistics generated by the previous updates; self-averaging means that independent runs converge to the same typical effective learning-rate trajectory. That typical trajectory retains its adaptive nature, as it remains coupled to the optimization trajectory. We also note that the concentration of the denominator is a result of our analysis, rather than an assumption that simplifies it away. Our analysis therefore derives a deterministic description of the stochastic adaptive feedback, rather than replacing it with an externally prescribed learning-rate schedule.
Additionally, the fact we model theoretically a setup that is used in more practical papers is a strength in our mind rather than a weakness.
We thank the reviewer for noting we provided an interesting model and analysis. Recent studies have demonstrated random-feature models and quadratic models are good models for practical training. We would like to draw the reviewer's attention to the work of Meterez et al. (2026c) "A Defense of the Quadratic Model," which shows experimentally on LLMs that during the second half of pre-training, a quadratic model with frozen features can describe stretches of training as long as 10% of chinchilla recipe while remaining close to the true dynamics, bridging our study and practical pre-training.
Many studies have successfully built on this connection. For example: [1] used random-feature models to identify the cause of loss spikes in LLMs and introduce an Adam β2 scheduler; [2] used a quadratic model to show that only the top directions of the Hessian are the ones that limit the linear benefit of scaling up batchsize, and then demonstrated this on LLMs. [3] Used a quadratic model to improve warmup in LLMs. [4] Used a quadratic model to derive a a coupled batchsize-learning rate schedule that improves performance in LLMs.
Our contribution is to extend this line of work by analytically characterizing how adaptive update magnitudes interact with stability, spectral structure, batch size, and model width.
[1] Tian R and Paquette C 2026 Simulating Training Spikes of Deep Neural Networks by Layer-wise Random Feature Models Online: http://arxiv.org/abs/2610.13524
[2] Dang X, Wen K and Malladi S 2026 The Best Optimizer Depends on Batch Size Online: http://arxiv.org/abs/2610.08975
[3] Topollai K and Choromanska A 2026 Balancing Early Performance Sacrifices with Long-Term Gains: Scaling Learning-Rate Warmup Duration Across Training Horizons Online: http://arxiv.org/abs/2609.33041
[4] Meterez A, Morwani D, Wu J, Oncescu C-A, Pehlevan C and Kakade S 2026 Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling Online: http://arxiv.org/abs/2510.14717