acceptodds
Under review as a conference paper at ICLR 2027

A Solvable Model of Adaptive Learning Rate Rescaling: Acceleration, Stability & Scaling

Abstract

A recurring design principle in modern optimizers is to decouple update magnitude from the raw gradient norm, yet its consequences for learning-curve and resource scaling remain unclear. We isolate this mechanism by studying normalized SGD in a random-feature model with power-law teacher and data covariance. Fixed-norm updates induce an effective learning rate that grows as gradients shrink. We derive a dynamical mean-field theory (DMFT) describing the joint dependence of the loss on training time, model width and batch size. Normalization initially accelerates SGD, mapping the power-law exponent to , with exponential convergence at and formal finite-time convergence for . At finite step size, however, the same feedback ultimately breaks the acceleration and leads to marginal stability. The late-time theory yields width-limited, edge-of-stochastic-stability (EoSS), and deterministic edge-of-stability (EoS) regimes. These phases determine when larger batches or wider models reduce serial training time at comparable compute. We quantify in which of these phases increased batch size or width can compensate the excess compute use per step by fewer optimization steps to target loss. Linearized ResNet experiments on CIFAR-5M support the predicted acceleration, breakdown, and resource-scaling trends. Together, these results connect normalization-induced acceleration, EoS effects, and width–batch allocation within a solvable theory.

Primary area:
optimization
Keywords:
Edge of stability, Stochastic edge of stability, Normalized SGD, High-dimensional optimization, Scaling laws, Random-feature models, Dynamical mean-field theory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dkrSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies normalized SGD in a power-law random-feature model to understand how gradient normalization affects convergence, stability, and resource allocation. Using dynamical mean-field theory, it characterizes transient acceleration and three late-time regimes: width-limited, edge-of-stochastic-stability, and deterministic edge-of-stability. It further analyzes how batch size and model width affect effective learning rates and computational efficiency, with numerical validation on synthetic models and linearized ResNets trained on CIFAR-5M.

Critical strengths
  1. A coherent theoretical framework. The paper connects normalization acceleration, stability, and scaling within a single tractable model.

  2. Explicit, interpretable predictions. The analysis derives a clear transformation of the convergence exponent and identifies conditions governing batch and width efficiency. Distinguishing the stability threshold from the threshold for approximately linear batch scaling is particularly useful.

Critical weaknesses
  1. The degenerate model makes the adaptivity lose the stochasticity. In the high-dimensional limit, the gradient norm denominator self-averages, reducing normalization to a deterministic, scalar lr rescaling. It is true that some more practical papers use this setup, but this is a gap for such a purely theoretical paper. The model therefore captures a narrow adaptivity mechanism.

  2. Irrelevance to Reality The paper does propose an interesting theoretical model and analysis, but the question is: how is it relevant to reality?

    • The paper can only attribute the benefit of methods such as RMSProp, Adam, SOAP, and Muon to learning rate rescaling. While this benefit is especially doubtful under a non-constant learning rate.
    • The paper's real-data experiment is too idealistic: linearized ResNet, far from a practical setting. Even under this setup, the experiment cannot cover the EoSS phenomenon as well as the EoSS–EoS transition or the predicted regimes in which increasing batch size or width improves compute efficiency.
  3. I suggest that the author compare with the works below. SignGD/SignSGD is a more practical proxy for adaptive methods.
    Power-law models with scaling analyses of SignSGD, both have more detailed experimental evidence.

    • Scaling Laws of SignSGD in Linear Regression: When Does It Outperform SGD?
    • Scaling-Law Analysis of SignSGD: From Feature-Space Linear Regression to LLM Pre-training.

    Adam (SignGD) dynamical mean-field theory (DMFT):

    • Adam Optimizer Dynamics in the High-dimensional Linear Regression: A Dynamical Mean Field Theory Analysis.
  4. Less minor, but I find that some asymptotics are weird: The paper fixes B/d>0B/d>0 while taking d→∞d\to\infty, so BB must scale linearly with dd. However, for b>1/2b>1/2 at fixed N/dN/d, the EoSS boundary scales as Bc≍dB_c\asymp\sqrt d. Consequently, EoSS disappears in this strict fixed-ratio limit. The paper should clarify the scaling assumptions under which all three regimes remain observable.

Questions for authors

See above.

9 comments

We thank the reviewer for taking the time to review our work and for finding out paper presents a coherent theoretical framework with explicit interpretable prediction. Below, we respond to the weaknesses raised by the reviewer.

(W1) The degenerate model makes the denominator lose the adaptivity. In the high-dimensional limit, the gradient norm denominator self-averages, reducing normalization to a deterministic, self-consistent scalar learning-rate schedule. It is true that some more practical papers use this setup, but this is a gap for such a purely theoretical paper. The model therefore captures a narrow mechanism.

We thank the reviewer for raising this point. We would like to clarify that, in our model determining the effective learning rate requires knowing the gradient statistics generated by the previous updates; self-averaging means that independent runs converge to the same typical effective learning-rate trajectory. That typical trajectory retains its adaptive nature, as it remains coupled to the optimization trajectory. We also note that the concentration of the denominator is a result of our analysis, rather than an assumption that simplifies it away. Our analysis therefore derives a deterministic description of the stochastic adaptive feedback, rather than replacing it with an externally prescribed learning-rate schedule.

Additionally, the fact we model theoretically a setup that is used in more practical papers is a strength in our mind rather than a weakness.

Irrelevance to Reality The paper does propose an interesting theoretical model and analysis, but the question is: how is it relevant to reality?

We thank the reviewer for noting we provided an interesting model and analysis. Recent studies have demonstrated random-feature models and quadratic models are good models for practical training. We would like to draw the reviewer's attention to the work of Meterez et al. (2026c) "A Defense of the Quadratic Model," which shows experimentally on LLMs that during the second half of pre-training, a quadratic model with frozen features can describe stretches of training as long as 10% of chinchilla recipe while remaining close to the true dynamics, bridging our study and practical pre-training.

Many studies have successfully built on this connection. For example: [1] used random-feature models to identify the cause of loss spikes in LLMs and introduce an Adam β2\beta_2 scheduler; [2] used a quadratic model to show that only the top directions of the Hessian are the ones that limit the linear benefit of scaling up batchsize, and then demonstrated this on LLMs. [3] Used a quadratic model to improve warmup in LLMs. [4] Used a quadratic model to derive a a coupled batchsize-learning rate schedule that improves performance in LLMs.

Our contribution is to extend this line of work by analytically characterizing how adaptive update magnitudes interact with stability, spectral structure, batch size, and model width.

[1] Tian R and Paquette C 2026 Simulating Training Spikes of Deep Neural Networks by Layer-wise Random Feature Models Online: http://arxiv.org/abs/2610.13524

[2] Dang X, Wen K and Malladi S 2026 The Best Optimizer Depends on Batch Size Online: http://arxiv.org/abs/2610.08975

[3] Topollai K and Choromanska A 2026 Balancing Early Performance Sacrifices with Long-Term Gains: Scaling Learning-Rate Warmup Duration Across Training Horizons Online: http://arxiv.org/abs/2609.33041

[4] Meterez A, Morwani D, Wu J, Oncescu C-A, Pehlevan C and Kakade S 2026 Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling Online: http://arxiv.org/abs/2510.14717

The paper can only attribute the benefit of methods such as RMSProp, Adam, SOAP, and Muon to learning rate rescaling. While this benefit is especially doubtful under a non-constant learning rate.

We thank the reviewer for the interesting question, we believe it has three different aspects:

  1. Applicability of the theory to practical optimizers:

We performed new experiments with RMSProp, Adam and Muon where we measured the effective learning rate in the same power-law random feature setting as in the paper. We find that both Adam and RMSProp follow the predicted ηeff∞\eta_{eff}^\infty of normSGD in the width-limited phase. We also find RMSProp's ηeff∞\eta_{eff}^\infty is accurately described by that of normSGD in the EoSS phase in our problem, and that Adam's ηeff∞\eta_{eff}^\infty is well approximated by it. We find that in the EoS phase all three optimizers achieve a larger ηeff∞\eta_{eff}^\infty, and that Muon's ηeff∞\eta_{eff}^\infty differs from normSGD throughout. We ablated the Muon experiment and found that momentum fully explains the difference between Muon and normSGD. This is expected since without momentum Muon approximate normSGD in this problem.

To facilitate these results, we give a more general definition of the effective learning rate ηeff:=∥Δw∥/∣∥u∥\eta_{eff} := \|\Delta w\|/ |\| u \| with uu the step before the preconditioner (the gradient for RMSProp and the first moment mm for Adam).

The figure can be found in the anonymized link: https://drive.google.com/file/d/1vB0vf1Gt-BsLt0HRLozRUZDfteYy_yuE/view?usp=sharing.

  1. How much of the behaviour of different practical optimizers can be attributed to learning rate rescaling:

This is an interesting question. First we note that the manuscript makes no claims on this topic and we think it is outside the scope of the current submission.

Having said that, there's a rich literature on this question and it does indicate the step magnitude/effective learning rate is important. The original paper that introduced grafting (mixing direction from one optimizer with magnitude from a different one) [1] found that magnitude plays a pivotal role in both image classification and NLP (often dictating the best performing optimizer) with evidence for the importance of both the direction and the magnitude of the optimizer. [2] Consider Adam-SGD grafting (their Fig.10) in the large batch regime for LLMs and found the direction to be more important. From a different angle, [3] investigated linear interpolation of model weights and considered Adam-SGD grafting and found that the magnitude, rather than the direction is the deciding factor.

Taken together, we believe these studies make a strong case for the importance of step magnitude / adaptive rescaling, and that its role is distinct from the step direction. In our mind, this makes it a worthy subject of study.

  1. The role of a learning rate scheduler when combined with an adaptive optimizer:

A learning rate schedule does not contradict an adaptive learning rate, and does not take away from it. For the case of normSGD the effective learning rate simply becomes ηefft=ηt/∥gt∥\eta_{eff}^t = \eta^t / \|g^t\| so the two effects compound. Our DMFT can accommodate that as we already consider a time-varying time integration operator Θ\Theta. Additionally, we do not expect a learning rate scheduler to change the existence of the 3 phases or the fundamental acceleration and its breakdown, though it will of course change other aspects of training.

[1] Agarwal N, Anil R, Hazan E, Koren T and Zhang C 2020 Disentangling Adaptive Gradient Methods from Learning Rates Online: http://arxiv.org/abs/2002.11803

[2] Srećković T, Geiping J and Orvieto A 2025 Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling Online: http://arxiv.org/abs/2506.12543

[3] Lucas J, Bae J, Zhang M R, Fort S, Zemel R and Grosse R 2021 Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes Online: http://arxiv.org/abs/2104.11044

The paper's real-data experiment is too idealistic: linearized ResNet, far from a practical setting. Even under this setup, the experiment cannot cover the EoSS phenomenon as well as the EoSS–EoS transition or the predicted regimes in which increasing batch size or width improves compute efficiency.

We thank the reviewer for the comment. We would like to emphasize the experiment corroborates our theory. Our theory predicts that there are no meaningful equal compute speedups in that setting and this is indeed what we find. Per the reviewer's suggestion, we performed new experiments with the same network and data where we train the full feature learning networks and carefully tune the learning rate for every batch size and width, and find the same result: no equal compute speedup in both axes. The results are available at: https://drive.google.com/drive/folders/1IX8JViQkshhfhd-03uBAyk7QTec9kVxa?usp=sharing.

The goal of our experiment it to show test our predictions for a transient accelerated power-law, and its predicted breakdown and degradation to the SGD power-law hold with a realistic but linearized model on a realistic dataset. They also confirm our width-limited stability criterion, which is, to the best of our knowledge, novel.

Additionally, the experiment in Fig. 5 shows our DMFT can track the dynamics in this case. This fact is highly non-trivial, as given the spectrum of the infinite-width kernel and the projection of the target on it, we are able to fully predict the dynamics with zero additional fitted parameters and without ever training the network. To the best of our knowledge, we are the first to demonstrate that the DMFT tracks the dynamics in this setting. Of independent interest, our experiment indicates that linearized networks can be treated as a random projection of the corresponding infinite-width NTK, an idea that had already been put forward by Bordelon et al.

Finally, since this experiment also confirms the power-law assumption we make on the data, it complements and strengthens the experiments on synthetic data which do show this transition.

I suggest that the author compare with the works below. SignGD/SignSGD is a more practical proxy for adaptive methods...

We thank the reviewer for pointing us to those interesting works. We will cite them alongside Xiao et al., 2025; Paquette et al., 2026 when we mention theories that resolve optimizer-dependent preconditioning. Regarding the experimental evidence provided by these papers compared to our own:

  • Kim et al. extensively validate scaling exponents in synthetic models, while their Transformer experiments primarily establish qualitative consistency with the theory rather than quantitative predictions of training curves.
  • Li et al. demonstrate impressive LLM-scale experiments and successful extrapolation to unseen training conditions, although their quantitative predictions rely on fitting a five-parameter scaling law to observed training losses.
  • Adam (SignSGD) DMFT demonstrates quantitative agreement between DMFT and simulations in synthetic linear regression, while its MLP and Transformer experiments provide qualitative evidence from single runs without quantitative theoretical predictions.

In contrast, our experiments systematically validate predictions of full learning dynamics, stability transitions, and batch- and width-dependent efficiency. Crucially, we also quantitatively test these predictions on CIFAR-5M using linearized ResNets with up to 4.1 million parameters, using independently measured NTK spectra and target alignment, without fitting the observed training-loss curves. We believe this provides a distinctive and complementary level of empirical validation connecting analytical theory to neural-network training.

Reviewer_wFCgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies normalized SGD in a power-law random-feature regression model. In continuous time, gradient normalization changes the time parameterization of gradient flow, producing accelerated power-law decay, exponential decay, or formal finite-time convergence depending on the teacher exponent. At finite step size, the authors use dynamical mean-field theory to derive stationary regimes governed by width limitations, stochastic stability, and deterministic stability. The resulting equations are used to analyze batch-size and width-dependent compute efficiency. Experiments with linearized ResNets on CIFAR-5M compare the predicted convergence trends with measured training dynamics.

Critical strengths

The continuous-time analysis identifies how gradient normalization changes the temporal decay exponent while preserving the underlying gradient-flow trajectory. The stationary DMFT equations provide an explicit description of the proposed width-limited, stochastic-stability, and deterministic-stability regimes.

Critical weaknesses

The paper introduces its DMFT in the proportional limit d,N,B→∞d,N,B\to\infty, with ν=N/d\nu=N/d and α=B/d\alpha=B/d fixed. However, Appendix F.2 derives the critical EoSS–EoS batch ratio and shows that αc→0\alpha_c\to0 for every fixed power-law exponent b>0b>0. In particular, for b>1/2b>1/2, Eq. (F.19) gives αc∼ν/d\alpha_c\sim\sqrt{\nu/d}. Consequently, at any fixed positive batch ratio α\alpha, the EoSS region disappears as d→∞d\to\infty. The three-regime diagram therefore does not survive as three nondegenerate regions in the strict proportional limit stated in Section 2. A finite-dimensional phase diagram or a boundary-layer scaling in which α\alpha vanishes with dd may still be meaningful, but that is a different asymptotic statement. The paper needs to specify the relevant order of limits and establish the regime in which each phase description applies. Section 5 uses the stationary effective learning rate divided by batch size or width as a measure of compute efficiency. This quantity does not by itself determine the number of optimization steps required to reach a prescribed loss. The argument also requires comparable learning trajectories under the effective-time clock and, where width changes, control of the differences in limiting loss. The paper presents empirical curve collapse and discusses some of these qualifications, but does not establish a general relationship between the stationary effective-rate ratio and compute-to-target performance. The distinction is consequential in the width-limited regime, where changing width also changes approximation error. The resulting compute-optimality statements are therefore conditional on assumptions that are not established by the stationary DMFT equations alone.

Questions for authors
  1. Appendix F.2 shows that αc→0\alpha_c\to0 for every fixed b>0b>0, while the main DMFT is introduced at fixed positive α=B/d\alpha=B/d. In what precise asymptotic scaling does the EoSS region remain nondegenerate? Are the three stationary phases intended as finite-dimensional approximations rather than distinct phases of the strict proportional limit?
  2. What conditions ensure that the relevant stationary DMFT branches are selected by the original normalized-SGD dynamics? Does the pole analysis establish stability of these branches, or only identify candidate stability boundaries?
  3. Under what assumptions does maximizing ηeff∞/B\eta_{\mathrm{eff}}^\infty/B or ηeff∞/N\eta_{\mathrm{eff}}^\infty/N minimize the compute required to reach a fixed loss? Can the authors establish this connection while accounting for transient dynamics and width-dependent loss floors?
  4. Since the linearized ResNet experiments use the dimension-free limit of Appendix H, in which the EoSS phase is absent, what empirical evidence independently validates the EoSS–EoS transition predicted by the proportional-regime analysis?
10 comments

We thank the reviewer for the careful and thoughtful review of our work. Below we respond point-by-point to the weaknesses and questions raised by the reviewer.

(W1) The paper introduces its DMFT in the proportional limit (d,N,B\to\infty), with (\nu=N/d) and (\alpha=B/d) fixed.... The paper needs to specify the relevant order of limits and establish the regime in which each phase description applies. (Q1) Appendix F.2 shows that (\alpha_c\to0) for every fixed (b>0), while the main DMFT is introduced at fixed positive (\alpha=B/d). In what precise asymptotic scaling does the EoSS region remain nondegenerate? Are the three stationary phases intended as finite-dimensional approximations rather than distinct phases of the strict proportional limit?

We thank the reviewer for the attention to detail and the thoughtful comment. The reviewer is correct that the limit was misspecified and we followup with a correction below. We would like to note that the experimental results and their agreement with the theory remain unchanged, as they all used finite for dd matching the experiment. Their validity and agreement require only d,N,B≫1d,N,B \gg 1.

The reviewer’s observation changes the large-dd interpretation of those comparisons. To address this point, we analyze a second limit in which d→∞d\to\infty with ν:=N/d=Od(1)\nu := N/d=O_d(1), α:=B/Tr[Λ]=Od(1)\alpha := B/{\rm Tr} [\Lambda]=O_d(1), and ηˉ=ηd12min⁡(b,2a+b−1)=Od(1)\bar{\eta} = \eta d^{\tfrac{1}{2} \min(b, 2a+b−1)}=O_d(1) with the update taken without the d\sqrt{d} multiplier; in addition we require the d,N,B≫1d,N,B\gg 1 regardless of the scaling with dd. Here is the rationale for this choice:

  • The SGD noise depends on α=TrΛ/B\alpha={\rm Tr} \Lambda / B (see equation in line 1162 in the manuscript) so fixing it ensures the limit does not suppress or amplify this effect as dd grows.
  • Over- and under-parameterization is dictated by the ability of the model to represent the relation between x,yx,y. This is fully decided by the dimension in this problem so we must fix ν=N/d\nu=N/d.
  • In the width limited regime, the loss floor and effective learning rate ceiling are controlled by both irreducible and sgd noise. To keep them both in the same order of magnitude, such that neither of them dominates infinitely over the other we must fix ηˉ=ηd12min⁡(b,2a+b−1)\bar{\eta} = \eta d^{\tfrac{1}{2}\min(b, 2a+b−1)} (that is, downscale the learning rate with the dimension). This can be seen from Equations D.4, D.12 and D.13, which give the width-limited loss floor L∞\mathcal{L}^\infty as the residual target power Pν=∑kλkβk21+λkηeff∞ρP_\nu = \sum_k \frac{\lambda_k\beta_k^2}{1+\lambda_k\eta_{\mathrm{eff}}^\infty\rho} plus a noise floor ∝η2/ηeff∞\propto \eta^2/\eta_{\mathrm{eff}}^\infty. At fixed ν=N/d\nu=N/d the residual target power decays as d−min⁡(b,,2a+b−1)d^{-\min(b,,2a+b-1)}, while ηeff∞=O(1)\eta_{\mathrm{eff}}^\infty=O(1), so keeping them at the same scale requires η2\eta^2 decays at the same rate.

(cont'd below)

(W1+Q1)

Finally, the remaining question is whether all three phases are present in the limit.

  • In the new parametrization and limit we have
\alpha_c^{\mathrm{EoSS\to EoS}}\longrightarrow \begin{cases} 1, &0<b<1/2;\quad\\ 1+\sqrt{\nu}/2, &b=1/2;\quad\\ \infty, &b>1/2. \end{cases} \end{aligned}$$ So the EoS phase disappears for $b>1/2$; by that we mean the solution boundary between the solution represented by the EoS and EoSS poles goes to infinity. However, at this limit the EoSS phase itself contains a new phase transition for $0<b<1$ at $\alpha=1$ (i.e. it co-occurs with the EoS phase transition for $0<b<1/2$) $$\eta_{eff}^{\infty}=\begin{cases} 2/\lambda_{1} & \alpha>1 & \text{EoSS-b};\quad\\ 2\alpha/\lambda_{1} & \alpha<1 & \text{EoSS-a} \end{cases}~,$$ separating EoSS-a where the edge is sustained by collective noise, and EoSS-b where the edge is sustained by collective noise *and* top mode marginal stability, combining both the mechanism of EoSS-a and of EoS. For $b>1$, $B$ no longer goes to $\infty$ with $d \to \infty$ and the phase transition dissolves into a crossover and EoSS-a and EoSS-b are combined back to a single EoSS phase. - A similar phenomenon happens for the width-limited and EoS phases: the EoS boundary is pushed to infinity for $b>1/2$, however width-limited (WL) splits into two for $1/2<b<1$: WL-a where the effective learning rate is limited by approximation error and SGD noise and WL-b where the effective learning rate is limited by approximation error, SGD noise, and top mode marginality. Similarly, the boundary between WL-a and WL-b dissolves for $b>1$. - The phase transition between width-limited and EoSS phases remains at $\nu=1$ for all $b$. To summarize, for $b<1/2$ there are three phases: WL, EoSS, and EoS. For $1/2<b<1$, there are four phases: WL-a, the regular width-limited phase; WL-b, an EoS-like phase where bias and SGD noise are still important; EoSS-a, the regular EoSS; and EoSS-b, an EoS variant of EoSS. Finally, for $b>1$ only EoSS and WL exist, and the crossover between the '-a' and '-b' phases smooths. We present **new figures** to confirm this choice, which are available at the anonymized link https://drive.google.com/drive/folders/1CQN8e13SMdREqsqC3-jARDU_spegwuo1?usp=sharing. The figures show a phase diagram similar to Figure 2, for different values of $b$ when scaling $d\to\infty$ according to the old and new recipes. It can be seen that the new scaling is stable and keeps all phases. We provide a derivation in a new appendix that is available at https://drive.google.com/file/d/1Vyr49Iurdn_Y5wqtCAV6TQAjUxS79Yp6/view?usp=sharing. We thank the reviewer again for helping us improve the manuscript and believe this change will greatly contribute to its clarity.

(W2) Section 5 uses the stationary effective learning rate divided by batch size or width as a measure of compute efficiency...

We thank the reviewer for prompting us to give a clearer and sharper statement of our results. While we believe original presentation gave appropriate qualifications, we agree clarity can be improved.

We will add the following sentence to the opening paragraph of the section "... speed or speedup. We study this question in the regime of long training horizon and far from the loss floor (quantified below). The three phases...". This is a natural regime for the question as characterizing the effect of the loss floor has already been done in many works on compute optimality.

Concretely: define tst_s, such that for t>tst>t_s we have ηeff(t)≃ηeff∞\eta_{eff} (t) \simeq \eta_{eff}^\infty, and assume tt large enough ts≪tt_s \ll t such that the cumulative steps τ(t)≃τ(ts)+ηeff∞(t−ts)≈ηeff∞t\tau(t) \simeq \tau(t_s) + \eta_{eff}^\infty (t-t_s)\approx \eta_{eff}^\infty t. Further define tlt_l as the number of steps it takes to reach close to the loss floor L(τ(tl))≈L∞L(\tau(t_l)) \approx L^\infty and assume t≪tlt\ll t_l such that L(τ(t))−L∞≈L(τ(t))L(\tau(t))-L^\infty \approx L(\tau(t)). Combining these assumptions with our empirical results that τ\tau determines the excess loss gives us control of the total loss as a function of ηeff∞\eta_{eff}^\infty.

We will add this as an explicit assumption in the text and make it transparent that we build on an empirical observation of curve collapse with τ\tau.

(Q3) Under what assumptions does maximizing ηeff∞/B\eta_{\mathrm{eff}}^\infty/B or ηeff∞/N\eta_{\mathrm{eff}}^\infty/N minimize the compute required to reach a fixed loss? Can the authors establish this connection while accounting for transient dynamics and width-dependent loss floors?

We thank the reviewer for the concrete question. Building on our answer to (W2) above and under the same assumptions as before, fixing τ\tau fixes the loss so the time to a target loss is t=τ/ηeff∞t = \tau/\eta_{eff}^\infty. Therefore the compute to target loss is C=τNBηeff∞C=\tau \tfrac{NB}{\eta_{eff}^\infty}, so minimizing compute to a fixed loss is equivalent to maximizing ηeff∞NB\tfrac{\eta_{eff}^\infty}{NB}.

While we can account for the loss floor, as our framework does predict it in closed form, the regime where it is important has been studied before and leads to the compute-optimal frontier. Here we are interested in speed (smallest tt) without increasing compute by much.

(Q2) What conditions ensure that the relevant stationary DMFT branches are selected by the original normalized-SGD dynamics? Does the pole analysis establish stability of these branches, or only identify candidate stability boundaries?

We thank the reviewer for the question. Regarding the first part, the DMFT tracks the dynamics throughout, not only asymptotically; thus, a stationary state of the DMFT is a stationary state of the original dynamics.

Regarding the second part, the three conditions in Eq. 3 are three necessary conditions for stability. So given three candidates, they are uniquely ordered by the value of ηeff∞\eta_{eff}^\infty they support. The smallest ηeff∞\eta_{eff}^\infty is the one that constrains the system. If one of the conditions is violated the dynamics become unstable, triggering a feedback mechanism that leads to marginal stability. At a phase boundary, the order swaps, and a different condition becomes the most limiting condition.

(Q4) Since the linearized ResNet experiments use the dimension-free limit of Appendix H, in which the EoSS phase is absent, what empirical evidence independently validates the EoSS–EoS transition predicted by the proportional-regime analysis?

We thank the reviewer for the question. First, our own experiments in Fig.2 and Fig.3 show theory alongside experiments on EoSS, where our DMFT description tracks the dynamics throughout, and our stationary closed-form solutions predict the long-time values. Second, we note that EoSS is a well-established phenomenon. Andreyev & Beneventano (2024) measured it empirically in real-world settings, and Agarwala & Pennington (2025) studied it theoretically and empirically.

The linearized ResNet experiments provide complementary validation of the DMFT’s time-dependent predictions against real data and networks. Additionally, it supports the view of real (finite) linearized networks as random projections of the infinite-width NTK.


Additionally, addressing the paper summary and strengths, we would like to emphasize that our DMFT result gives a full dynamical description. Our DMFT results accurately track the stochastic dynamics as can be seen in Fig. 2 right panel, Fig. 3 middle panel and Fig. 5. The stationary analysis is a corollary of this contribution.

Reviewer_em2oOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the effects of gradient normalization on the convergence speed, training stability, and computational efficiency of SGD. Using dynamical mean-field theory (DMFT), the authors derive convergence dynamics under different data spectral conditions. The theoretical results show that gradient normalization can accelerate early-stage convergence. However, as the effective learning rate increases, training eventually becomes constrained by model width, EoSS, or EoS. The authors further characterize how batch size and model width affect training speed and computational efficiency across different stability regimes. The main theoretical predictions are validated through synthetic experiments and experiments with a linearized ResNet on CIFAR-5M.

Critical strengths
  1. Strong theoretical contributions. The paper develops an analytically tractable dynamical model of Normalized SGD using DMFT. It derives the convergence acceleration induced by gradient normalization and identifies the conditions under which this acceleration breaks down at finite step sizes.

  2. A unified framework for convergence acceleration and stability analysis. The paper incorporates three regimes, EoS, EoSS, and Width-limited, into a unified theoretical framework. It derives the phase transition conditions between these regimes and characterizes their relationships with the data spectrum, batch size, and model width.

  3. Practical implications for training efficiency. The theoretical analysis further characterizes the computational efficiency of different batch sizes and model widths across stability regimes, providing quantitative guidance for allocating training resources. The authors also validate their main theoretical predictions through synthetic experiments and experiments with linearized ResNets.

Critical weaknesses
  1. Strong theoretical assumptions. The analysis is primarily based on a random feature model with a power-law spectrum, assuming fixed features without Feature Learning. These assumptions make it difficult to directly generalize the theoretical results to the nonlinear training dynamics of real deep neural networks.

  2. Limited experimental validation. The experiments on real data mainly rely on linearized ResNets operating in the NTK/Lazy Training regime on CIFAR-5M. The paper lacks validation on end-to-end trained networks and with modern adaptive optimizers such as Adam and Muon.

  3. Some theoretical phenomena have been previously studied. EoS, EoSS, and Critical Batch Size have already been investigated in prior work. The main contribution of this paper lies in establishing a unified quantitative theory for Normalized SGD, rather than introducing fundamentally new stability mechanisms.

Questions for authors
  1. Can the theoretical framework be extended to real nonlinear neural networks? The analysis relies on fixed random features and the NTK/Lazy Training assumption. However, in real neural networks, the learned features, Hessian spectrum, and alignment with the target evolve throughout training. Which theoretical conclusions do the authors expect to remain valid in the Feature Learning regime?

  2. To what extent can the proposed DMFT framework be generalized to more complex adaptive optimizers, such as RMSProp and Adam? Could the authors further extend the current theoretical framework to analyze the convergence acceleration and EoS/EoSS stability behavior of these optimizers, and experimentally validate the resulting theoretical predictions?

  3. What are the novel contributions compared to existing EoS/EoSS theories? Previous studies have already investigated deterministic and stochastic stability boundaries. Does the phase diagram derived in this paper reveal new phenomena that cannot be explained by existing theories, or does it primarily extend known stability conditions to Normalized SGD?

  4. Does early-stage convergence acceleration translate into improved overall training efficiency? The authors show that Normalized SGD achieves a faster convergence exponent in the early stages, but this acceleration eventually disappears due to stability constraints. Compared with standard SGD with carefully tuned hyperparameters or learning rate schedules, can Normalized SGD still reach the same target Loss with lower overall computational cost?

6 comments

We thank the reviewer for reviewing our paper and for recognizing the strong theoretical contribution and its ability to unify multiple observed phenomena in a single solvable model. We respond to the weaknesses and questions below.

Critical Weaknesses:

(W1) Strong theoretical assumptions. The analysis is primarily based on a random feature model with a power-law spectrum, assuming fixed features without Feature Learning...

(Q1a) Can the theoretical framework be extended to real nonlinear neural networks?...

We agree that extending the results to feature-learning networks is an important open problem, especially since their Hessian and NTK are not frozen in time, and their change creates the same approach to edge of stability even with standard SGD and no adaptivity. Studying adaptive optimizers with feature learning is an outstanding theoretical challenge, which we plan to address in future work. There are a number of approaches that could facilitate that: first, Bordelon et al. (2024b) use closely related spectral and dynamical mean-field tools to study scaling laws with feature learning, without accounting for adaptive learning rates or stability; a natural next step is to combine our approaches. Second, Lauditi et al. (2025) derive adaptive NTK and NNGP predictors in feature-learning regimes, showing that kernel descriptions remain useful when the kernels themselves are learned from data, so even at the end of training a random feature model might be appropriate when the distribution of features is correctly chosen.

Second, the paper’s stated contribution is a solvable theory of how gradient normalization jointly affects acceleration, stability, and resource scaling. Fixed features provide a controlled setting in which these effects can be isolated and quantitatively characterized. A solvable theory need not capture the full nonlinear dynamics of deep networks to provide useful insight into their optimization. Power-law random-feature models are an established framework for studying neural scaling laws and compute allocation (Maloney et al., 2022; Bordelon et al., 2024a; Paquette et al., 2024). Our work advances this framework by characterizing the interaction between adaptive learning rescaling, acceleration, and stability. Finally, the relevance of quadratic optimization models also extends beyond the fixed-feature regime: Meterez et al. (2026c) show that local quadratic approximations can predict optimization dynamics over substantial windows of LLM pretraining.

We think the current literature supports both the value of the present modeling approach and a concrete research program for extending it to learned representations.

(Q1b) Which theoretical conclusions do the authors expect to remain valid in the Feature Learning regime?

We thank the reviewer for reiterating this important point. Meterez et al. (2026c) “A Defense of the Quadratic Model” showed experimentally on LLMs that during the second half of pre-training a quadratic model with frozen features can describe stretches of training as long as 10% of chinchilla recipe while remaining close to the true dynamics. In this regime, we expect the insights about the limits of stability, the phases we describe in this work, and their dependence on width and batch size to carry over. Namely, we expect the model to be in one of these three phase and we expect that the phase to dictate whether speedup with equal compute is possible. The experiments of Meterez et al. already support this: Fig. 3 supports our power-law covariance assumption; Fig. 5 supports our power-law target assumption; and Figures 6,7 show both a regime where increased batch size supports a larger learning rate and a regime where it does not.

Additionally, we note that insight from quadratic models has been repeatedly proven useful in improving LLM pre-training; for example: [1] used random-feature models to identify the cause of loss spikes in LLMs and introduce an Adam β2\beta_2 scheduler, [2] used a quadratic model to show that only the top directions of the Hessian are the ones that limit the linear benefit of scaling up batchsize, and then demonstrated this on LLMs. [3] Used a quadratic model to improve warmup in LLMs. [4] Used a quadratic model to derive a coupled batchsize-learning rate schedule that improves performance in LLMs.

[1] Tian R and Paquette C 2026 Simulating Training Spikes of Deep Neural Networks by Layer-wise Random Feature Models Online: http://arxiv.org/abs/2610.13524

[2] Dang X, Wen K and Malladi S 2026 The Best Optimizer Depends on Batch Size Online: http://arxiv.org/abs/2610.08975

[3] Topollai K and Choromanska A 2026 Balancing Early Performance Sacrifices with Long-Term Gains: Scaling Learning-Rate Warmup Duration Across Training Horizons Online: http://arxiv.org/abs/2609.33041

[4] Meterez A, Morwani D, Wu J, Oncescu C-A, Pehlevan C and Kakade S 2026 Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling Online: http://arxiv.org/abs/2510.14717

(W2) Limited experimental validation. The experiments on real data mainly rely on linearized ResNets operating in the NTK/Lazy Training regime on CIFAR-5M. The paper lacks validation on end-to-end trained networks and with modern adaptive optimizers such as Adam and Muon.

We thank the reviewer for this comment. The role of the experiments in our work is to test whether the dynamical DMFT equations track the dynamics on real data and linearized realistic networks. The experiment also confirms the accelerated exponent, its breakdown at marginal stability, and the crossover to SGD-like scaling.

Linearized ResNets on CIFAR-5M meet this purpose. They involve a real architecture and real data, yet stay within the class of models the theory describes. The test is also demanding. The DMFT takes only the measured NTK spectrum, the target coefficients, N and B as input, with no fitted parameters from training. From these, it predicts the full trajectories of the loss and the effective learning rate. This includes the accelerated exponent, its breakdown at marginal stability, and the crossover to SGD-like scaling. The theory tracks these across 241 runs spanning widths from roughly 10³ to 4×10⁶ parameters and batch sizes from 2 to 10⁴ (Fig. 5). We thus view it as strong rather than limited experimental validation.

Having said that, per the reviewer's suggestion, we performed new experiments with the same network and data where we train the full feature learning networks and carefully tune the learning rate for every batch size and width, and find the same result: no equal compute speedup in both axes. The results are available in: https://drive.google.com/drive/folders/1IX8JViQkshhfhd-03uBAyk7QTec9kVxa?usp=sharing.

Regarding Adam and Muon, our paper studies normalized SGD by design. Normalized SGD keeps the stochastic-gradient direction and changes only the update magnitude. This isolates one ingredient shared by modern adaptive optimizers. Adam and Muon additionally change the update direction through preconditioning, a separate effect that the grafting perspective makes explicit (Agarwal et al., 2020). Nevertheless, we can test their learning rate rescaling experimentally. Following the reviewer's suggestion, we performed new experiments with RMSProp, Adam and Muon where we measured the effective learning rate in the same power-law random feature setting as in the paper. We find that both Adam and RMSProp follow the predicted ηeff∞\eta_{eff}^\infty of normSGD in the width-limited phase. We also find RMSProp's ηeff∞\eta_{eff}^\infty is accurately described by that of normSGD in the EoSS phase in our problem, and that Adam's ηeff∞\eta_{eff}^\infty is well approximated by it. We find that in the EoS phase all three optimizers achieve a larger ηeff∞\eta_{eff}^\infty, and that Muon's ηeff∞\eta_{eff}^\infty differs from normSGD throughout. We ablated the Muon experiment and found that momentum fully explains the difference between Muon and normSGD. This is expected since, without momentum, Muon approximates normSGD in this problem.

To facilitate these results, we give a more general definition of the effective learning rate ηeff:=∥Δw∥/∣∥u∥\eta_{eff} := \|\Delta w\|/ |\| u \| with uu the step before the preconditioner (the gradient for RMSProp and the first moment mm for Adam).

The figure can be found at the anonymized link: https://drive.google.com/file/d/1vB0vf1Gt-BsLt0HRLozRUZDfteYy_yuE/view?usp=sharing.

We believe such scoping is appropriate in theoretical work on optimizers, where each analysis targets a single update rule. Extending the analysis to direction-modifying optimizers is a natural next step, which we are currently working on.

Comments on the paper

1 comment

We thank the reviewers for their thoughtful feedback and continued engagement in the discussion.

The reviewers recognized the strong, unified theoretical framework connecting acceleration, stability, and scaling (em2o, dkrS), the explicit characterization of stability regimes (em2o, wFCg), and the interpretable predictions for training efficiency and resource scaling (em2o, dkrS).

We have uploaded a new version of the manuscript. In this version, we address a common concern across the reviewers about the d→∞d \to \infty limit. We analyzed an additional limit where α:=B/trλ\alpha:=B/{\rm tr} \lambda is Od(1)O_d(1) instead of α:=B/d\alpha:=B/d. This limit produces a richer phase diagram at the limit d→∞d \to \infty. Experimental results and their match to the theory are unchanged by this choice. We have also sharpened and clarified some of the statements regarding speedup with equal compute at the width-limited phase.

We look forward to further exchanges during the remaining discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.