acceptodds
Under review as a conference paper at ICLR 2027

Optimizer-dependent training dynamics converge to the same one-third optimal data scaling

Abstract

Neural scaling, in which loss falls as a power law with training, is central to large language models, and one recent proposal is that a exponent emerges from learning peaked distributions. That account describes SGD, but models in practice are trained with adaptive optimizers. Here we separate two exponents the account does not distinguish: how fast the loss falls with training steps along a single run, and how fast the optimally tuned loss falls with dataset size . We show that the first, a dynamic exponent, is optimizer-specific while the second, an optimal data exponent, converges to across optimizers. In an online teacher–student model we decompose the loss into norm growth (radial) and alignment toward the teacher direction (tangential), each decaying as a power law with dynamic exponents and . Under SGD, both are close to , so the data exponent is also across different learning rates. Under Adam the two separate: but . Since the total loss is minimized when these two parts are balanced, the optimal learning rate is optimizer-dependent: -independent for SGD but falls with for Adam. Yet tuned to that optimum, the loss returns to for both. A stochastic-dynamics analysis explains why: the optimizers can trade decay speed between the two channels, but they all fall on a single dynamic exponent relation, , which fixes the optimal data exponent at . Across seven optimizers, including Muon, the measured exponents are consistent with this relation, and the optimal-loss envelopes agree with across them. The optimizer sets how fast a model learns per step; tuned optimally, it changes the prefactor but not the rate at which loss falls per sample.

Primary area:
optimization
Keywords:
optimizer, neural scaling, hyperparameter scaling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UemVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how optimizer choice affects training dynamics and data scaling in an online, hard-label teacher–student softmax classification task. It distinguishes loss decay along fixed-hyperparameter training trajectories from the optimal loss envelope obtained by tuning the learning rate for each sample budget. Under an effective preconditioner scaling assumption, the authors derive a relation between radial and tangential decay exponents and predict an optimizer-independent optimal data scaling of D−1/3D^{-1/3}. Experiments with SGD, PowerAdam, Adam, Muon, and SignGD show different training dynamics but similar optimal data-scaling exponents.

Critical strengths
  • The distinction between trajectory scaling and optimal-envelope scaling is meaningful. A fixed learning rate may lead to slow late-stage loss decay, even when the optimizer achieves good scaling after tuning for each training horizon. The radial–tangential decomposition provides a useful explanation for this difference. In particular, Adam's faster radial decay and slower tangential decay explain both the shape of its loss curves and why its optimal learning rate decreases with the training budget. Connecting these observations helps explain how optimal hyperparameters scale with training.
  • The framework connects several scaling relations and provides useful empirical evidence. It links the channel exponents, optimal learning-rate scaling, and the loss balance at the optimum. The PowerAdam sweep is particularly useful, as it shows a smooth transition between different dynamics on the same task. Muon and SignGD provide additional evidence beyond diagonal second-moment preconditioning. Proposition 1 also gives an elegant trace-based argument for the tangential loss without requiring the gradient-noise covariance to be proportional to, or commute with, the Hessian.
Critical weaknesses
  • The assumptions behind Theorem 2 need further justification. The theorem optimizes the two-channel loss while treating its coefficients as independent of z=ηBp−1z=\eta B^{p-1}. However, tangential fluctuations also affect radial growth, and this feedback depends on zz. It is unclear whether the assumed loss expression remains valid when optimizing over zz.
  • The radial correction in Appendix B.3 appears inconsistent. The paper differentiates the stationary relation Lt∝zβq−1L_t\propto z\beta^{q-1}, rather than taking the radial derivative at a fixed parameter direction. From H≃H0/βH\simeq H_0/\beta and e≃βθe\simeq\beta\theta, we instead obtain Lt≃(β/2)θ⊤H0θL_t\simeq(\beta/2)\theta^\top H_0\theta, giving a radial gradient of approximately (Lt−Lr)/β(L_t-L_r)/\beta. This correction is not negligible at the claimed optimum Lt/Lr=1/2L_t/L_r=1/2 and may affect the predicted optimal loss balance. A similar residual-dependent drift appears in arXiv:2605.22341.
  • The effective scaling exponent qq is not consistently supported by the measurements. The analysis assumes the same qq governs radial drift and tangential noise, which is not obvious for adaptive or normalized optimizers. For Adam, Remark 2 gives q≃0.74q\simeq0.74 from norm growth, 0.920.92 from radial loss, and 1.051.05 from optimal learning-rate scaling. The finite-norm correction explains the first discrepancy but not the last. In particular, the measured norm exponent 0.440.44 predicts sD≃−0.24s_D\simeq-0.24, compared with the observed −0.35-0.35.
  • The drift-dominated regime is not directly verified. Appendix G mainly infers drift dominance from agreement with the predicted exponent relation, rather than separately measuring deterministic radial drift and stochastic norm growth. Thus, the experiments support the common scaling pattern more strongly than the proposed mechanism. Direct measurements of these update statistics would provide stronger evidence.
Questions for authors

See Weaknesses.

1 comment
AuthorsOpenReview ↗

First reply to reviewer UemV

We thank the reviewer for the careful reading. We agree with the reviewer’s suggestions, and addressing them significantly improved the paper. Here we post the summary of changes: for details, please take a look at our revised manuscript with changes highlighted in orange.

Including feedback from tangential fluctuations. Appendix B.3 differentiated the stationary relation rather than taking the gradient at fixed direction; the correct radial gradient is (Lt−Lr)/β(L_t - L_r)/\beta, as the reviewer notes. We now resolve the dynamics by separating the norm β\beta from the overlap ω=⟨WS,W^T⟩F\omega = \langle W_S, \widehat W_T\rangle_F: ω\omega is linear in WSW_S and receives only drift, driven by the full loss LL, while β\beta receives the drift (Lr−Lt)/β(L_r - L_t)/\beta plus a noise term 2Lt/β2L_t/\beta, again summing to L/βL/\beta. Tangential fluctuations thus feed back on radial growth through R=Lt/Lr∝zβqR = L_t/L_r \propto z\beta^q. In the notation of Kühn et al. (2026), ω\omega, β2\beta^2 and ∣e∣2=β2−ω2|e|^2 = \beta^2 - \omega^2 correspond to DD, QeffQ_{\rm eff} and Δ\Delta, with R=6Δ/π2R = 6\Delta/\pi^2; for SGD at batch size one, their equations and ours coincide. Because the feedback depends on the state only through RR, the scaling symmetry, and hence αD=1/3\alpha_D = 1/3 and sD=−q/3s_D = -q/3, survive up to finite-norm corrections of order 1/β1/\beta. With no fitted parameters, the theory with feedback predicts SGD's optimal η/B\eta/B (0.06370.0637 against 0.06250.0625 measured), the optimal channel ratio (Lt/L=0.30L_t/L = 0.30 against 0.2960.296; the value without the feedback is 1/31/3), and the point where ∂L/∂ln⁡β\partial L/\partial \ln\beta changes sign (zc=0.100z_c = 0.100 against 0.1010.101–0.1040.104; the value without the feedback is 0.2660.266). We have replaced the original derivation in Appendix B.3 and collected the consequences of the feedback in Appendix E.7.

Inconsistent qq. Part of the inconsistency was a grid-conversion error in sDs_D (general comment). The rest is the feedback above: for q>0q > 0, RR grows along a run and shifts the trajectory exponents. With the feedback included, αβ\alpha_\beta, αr\alpha_r, αt\alpha_t and sDs_D give q=0.65q = 0.65–0.680.68, against 0.680.68–0.920.92 without it. The sum rule 2αr+αt=12\alpha_r + \alpha_t = 1 accordingly holds exactly only without the feedback; with it, the corrections are within the measured few-percent excess. (App. E.7, revised Table 1)

Drift-dominated regime. We measured both terms directly. For SGD, the drift and noise contributions to Δβ2\Delta\beta^2 match −2ηG-2\eta G and 4ηLt4\eta L_t within 1%, and the noise term is 1.25 times the drift, so "drift-dominated" was incorrect in β\beta; in ω\omega, growth is pure drift and matches 2ηL2\eta L within 3%, the measured finite-norm correction. For adaptive optimizers the noise term is a few percent of the drift. We have revised the text accordingly. (App. G.2)

Reviewer_8t5LOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper analyzes a toy model where the a universal 1/3 scaling exponent with respect to data is observed across optimizers. Experiments confirm the validity of the theory.

Critical strengths
  1. The paper presents clean and well-motivated toy model that gave rise to nontrivial dynamics that reproduces the 1/3 exponent sometimes observed in practice.
  2. The theory is well supported by the experiments.
Critical weaknesses
  1. The main limitation of this model is that preconditioned optimizers simply don't do anything that interesting. In particular, they act as an overall, isotropic rescaling of time in each direction. This explains the whole universality of the exponent, yet there is ample evidence that in real LLM training the adaptive optimizers accelerate training precisely by adapting to anisotropic curvature or variance in each direction [1,2].
  2. Learning rate schedule is known to have a major effect on scaling laws in practice [3,4], but is omitted from this paper. It is very likely that the conclusion from this model will change if the learning rate is annealed since the noise contribution will become negligible by the end of training [4].

[1] Xie et al., Adam Exploits ℓ∞-geometry of Loss Landscape via Coordinate-wise Adaptivity

[2] Kunstner et al., Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models

[3] Hoffmann et al., Training Compute-Optimal Large Language Models

[4] Qiu et al., Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks

Questions for authors
  1. How do your results change if the input is not isotropic?
  2. How do your results change if learning rate is annealed?
1 comment
AuthorsOpenReview ↗

First reply to reviewer 8t5L

We thank the reviewer for the positive assessment and the two questions. We agree with the reviewer’s suggested extensions, and addressing them has significantly improved the paper. Here we post the summary of changes: for details, please take a look at our revised manuscript with changes highlighted in orange.

Anisotropy. We repeated the constant-learning-rate sweeps with power-law input spectra λk∝k−s\lambda_k \propto k^{-s}, s=1,2s = 1, 2 (effective dimension about 10 and 2.4 out of 32), both along the coordinate axes and in a random rotated basis (the latter for Adam, whose preconditioning is per coordinate). Mild anisotropy leaves αD\alpha_D unchanged for both optimizers. Strong anisotropy, which also makes the class frequencies very unequal, delays the approach to 1/31/3 without changing the exponent: for SGD the exponent is 0.410.41 fitted over all budgets but 0.340.34 over the final decade, and for Adam with a rotated spectrum it is 0.350.35 up to D=107D = 10^7 but 0.323±0.0140.323 \pm 0.014 between 10710^7 and 8×1078 \times 10^7, matching the isotropic 0.326±0.0090.326 \pm 0.009 over the same range. (App. J)

Schedules. The reviewer's intuition is correct, with a qualification: annealing lowers the loss by removing noise at the end of training, but with the peak rate bounded, as stability bounds it in deep networks, the exponent stays at 1/31/3 (within 0.03 for SGD and Adam, while the loss drops by 30–60% for cosine and WSD). A steeper envelope requires two conditions together: annealing all the way to zero rather than to a finite floor, and a peak rate that grows without bound with the budget, to orders of magnitude above the constant-rate optimum. We have replaced our conjecture with this measured scope. (new Fig. 5, App. I)

Reviewer_igpTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how optimizer choice affects learning dynamics and data scaling in an online teacher–student softmax classification model. The authors decompose the loss into radial and tangential components, whose temporal exponents vary across optimizers. They propose the relation �0��0� and argue that, after tuning the constant learning rate separately for each sample budget, the optimal loss scales as �1��1�. Experiments with seven optimizer variants provide supporting evidence.

Critical strengths

The investigated problem is interesting.

Critical weaknesses

However, I have substantial concerns about the scope and support of the central claims.

  1. Learning-rate schedules are missing from the main analysis. All experiments use constant learning rates. Selecting a different constant learning rate for each budget is distinct from using a schedule within a training run. Scheduling directly affects both optimization progress and stochastic fluctuations, which are central to the proposed mechanism. The discussion conjectures that self-similar schedules such as cosine decay and warmup–stable–decay preserve the scaling relation, but provides neither a derivation nor experimental validation. This omission limits the connection to practical language-model training. A particularly informative experiment would compare optimized constant-rate, cosine, and WSD runs, tuning the peak learning rate separately at each budget.

  2. The connection between actual optimizer updates and the effective theory remains incomplete. The analysis assumes an effective preconditioner scaling �0��0�, together with local stationarity and timescale separation. These conditions are not established from the update rules of Adam, Muon, or SignGD. Consequently, the theory provides conditional relations among measured quantities, while the optimizer-specific exponents remain unexplained. The discrepancy between nominal Adam predictions and measured exponents is particularly relevant. Appendix E also reports different estimates of Adam’s effective �1��1� from different observables. The proposed finite-norm corrections are plausible, but a quantitative demonstration that they reconcile these estimates would strengthen the argument.

Questions for authors

The abstract would benefit from stating the experimental setting and optimization domain earlier. Current abstract is full of terminology and broad concluding language. This make the abstract hard to understand.

1 comment
AuthorsOpenReview ↗

First reply to reviewer igpT

We thank the reviewer for the concrete requests. We agree with the reviewer’s concerns, and addressing them has significantly improved the paper. Here we post the summary of changes: for details, please take a look at our revised manuscript with changes highlighted in orange.

Learning-rate schedules. We ran the requested comparison: constant, cosine (to 20%, 10%, 1% and 0 of the peak) and WSD, for SGD (B=256B = 256) and Adam (B=64B = 64), with the peak tuned at each of nine budgets and five teachers per setting. With the peak bounded by 2.5 times the constant-rate optimum, a stand-in for the stability limits of deep networks, all schedules keep αD\alpha_D within 0.03 of 1/31/3 and lower the loss by 30–60% for cosine and WSD (new Fig. 5). Without a bound, a schedule with a finite floor, such as Chinchilla's 10%, still gives 1/31/3. Steeper envelopes appear only when two conditions hold together: the schedule anneals all the way to zero, and its peak rate grows without bound with the budget, reaching orders of magnitude above the constant-rate optimum. We therefore withdraw the conjecture that cosine and WSD preserve 1/31/3 unconditionally, and derive 1/31/3 for a finite floor or a bounded peak. (new Fig. 5, App. I)

Nominal versus effective pp and qq. We found two sources for the inconsistency among Adam's qq estimates: the tangential feedback on norm growth (see our response to Reviewer UemV), and a grid-conversion error in sDs_D and sBs_B (general comment). After both corrections, the trajectory exponents and sDs_D agree at q=0.65q = 0.65–0.680.68. The corrected pp also tracks the nominal power across the PowerAdam family (p≈0.85ap \approx 0.85a), while qq exceeds it by 0.140.14–0.180.18; the origin of this excess remains open, and we say so. (App. E.7, H.3, revised Table 1)

Abstract. As suggested, the abstract now states the setting (a single softmax layer trained online on a teacher's hard labels) and the optimizers compared before presenting results, and defines each term where it first appears.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of rebuttal

Dear reviewers,

We thank all reviewers for their constructive comments. We have uploaded a revised manuscript with changes highlighted in orange, and respond to each reviewer individually. The main changes:

  1. Including feedback from tangential fluctuations (UemV, igpT). Separating the norm β\beta from the overlap ω\omega shows how tangential fluctuations feed back on radial growth. The feedback preserves the scaling symmetry, so αD=1/3\alpha_D = 1/3 survives, up to finite-norm corrections; in addition, the theory now predicts SGD's optimal learning rate with no fitted parameters and makes Adam's different qq estimates consistent (App. B.3, E.7).

  2. Learning-rate schedules (8t5L, igpT). With the peak rate tuned at every budget within 2.5 times the constant-rate optimum, a stand-in for the stability limits of deep networks, constant, cosine and WSD schedules all give αD≈1/3\alpha_D \approx 1/3 for SGD and Adam and change only the coefficient. The envelope becomes steeper only when two conditions hold together: the schedule anneals all the way to zero, rather than to a finite floor such as Chinchilla's 10%, and its peak rate grows without bound with the budget, to orders of magnitude above the constant-rate optimum. We therefore withdraw the conjecture that cosine and WSD preserve 1/3 unconditionally (new Fig. 5, App. I). We also state the schedule result in the abstract, the contributions, the main results (with the new Figure 5) and the Discussion.

  3. Correction. We found errors in computing sDs_D and sBs_B: grid indices were converted to learning rates with a single grid for all optimizers, although four were swept on finer grids, and the plane fit used a window differing from the one stated in Appendix H.3. The corrected values remove the inadmissible q>1q > 1; αD\alpha_D and the channel exponents are unaffected (App. H.3).

We have also added a direct measurement of drift and diffusion (UemV, App. G.2) and experiments with anisotropic inputs (8t5L, App. J).

AuthorsOpenReview ↗

An addition: late-time exponents

An addition: late-time exponents.

We have uploaded a revised manuscript that clarifies a question left open in the first revision and extends Figure 5. The new material is in Appendix H.7, with short updates to the captions of Figures 4 and 5, Appendix E.7 and the Discussion; like the earlier changes, it is marked in orange.

  1. Late-time exponents (igpT, UemV). The new Appendix H.7 reports eight-batch sweeps with runs ten times longer. Late in training, the effective exponents of the PowerAdam family and Adam approach the nominal p=q=ap = q = a of an idealized A=V−aA = V^{-a} preconditioner, while αD=1/3\alpha_D = 1/3 and the sum rule hold throughout. The excess of qq in the earlier window is therefore pre-asymptotic; it lasts longer when the memory of Adam's second moment is longer. Muon's exponents change little, and their origin remains open. Figure 4's caption, Appendix E.7 and the Discussion are updated accordingly.
  2. Extended Figure 5. Adam's schedule envelopes now extend to D≈8×107D \approx 8 \times 10^7; with the bounded peak rate, they stay consistent with D−1/3D^{-1/3} (fitted exponents 0.31–0.33 over D=107D = 10^7–8×1078 \times 10^7).

We would be grateful if the reviewers could let us know whether our responses address their concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.