- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how optimizer choice affects training dynamics and data scaling in an online, hard-label teacher–student softmax classification task. It distinguishes loss decay along fixed-hyperparameter training trajectories from the optimal loss envelope obtained by tuning the learning rate for each sample budget. Under an effective preconditioner scaling assumption, the authors derive a relation between radial and tangential decay exponents and predict an optimizer-independent optimal data scaling of . Experiments with SGD, PowerAdam, Adam, Muon, and SignGD show different training dynamics but similar optimal data-scaling exponents.
Critical strengths
- The distinction between trajectory scaling and optimal-envelope scaling is meaningful. A fixed learning rate may lead to slow late-stage loss decay, even when the optimizer achieves good scaling after tuning for each training horizon. The radial–tangential decomposition provides a useful explanation for this difference. In particular, Adam's faster radial decay and slower tangential decay explain both the shape of its loss curves and why its optimal learning rate decreases with the training budget. Connecting these observations helps explain how optimal hyperparameters scale with training.
- The framework connects several scaling relations and provides useful empirical evidence. It links the channel exponents, optimal learning-rate scaling, and the loss balance at the optimum. The PowerAdam sweep is particularly useful, as it shows a smooth transition between different dynamics on the same task. Muon and SignGD provide additional evidence beyond diagonal second-moment preconditioning. Proposition 1 also gives an elegant trace-based argument for the tangential loss without requiring the gradient-noise covariance to be proportional to, or commute with, the Hessian.
Critical weaknesses
- The assumptions behind Theorem 2 need further justification. The theorem optimizes the two-channel loss while treating its coefficients as independent of . However, tangential fluctuations also affect radial growth, and this feedback depends on . It is unclear whether the assumed loss expression remains valid when optimizing over .
- The radial correction in Appendix B.3 appears inconsistent. The paper differentiates the stationary relation , rather than taking the radial derivative at a fixed parameter direction. From and , we instead obtain , giving a radial gradient of approximately . This correction is not negligible at the claimed optimum and may affect the predicted optimal loss balance. A similar residual-dependent drift appears in arXiv:2605.22341.
- The effective scaling exponent is not consistently supported by the measurements. The analysis assumes the same governs radial drift and tangential noise, which is not obvious for adaptive or normalized optimizers. For Adam, Remark 2 gives from norm growth, from radial loss, and from optimal learning-rate scaling. The finite-norm correction explains the first discrepancy but not the last. In particular, the measured norm exponent predicts , compared with the observed .
- The drift-dominated regime is not directly verified. Appendix G mainly infers drift dominance from agreement with the predicted exponent relation, rather than separately measuring deterministic radial drift and stochastic norm growth. Thus, the experiments support the common scaling pattern more strongly than the proposed mechanism. Direct measurements of these update statistics would provide stronger evidence.
Questions for authors
See Weaknesses.
First reply to reviewer UemV
We thank the reviewer for the careful reading. We agree with the reviewer’s suggestions, and addressing them significantly improved the paper. Here we post the summary of changes: for details, please take a look at our revised manuscript with changes highlighted in orange.
Including feedback from tangential fluctuations. Appendix B.3 differentiated the stationary relation rather than taking the gradient at fixed direction; the correct radial gradient is (Lt−Lr)/β, as the reviewer notes. We now resolve the dynamics by separating the norm β from the overlap ω=⟨WS,WT⟩F: ω is linear in WS and receives only drift, driven by the full loss L, while β receives the drift (Lr−Lt)/β plus a noise term 2Lt/β, again summing to L/β. Tangential fluctuations thus feed back on radial growth through R=Lt/Lr∝zβq. In the notation of Kühn et al. (2026), ω, β2 and ∣e∣2=β2−ω2 correspond to D, Qeff and Δ, with R=6Δ/π2; for SGD at batch size one, their equations and ours coincide. Because the feedback depends on the state only through R, the scaling symmetry, and hence αD=1/3 and sD=−q/3, survive up to finite-norm corrections of order 1/β. With no fitted parameters, the theory with feedback predicts SGD's optimal η/B (0.0637 against 0.0625 measured), the optimal channel ratio (Lt/L=0.30 against 0.296; the value without the feedback is 1/3), and the point where ∂L/∂lnβ changes sign (zc=0.100 against 0.101–0.104; the value without the feedback is 0.266). We have replaced the original derivation in Appendix B.3 and collected the consequences of the feedback in Appendix E.7.
Inconsistent q. Part of the inconsistency was a grid-conversion error in sD (general comment). The rest is the feedback above: for q>0, R grows along a run and shifts the trajectory exponents. With the feedback included, αβ, αr, αt and sD give q=0.65–0.68, against 0.68–0.92 without it. The sum rule 2αr+αt=1 accordingly holds exactly only without the feedback; with it, the corrections are within the measured few-percent excess. (App. E.7, revised Table 1)
Drift-dominated regime. We measured both terms directly. For SGD, the drift and noise contributions to Δβ2 match −2ηG and 4ηLt within 1%, and the noise term is 1.25 times the drift, so "drift-dominated" was incorrect in β; in ω, growth is pure drift and matches 2ηL within 3%, the measured finite-norm correction. For adaptive optimizers the noise term is a few percent of the drift. We have revised the text accordingly. (App. G.2)