- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces a new anytime training method by using a muon-variant as a baseline optimizer in a schedule-free scheme. Analytical results in a relatively simplified case are given to support the proposal, and empirics provide strong validation that the method works. Authors also provide analytical and empirical arguments to argue why weight decay is needed in long horizon training.
Critical strengths
I think this is a very good paper within the standards of this subfield. The main strength is the empirical results. The proposed method seem to work well in a reasonable set of benchmarks.
Critical weaknesses
The proofs are done using what authors call SF-SpectralSGD with Momentum, but the performant method is SF-NorMuon which has extra features. There is no SF-SpectralSGD with Momentum benchmark in simulations as far as I can see to judge how much gap there is between theory and practice.
Questions for authors
-
How big is the a gap between theory and practice? Which of the extra features introduced in SF-NorMuon over SF-SpectralSGD with Momentum is necessary to make the method work? How much do each of these contribute?
-
Relatedly, the logic behind the theory could be better explained. The authors claim that their SF-SpectralSGD with Momentum has the same convergence rates with muon. What is the significance of this result? Is the idea that the muon variant that is being cited is not schedule-free?
-
Corollary 2.1 is a big-O-with-tilde result. Should we not worry about the coefficients? Since the goal of this kind of a study is to beat the sota, and coefficients can matter, what is the value of this kind of a theory? I know the authors are not the only ones who are doing this in this literature, this is just me trying to see how they think about this.
4 comments
Q: What is the significance of matching Muon's rates? Is the cited Muon variant not schedule-free?
A: Yes, the cited analyses are for Muon, which is not schedule-free. Our result shows that the schedule-free mechanism preserves Muon's convergence rate. That mechanism, evaluating gradients at interpolated points and averaging online, is what enables anytime training without a learning-rate schedule in practice. This is in the same spirit as Defazio et al. [2], where Schedule-Free SGD attains the optimal rate of SGD without a schedule. One cannot hope for a faster rate: the rates (stochastic) and (deterministic) match the lower bounds in for first-order methods on smooth non-convex problems [3, 4]. The point is that the schedule-free modification comes at no cost in the rate, up to a mild logarithmic factor (see next answer).
Q: Corollary 2.1 is an result. What is the value of such theory when constants matter?
A: Thank you for this question. All coefficients are explicit in Theorem 2.1, and the precise rate is (Corollary C.2). Concretely, the leading term is with . The therefore hides three things:
- Problem constants: , , , , and the rank .
- The interpolation factor .
- A logarithmic factor from online averaging, at most . This equals 1.83–1.95 across all our runs ( 5k–118k steps) and vanishes for (Corollary C.2(iii)).
Hence the logarithmic factor is mild, whereas the problem constants cannot be measured for LLMs, as in essentially all such analyses. We fully agree that such bounds cannot rank optimizers. Even for smooth convex minimization, the exact worst-case constants of gradient descent and the exactly optimal first-order method were established only through the performance-estimation line of work [5–8], which is largely separate from rate analyses. Therefore, the role of our theory is soundness, and SOTA comparisons are made empirically (Section 4 and the new results above). More broadly, the gap between worst-case theory and deep learning practice remains a major open problem. Notably, where constants do matter in practice, our analysis provides them: Lemma 3.2 predicts the steady-state norm of including its constant, which closely matches the measured values (Figure 4).
We hope the above answers clarify your questions. Thank you again for your time in reviewing our manuscript.
[1] Kim et al. (2026) "AMUSE: Anytime Muon with Stable Gradient Evaluation", arXiv:2605.22432
[2] Defazio et al. (2024) "The Road Less Scheduled", NeurIPS 2024
[3] Arjevani et al. (2023) "Lower Bounds for Non-Convex Stochastic Optimization", Math. Programming
[4] Carmon et al. (2020) "Lower Bounds for Finding Stationary Points I", Math. Programming
[5] Drori and Teboulle (2014) "Performance of first-order methods for smooth convex minimization: a novel approach", Math. Programming
[6] Kim and Fessler (2016) "Optimized first-order methods for smooth convex minimization", Math. Programming
[7] Drori (2017) "The exact information-based complexity of smooth convex minimization", J. Complexity
[8] Rotaru et al. (2024) "Exact worst-case convergence rates of gradient descent: a complete analysis for all constant stepsizes over nonconvex and convex functions", arXiv:2406.17506
Thanks for your answers.
Thank you for reading our responses. We appreciate your feedback and engagement throughout the discussion; your questions helped us clarify the connection between our theory and practice in the revised manuscript. Thank you again for taking the time to carefully review our paper.
Dear Reviewer GLsZ,
Thank you for your valuable comments and feedback. We are encouraged that you found our paper very good within the standards of the subfield, with strong empirical results. We also kindly refer you to our new result: combined with the time-varying interpolation of the concurrent work AMUSE [1] and a norm-preserving variant of our update scaling, SF-NorMuon outperforms both per-horizon tuned NorMuon and AMUSE (125M model, validation loss):
SF-NorMuon uses a single configuration for all horizons, with all hyperparameters set to the values tuned for AMUSE except for the learning rate. AMUSE is run with its released code and the best configuration reported in its paper for the 124M model (η=0.01, βinit=0.6, ρ=0.8), with its 6,000 warmup steps converted to 1,500 steps at our 4× larger batch (the same number of warmup tokens). NorMuon, AdamW, and SF-AdamW are from Table 1.
Due to limited GPU availability during the rebuttal period, we report the 125M model here; we will include the 772M results with the new setting in the final manuscript.
Q: How big is the gap between theory and practice? Which features of SF-NorMuon are necessary, and how much does each contribute?
A: We kindly refer the reviewer to Figure 3, which is exactly this ablation (125M model, 8× Chinchilla):
We have made the connection between Figure 3 and Theorem 2.1 explicit in Section 2.2 of the revision, and added the discussion of the rates below after Corollary 2.1 (in blue).