- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the convergence of the practical Muon optimizer with practical Newton–Schulz (NS) iterations and decoupled weight decay. The authors interpret Muon as a right-preconditioned optimization method with a dynamic weighted l2 regularizer, whose influence becomes negligible as the algorithm approaches stationarity. Under standard assumptions, they establish an convergence rate in terms of the expected Frobenius norm of the gradient. The analysis also shows that convergence does not require accurate matrix orthogonalization, but instead relies on the positive alignment between the NS update direction and momentum. Finally, experiments on ResNet and language models support the theoretical findings and demonstrate the practical advantages of Muon over AdamW.
Critical strengths
This paper provides a convergence analysis of practical Muon with practical Newton–Schulz (NS) iterations. In particular, it establishes an convergence rate, eliminating the dependence on present in previous convergence bounds.
Critical weaknesses
I think the authors' claim that Muon theoretically achieves an improvement in convergence rate over AdamW is somewhat overstated. The claimed improvement is obtained by converting an existing AdamW l1-stationarity upper bound. This is a valid comparison with the resulting Frobenius upper bound, but the conversion itself can lose a factor of . Comparing these upper bounds does not establish a separation between the optimizers’ actual convergence behavior.
Questions for authors
Can the convergence guarantee be extended to a constant weight-decay coefficient? In practical large-scale training, weight decay is typically chosen independently of the training horizon. How essential is this horizon-dependent choice to the convergence analysis? Can the authors establish convergence guarantees with a fixed, non-vanishing weight-decay coefficient, or characterize the residual stationarity error in this setting?
3 comments
I thank the authors for their response. I will maintain my current positive rating.
We sincerely appreciate your time and thoughtful questions.
We appreciate the reviewer's positive assessment of our theoretical contributions and constructive feedback. We address the concern and question below.
W1. Support for the d Improvement
Thank you for this important comment. We agree that comparing convergence upper bounds alone does not establish a separation between the actual convergence behavior of Muon and AdamW. However, our claim is supported by both experimental and theoretical evidence.
Fig.2 (right) on p.10 examines ResNet-32 models of varying dimensions and shows that the normalized gradient norm ratio d∥∇f(WMuon)∥F/∥∇f(WAdamW)∥F remains broadly stable across dimensions, providing empirical support for the predicted d improvement in the Frobenius norm.
Moreover, our comparison is based on the best-known convergence guarantee for AdamW established by Li et al. (2026), who state that "the only remaining uncertainty concerns the tightness of the dimension d" (Section 2.1), leaving the d dependence an open problem.
Finally, to avoid overstating our results, we consistently specify that the d advantage concerns the Frobenius norm (p. 1, line 20; p. 2, line 97; p. 6, line 281; p. 7, line 326; p. 10, lines 487 and 531), and also emphasize that Muon and AdamW achieve the same order of convergence rate under the entrywise ℓ1 norm (p. 2, line 102; p.6, line 284).
Q1. Extension to Constant Weight Decay
Thank you for this question. Our stochastic nonconvex analysis can be extended to a fixed, iteration-independent weight decay (WD) coefficient, yielding a residual stationarity error proportional to λ. Specifically, choose
1-\beta=\frac{1}{(\lambda T)^{2/3}}, \varepsilon\in\Big(0,\frac{1}{(\lambda T)^{1/3}}\Big], \lambda\in\Big(0,\frac{C_2}{4\|W_0\|_F}\Big], T\ge\frac{1}{\lambda},$$ then $$\frac1T\sum\nolimits_{t=0}^{T-1} \mathbb E\lVert \nabla f(W_t)\rVert_F \le \frac{8C_3(f(W_0)-f_{\inf})}{C_2^2}\lambda +\mathcal O\Big( \frac{\sigma/\sqrt{b_{\mathrm{mb}}}+L+1} {\lambda^{1/3}T^{1/3}} + \frac{\lambda^{2/3}\sqrt{L(f(W_0)-f_{\inf})}} {T^{1/3}} \Big).$$ Similarly, choosing $$\eta=\frac{C_2}{4C_3\lambda T}, 1-\beta=\lambda^2, \varepsilon\in\Big(0,\frac{L}{\lambda T}\Big], \lambda\in\Big(0,\min\Big\lbrace 1,\frac{C_2}{4\lVert W_0\rVert_F}\Big\rbrace\Big], T\ge\frac{1}{\lambda},$$ yields $$\frac1T\sum_{t=0}^{T-1} \mathbb E\lVert \nabla f(W_t)\rVert_F \le \Big( \frac{8C_3(f(W_0)-f_{\inf})}{C_2^2} +\frac{2\sqrt{2}(C_2+C_3)\sigma} {C_2\sqrt{b_{\mathrm{mb}}}} \Big)\lambda +\mathcal O\Big( \frac{\sqrt{L(f(W_0)-f_{\inf})}}{\lambda^2T} +\frac{L}{\lambda^3T} \Big).$$ The original choice $\lambda=O(T^{-1/4})$ is used to control $\lambda\lVert W_t\rVert_F$. For a iteration-independent $\lambda$, we instead choose $\eta=O(T^{-1})$ to maintain this bound, resulting in a residual stationarity error. We further emphasize that a iteration-dependent $\lambda$ is common in convergence analyses of algorithms with decoupled WD, and remains compatible with practical training, as we have now clarified in the revised manuscript (p. 6, line 286): "Existing convergence results for algorithms with decoupled WD commonly impose iteration-dependent requirements on $\lambda$, as in our condition $\lambda\le\mathcal{O}(T^{-1/4})$. For AdamW, Zhou et al. (2024) adopts exponentially decaying WD coefficient, while Li et al. (2026) requires $\lambda\le\mathcal{O}(\sqrt{d}/T^{3/4})$. Importantly, our relatively mild dependence on $T$ allows a sufficiently large WD coefficient to retain its regularization effect in practice. For example, Moonlight pretraining with Muon takes approximately $1.76\times10^5$ updates (Liu et al., 2025), giving $T^{-1/4}\approx0.05$, with larger values for shorter iteration horizons".