acceptodds
Under review as a conference paper at ICLR 2027

Convergence of Practical Muon

17 views

Abstract

Muon is emerging as a promising alternative to AdamW for large-scale neural network training, yet theoretical understanding of its practical implementation remains incomplete, as existing analyses often simplify or omit two key components: (i) practical Newton–Schulz iterations with empirically tuned polynomial coefficients ; and (ii) decoupled weight decay for regularization. In this paper, we provide an optimization interpretation and establish convergence for practical Muon, jointly accounting for both components. Specifically, we interpret practical Muon as right-preconditioned optimization of the original loss with a dynamic weighted regularizer that vanishes as stationarity is approached, so that the optimization target remains the original objective. We then establish, to our best knowledge, the first convergence guarantee for practical Muon in the stochastic nonconvex setting, with an convergence rate in terms of the expected Frobenius norm of the gradient, improving the dimension dependence of the best known AdamW's convergence rate by a factor of , where is the iteration horizon and is the parameter dimension. Experiments further support the theoretical convergence results.

Primary area:
optimization
Keywords:
Muon, practical Newton-Schulz, weight decay, stochastic nonconvex optimization, convergence analysis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vGRuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the convergence of the practical Muon optimizer with practical Newton–Schulz (NS) iterations and decoupled weight decay. The authors interpret Muon as a right-preconditioned optimization method with a dynamic weighted l2 regularizer, whose influence becomes negligible as the algorithm approaches stationarity. Under standard assumptions, they establish an O(T−1/4)O(T^{-1/4}) convergence rate in terms of the expected Frobenius norm of the gradient. The analysis also shows that convergence does not require accurate matrix orthogonalization, but instead relies on the positive alignment between the NS update direction and momentum. Finally, experiments on ResNet and language models support the theoretical findings and demonstrate the practical advantages of Muon over AdamW.

Critical strengths

This paper provides a convergence analysis of practical Muon with practical Newton–Schulz (NS) iterations. In particular, it establishes an O(T−1/4)O(T^{-1/4}) convergence rate, eliminating the dependence on rr present in previous convergence bounds.

Critical weaknesses

I think the authors' claim that Muon theoretically achieves an O(d)O(\sqrt{d}) improvement in convergence rate over AdamW is somewhat overstated. The claimed O(d)O(\sqrt{d}) improvement is obtained by converting an existing AdamW l1-stationarity upper bound. This is a valid comparison with the resulting Frobenius upper bound, but the conversion itself can lose a factor of d\sqrt{d}. Comparing these upper bounds does not establish a separation between the optimizers’ actual convergence behavior.

Questions for authors

Can the convergence guarantee be extended to a constant weight-decay coefficient? In practical large-scale training, weight decay is typically chosen independently of the training horizon. How essential is this horizon-dependent choice to the convergence analysis? Can the authors establish convergence guarantees with a fixed, non-vanishing weight-decay coefficient, or characterize the residual stationarity error in this setting?

3 comments

We appreciate the reviewer's positive assessment of our theoretical contributions and constructive feedback. We address the concern and question below.

W1. Support for the d\sqrt{d} Improvement

Thank you for this important comment. We agree that comparing convergence upper bounds alone does not establish a separation between the actual convergence behavior of Muon and AdamW. However, our claim is supported by both experimental and theoretical evidence.

Fig.2 (right) on p.10 examines ResNet-32 models of varying dimensions and shows that the normalized gradient norm ratio d∥∇f(WMuon)∥F/∥∇f(WAdamW)∥F\sqrt{d}\lVert\nabla f(W_{\mathrm{Muon}})\rVert_F/\lVert\nabla f(W_{\mathrm{AdamW}})\rVert_F remains broadly stable across dimensions, providing empirical support for the predicted d\sqrt{d} improvement in the Frobenius norm.

Moreover, our comparison is based on the best-known convergence guarantee for AdamW established by Li et al. (2026), who state that "the only remaining uncertainty concerns the tightness of the dimension dd" (Section 2.1), leaving the d\sqrt{d} dependence an open problem.

Finally, to avoid overstating our results, we consistently specify that the d\sqrt{d} advantage concerns the Frobenius norm (p. 1, line 20; p. 2, line 97; p. 6, line 281; p. 7, line 326; p. 10, lines 487 and 531), and also emphasize that Muon and AdamW achieve the same order of convergence rate under the entrywise ℓ1\ell_1 norm (p. 2, line 102; p.6, line 284).

Q1. Extension to Constant Weight Decay

Thank you for this question. Our stochastic nonconvex analysis can be extended to a fixed, iteration-independent weight decay (WD) coefficient, yielding a residual stationarity error proportional to λ\lambda. Specifically, choose

1-\beta=\frac{1}{(\lambda T)^{2/3}}, \varepsilon\in\Big(0,\frac{1}{(\lambda T)^{1/3}}\Big], \lambda\in\Big(0,\frac{C_2}{4\|W_0\|_F}\Big], T\ge\frac{1}{\lambda},$$ then $$\frac1T\sum\nolimits_{t=0}^{T-1} \mathbb E\lVert \nabla f(W_t)\rVert_F \le \frac{8C_3(f(W_0)-f_{\inf})}{C_2^2}\lambda +\mathcal O\Big( \frac{\sigma/\sqrt{b_{\mathrm{mb}}}+L+1} {\lambda^{1/3}T^{1/3}} + \frac{\lambda^{2/3}\sqrt{L(f(W_0)-f_{\inf})}} {T^{1/3}} \Big).$$ Similarly, choosing $$\eta=\frac{C_2}{4C_3\lambda T}, 1-\beta=\lambda^2, \varepsilon\in\Big(0,\frac{L}{\lambda T}\Big], \lambda\in\Big(0,\min\Big\lbrace 1,\frac{C_2}{4\lVert W_0\rVert_F}\Big\rbrace\Big], T\ge\frac{1}{\lambda},$$ yields $$\frac1T\sum_{t=0}^{T-1} \mathbb E\lVert \nabla f(W_t)\rVert_F \le \Big( \frac{8C_3(f(W_0)-f_{\inf})}{C_2^2} +\frac{2\sqrt{2}(C_2+C_3)\sigma} {C_2\sqrt{b_{\mathrm{mb}}}} \Big)\lambda +\mathcal O\Big( \frac{\sqrt{L(f(W_0)-f_{\inf})}}{\lambda^2T} +\frac{L}{\lambda^3T} \Big).$$ The original choice $\lambda=O(T^{-1/4})$ is used to control $\lambda\lVert W_t\rVert_F$. For a iteration-independent $\lambda$, we instead choose $\eta=O(T^{-1})$ to maintain this bound, resulting in a residual stationarity error. We further emphasize that a iteration-dependent $\lambda$ is common in convergence analyses of algorithms with decoupled WD, and remains compatible with practical training, as we have now clarified in the revised manuscript (p. 6, line 286): "Existing convergence results for algorithms with decoupled WD commonly impose iteration-dependent requirements on $\lambda$, as in our condition $\lambda\le\mathcal{O}(T^{-1/4})$. For AdamW, Zhou et al. (2024) adopts exponentially decaying WD coefficient, while Li et al. (2026) requires $\lambda\le\mathcal{O}(\sqrt{d}/T^{3/4})$. Importantly, our relatively mild dependence on $T$ allows a sufficiently large WD coefficient to retain its regularization effect in practice. For example, Moonlight pretraining with Muon takes approximately $1.76\times10^5$ updates (Liu et al., 2025), giving $T^{-1/4}\approx0.05$, with larger values for shorter iteration horizons".

I thank the authors for their response. I will maintain my current positive rating.

We sincerely appreciate your time and thoughtful questions.

Reviewer_j4FXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work studies the convergence of the practical Muon algorithm, which employs Newton-Schulz iterations for approximate orthogonalization and decoupled weight decay. Specifically, the authors provide a right-preconditioning interpretation of the Muon update and introduce a dynamically regularized objective, for which they show an exact equivalence with preconditioned gradient descent when no momentum is applied. Furthermore, the authors present experimental results comparing Muon with the Newton–Schulz coefficients used in practice (practical Muon), Muon with the standard Newton–Schulz coefficients from the Taylor expansion (standard Muon), and AdamW on ResNet-32 and GPT-2.

Critical strengths
  • This work addresses a very relevant issue of the convergence of the Muon algorithm with inexact polar factorization that is implemented in practice. Much of the prior work on the convergence of Muon focuses on convergence with exact SVD or does not account for the weight decay component. Additionally, compared to prior work (Kim and Oh (2026), Choudhury et al. (2026), Qian and Richtárik (2026)), which analyzes convergence using the orthogonalization error, this work directly analyzes the Newton-Schulz direction through the right-preconditioned interpretation, without explicitly bounding the orthogonalization error.
  • The paper is overall nicely written and pleasant to read.
Critical weaknesses
  • One minor weakness is that while the authors provide some explanation/intuition on the advantage of practical NS over standard NS in Section 6 through a specific example and they also illustrate this through the experimental results, the provided convergence guarantee does not capture this distinction.
  • One other minor point is that the implications of the dynamic regularized objective are not sufficiently explored, and more specifically whether this interpretation provides further insights into the optimization dynamics or explains concrete convergence advantages, similar to those established by Ma et al. (2026) for exact orthogonalization.
Questions for authors
  • Could the authors clarify whether the benefits of the decoupled weight decay are reflected in the convergence guarantees?
3 comments

We sincerely appreciate the reviewer's positive assessment of our contributions and the clarity of our presentation, as well as the constructive feedback. We address each comment and question below.

W1. We appreciate the reviewer's observation. We agree that our convergence guarantee does not capture the advantage of practical NS over standard NS. However, a general convergence theorem should cover as many settings as possible, where practical NS does not necessarily outperform standard NS. We therefore devote Section 6 (p. 9) to explaining why practical NS can offer advantages in practice. To clarify this point, we have added the following discussion in the manuscript (p. 9, line 329):

"Indeed, practical NS is not necessarily superior. For example, in the scalar case, the two NS outputs differ only by a constant factor after normalization, allowing identical updates with rescaled stepsizes. However, in practical settings such as large language model training, singular values can span several orders of magnitude, with the smallest ones dominating the orthogonalization error (Wu et al., 2026). Practical NS amplifies these small singular values faster, reducing the orthogonalization error and potentially improving optimization (Jordan et al., 2024). Section 6 further illustrates this advantage through an ill-conditioned quadratic example".

W2. Thank you for this suggestion. Optimization dynamics and concrete convergence advantages can be studied in specific structured settings, as exemplified by the linear transformer analysis of Ma et al. (2026). Our analysis of an ill-conditioned quadratic objective in Section 6 (p. 9) takes a similar approach. However, a general characterization of how the dynamic regularizer affects optimization dynamics and convergence speed is more challenging and warrants a dedicated study, whereas our interpretation primarily aims to clarify what Muon actually optimizes when practical NS and WD are combined and to motivate the subsequent convergence analysis.

Q1. Thank you for raising this question. Our convergence guarantees do not explicitly quantify the benefits of decoupled weight decay (WD). Similar to the advantage of practical NS over standard NS, the benefits of WD cannot be guaranteed by a general convergence theorem intended to cover as many settings as possible. Nevertheless, Proposition 1 shows that WD induces a strongly convex weighted ℓ2\ell_2 regularizer in the dynamic objective. This may enhance the convexity of the regularized objective and facilitate faster convergence in suitable settings.

Thank you for your responses! I will maintain my initial rating.

Many thanks for your valuable time and constructive feedback.

Reviewer_tcuNOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper provides a convergence proof of Muon with practical Newton-Schulz iteration and decoupled weight decay, motivated by implementations of the algorithm in practice. The precise result is a O(T−1/4)O(T^{-1/4}) convergence rate on the average expected Frobenius norm of the gradient over TT iterations under bounded variance and smoothness assumptions.

Critical strengths

The paper is generally well-written and easy to read. I have checked the proofs, and they appear to be mostly correct. Theorem 1 is indeed the first convergence result for Muon with practical NS and decoupled weight decay in the general stochastic, smooth, and nonconvex setting, to the best of my knowledge.

Critical weaknesses
  • The discussion of parameter choices from Lines 286 to 294 is not convincing. In particular, it is evident from its definition in (8) that λ\lambda is not merely T−1/4T^{-1/4}, which invalidates the given calculation as a justification of this parameter choice. Indeed, having λ\lambda depend on TT is already problematic, since most, if not all, practical implementations use a constant λ\lambda independent of TT. The same can also be said of the momentum coefficient β\beta, although I find this to be less problematic. This is not an issue from a mathematical point of view, but given the practical motivations for studying this problem in the first place, I see this to be a serious limitation.
  • The claim that Muon with weight decay "still targets the original objective" in Lines 89 and 90 is in direct contradiction with known results [1, 2, 3]. In fact, this discrepancy can be attributed to the fact that the presented analysis allows λ\lambda to depend on TT. I suggest making this point precise.
  • It should be noted that the experiments do not actually prove a O(T−1/4)O(T^{-1/4}) convergence rate or a d\sqrt{d} advantage over AdamW, and changing width does not necessarily isolate the dependence between parameters and dimensions. However, this is not of critical importance for a theory paper.
  • (minor)
    • "Contributions" instead of "Controbutions" in Line 86.
    • Theorem 1 is stated with σ=0\sigma=0 possible, but (20) requires division by σ\sigma.

References

[1] Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, Volkan Cevher. Training Deep Learning Models with Norm-Constrained LMOs. ICML 2025.

[2] Lizhang Chen, Jonathan Li, Qiang Liu. Muon Optimizes Under Spectral Norm Constraints. TMLR 2026.

[3] Maria-Eleni Sfyraki, Jun-Kun Wang. Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noise. ICML 2026.

Questions for authors
  1. Why do the authors bound ∥∇f(Wt)∥F\lVert\nabla f(W_t)\rVert_\text{F} when ∥∇f(Wt)∥tr\lVert\nabla f(W_t)\rVert_\text{tr} is more natural for Muon?
  2. Similarly, LL-smoothness with respect to ∥⋅∥op\lVert\cdot\rVert_\text{op} is more natural than LL-smoothness with respect to ∥⋅∥F\lVert\cdot\rVert_\text{F}. How does the analysis change under this assumption?
6 comments

We appreciate the reviewer's careful reading and constructive feedback. We address each comment and question below.

W1. Practicality of the Parameter Choice for $\lambda$

We appreciate the reviewer's concern.

Regarding the calculation as a justification of this parameter choice, under standard initialization schemes, such as the PyTorch nn.Linear default initialization and He/Kaiming initialization, the Frobenius norm ∥W0∥F\lVert W_0\rVert_{\mathrm F} of the initial weight matrix W0∈Rm×nW_0\in\mathbb R^{m\times n} typically scales as O(m)O(\sqrt m), which can be comparable to or smaller than T1/4T^{1/4}. Moreover, c1c_1 can be freely chosen, for example, as c1=1/C3c_1=1/C_3, while C2C_2 is also approximately one. Therefore, it is reasonable to use T−1/4T^{-1/4} to roughly estimate the order of magnitude of λ\lambda.

Regarding the practicality of the theoretical requirement on λ\lambda, we believe that its dependence on TT does not necessarily make it impractical. The key consideration is whether the constant weight decay (WD) coefficients commonly used in practical implementations satisfy our theoretical condition. In fact, dependence on TT is common in convergence analysis. To clarify this point, we have revised the discussion of parameter choice (p. 6, line 286) as follows: "Existing convergence results for algorithms with decoupled WD commonly impose iteration-dependent requirements on λ\lambda, as in our condition λ≤O(T−1/4)\lambda\le\mathcal{O}(T^{-1/4}). For AdamW, Zhou et al. (2024) adopts exponentially decaying WD coefficient, while Li et al. (2026) requires λ≤O(d/T3/4)\lambda\le\mathcal{O}(\sqrt{d}/T^{3/4})". The key practical consideration is whether our condition permits sufficiently large WD coefficients. We have therefore further clarified (p.6, line 289): "Importantly, our relatively mild dependence on TT allows a sufficiently large WD coefficient to retain its regularization effect in practice. For example, Moonlight pretraining with Muon takes approximately 1.76×1051.76\times10^5 updates (Liu et al., 2025), giving T−1/4≈0.05T^{-1/4}\approx0.05, with larger values for shorter iteration horizons".

W2. No Contradiction with [1–3] and Distinct Optimization Dynamics

Thank you for raising this point. However, we respectfully disagree that our result contradicts [1–3], since the algorithms analyzed are fundamentally different. Specifically, [1–3] all consider Muon with exact orthogonalization, achieved through linear minimization oracles under spectral norm constraints in [1] and [3], and explicit use of the matrix sign function in [2]. In contrast, our work considers Muon with finite-step Newton–Schulz (NS) iterations, whose outputs are generally not exactly orthogonal. Consequently, the constrained optimization interpretations in [1–3], which rely on exact orthogonalization, do not directly apply to the algorithm studied in this work.

In fact, Section 3 (pp. 4–5, Optimization Interpretation) of our manuscript explicitly investigates the distinct optimization dynamics arising from the combination of NS and WD. We show that Muon with NS and WD can be interpreted as right-preconditioned optimization of the original loss with a dynamic weighted ℓ2\ell_2 regularizer. In the deterministic setting, this regularizer vanishes up to an O(10−7)O(10^{-7}) residual as ∥∇f(Wt)∥F→0\lVert \nabla f(W_t)\rVert_F\to 0, indicating that Muon still targets the original objective, regardless of whether λ\lambda depends on TT or not (p. 5, lines 249–258).

W3. Experimental Validation

Thank you for this comment. Our experiments aim to empirically support the theoretical predictions. We maintain a consistent model architecture to isolate the dependence between parameters and dimensions as much as possible. A similar approach is adopted by Kim and Oh (2026), who change width to compare convergence across different ranks and demonstrate Muon's advantage over SGD.

W4. Minor

Thank you for pointing out these issues. We have corrected the typo and added the following clarification on p. 9, Line 445: "Here we consider σ>0\sigma>0, as σ=0\sigma=0 follows similarly".

Q1+Q2. The Frobenius Norm Is Also a Natural Choice

Thank you for these questions. The main reason for using the Frobenius norm is to facilitate direct comparison of convergence rates with AdamW, SGD, and other methods. Which norm is more natural depends on the perspective. Muon can be viewed as steepest descent under the spectral norm, whose dual is the trace norm. However, the Frobenius norm corresponds to the Euclidean norm commonly used in convergence analysis and is also adopted in many existing theoretical studies of Muon.

An analysis based on spectral- and trace-norm smoothness is also possible in principle, with our convergence analysis remaining largely unchanged. However, we prefer the Frobenius norm in this work, as one of our main goals is to compare Muon with AdamW and highlight Muon's advantages.

I thank the authors for their response.

I maintain some concerns regarding the parameter choices. In particular, for practical large-scale training, matrix dimensions are typically in the thousands, which suggests m\sqrt{m} is actually several times larger than T−1/4T^{-1/4} when T≈105T\approx10^5. As a conservative estimate, the stated range for λ\lambda in (8) is perhaps around (0,0.01](0,0.01], which is at least an order of magnitude smaller than a common value of λ=0.1\lambda=0.1. I also find the appeal to other works using a decaying weight decay coefficient to be unconvincing, since those works focus on a theoretical bound while the main selling point of this paper is that it claims to analyze Muon as implemented in practice. The analysis in the response to Reviewer vGRu below is also unsatisfactory in that there is a residual error term depending on λ\lambda.

Regarding the contradiction with known results, it can be shown (and seen via a few lines of Python) that even for the scalar objective f(w)=12(w−20)2f(w)=\frac12(w-20)^2, with initialization w0=0w_0=0, λ=0.1\lambda=0.1, and all other parameters specified as in the paper, the practical implementation of Muon does not converge to the optimal parameter of 2020. Instead, it converges to a parameter around 6.96.9, at which the gradient does not vanish. The error in the argument in Lines 249 to 258 lies in the assumption that the gradient vanishes, which need not happen in the presence of decoupled weight decay (again, cf. [1, 2, 3]). Thus, the claim that (the practical implementation of) Muon with decoupled weight decay targets the original objective regardless of whether λ\lambda depends on TT or not is false. It can be reconciled by allowing λ\lambda to depend on TT, which is why I suggested making this point clear in the original review.

I therefore maintain my rating.

1. Contributions of Practical Muon and Shared Limitation in Choosing λ\lambda

We appreciate the reviewer's concern and would like to clarify the scope of our contribution. Here, "practical" primarily refers to retaining two key components of Muon's implementation: (i) practical Newton--Schulz (NS) iterations with empirically tuned polynomial coefficients (3.4445,−4.7750,2.0315)(3.4445,-4.7750,2.0315), and (ii) decoupled weight decay (WD) for regularization. We establish the first convergence guarantee for practical Muon in the stochastic nonconvex setting, eliminating the dependence on rr (the smaller dimension of the weight matrix) present in previous convergence rates, and improving the dimension dependence by a factor of d\sqrt d over the best-known convergence rate for AdamW in terms of the expected Frobenius norm of the gradient.

This represents a substantial advance over prior work in the convergence analysis of practical Muon. While Qian & Richtárik (2026) also cover practical NS with WD, their guarantees rely on star-convexity and parameter choices that depend on the unknown norm ∥W∗∥\lVert W^* \rVert of a global minimizer W∗W^*. Other analyses either use exact orthogonalization when accounting for WD (Pethick et al., 2025, and Chen et al., 2026), or omit WD while studying practical NS (Do et al., 2026; Qian & Richtárik, 2026), standard NS (Kim and Oh, 2026, and Choudhury et al., 2026), or exact orthogonalization(Shen et al., 2026). Importantly, to the best of our knowledge, neither exact SVD-based orthogonalization nor standard NS with Taylor-based coefficients has been publicly validated in large-scale commercial model training (Liu et al., 2025; Team et al., 2025; Xu et al., 2026; Team et al., 2026; Zeng et al., 2026; Hong et al., 2026), whereas WD has been shown by the Kimi team (Liu et al., 2025) to be important for scaling Muon and is enabled by default in PyTorch's Muon implementation.

Regarding λ\lambda, we agree that the dependence on the initialization norm or iteration horizon TT in (8) can restrict the sufficient range of λ\lambda to values one or two orders of magnitude smaller than the commonly used λ=0.1\lambda=0.1. However, this is an inherent limitation shared by both AdamW and Muon when analyzing their theoretical performance. For AdamW, Zhou et al. (2024) adopt an exponentially decaying WD coefficient, while Li et al. (2026) require λ≤O(d/T3/4)\lambda\le\mathcal{O}(\sqrt{d}/T^{3/4}), both imposing stronger decay requirements than our condition. The reviewer's example f(w)=12(w−20)2f(w)=\frac{1}{2}(w-20)^2, together with our analysis in the response to Reviewer vGRu establishing a residual stationarity error proportional to λ\lambda, further illustrates this inherent limitation. Our analysis in the response to Reviewer vGRu captures a genuine effect of decoupled WD rather than reflecting a weakness of the proof, while our main theorem provides sufficient parameter conditions for exact convergence to stationarity.

As the reviewer has "substantial direct research experience on this topic", we think the reviewer would understand the progress and importance of our contributions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.