acceptodds
Under review as a conference paper at ICLR 2027

Muon with Finite Newton–Schulz: The Smoothing Benefit in Nonsmooth Nonconvex Optimization

Abstract

Muon has emerged as a strong optimizer for the matrix-valued parameters in large language model pretraining, approximately orthogonalizing its momentum with a few Newton–Schulz iterations. Existing theory either replaces this iteration with the exact polar factor it approximates, or treats it as a source of approximation error, and thus the iteration Muon actually runs can only hurt the guarantees. We show that finite Newton–Schulz can instead be beneficial for nonsmooth nonconvex optimization. To this end, we analyze Muon through the online-to-nonconvex conversion, which views the update rule of an optimizer as an online learner and converts its regret bound into a stationarity guarantee. We observe that the finite Newton–Schulz iteration smooths the discontinuous polar map into a Lipschitz map of the singular values, and Muon with finite Newton–Schulz can be regarded as an online learner with a smoothed spectral potential. Building on this observation, we prove that a Newton–Schulz depth growing only logarithmically in the target accuracy suffices for convergence to stationary points in nonsmooth nonconvex optimization, whereas Muon with the exact-polar update may fail to converge. The resulting sample complexity bounds match the best-known guarantees for nonsmooth nonconvex optimization and are optimal for smooth nonconvex optimization up to problem-dependent factors. The argument extends beyond Newton–Schulz to general spectral maps that are Lipschitz smoothings of the polar map.

Primary area:
optimization
Keywords:
Muon optimizer, Newton–Schulz, spectral optimizer, nonsmooth nonconvex optimization, onlinle-to-nonconvex conversion

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_nYbiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors give a general regret bound for momentum updates with smooth spectral maps, such as finite Newton-Schulz, to characterize the tradeoff between polar-approximation accuracy and update stability and derive stationarity guarantees for nonsmooth nonconvex optimization. They show that q=O(log⁡(1/ε))q=O(\log(1/\varepsilon)) iterations is sufficient for expected relaxed (ρ,ε)(\rho, \varepsilon)-stationary on nonsmooth nonconvex objectives with complexit as O(ρ−1ε−3+ε−2)O(\rho^{-1}\varepsilon^{-3}+\varepsilon^{-2}). Additionally, they recover optimal accuracy dependence for smooth deterministic and stochastic optimization. In other words, they show that the analyzed algorithm achieves the optimal ε−2\varepsilon^{-2} deterministic and ε−4\varepsilon^{-4} stochastic gradient-oracle dependence, up to problem dependent factors for smooth objectives.

Critical strengths
  1. The paper introduces a smoothing mechanism, which explains how finite Newton–Schulz implicitly regularizes the update and shows that more iterations improve the polar approximation while making updates more sensitive to changes in momentum (Theorem 5.3 and Lemma 5.2). This explains why stopping Newton-Schulz early can help Muon.

  2. The analysis of this paper covers a broad class of Lipschitz spectral maps and shows that logarithmic Newton–Schulz depth suffices for nonsmooth stationarity guarantees. The proposed mechanism yields concrete guarantees and extends beyond a single polynomial or a synthetic observation, providing a foundation for further optimizer analysis.

Critical weaknesses
  1. The version of Muon used to prove convergence has several differences from its practical implementations. Theorems 5.4 and 5.7 analyze Algorithm 1 within the randomized O2NC protocol of Algorithm 2. This setup uses a known fixed normalization bound (Assumption 2.3), classical Newton–Schulz coefficients (lines 206 - 209), randomized gradient queries, and an averaged output (lines 235 - 251). The authors acknowledge several differences from practical Muon in the conclusion section. Existing guarantees remain insufficient to show whether this mechanism guarantees convergence under the practical normalization, coefficient, and gradient evaluation configurations of Muon.

  2. The empirical results are not clearly sufficient to demonstrate the practical importance of the Newton–Schulz depth tradeoff. First, this paper does not include many experimental results. Second, Appendix H reports one synthetic problem instance and two learning rates. It uses current-iterate gradients rather than the analyzed O2NC protocol (lines 1517 - 1521), and Figure 3 reports loss curves without directly evaluating the theorem’s stationarity criterion. Additionally, increasing the Newton–Schulz depth can make updates both larger and more sensitive to changes in momentum, since hq(x)≈(15/8)qxh_q(x)\approx(15/8)^q x for small xx. Comparing depths at the same learning rate also compares different update sizes.

Questions for authors

Refer to the weaknesses:

  1. Can the authors clarify and explain the differences between the version of Muon used to prove convergence and its practical implementations? And what prevents extending the guarantees to practical Muon implementations?
  2. Can the authors explain the second weakness: the empirical results are not clearly sufficient to demonstrate the practical importance of the Newton–Schulz depth tradeoff?
2 comments

Thank you for your time and thoughtful comments on our work.

Can the authors clarify and explain the differences between the version of Muon used to prove convergence and its practical implementations?

The analyzed algorithm differs from practical Muon in its Newton–Schulz coefficients, normalization, gradient query points, and output rule. We think these differences largely arise from the difficulty of obtaining convergence guarantees in the nonsmooth nonconvex setting.

The classical Newton–Schulz coefficients obtained from a Taylor expansion give a nondecreasing singular-value map (see Assumption 4.1). This nondecreasing property allows us to construct a convex potential. The empirically tuned coefficients used in practical Muon do not satisfy this property, so our proof does not directly apply to them.

Fixed normalization satisfies the condition Ct+1≥βCtC_{t+1}\geq\beta C_t needed in our proof (see Lemma 4.7). Momentum-dependent normalization does not generally satisfy this condition and is therefore not covered by our analysis.

Randomized queries are motivated by the fact that deterministic local algorithms can require exponentially many queries in the dimension to find Goldstein stationary points in the nonsmooth nonconvex setting ([1], Theorem 3.1). We also use relaxed stationarity and output rule because ordinary approximate stationarity cannot in general be guaranteed in finite time in this setting ([2], Theorem 5).

Can the authors explain the second weakness: the empirical results are not clearly sufficient to demonstrate the practical importance of the Newton–Schulz depth tradeoff?

Thank you for this suggestion. We have added experiments to Appendix H of the revised paper and marked the additions in red. Figures 4–5 report stationarity measurements, Figure 6 shows how they depend on Newton–Schulz depth and learning rate, and Figures 7–8 present experiments with O2NC randomized query points.

We agree that equal learning rates do not imply equal update sizes. However, the example of Parshakova et al. [3] shows that exact polar can fail regardless of the learning rate. On this example, our experiments show that finite Newton–Schulz makes progress while exact polar remains away from stationarity.

References

[1] Guy Kornowski. The Exponential Price of Determinism in Nonsmooth Nonconvex Optimization. arXiv:2609.23837, 2026.

[2] Jingzhao Zhang, Hongzhou Lin, Stefanie Jegelka, Suvrit Sra, and Ali Jadbabaie. Complexity of Finding Stationary Points of Nonconvex Nonsmooth Functions. ICML, 2020.

[3] Tetiana Parshakova, Ahmed Khaled, Michael Crawshaw, Guillaume Garrigos, and Robert M. Gower. Muon Does Not Converge on Convex Lipschitz Functions. arXiv:2605.08980, 2026.

Thank the authors for their detailed responses, solves my quesitons. I will maintain my score.

Reviewer_hpkmOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper considers the convergence of the Muon optimizer, which employs momentum and a few Newton-Schulz (NS) iterations. While prior work treats the finite NS approximation as an additional source of error in the convergence analysis, the authors show that this approximation actually provides a beneficial smoothing effect for nonsmooth nonconvex optimization. The key idea is to analyze Muon through the online-to-nonconvex conversion framework and interpret Muon with finite NS as an online learner with a smoothed spectral potential. The number of NS iterations determines the tradeoff between the approximation error to the exact polar map and the Lipschitz constant of the resulting mapping. The authors show that choosing it to be Θ(log⁡(1/ϵ))\Theta(\log (1/\epsilon)) balances the two and leads to sample complexity bounds optimal in the target accuracy for both nonsmooth nonconvex optimization and smooth nonconvex optimization.

Critical strengths
  • This paper provides a novel perspective on understanding the effect of finite NS iterations in the Muon optimizer. Rather than treating NS as an undesirable approximation that introduces extra error terms in the convergence analysis, the authors show that it can provide the necessary smoothing for nonsmooth nonconvex optimization. In my view, researchers have a general understanding that Newton-Schulz can be viewed as a smooth approximation of the polar map, but it is nice to formalize this intuition in a theoretical framework.
  • As a side technical contribution, the authors also propose a different analysis of the gradient-based prediction algorithm from (Jiang et al., 2026a), where the gradient of the potential function is evaluated at the momentum instead of the cumulative gradient. This enables them to also establish a convergence rate for the smooth nonconvex optimization setting.
Critical weaknesses
  • As acknowledged by the authors, the Newton-Schulz iteration analyzed in this paper departs from the practice in several ways. In particular, the authors fix the normalization constant in advance, whereas in practice it is typically chosen based on the Frobenius norm of the momentum matrix. Also, the NS coefficients are generally different from the ones in the paper, which are obtained by truncating a Taylor expansion.
  • The authors claim that their sample complexity bounds match the best-known guarantees for nonsmooth nonconvex optimization and for smooth nonconvex optimization. In my opinion, this is somewhat imprecise, since the complexity bounds in Theorems 5.4 and 5.7 also incur additional explicit dimensional factors, which may be improved. Hence, it would be better to clarify that the optimality is in terms of the dependence on the target accuracy ϵ\epsilon.
Questions for authors

None

1 comment

Thank you for taking the time to read our paper and provide us with valuable feedback.

Hence, it would be better to clarify that the optimality is in terms of the dependence on the target accuracy ϵ\epsilon.

Thank you for pointing this out. We have clarified in the revised version that the optimality is in terms of the dependence on ϵ\epsilon and ρ\rho.

Reviewer_sMQhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper interprets the finite Newton-Schulz transformation of momentum in Muon as a way of smoothing for the discontinuous polar map. A discounted online-regret analysis quantifies a tradeoff between approximation to the exact polar map and stability, yielding an O(log⁡(1/ϵ))O(\log(1/\epsilon)) depth choice. Combined with an online-to-nonconvex conversion, this gives O(ρ−1ϵ−3+ϵ−2)O(\rho^{-1}\epsilon^{-3}+\epsilon^{-2}) oracle complexity for nonconvex non smooth objective. A similar argument recovers the usual ϵ−2\epsilon^{-2} deterministic and ϵ−4\epsilon^{-4} stochastic dependences for smooth nonconvex objectives. The paper also develops an FTRL interpretation and illustrates the depth tradeoff on a synthetic objective.

Critical strengths
  1. One of the main contribution is that the authors provide an informative interpretation of finite orthogonalization. The analysis treats finite Newton-Schulz as a source of regularization and stability, providing a new explanation beyond its usual role as an inaccurate polar computation.

  2. Another strength of the paper is that the analyses extend to general spectral maps that are Lipschitz smoothings of the polar map.

Critical weaknesses
  1. The analyzed algorithm differs from practical Muon through its fixed normalization, classical Newton–Schulz coefficients, randomized gradient queries, and output rule. Although acknowledged in Section 6, these differences limit the guarantee’s applicability to practical implementations. Moreover, the deterioration of the regret bound as the Newton–Schulz depth increases shows a limitation of the analysis; it does not itself imply failure of the exact-polar Muon. Consequently, the results support finite Newton–Schulz as a mechanism enabling a convergence guarantee within this framework, but provide limited evidence that finite depth itself improves convergence over exact orthogonalization. From a theoretical viewpoint, we still cannot make conclusion about the comparison between finite NS and exact polar map in nonconvex non smooth setting.
Questions for authors

See weakness.

2 comments

Thank you for your time and helpful comments.

The analyzed algorithm differs from practical Muon through its fixed normalization, classical Newton–Schulz coefficients, randomized gradient queries, and output rule. Although acknowledged in Section 6, these differences limit the guarantee’s applicability to practical implementations.

We agree that these differences limit the applicability of our guarantee to practical Muon. We study the classical Newton–Schulz iteration obtained from a Taylor expansion, with coefficients$\left(\frac{15}{8},-\frac54,\frac38\right).$ Practical implementations often tune these coefficients for empirical performance, while our analysis explains the regularization introduced by the classical iteration. Extending the guarantee to practical variants with tuned coefficients, momentum-dependent normalization remains open.

Regarding the output rule, a fundamental difficulty arises in nonsmooth nonconvex optimization, where ordinary approximate stationarity cannot in general be guaranteed in finite time ([1], Theorem 5). This motivates our use of a relaxed stationarity criterion and an output rule based on averaging.

Consequently, the results support finite Newton–Schulz as a mechanism enabling a convergence guarantee within this framework, but provide limited evidence that finite depth itself improves convergence over exact orthogonalization. From a theoretical viewpoint, we still cannot make conclusion about the comparison between finite NS and exact polar map in nonconvex non smooth setting.

Parshakova et al. [2] show that Muon with the exact polar update can fail to converge even on convex Lipschitz objectives. The example in [2] also extends to randomized line-segment queries. For sufficiently small momentum, exact-polar Muon fails to converge, while finite Newton–Schulz reaches relaxed stationary points. This counterexample shows that exact-polar Muon can fail, independently of our regret analysis.

Our analysis further explains the role of finite Newton–Schulz iterations. Exact-polar Muon corresponds to discounted FTL (Appendix F), whereas Muon with finite Newton–Schulz corresponds to FTRL with an explicit regularizer (Theorem 5.3). This induced regularizer yields the regret bound underlying our stationarity guarantee. As additional motivation for randomized queries, we note that deterministic local algorithms can require exponentially many queries in the dimension to find Goldstein stationary points in nonsmooth nonconvex problems ([3], Theorem 3.1).

References

[1] Jingzhao Zhang, Hongzhou Lin, Stefanie Jegelka, Suvrit Sra, and Ali Jadbabaie. Complexity of Finding Stationary Points of Nonconvex Nonsmooth Functions. ICML, 2020.

[2] Tetiana Parshakova, Ahmed Khaled, Michael Crawshaw, Guillaume Garrigos, and Robert M. Gower. Muon Does Not Converge on Convex Lipschitz Functions. arXiv:2605.08980, 2026.

[3] Guy Kornowski. The Exponential Price of Determinism in Nonsmooth Nonconvex Optimization. arXiv:2609.23837, 2026.

Thank you for your clarification and I have no more question. I'll maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.