acceptodds
Withdrawn from ICLR 2027

Training-Loss Guarantees for Muon with Finite-Step Newton-Schulz Orthogonalization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Existing convergence analyses of Muon assume exact orthogonalization and guarantee only stationarity, so it is unresolved what Muon's five tuned Newton-Schulz steps preserve and whether that suffices to reach a prescribed neural-network training loss. We establish a finite-time training guarantee that accounts for both momentum accumulation before orthogonalization and the tuned finite-step update. For full-batch training of a sufficiently wide two-layer ReLU network with fixed random output weights and a positive-definite limiting neural tangent kernel, we prove that Muon reaches any target empirical squared loss with high probability over initialization. For every momentum parameter , a target-dependent constant learning rate proportional to yields a hitting-time bound of , with other problem parameters fixed. The sufficient width is independent of both target accuracy and momentum. The analysis shows that the tuned Newton–Schulz map preserves alignment with the momentum buffer while bounding the update's spectral norm. Control of gradient variation near initialization transfers this alignment to the current gradient, ensuring descent until the target is reached without requiring exact orthogonalization. Numerical experiments support these mechanisms at widths below the sufficient theoretical threshold: gradient-update alignment remains above the analytical reference, and all 30 runs across six widths and five student initializations on a fixed teacher-student dataset reach the target loss while maintaining kernel positivity.

Primary area:
optimization
Keywords:
Muon, Neural Net, Matrix Optimizers, Gradient Descent

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6LouOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper establishes theoretical guarantees regarding training loss for the Muon algorithm, which employs a momentum mechanism and a five-step Newton–Schulz orthogonalization method. It demonstrates that this finite-step mapping maintains alignment with the momentum buffer and bounds the spectral norm of the update; leveraging gradient stability near initialization, this ensures descent along the current gradient direction. For sufficiently wide two-layer ReLU networks with fixed output weights trained via full-batch gradient descent, the Muon algorithm—using a target-dependent learning rate—attains any target loss ϵ\epsilon within O((1−μ)−1ϵ−1/2)O((1-\mu)^{-1}\epsilon^{-1/2}) steps with high probability. Results from synthetic experiments further confirm that these mechanisms of alignment and kernel stability hold true even when the network width is small.

Critical strengths
  1. This paper examines the significant gap between exact orthogonalization analysis and the five-step update method employed by the Muon algorithm. It establishes the sufficient properties required for the tuned map, without necessitating that the map strictly approximate the exact polar factor.
  2. The proof follows a clear logical progression: it first establishes trajectory containment via update bounds, then utilizes kernel stability to control momentum lag and derive loss descent, thereby avoiding circular reasoning.
Critical weaknesses
  1. There remains a significant gap between this theory and practical training protocols. The theory relies on assumptions such as two-layer networks, fixed random output weights, full-batch updates, and exact arithmetic operations. However, such theoretical analysis is difficult to scale to truly large-scale models.
  2. This framework is equally applicable to the normalized momentum method without orthogonalization (i.e., K=0K=0); thus, the theorem does not establish an advantage unique to the Newton–Schulz update. An end-to-end comparison—tracking both loss values ​​and runtime—among the K=0K=0 baseline, tuned NS, classic NS, and the exact polar factor would help clarify the specific contribution of orthogonalization.
  3. I am personally unclear on the actual value provided by this type of theoretical analysis in the papers related to Muon. Furthermore, current observations do not offer sufficient insight; I would find empirical results from pre-trained models in the range of hundreds of millions to billions of parameters more informative.
Questions for authors

No

Reviewer_qDMcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper claims to be about Muon and and analysing it's convergence with an inexact Newton-Schulz and on two layer neural networks. I think this misframes the work, and really the paper does not give any insight into Muon, or inexact Newton-Schulz. Instead, the paper is about the analysis of inexact momentum methods on two layer networks. In particular it is an extension of  [SimonDu2019] work on analysis full batch gradient descent on two layer neural networks. They use the same near-initialization framework but for bounded, approximate transformations of momentum. The question now, is why do this and what does it say about Muon?

[SimonDu2019] Simon S. Du, Xiyu Zhai, Barnabas Poczos, Aarti Singh, Gradient Descent Provably Optimizes Over-parameterized Neural Networks, ICLR 2019

Critical strengths

The paper does propose an analysis of approximate gradient descent for two layer neural networks. Though unclear what this analysis could tell us

Critical weaknesses

A mismatch between the story and objective, and what the paper really achieves.

Questions for authors

I would say this analysis says very little about Muon specifically, since it bounds away the effects of approximating the polar factor. It is really about any update of the form

Xt+1=Xt−γΦ(Mt∥Mt∥F),X_{t+1} = X_t - \gamma \Phi\left(\frac{M_t}{\|M_t\|_F}\right),

where Φ\Phi is a generalized matrix function acting on the singular values through a scalar function ϕ\phi satisfying

\qquad \text{and} \qquad \sup_{0 \leq x < 1} \phi(x) < \infty.$$ Although this includes the Newton–Schulz map—specifically, the original approximate method, rather than the newer methods implemented in the modded-nanoGPT speedruns—it provides little insight into Muon. Indeed, according to the resulting bounds, the fastest method is full-batch gradient descent. For Muon, this amounts to a very rough sanity check. Because of this misalignment between the paper's stated objective and its actual contributions, I cannot recommend acceptance. I would instead suggest rewriting the work to focus on insights into approximate gradient methods. Furthermore, the paper should compare its results with other work analysing Muon with an approximate polar map, such as [Shulgin2025]. [Shulgin2025] Egor Shulgin, Sultan AlRashed, Francesco Orabona, and Peter Richtárik. *Beyond the Ideal: Analyzing the Inexact Muon Update*. 2025. https://arxiv.org/abs/2510.19933 Minor comments: 1. The bibliography needs substantial formatting improvements. It omits publication venues for many papers and uses Semantic Scholar URLs rather than links to the papers or their publication pages. 2. The introduction is confusing, particularly in its description of the contributions. I believe this reflects the main issue above: the limited insight into Muon makes it difficult to articulate the paper's contribution as currently framed. 3. "We assume that this normalized limiting kernel is positive definite" You should mention that first (7) is always positive semi-definite, and that assuming positive definite is a very weak assumption. It just requires having inputs such that $x_i \neq x_j$ or $x_i \neq -x_j$.
Reviewer_Rz31OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether Muon can reach a target training loss when using momentum and five Newton–Schulz steps instead of exact orthogonalization. The authors prove a finite-time guarantee for sufficiently wide two-layer ReLU networks, with an iteration bound of O((1−μ)⁻¹ε⁻¹/²). The proof shows that the Newton–Schulz update maintains enough alignment with the gradient to reduce the training loss. Experiments on a teacher–student setup show that Muon reaches the target loss across different widths, with gradient alignment and kernel stability consistent with the theoretical analysis.

Critical strengths
  1. The paper goes beyond existing stationarity guarantees by proving that Muon with momentum and five Newton–Schulz steps can reach a target training loss.
  2. The proof identifies two simple properties of Newton–Schulz updates, positive alignment and bounded spectral norm, and connects them to training loss reduction. The experiments also assess gradient alignment, kernel stability, and convergence across different widths and momentum values.
Critical weaknesses
  1. One main problem is that the theorem applies to simple Frobenius-normalized momentum without Newton-Schulz (K = 0), which actually gives better theoretical constants. The bound also gets worse as momentum increases. So while the paper proves that Muon can reach the target loss, it does not really explain why orthogonalization or momentum is useful. The analysis essentially shows that these components do not prevent convergence, instead of explaining their benefits.
  2. The experiments show that Muon can reach the target loss, but it is unclear what they tell us beyond that. There are no training comparisons with simpler methods such as GD or normalized GD. Also, the scaling trends in Figure 1 are partly driven by the learning rates, and the theoretical width requirement is far beyond the tested settings. Therefore, the experiments fail to show what is specific to Muon or how informative the theoretical guarantee is in practice.
Questions for authors
  1. Since the same guarantee holds without Newton–Schulz (K = 0), and the bound gets worse with larger momentum, what does the analysis tell us about the benefits of these two components in Muon?
  2. Could the authors compare Muon with simpler methods such as GD, Frobenius-normalized GD, and classical Newton–Schulz in actual training? Also, do the scaling trends in Figure 1 still hold without prescribing the learning rate scaling in advance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.