acceptodds
Under review as a conference paper at ICLR 2027

When Muon Becomes Gradient Descent: Finite-Time Flows and Noise Smoothing Near the Optimum

Abstract

Muon is a recently proposed optimizer whose matrix update applies the polar factor (matrix sign) of a gradient or momentum matrix. We study an idealized no-momentum, no-weight-decay Muon update on matrix parameters in both deterministic and stochastic regimes. In the deterministic continuous-time limit, Muon is steepest descent with respect to the spectral norm and obeys the dissipation identity ; under a nuclear Łojasiewicz-type error bound this yields *finite-time* convergence, in contrast to the merely exponential rate of gradient flow. In the minibatch case, the correct continuous model is an Itô SDE whose drift is the noise-smoothed polar map , not . For Gaussian noise , the drift linearizes near stationarity as with , and for square matrices. Thus stochasticity smooths the matrix sign and locally turns Muon into a scaled gradient method with an Ornstein–Uhlenbeck noise neighborhood. This suggests a simple hybrid: use Muon away from stationarity to remove anisotropy and switch to SGD near the optimum, where Muon's finite-time advantage disappears and, at a matched local contraction rate, its stationary noise floor is never lower than that of SGD.

Primary area:
optimization
Keywords:
muon, optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FTJCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies an idealized Muon update using the exact polar factor, without momentum or weight decay. It derives a nuclear-norm dissipation identity and a finite-time convergence bound under an error-bound assumption. For isotropic Gaussian gradient noise, it calculates the local linear response of the smoothed polar map and its dependence on matrix shape. A local Ornstein–Uhlenbeck approximation then gives a stationary-variance comparison with SGD at matched contraction rates. Synthetic quadratic experiments illustrate the predicted drift, noise floor, and a proposed switch from Muon to SGD near the optimum.

Critical strengths
  1. The Gaussian response coefficient is exact and useful. Theorem 6.1 identifies the local coefficient as E∥Z∥∗/(mn s)\mathbb E\|Z\|_*/(mn\,s). Its derivation through Gaussian differentiation and orthogonal invariance is sound, and the matrix-shape dependence provides a clear quantitative insight.

  2. The matched-rate comparison addresses a meaningful question. Appendix A.11 accounts for the different natural learning-rate scales and derives an interpretable variance ratio for the local OU models. Figure 4 directly measures the predicted drift. These controlled experiments suit the paper's theoretical scope.

Critical weaknesses
  1. The deterministic ODE can fail to have a solution. Definition 3.1 and equation (8) select the compact polar factor at rank-deficient gradients. Consider F(W)=x2+xy+y2F(W)=x^2+xy+y^2, with x=W11x=W_{11}, y=W22y=W_{22}, and irrelevant off-diagonal entries. This smooth convex objective satisfies ∥∇F∥∗≥2F\|\nabla F\|_*\geq\sqrt{2F}. Starting at (x,y)=(−1/3,2/3)(x,y)=(-1/3,2/3), the gradient is diag⁡(0,1)\operatorname{diag}(0,1). Locally, u=2x+yu=2x+y must satisfy u(0)=0u(0)=0 and u˙=−2sign⁡(u)−1\dot u=-2\operatorname{sign}(u)-1, with sign⁡(0)=0\operatorname{sign}(0)=0. The identity (u2/2)′=−2∣u∣−u≤0(u^2/2)'=-2|u|-u\leq0 forces u≡0u\equiv0, contradicting the prescribed derivative there. The energy calculation is valid along an existing trajectory. The flow formulation needs an existence argument, potentially through a differential inclusion. This is the main obstacle to acceptance.

  2. The stationary comparison needs precise local-model scope. Theorem 6.1 is a small-gradient expansion, and Appendix A.11 derives stationary variances for the linearized OU model. The statement should specify positive curvature and the approximation involved. Figure 3 uses equal numerical learning rates and thus checks equation (35). The matched-contraction result is equation (36). At the stated parameters, the paper's OU approximation and square-matrix asymptotic predict RMS ∥g∥F/s≈3.33\|g\|_F/s\approx3.33, leaving the accuracy of the local approximation uncertain. A measured regime diagnostic would help interpret this experiment. The exact local calculation remains a credible contribution.

Questions for authors
  1. Clarification and mathematical repair: What solution concept addresses the rank-deficient example above? Please give the corresponding existence and finite-time statements and explain the relationship to the discrete polar update. A convincing repair is necessary for my support.

  2. Clarification of scope: Can you state the stationary comparison explicitly at the approximation level justified by the proof, including positive-curvature assumptions? Please report a diagnostic of Figure 3's local regime and the hybrid switching threshold. Resolving this alongside Question 1 could move my recommendation to weak acceptance. A general invariant-measure theorem is optional.

  3. Clarification of novelty: How do the matrix results extend Monzio Compagnoni et al., Adaptive Methods through the Lens of SDEs (ICLR 2025), which develops Gaussian-smoothed signSGD and local covariance analysis? A direct comparison would help assess the significance of the matrix-specific coefficient and shape dependence.

Reviewer_AFkmOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies several toy models for the convergence of the Muon optimizer. Several toy, continuous-time models for the behavior of Muon are derived by ignoring aspects of the algorithm such as momentum and weight decay: the authors present a deterministic model ignoring noise in the gradient updates, and several SDE based models that do not ignore the noise in the gradient updates. Global convergence guarantees are proven for the deterministic model under a Łojasiewicz-type condition, and gradient norm convergence guarantees are proven for one of the SDE based models. Synthetic simulations are presented which match some of the mathematical results.

Critical strengths

-The paper studies a timely and practically relevant topic. Analyzing Muon from a matrix perspective and comparing to some more traditional ideas in optimization theory is relevant and to my knowledge, novel

-The results seem correct and plausible, although I did not check the proofs carefully

Critical weaknesses

The biggest weakness of this paper is its significance. The results are on toy models of Muon that strip away many subtleties, such as momentum, weight decay, and time discretization. Even in these toy settings, I unfortunately believe that little of substance is shown. The global convergence result Theorem 4.3 is not surprising, it is essentially a direct analogue of global convergence results of gradient flow for PL (gradient domination) functions. The result (19) is similarly a direct analogue of small expected norm convergence results for stochastic gradient descent (in continuous time, no less). Theorems 6.1 and 7.1 are simple manipulations.

Furthermore, the paper is poorly written and I found it extremely difficult to understand.

Questions for authors

Could the authors explain why it is reasonable to ignore momentum, weight decay, and time discretization in their theoretical models to analyze Muon?

Reviewer_oPeqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the deterministic and stochastic dynamics of Muon. The authors establish finite-time convergence under certain assumptions and show that stochastic Muon behaves similarly to scaled gradient descent near the optimum due to noise smoothing. Based on this observation, they propose switching from Muon to SGD near convergence.

Critical strengths
  1. Clear presentation. The paper is well written, well organized, and easy to follow. The theoretical analysis is clear.
  2. Meaningful research direction. Studying stochastic Muon and understanding how gradient noise affects its optimization dynamics is a meaningful direction.
Critical weaknesses

1. Highly idealized assumptions.

The analysis relies on simplified Muon dynamics and isotropic Gaussian noise, which may not reflect practical LLM training. More importantly, the paper suggests switching from Muon to SGD when the gradient enters the noise-dominated regime. However, how can we identify this regime in practice? Gradient noise in LLM training can be highly structured and vary across layers, making it difficult to define a clear and practical switching criterion.

2. The analysis may not capture the main advantages of Muon.

One important reason Muon may work well in LLM training is its ability to handle complex optimization landscapes. SGD follows the first-order gradient direction, which may not provide the best finite-step decrease when second-order curvature effects are significant. Muon's matrix-orthogonalized updates may offer a better trade-off between first-order descent and curvature-related effects.

However, the current analysis mainly focuses on local quadratic behavior and does not sufficiently capture these potential advantages of Muon on complex neural network landscapes.

3. Overly simplified experiment.

The experiment focuses entirely on synthetic quadratic objectives, where SGD is naturally effective. For example, on simple linear regression problems, SGD can achieve very small optimization errors, while fixed-step Muon may oscillate near the optimum. Therefore, observing a lower stationary loss for SGD in these settings is not particularly surprising.

The main concern is whether these experiments reflect the optimization regimes where Muon is actually beneficial. Similar limitations arise when evaluating adaptive optimizers such as Adam exclusively on simple quadratic problems.

4. Lack of practical validation.

The paper proposes switching from Muon to SGD near convergence, but provides no experiments on realistic neural networks. Even small-scale Transformer experiments comparing Muon, SGD, and the proposed hybrid strategy would help establish whether the theoretical findings are practically meaningful.

Questions for authors
  1. How can the noise-dominated regime be efficiently identified during practical LLM training?
  2. Can the authors provide experimental evidence that the proposed switching strategy improves optimization on LLM or Diffusion model tasks?

The paper is clearly written and provides interesting theoretical insights into stochastic Muon. However, the highly idealized assumptions and simplified quadratic experiments limit the practical relevance of its conclusions. In particular, the proposed Muon-to-SGD switching strategy lacks validation on the complex optimization landscapes that motivate Muon's use in LLM training.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.