acceptodds
Under review as a conference paper at ICLR 2027

On the Convergence and Divergence of Muon: The Role of Momentum

33 views

Abstract

Muon has emerged as an effective optimizer for training large language models. Previous studies have established momentum-dependent convergence guarantees for Muon. Some obtain convergence by choosing approaching one as the training horizon grows. Other analyses give guarantees for arbitrary fixed , but often require growing batch sizes or additional conditions such as noise that vanishes with the gradient norm. These results do not establish both a convergence threshold independent of the training horizon and small-momentum divergence on the same fixed problem. To our knowledge, we provide the first joint analysis of these two behaviors for Muon within a fixed smooth finite-sum setting. We establish sufficient high-momentum thresholds and for best-iterate stationarity bounds under the affine growth condition and diminishing stepsizes: pathwise for arbitrary epochwise permutations and in expectation for i.i.d. uniform sampling. Under the strong growth condition, these bounds tend to zero as the horizon grows. We also establish a sufficient low-momentum threshold and a smooth, strongly convex quadratic for which, from suitable initializations, Muon diverges to infinity for all : on every permutation path and on an event of arbitrarily high probability under i.i.d. sampling. Within each sampling scheme, the high-momentum guarantees apply to the same quadratic with the initialization and stepsize schedule unchanged. Together, these results show that changing momentum alone can separate triple divergence from vanishing stationarity bounds on the same fixed problem.

Primary area:
optimization
Keywords:
Muon, convergence analysis, divergence, momentum, stochastic optimization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_4bUDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how momentum affects the convergence of the Muon optimizer. The authors prove convergence guarantees for sufficiently large momentum and construct a strongly convex quadratic problem on which Muon diverges for small momentum. They consider two sampling strategies and show that both behaviors can occur on the same problem. The paper also includes experiments on quadratic problems, MNIST, and CIFAR-10.

Critical strengths

The main strength of the paper is the divergence counterexample. The authors show that Muon can diverge even on a smooth, strongly convex quadratic problem where all component functions share the same minimizer. They also establish a stationarity guarantee for sufficiently large momentum on the same problem.

Critical weaknesses

The theoretical results cover only narrow ranges of momentum. For the quadratic problem with n=8,n = 8, divergence is guaranteed below 0.00180.0018 and stationarity above 0.9995,0.9995, but the experiments show a transition around 0.44.0.44. Thus, the theory does not explain the observed behavior. The convergence guarantees are for the best iterate and require additional assumptions for vanishing stationarity bounds. This makes the practical relevance of the theoretical results unclear.

The divergence example uses a specific quadratic problem where Muon behaves like SignSGD. It is unclear whether the same behavior occurs in real neural-network training. The MNIST and CIFAR-10 experiments mainly show that higher momentum reduces training loss, but do not directly support the theoretical results. The models are relatively small, and the CIFAR-10 experiments use approximate orthogonalization instead of the exact from the theory.

Questions for authors

I do not have specific questions for the authors. My main concerns are about how useful the theoretical results are in practice and whether they apply to other problems.

Reviewer_TzkvOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors identify regimes for which the momentum parameter β\beta causes Muon to converge or diverge. In contrast to existing analyses, the authors fix β\beta and establish convergence bounds under the affine growth condition and decreasing stepsizes.

Critical strengths
  • The paper provides several convergence and divergence guarantees for Muon under a fixed momentum parameter and certain assumptions. I have not carefully checked the proofs, but they appear to be correct. This is in contrast to prior analyses, which either varied the momentum parameter, batch size, or noise assumptions.
Critical weaknesses
  • The theorem statements are extremely hard to interpret, which severely undermines any goal of practical relevance. To me, the biggest argument for writing additional Muon convergence proofs (of which, by the way, some important ones are not cited in the paper, e.g. [1,2]) is that existing analyses only apply in parameter regimes that are unrealistic in practice. However, the paper offers no interpretation of β+perm\beta_+^\text{perm} or what typical values in practice would look like. (Does β=0.95\beta=0.95 converge? β=0.9\beta=0.9?)
  • The stated rates, when taken at face value, are O(log⁡TT)O\left(\frac{\log T}{\sqrt{T}}\right), which is suboptimal compared to the known, achievable rate of O(1T)O\left(\frac{1}{\sqrt{T}}\right) under certain conditions.
  • In addition to violating the formatting policy, the decreased space between paragraphs and certain lines also makes the text hard to read.

References

[1] Lizhang Chen, Jonathan Li, Qiang Liu. Muon Optimizes Under Spectral Norm Constraints. TMLR 2026.

[2] Egor Shulgin, Sultan AlRashed, Francesco Orabona, Peter Richtárik. Beyond the Ideal: Analyzing the Inexact Muon Update. CPAL 2026.

Questions for authors

Address the weaknesses above. Do typical values of β\beta used in practice fall under the convergent regime? The theorem statements are quite convoluted; how should they be interpreted?

Reviewer_pUwNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper's theorems show two thresholds for Muon momentum β\beta: for β\beta above β+\beta^+ they prove closeness (to stationary point) guarantees, and for β\beta below β−\beta^- they prove divergence. Closeness (to stationary point) guarantee means an upper bound on a minimum (over steps) full-batch gradient norm. They call these best-iterate stationary bounds.

The two settings that the paper proves are: multi-epoch mini-batch optimization, and i.i.d. sampling with replacement from a pool of nn batches. Both methods assume fixed batches, no reshuffling. The assumptions include smoothness of the full-batch and mini-batch losses, and affine growth condition. The divergence proofs construct a single quadratic example of a loss function on which optimization diverges.

Critical strengths

The paper proves the results on the two settings of multi-epoch training, and i.i.d. batch sampling with replacement. The main text is written clearly.

I'm not qualified enough to comment on the novelty of the proof techniques used.

Critical weaknesses

There are several assumptions that make the results either 1) not applicable to the usual deep learning training (e.g. keeping batches the same) or 2) give too large of constants for any realistically useful bound (e.g. realistically D_0 and D_1 would grow with nn, meaning the bounds are not useful for e.g. LLM training where each batch is seen once and nn is large). Additionally, it's not clear how the results would apply to per-matrix Muon updates in deep learning; only a single parameter matrix is considered.

To somewhat connect the theory results from the paper to practical usage of Muon, the authors could provide the estimates of the constants in the bounds in a realistic setting, and show (if it is the case) how the bounds are somewhat tight and predictive of Muon optimization. What would be the constants β±\beta^\pm in realistic settings?

The paper (abstract, section 2.3) frames their contributions as more general than the prior work (e.g. [1]). To me they just seem to be different settings, and one is not clearly better than another. E.g. [1] seems to better capture a single epoch training.

[1] On the Convergence Analysis of Muon

Questions for authors
  1. See weaknesses
  2. In the CIFAR-10 ResNet and CNN experiments (Figure 3e,f), there is not a clear threshold when the loss drops, i.e. the drop seems more gradual. Where would you expect β±\beta^\pm constants to be in this case? How do these plots connect to your theory?
  3. In both settings the batches, and fif_is stay fixed. That's usually not the case in real training, have you thought of relaxing it? Could one just relax assumptions (1) (2) (3) to hold on any batch?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.