acceptodds
Under review as a conference paper at ICLR 2027

On the convergence analysis of the AdaMuon optimizer

18 views

Abstract

In recent years, Muon has demonstrated excellent experimental performance in neural networks. This success has given rise to numerous Muon variants, among which AdaMuon introduces a dynamic adaptive mechanism for large-scale neural network training. However, despite its strong empirical performance, the convergence guarantees hold only under some specific assumptions and are not fully established. To to further narrow the theoretical gap, this paper provides a systematic convergence analysis of AdaMuon. Specifically, we establish convergence analysis under three distinct settings. For nonconvex case under and without Frobenius norm Lipschitz smoothness, we prove the convergence rate is \(O(T^-1/4)\). For star-convex case under Frobenius norm Lipschitz smoothness, we prove the convergence rate to reach the precision \(\epsilon\) is \(O(\epsilon^-3\log(\epsilon^-1))\). These results extend the theoretical scope of the original analysis, offering a more complete characterization of AdaMuon across different optimization assumptions.

Primary area:
optimization
Keywords:
Optimization, Convergence analysis, Smoothness

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_n4y1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The success of Muon has led to the emergence of many Muon variants, and one of which is AdaMuon. AdaMuon introduces a dynamic adaptive mechanism for large-scale neural network training. This paper provides the convergence analysis of AdaMuon in three settings: the nonconvex case under and without Frobenius norm Lipschitz smoothness, and the star-convex case under Frobenius norm Lipschitz smoothness.

Critical strengths

This paper provides a convergence analysis of AdaMuon and establishes the convergence rate for nonconvex functions. It also shows that the cosine similarity between the momentum matrix and update matrix has a positive global minimum and increases monotonically during training.

Critical weaknesses
  1. This paper provides a theoretical analysis for one existing Muon variant AdaMuon, however, it does not clearly explain why this particular variant was chosen. How could this paper advance the understanding of Muon (and its variants), rather than only the AdaMuon algorithm itself? The motivation and novelty of this paper seem not fully convincing.

  2. The experiments mainly illustrate cosine values. The experiments are not sufficient to support the claimed improvements in empirical performance or efficiency of AdaMuon. More experiments are needed to support these claims.

Questions for authors
  1. Given that some papers have already analyzed different Muon variants, why the theoretical convergence analysis of these methods remain insufficiently understood? The related work section on Muon variants does not adequately explain the exact limitations of current theoretical understanding.

  2. Why does this paper focus on the variant AdaMuon? Why is this variant theoretically more important than other Muon variants? Can the analysis in this paper being extend to other variants?

  3. What about training efficiency and performance improvement of AdaMuon in the experiments? How does the claimed "strong empirical performance" can be demonstrated?


Minor: The presentation of this paper could be improved, for example:

Since “AdaMuon” is an algorithm proposed in the previous paper, an appropriate reference should be provided the first time it appears (Line 13);

Many incorrectly formatted references in Section 1.1 (Line 28-38);

“To to further narrow the theoretical gap” (Line 15);

No conclusion section.

1 comment
AuthorsOpenReview ↗

On the Motivation and Contributions of Our Work

We thank the reviewer for the comment.

1. Why AdaMuon and its connection to other Muon variants.

We focus on AdaMuon because it combines Muon's matrix orthogonalization with sign transformation and element-wise adaptive scaling. These modifications make its convergence analysis more challenging. In particular, for standard Muon, exact polar decomposition gives a direct relationship between the momentum and the update direction. This relationship no longer holds directly for AdaMuon, making it difficult to apply analyses that rely on this property.

Our work addresses this difficulty by studying the alignment between the momentum and the final update direction. We establish convergence bounds under three different settings, providing a way to analyze the effects of these additional operations. This approach may also be useful for other Muon variants that modify the orthogonalized update, provided that the corresponding assumptions hold.

2. Experimental performance and efficiency.

Our paper focuses on the theoretical convergence properties of AdaMuon rather than proposing a new optimizer or demonstrating new performance improvements. The experiments are mainly designed to examine the alignment assumption used in our analysis.

The original AdaMuon paper (Si et al., 2025) has already evaluated its training performance and efficiency across GPT-2 and Qwen2.5 models of different sizes, with comparisons against Muon and AdamW.

Reviewer_d7ewOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the AdaMuon problem with theoretical convergence rate. Moreover, the results include three distinct settings. One of the contributions the authors claim is the observation that cosine similarity between momentum matrix and update matrix has a positive minimum. And all the theoretical results rely on this observation.

Critical strengths

This paper studies the convergence of AdaMuon, an important and interesting problem. The authors analyze the optimizer under three distinct settings and derive convergence rates for each.

Critical weaknesses

My main concern is Assumption 2.4, on which the convergence analyses in all three settings rely. The authors state that they support this assumption through theoretical intuition and experimental results. Given its central role, however, the design intuition behind Muon is not sufficient justification. I would like to see a more quantitative analysis: under what conditions does a positive lower bound κ\kappa exist, and which properties of the problem or optimizer affect its value?

The experimental evidence is also limited to two experiments with a small language model. It is unclear whether the observed behavior extends to other models and training settings, or whether it is sufficiently robust to support such a central assumption in a theoretical paper.

I also have two additional comments. First, the authors should make the dependence of the convergence bounds on κ\kappa explicit and discuss how sensitive the guarantees are to small values of κ\kappa. Second, where comparable results exist, they should report the convergence rates for Muon under the same three settings to put the obtained rates in context.

Questions for authors

My main concern is about the Assumption 2.4. If κ\kappa only exists for some specifical settings, the theoretical contributions obtained in this paper could be really limited.

2 comments
AuthorsOpenReview ↗

Response to the Concern on Assumption 2.4

We thank the reviewer for the comment.

For Muon, exact polar decomposition guarantees a positive inner product between the momentum and the update matrix. AdaMuon further introduces sign transformation and element-wise adaptive scaling. While these operations make a direct theoretical guarantee difficult, our experiments suggest that positive alignment is still maintained. Assumption 2.4 formalizes this observation with a uniform lower bound κ>0\kappa>0.

We do not require κ\kappa to be the same for different models or training settings. It depends on the optimization trajectory and measures the minimum alignment between the momentum and the update direction. In our GPT-2 Small experiments, the minimum cosine similarity across parameter matrices remains positive over all observed iterations. We acknowledge that these experiments do not guarantee the same behavior for other models, and further validation would be useful.

As for the convergence bounds, κ\kappa appears through the factor 1/κ1/\kappa. A smaller κ\kappa gives a looser bound, but does not change the convergence rate with respect to TT, provided that κ\kappa remains positive and independent of TT. We will make this dependence clearer in the revised paper.

I agree with Reviewer wr8Q and still have concerns about Assumption 2.4. Thus, I decide to keep my score.

Reviewer_wr8QOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper provides a systematic convergence analysis of AdaMuon. Specifically, the authors establish convergence analysis under three distinct settings. For nonconvex case under and without Frobenius norm Lipschitz smoothness, we prove the convergence rate is O(T^{-1/4}). For star-convex case under Frobenius norm Lipschitz smoothness, we prove the convergence rate to reach the precision $ \varepsilon$ is O(ε−3log⁡(ε−1))O(\varepsilon^{-3}\log(\varepsilon^{-1})), which extend the theoretical scope of the original analysis, offering a more complete characterization of AdaMuon across different optimization assumptions.

Critical strengths
  1. The motivation behind this paper is clear and meaningful, addressing a gap between theory and practice in the field, namely: AdaMuon's strong empirical performance has outpaced its theoretical understanding, and it covers three distinct assumption regimes. 2 This paper has a crucial and important discovery, namely: a positive, monotonically increasing cosine alignment between momentum and update, which is interesting and is illustrated with per-matrix statistics on GPT-2 Small.
  2. This paper also extends the theory to non smooth situations and provides complete theoretical analysis results.
  3. In this paper, the author provides a perspective that the per step cost minimum is an extreme order statistic, which is a key finding that helps explain why naive per-step checks can be misleading.
Critical weaknesses
  1. The author needs to provide an intuitive explanation for this core Assumption 2.4 (\cos \theta_t ≥ k) or add some discussion in their paper, or use experiments to verify whether this hypothesis satisfies practical problems and how it affects convergence rate.
  2. There are some inconsistencies in the theoretical presentation section of this paper. Please review and revise: the abstract and the Theorem 3.12 remark report the star-convex complexity as O(\varepsilon^{−3} log(\varepsilon^{−1})), while the accompanying text states O(\varepsilon^{−1/3} log(\varepsilon^{−1})).
  3. This paper should provide more experimental analysis and include more experimental scenarios, 1D parameters, embeddings, and LayerNorms. The current experimental results only include the case of 2D matrices with the parameter m>n
  4. This paper needs to discuss why the convergence rate O (T ^ {-1/2}) slows down compared to the existing SGD or Adam, and the role of the "weak" assumption 3.5.
Questions for authors
  1. he author needs to provide an intuitive explanation for this core Assumption 2.4 (\cos \theta_t ≥ k) or add some discussion in their paper, or use experiments to verify whether this hypothesis satisfies practical problems and how it affects convergence rate.
  2. There are some inconsistencies in the theoretical presentation section of this paper. Please review and revise: the abstract and the Theorem 3.12 remark report the star-convex complexity as O(\varepsilon^{−3} log(\varepsilon^{−1})), while the accompanying text states O(\varepsilon^{−1/3} log(\varepsilon^{−1})).
  3. This paper should provide more experimental analysis and include more experimental scenarios, 1D parameters, embeddings, and LayerNorms. The current experimental results only include the case of 2D matrices with the parameter m>n.
1 comment
AuthorsOpenReview ↗

Clarifications on Theoretical Assumptions and Convergence Guarantees

We thank the reviewer for the comment.

1. On Assumption 2.4 and the role of κ\kappa.

Assumption 2.4 quantifies the alignment between the momentum and the final update direction. Our experiments on GPT-2 Small show that the minimum cosine similarity across parameter matrices remains positive throughout the observed training process. We agree that this does not establish a universal lower bound across different architectures or training settings. The constant κ\kappa depends on the optimization trajectory, and our convergence bounds scale with 1/κ1/\kappa. We will clarify this dependence and the scope of the assumption in the revised paper.

2. On the star-convex complexity and learning rate.

We thank the reviewer for identifying the inconsistency between the reported complexity bounds. We will correct the erroneous exponent and ensure that the abstract, theorem, and accompanying discussion are consistent. The learning rate choice η∝ϵ3\eta\propto\epsilon^3 is used to establish the theoretical complexity bound and is not intended as a recommendation for practical learning-rate selection. We will make this distinction explicit.

3. On parameter shapes.

Our analysis focuses on two-dimensional matrix parameters, which are the main parameters updated by AdaMuon. In practical training, other parameters are typically handled by AdamW, and their joint dynamics are not covered by our current results.

The restriction m≥nm\geq n is mainly used in the norm inequalities in our proofs. For example, in Lines 608–611, we use ∥Nt∥∗≤n∥Nt∥F\|N_t\|_*\leq\sqrt{n}\|N_t\|_F, which follows from the fact that the rank of NtN_t is at most nn. When $m<n$, the same argument applies with mm and nn interchanged. Thus, the analysis can be extended to this case with the corresponding changes to the dimension-dependent factors.

4. On smoothness assumptions and convergence rates.

We agree that bounded third-order derivatives should not be described as an unconditionally weaker assumption than Frobenius-norm Lipschitz smoothness. The two settings impose different regularity requirements, and we will revise the wording accordingly.

Regarding the convergence rate, the apparent difference from the O(T−1/2)\mathcal{O}(T^{-1/2}) rates commonly reported for SGD and Adam comes from the choice of indicator. When expressed in terms of the same indicator, the leading-order dependence on TT is consistent.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.