- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The success of Muon has led to the emergence of many Muon variants, and one of which is AdaMuon. AdaMuon introduces a dynamic adaptive mechanism for large-scale neural network training. This paper provides the convergence analysis of AdaMuon in three settings: the nonconvex case under and without Frobenius norm Lipschitz smoothness, and the star-convex case under Frobenius norm Lipschitz smoothness.
Critical strengths
This paper provides a convergence analysis of AdaMuon and establishes the convergence rate for nonconvex functions. It also shows that the cosine similarity between the momentum matrix and update matrix has a positive global minimum and increases monotonically during training.
Critical weaknesses
-
This paper provides a theoretical analysis for one existing Muon variant AdaMuon, however, it does not clearly explain why this particular variant was chosen. How could this paper advance the understanding of Muon (and its variants), rather than only the AdaMuon algorithm itself? The motivation and novelty of this paper seem not fully convincing.
-
The experiments mainly illustrate cosine values. The experiments are not sufficient to support the claimed improvements in empirical performance or efficiency of AdaMuon. More experiments are needed to support these claims.
Questions for authors
-
Given that some papers have already analyzed different Muon variants, why the theoretical convergence analysis of these methods remain insufficiently understood? The related work section on Muon variants does not adequately explain the exact limitations of current theoretical understanding.
-
Why does this paper focus on the variant AdaMuon? Why is this variant theoretically more important than other Muon variants? Can the analysis in this paper being extend to other variants?
-
What about training efficiency and performance improvement of AdaMuon in the experiments? How does the claimed "strong empirical performance" can be demonstrated?
Minor: The presentation of this paper could be improved, for example:
Since “AdaMuon” is an algorithm proposed in the previous paper, an appropriate reference should be provided the first time it appears (Line 13);
Many incorrectly formatted references in Section 1.1 (Line 28-38);
“To to further narrow the theoretical gap” (Line 15);
No conclusion section.
On the Motivation and Contributions of Our Work
We thank the reviewer for the comment.
1. Why AdaMuon and its connection to other Muon variants.
We focus on AdaMuon because it combines Muon's matrix orthogonalization with sign transformation and element-wise adaptive scaling. These modifications make its convergence analysis more challenging. In particular, for standard Muon, exact polar decomposition gives a direct relationship between the momentum and the update direction. This relationship no longer holds directly for AdaMuon, making it difficult to apply analyses that rely on this property.
Our work addresses this difficulty by studying the alignment between the momentum and the final update direction. We establish convergence bounds under three different settings, providing a way to analyze the effects of these additional operations. This approach may also be useful for other Muon variants that modify the orthogonalized update, provided that the corresponding assumptions hold.
2. Experimental performance and efficiency.
Our paper focuses on the theoretical convergence properties of AdaMuon rather than proposing a new optimizer or demonstrating new performance improvements. The experiments are mainly designed to examine the alignment assumption used in our analysis.
The original AdaMuon paper (Si et al., 2025) has already evaluated its training performance and efficiency across GPT-2 and Qwen2.5 models of different sizes, with comparisons against Muon and AdamW.