Muon, an optimizer designed for matrix parameters, has recently demonstrated strong empirical performance in large language model pretraining. However, Muon's underlying mechanisms remains elusive, particularly in practical settings that combine it with other optimizers. In this paper, we study the convergence and generalization of joint Muon–Adam training for overparameterized ReLU networks with two layers. Specifically, we apply Muon to the hidden weight matrix and Adam to the output weights. We show that, with paired random initialization, the empirical loss converges exponentially to zero. Our analysis reveals and separately characterizes the contributions of Muon and Adam to convergence. Furthermore, we establish a generalization bound throughout training, which is determined by the kernel complexity of the training data and reduces to O(1/n) after sufficiently many iterations when this complexity is uniformly bounded. Our results shed light on the roles of Muon and Adam in joint neural network training from a theoretical perspective.
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.Reviewer_Z5gVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
This paper studies the convergence and generalization of joint Muon–Adam training for overparameterized two-layer ReLU networks, where Muon is applied to the hidden layer weight matrix and Adam is applied to the output weights. It is shown that, under paired random initialization, the empirical loss converges exponentially to zero. This paper characterizes the contributions of Muon and Adam to convergence separately.
Critical strengths
This paper studies the empirical loss of the overparameterized two-layer ReLU networks, trained with joint Muon–Adam with paired initialization, and establishes a generalization bound with high probability for all iterates.
Experiments on synthetic data, MNIST, and CIFAR-10 datasets are provided to verify the theoretical results.
Critical weaknesses
The scope of this paper is limited. It only analyzes the two-layer ReLU networks. This makes it unclear how the theoretical results could be extended to more practical settings, particularly those large language models where Muon has shown success. This paper provides limited insight into more complex models or practical scenarios.
The empirical validation is also limited to experiments on two-layer ReLU neural networks.
Questions for authors
This paper’s analysis is based on two-layer ReLU networks. Can the theoretical results be extended to deeper networks or more complex models, such as transformers?
Why does this paper assume paired initialization with exactly zero initial predictions? Can this assumption be relaxed?
How does this paper differ from Gronich & Vardi (2026)? Could the author provide a more detailed comparison?
This paper analyzes joint Muon–Adam training of overparameterized two-layer ReLU networks, applying Muon to the hidden weight matrix and Adam to the output weights. Under paired initialization, positive-definite kernel assumptions, sufficiently large width, and restrictive learning-rate conditions, it establishes exponential convergence of the empirical loss and a data-dependent population-loss bound.
Critical strengths
The analysis retains momentum and finite Newton–Schulz iterations, making the optimization problem more substantive than an exact-polar, momentum-free idealization. Separating the two optimizers’ contributions within a joint convergence analysis is technically interesting. Reporting both iteration count and elapsed time is also appropriate.
Critical weaknesses
1. The main result establishes convergence but provides limited insight into the distinctive benefits of Muon.
The paper operates in a strongly overparameterized regime with controlled movement around initialization, where convergence guarantees already exist for simpler optimization methods. Extending such guarantees to a new optimizer combination can be worthwhile, but the paper does not establish what qualitatively new phenomenon this extension reveals.
In particular, the convergence bound contains an Adam contribution that is already sufficient to yield exponential decay after the nonnegative Muon contribution is discarded. This does not establish convergence of a separate Adam-only trajectory, but it raises an important question: what essential role does Muon play in the guarantee beyond contributing an additional descent term?
There is no comparative theorem showing improved dependence on sample size, kernel conditioning, iteration complexity, or computational cost relative to an appropriate baseline. An additive contribution within one algorithm's bound is not, by itself, evidence of an advantage over another algorithm.
2. The assumptions substantially limit the connection to practical Muon training.
The sufficient width scales quartically with sample size, the initialization is specially paired, and the main convergence theorem uses a Muon learning rate proportional to the square root of the current loss. The generalization theorem imposes still more restrictive step sizes, ensuring very small hidden-weight movement.
These assumptions may be sufficient for the proof, but they leave a considerable gap between the analyzed dynamics and the practical motivation. The paper should explain which conclusions depend essentially on Muon's spectral transformation, rather than on generic descent and trajectory-control properties that other methods could also satisfy.
3. The generalization result does not establish a Muon-specific generalization benefit.
The statistical term is governed by Ca=y⊤Ka−1y, a complexity associated with the random ReLU feature kernel. The advertised O(1/n) rate requires this quantity to remain bounded; bounded labels alone do not suffice.
Moreover, the proof tightly restricts hidden-weight movement. The result therefore gives limited evidence about feature learning or an advantageous implicit bias induced by Muon. The paper does not show that Muon improves the relevant complexity, achieves a better bound than simpler optimizers, or reaches predictors with better population risk. The generalization theorem is meaningful as a conditional guarantee, but its explanatory value for Muon remains unclear.
4. The experiments largely demonstrate compatibility with the theory rather than test its distinctive predictions.
The experimental evidence is particularly weak relative to the paper's motivation:
Positive kernel eigenvalues on spherical synthetic data illustrate the assumptions but do not establish an optimization advantage.
Approximately geometric loss decay is consistent with the theorem but does not distinguish this method from standard algorithms in the same regime.
The generalization experiment uses only three training sizes and three seeds. Showing that test error decreases with more data does not meaningfully validate the proposed kernel-complexity dependence or the O(1/n) claim.
The experiments use width m=512, far below the scale suggested by the theorem's quartic sufficient-width requirement. They may illustrate behavior beyond the guarantee, but they do not directly validate predictions within the guaranteed regime.
As presented, the experiments would look broadly plausible under several simpler explanations. They provide little evidence that the proposed analysis captures something specific and predictive about Muon–Adam training.
Questions for authors
Which main conclusion would fail, or become quantitatively worse, if the hidden-layer update were replaced by a suitably tuned SGD update?
Can the authors provide a fair comparison demonstrating an advantage over pure Adam or Muon–SGD, including tuning and computational cost?
The paper studies the convergence rate of jointly training with Muon and Adam for two-layer ReLU networks with quadratic loss. The paper establishes that the empirical loss converges exponentially to zero. Finally, the paper proves a generalization bound throughout training.
Critical strengths
Clarity: The paper is well written and easy to follow, which is appreciated.
Relevance & impact:
The paper addresses a timely topic of developing theory to track the convergence and generalization of Muon (in combination with Adam for non-matrix-shaped weights), which is a popular optimizer and therefore warrants understanding of these. Proving fast convergence of Muon can perhaps shed light as to why Muon converges so quickly in practice, and nonetheless generalizes similarly to Adam-trained models. I also think the analysis connecting the dependence of convergence on the Newton-Schulz depth is interesting.
Soundness: While I haven't carefully checked the proofs, everything seems reasonable at its face.
Critical weaknesses
Relevance to practice: I think theory of Muon is important, but there are a few assumptions that violate what would typically be chosen during training empirically, which limits the applicability of the theory to give practical predictions:
The learning rate schedule required by Theorem 4.1 is not typical (it depends on the loss itself at each step).
The sufficient width in Theorem 4.1 is quartic in the number of training examples, which isn't typical of practice.
Impact: I'd be interested to see to what degree the theory can actually make predictions about practice.
The empirical analysis only captures the dependence on the number of Newton-Schultz iterations
Empirical analysis:
The empirical analysis is lacking (which is fine for a paper focused on its theoretical contribution).
Figure 1 shows that additional NS iterations improve performance, and SVD polar is best, except it doesn't actually say anything about the rate (e.g., via an empirical estimate of the asymptotic behavior), which is what the theory is trying to show.
Figure 2 doesn't seem to contribute anything other than showing that test loss goes down with n.
As a suggestion, I think the details about the learning rate schedule would be interesting to investigate empirically.
In addition, testing the theory when you break the assumptions (e.g., use a much smaller-width network than the theory requires) would also be useful to back up your theoretical contributions with something beyond what can be stated with theory.
Questions for authors
Theorem 4.1 requires $$\eta_t$ = \eta \sqrt{2 L(\theta_t)}$ — could you elaborate on why this learning rate schedule is chosen? Also, should this learning rate schedule be chosen in practice, empirically?
From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.