acceptodds
Under review as a conference paper at ICLR 2027

Provable Advantages of Muon from Data-Induced Curvature

Abstract

Muon, a matrix-orthogonalization-based optimizer, has shown strong empirical performance in large-scale language-model training, yet when and why spectral normalization yields a sustained optimization advantage remain incompletely understood. Since Muon acts on matrix-valued parameter blocks, we study a one-hidden-layer quadratic network with a single trainable matrix, which retains the matrix-gradient structure while linking input moments explicitly to parameter curvature. Motivated by empirical feature spectra, we consider Gaussian inputs with geometrically decaying centered variances and a nonzero mean whose direction is close to an active feature. In this model, data geometry induces anisotropic curvature and an imbalanced gradient spectrum, creating a regime favorable to spectral normalization. Under explicit conditions, we prove that exact-polar Muon achieves an optimization advantage over both gradient descent and SignGD, a special case of Adam's coordinatewise normalization, and that this advantage persists along the nonlinear trajectories on a fixed empirical objective. These results establish a provable mechanism linking data-induced curvature to the optimization advantage of spectral normalization.

Primary area:
optimization
Keywords:
Muon, neural network optimization, spectral normalization, data-induced curvature, optimization theory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CFLfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the conditions under which Muon's spectral normalization improves the optimization process. Using quadratic networks, the authors derive parameter curvature based on the input distribution and identify specific conditions under which Muon—in its exact-polar form—achieves greater relative progress compared to gradient descent (GD) and, under specific coordinate constructions, SignGD.

Starting from a quadratic approximation, the analysis extends this advantage to finite-step optimization of non-linear empirical loss functions, assuming appropriate local initialization. Experiments with a 124-million-parameter NanoGPT model show that, across three paired runs, Muon ultimately achieves a lower validation loss than AdamW.

However, Muon does not consistently maintain a greater relative reduction in loss compared to AdamW throughout the entire training process.

Critical strengths
  1. This paper derives the relevant curvature based on input statistics, rather than simply assuming favorable properties for the Hessian matrix. This provides a concrete explanation of how the geometric structure of the data imparts practical utility to spectral normalization.

  2. The paper is clearly structured: before presenting the main theorem, it explains the underlying mechanisms using a quadratic form example and explicitly outlines the assumptions and experimental limitations.

Critical weaknesses
  1. The theoretical comparison is against GD and SignGD rather than AdamW, establishing a finite-step advantage in loss reduction under specific conditions. Furthermore, the theoretical setting is overly restrictive and fails to establish a sufficient connection to the actual training of NanoGPT.

  2. The parameter-distance experiments do not support the local regime required by the theorem. The initialization condition implies: ∥W0−W⋆∥Fσmin⁡(W∗)<14.\frac{\|W_0-W_{\star}\|_F}{\sigma_{\min}(W_{*})}<\frac14. Using the terminal checkpoint as a reference, the minimum ratio observed across 240 nonterminal combinations was approximately 3.57. Consequently, none of the tested checkpoints satisfied this necessary proximity condition. While this result does not falsify the theorem, it weakens the proposed connection to the actual training process.

  3. The mean-alignment experiments measured alignment but failed to demonstrate that the aligned directions contribute substantially to the fitted model’s response. In the quadratic model,f(x)=∑jσj2(uj⊤x)2,f(x)=\sum_j\sigma_j^2(u_j^\top x)^2, a high cosine value must be considered together with the corresponding response strength. Although the reported peak cosines approached 1, the corresponding relative singular values were approximately 10−9–10−1110^{-9}–10^{-11}. The paper also reports that most of the mean’s projection energy lies in the fitted model’s numerical nullspace, while the largest resolved-direction cosines were only 0.000487, 0.01030, and 0.08473. These results do not establish the active-feature alignment required by the theory.

Questions for authors

No

Reviewer_Fi8dOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors compares simplified Muon with GD and SignGD on an assumed quadratic network. Under a specific input distribution, a curvature-separation condition, and a chosen balanced initialization direction (z0), the authors claims that Muon is strickly better than GD (GP retains loss fraction at least q + g, while muon is q). The authors also run experiments on NanoGPT, comparing Muon with AdamW to provide support for their theory.

Critical strengths

The paper introduces a simple model (quadratic model) that analyzes the effect of spectrum normalization of Muon. The assumption about the input distribution is reasonable, and the quadratic analysis is clean.

Critical weaknesses

I have two main concerns about the paper.

  1. Equation 3.4 assumes a canonical balanced reference direction z0, and the following advantage of Muon over GD is based on this assumption. However, the authors provide no theoretical or empirical justification for this assumption.
  2. The experimental setting cannot fully reflect what is assumed in their theory. The theory is on the quadratic model + exact polar factors, while the experiments use NanoGPT + Newton-Schulz polar approximation. This experiment-to-theory gap also exists in other aspects, like the momentum.
Questions for authors

I will consider updating my evaluation if the author fully addresses my concerns:

  1. Can the author generalize the assumption in Equation 3.4 (e.g., random directions)?
  2. Can the author provide simulation experiments that match the theoretical setting?
Reviewer_GeahOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why Muon's spectral normalization can outperform gradient-based optimizers. Using a quadratic neural network, the authors derive data-induced curvature conditions under which Muon achieves a provable finite-step advantage over GD and SignGD. Experiments on NanoGPT provide empirical observations of representation anisotropy and optimization behavior.

Critical strengths

1. Theoretical contribution. The paper establishes a connection between input data statistics, Hessian curvature, and Muon's spectral normalization.

2. Rigorous analysis. The paper derives explicit curvature conditions and proves a finite-step advantage along nonlinear optimization trajectories.

Critical weaknesses

1. Highly restrictive initialization assumptions. The theoretical guarantees require initialization extremely close to the optimum and aligned with a specially constructed Hessian-dependent direction. These conditions are unrealistic for standard LLM pretraining, and the empirical diagnostics fail to verify the required local proximity condition.

2. Insufficient empirical validation of the proposed mechanism. The strongest observed mean-direction alignment occurs in numerical nullspaces rather than active singular directions. Moreover, the paper does not directly establish the required Hessian curvature separation in real LLMs, leaving the connection between the theoretical mechanism and practical Muon performance unclear.

Questions for authors
  1. Can the authors extend their theoretical results to more general initialization conditions, particularly standard random initialization?

  2. Appendix A.4 shows that the strongest mean-direction alignment occurs in numerical nullspaces rather than active singular directions. How does this support the theoretical assumptions?

  3. Could the authors directly verify the proposed Hessian curvature separation in real LLMs?

  4. Can the theoretical analysis be extended to Muon with momentum and practical learning-rate schedules, rather than relying on exact line search?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.