acceptodds
Withdrawn from ICLR 2027

The Spectral Dynamics and Noise Geometry of Muon

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Muon replaces a matrix gradient \(G=U\Sigma V^\top\) by its polar factor \(UV^\top\). This keeps the singular directions selected by the gradient, but makes the update spectrum flat. We study the optimization bias created by this operation. In an underdetermined regression model, we derive exact singular-value dynamics for a projected polar flow and identify a measurement-dependent condition under which the normalized spectrum moves toward equal nonzero singular values. At fixed Frobenius norm, the pairwise spectral functional is minimized by a flat spectrum, whereas nuclear-norm minimization favors spectral concentration. Controlled matrix-sensing experiments separate the effect from simple gradient rescaling, show that norm-matched gradient descent does not reproduce Muon, and recover the predicted flattening trend across broad ablations. In small NanoGPT pretraining, Muon preserves stable rank, has a broad learning-rate plateau, and improves validation loss relative to AdamW; in a matched small-ViT control, the ranking reverses. The resulting picture is regime-dependent: Muon is not universally superior, but its flat-spectrum bias can help when many spectral directions need to remain active.

Primary area:
optimization
Keywords:
Muon, Training Dynamics, Pretraining Theory, Implicit Regularization, Matrix Completion

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Ai4NOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The manuscript is so confusing that summarizing it thoroughly would require a significant effort. Therefore, I will restrict my summary to the part I tried to understand: motivated by the recent success of Muon, this work aims to explain the implicit bias of Muon through the analysis of the dynamics of a projected polar flow under a linear regression model.

Critical strengths

None

Critical weaknesses

The manuscript is almost unreadable to me, and its main claim seems either tautological or insufficiently supported, as elaborated below.

  1. The first motivation for this study comes from the success of the Muon optimizer. In the second sentence of the manuscript, it is defined as the polar factor of the gradient. This does not match the definition given by Jordan et al. (2024), which use momentum and the Newton–Schulz approximation. This is particularly confusing since, in Section 3, one refers to a momentum buffer mtm_t that is not defined. It is difficult to assess claims about algorithms that are not defined consistently.

  2. The projected polar flow seems to be a new algorithm introduced in this work. It is unclear why this algorithm is relevant to understanding Muon's properties. One motivation to replace the gradient GG with the weights WW is the SF condition under which P(G)=P(W)P(G)=P(W). But we have P⊥P(G)=0P_\perp P(G)=0, so under this condition the projected polar flow is constant... Also, if the goal is to study what interpolating solution is reached by Muon on a regression problem, using at each step a projection on the interpolating step considerably changes the problem.

  3. The motivations of this work are very confusing. In the introduction, to explain the problem they are addressing, the authors wrote

Muon has become a practical optimizer for transformer training (Liu et al., 2025), but empirical acceleration alone does not explain what the polar map selects. Two natural explanations are tempting. One is that Muon is just matrix-gradient normalization; another is that spectral-norm geometry should lead to nuclear-norm, hence low-rank, implicit bias. We show that both are incomplete. Muon is better understood as a flat-update-spectrum method: it keeps the gradient singular directions while allocating equal update amplitude across them.

Where do these "natural" explanations come from? Are they coming from the authors or previous work? Also, it is unclear what matrix normalization refers to, and why this alone would explain the interpolator solution reached by Muon (should one guess that it would be the same as GF since "matrix normalization" would only induce a reparametrization of GF? But Muon is clearly not a time-reparametrized version of GF...). The second explanation is also unclear to me. Why should Muon reach a minimal nuclear norm interpolant? This would be an implicit bias theorem and not trivial. It is unclear to me how the previous statement explains anything. More importantly, the new interpretation provided by the authors: "Muon is better understood as a flat-update-spectrum method" is essentially tautological and not related to the previous problem of characterizing the reached interpolant.

  1. The first stated contribution is "A one-step spectral-bias theorem. Under explicit alignment assumptions, we prove that the polar profile maximizes first-order spectral-entropy gain among bounded updates that follow the gradient singular directions and ignore the current weight spectrum". I didn't find any corresponding result in the main text.
Questions for authors

According to Theorem 1, i) σi\sigma_i cannot increase; isn't it in contradiction with the left part of Figure 2, where σ3\sigma_3 and σ4\sigma_4 seem to increase?

Reviewer_EGwQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the spectral bias and noise geometry of Muon. The central motivation is that Muon replaces a matrix gradient G=UΣV⊤G=U\Sigma V^\top by its polar factor UV⊤UV^\top, preserving the singular directions while assigning equal magnitude to all active directions.

The main theoretical object is a projected self-polar flow W˙=−ηP⊥P(W)\dot W=-\eta P_\perp P(W) on the interpolation manifold M\mathcal M, where P⊥P_\perp projects onto the tangent space of M\mathcal M, and P(W)P(W) is the polar factor of the weight matrix. This paper derives exact singular-value dynamics for this flow, a measurement-dependent sign criterion for changes in normalized spectral shape, and a fixed-radius variational characterization in which a pairwise spectral functional is minimized by a flat nonzero spectrum.

This paper also provides local perturbation results relating the self-polar flow to a projected gradient / momentum polar flow, a finite-step approximation, and a linearized critical-batch-size formula based on the Fréchet derivative of the polar map. Experiments compare Muon with minimum-nuclear-norm solutions in matrix sensing and examine spectral entropy in small language, vision, and NanoGPT.

Critical strengths
  1. This paper asks an interesting and timely question.

Understanding what spectral transformation Muon induces beyond generic optimization-rate analyses, is important given its growing use in language-model training.

  1. The distinction between spectral direction and spectral magnitude is useful.

This paper correctly emphasizes that Muon is not simply norm-matched gradient descent. It actually preserves singular directions but removes their relative strengths.

  1. The singular-value calculation for the projected self-polar flow is clean.

The identity σ˙i=−η∣P⊥ui∣2\dot\sigma_i=-\eta|P_\perp u_i|^2 gives a transparent connection between measurement geometry and spectral evolution.

  1. The paper carefully separates flat-spectrum geometry from nuclear-norm geometry.

At fixed Frobenius norm, the proposed pairwise functional favors equal nonzero singular values, whereas nuclear-norm bias favors spectral concentration. The controlled matrix-sensing experiments are consistent with Muon on not selecting the minimum-nuclear-norm interpolant.

Critical weaknesses
  1. The main theorem analyzes a surrogate flow rather than Muon.

Literal Muon follows the polar factor of the gradient or momentum W˙=−ηP(G),G=∇L(W),\dot W=-\eta P(G), G=\nabla L(W), while Theorem 1 studies W˙=−ηP⊥P(W)\dot W=-\eta P_\perp P(W). The replacement of P(G)P(G) by P(W)P(W) is substantial. On the interpolation manifold, the literal gradient is zero, so gradient-driven Muon stops, while the analyzed self-polar flow generally continues to move. Theorem 3 attempts to bridge the two dynamics by assuming that δP=∣∣P(G)−P(W)∣∣F\delta_P=||P(G)-P(W)||_F is small. However, the paper does not prove that this condition arises during Muon training, and does not prove that the condition persists. Also, the result is only local in time. The reported singular-subspace angle is insufficient because relative left/right singular-frame gauge also affects the polar factor, as the paper itself acknowledges.

  1. The paper does not establish that the spectrum flattens.

Theorem 1 gives an exact sign criterion dR~pwdt≤0⇔∑iαiqi≤0.\frac{d\widetilde R_{\rm pw}}{dt}\leq 0\Leftrightarrow\sum_i\alpha_iq_i\leq0. This is a characterization of the derivative, but the paper does not establish broadly applicable conditions under which the inequality holds along the trajectory.

  1. The main matrix-sensing experiments do not directly validate Theorem 1.

Theorem 1 is exact for the special left-multiplication regression structure XW=YXW=Y, where the tangent projector acts only on the left singular vectors and cross-terms vanish. The experiments instead use a general Gaussian affine sensing operator A(vec⁡W)=y.\mathcal A(\operatorname{vec}W)=y. In this setting, the paper notes that off-diagonal cross-terms appear in the singular-value dynamics. The experiments are therefore only qualitative sanity checks, not direct validation of the theorem. This mismatch is important because the empirical flattening behavior could be caused by precisely the additional terms excluded from Theorem 1.

  1. The local bridge assumption is not empirically verified

Theorem 3 depends on gauge-invariant polar misalignment δP\delta_P, but the experiments only report a subspace-angle proxy. The manuscript explicitly states that these angles do not control δP\delta_P, and direct δP\delta_P trajectories are not included. Therefore, the experiment intended to connect the surrogate dynamics to actual Muon does not verify the theorem’s central hypothesis. Measuring δP\delta_P directly appears essential.

Questions for authors
  1. Can the authors identify a natural class of measurement geometries for which ∑iαiqi≤0\sum_i\alpha_iq_i\leq0 is guaranteed along the trajectory, rather than merely checked after the fact?

  2. Why should P(G)P(G) remain close to P(W)P(W) during Muon training? Can this be proved in the regression model or measured directly in the experiments?

  3. Can the authors provide direct trajectories of ∣∣P(Gt)−P(Wt)∣∣F||P(G_t)-P(W_t)||_F for the matrix-sensing and neural-network experiments?

  4. Can Theorem 1 be reformulated directly for a momentum-driven polar flow without replacing P(G)P(G) by P(W)P(W)?

Reviewer_BzYSOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors investigate polar flow, the continuous-time version of Muon, and derive several theoretical results on its dynamics. Experiments show that the polar flow dynamics may differ from those of Muon, in particular concerning nuclear norm minimizers.

Critical strengths

I could not identify any strengths in this paper. The writing is extremely difficult to read, to the point that it is not discernable what the main results are or what they are trying to say.

Critical weaknesses

The presentation of the paper needs to be significantly improved. It is possible that there are genuine contributions, but the present manuscript contains vague or contrived writing and introduces many unjustified quantities and assumptions in its theoretical results. As such, it is difficult to determine whether this work has potential impact, even if the results are assumed to be correct.

Questions for authors
  1. Could you improve the overall presentation, including better descriptions and interpretations of the main results?
  2. What is meant by "many spectral directions must remain active" and "the useful spectrum is already low-dimensional or well conditioned" in the takeaway?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.