acceptodds
Under review as a conference paper at ICLR 2027

Implicit Bias of Muon with Finite Newton–Schulz Iterations

Abstract

Muon has recently attracted growing interest as an effective optimizer for large-scale neural network training. However, the theoretical mechanisms underlying Muon, especially the role of Newton–Schulz (NS) iterations in practice, are not yet fully understood. We focus on Muon’s implicit bias with finite NS iterations in multi-class classification. We characterize the gap to the optimal spectral margin through a margin bound, which decomposes the gap into a vanishing training term and an additional path-weighted distortion arising from NS iterations. Persistent distortion can leave a nonvanishing margin gap, whereas vanishing weighted distortion guarantees recovery of the optimal spectral margin. This characterization clarifies when finite NS iterations alter or preserve Muon’s spectral implicit bias, providing a principled understanding of their role in practical Muon implementations. Finally, extensive experiments across synthetic and real-data settings validate our theoretical findings.

Primary area:
learning theory
Keywords:
Muon, Newton–Schulz, Implicit Bias, Classification

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_A4Q8OpenReview ↗
Rating
3(−1)
Confidence
3(−1)
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies the classifier selected by Muon with finitely many Newton–Schulz iterations on separable multiclass linear data. Its main bound connects the spectral-margin gap to directional error accumulated along the update path. A fixed-map norm construction yields datasets with persistent fixed-depth bias, and increasing-precision schedules recover the spectral max margin. Experiments compare exact SVD, fixed depths, and calibrated or adaptive depth using certified margin references. Additional neural experiments examine finite-time depth sensitivity.

Critical strengths
  1. The solution-selection result is meaningful and technically convincing. Lemma 4.6 identifies a norm whose supporting direction matches the finite spectral map, and Proposition 4.2 constructs a persistent margin gap of order (3/2)−q(3/2)^{-q}. This gives a concrete account of how finite depth changes implicit bias. Theorem 4.1 and the precision corollaries complement the lower bound with quantitative recovery guarantees.

  2. The validation is closely matched to the theory. The 24 paired linear datasets use certified spectral-margin references, with interval widths below 9.83×10−89.83\times10^{-8}. Table 1 reports median margin ratios of 0.9216 at depth 1, 0.9575 at depth 4, and 0.9680 for exact SVD. Directional-error measurements, horizon sweeps, equal-polynomial-computation comparisons, and norm-matched controls make the observed depth effect convincing. The proof audit supports the stated asymptotic claims.

Critical weaknesses

The main limitation is practical scope. The theory covers cubic and admissible Taylor maps after exact Frobenius normalization. Common implementations use a tuned quintic, a normalization stabilizer, and lower precision. Appendix A acknowledges these differences, and its quintic experiment uses Taylor coefficients. A fixed stabilizer can matter when momentum becomes small, which is relevant to asymptotic solution selection. This limits direct transfer to a particular training implementation. The stated theoretical contribution remains sufficient for clear acceptance.

Questions for authors

I have no unresolved questions that would materially affect my recommendation on the current evidence. The central claims are adequately supported, and the manuscript discloses its main implementation restrictions. Further discussion of the tuned polynomial or a normalization stabilizer is optional.

1 comment
AuthorsOpenReview ↗

Response to Reviewer A4Q8

Thank you for your positive evaluation and feedback. Your comments helped connect our results to practical Muon through the tuned quintic, additive stabilization, and lower precision.

1. Do our results hold for the tuned quintic polynomial? (W1)

Under the response bounds below, we can extend Thm. 4.1's pathwise bound to Jordan's tuned quintic P(s)=3.4445s−4.7750s3+2.0315s5P(s)=3.4445s-4.7750s^3+2.0315s^5 [1]. Yet it leaves a spectral-margin gap at fixed depth (cf. Prop. 4.2) in our three-class construction, and increasing qq alone cannot ensure Thm. 4.3's recovery guarantee.

  • Even without monotonicity, Thm. 4.1 holds under weaker assumptions. In the original setup, Thm. 4.1's pathwise bound and loss convergence hold if the response satisfies ψt(0)=0ψ_t(0)=0 and ms≤ψt(s)≤Kms≤ψ_t(s)≤K on [0,1][0,1], with fixed 0<m≤K<∞0 < m≤K < ∞. For unit-Frobenius inputs of rank ≤p≤p, the top singular value yields m/p≤∣Dt∣op≤Km/\sqrt{p}≤\\|D_t\\|_{\mathrm{op}}≤K, while shared singular vectors and ∑isi2=1\sum_i s_i^2=1 give ⟨Mt,Dt⟩/∣Mt∣\*≥m/p\langle M_t,D_t\rangle/\\|M_t\\|_\*≥m/\sqrt{p}. Jordan's quintic meets these bounds with (m,K)=(3/5,5/4)(m,K)=(3/5,5/4) for all q≥1q≥1, extending Thm. 4.1 with new constants.
  • The Jordan quintic can exhibit a spectral-margin gap at fixed depth (cf. Prop. 4.2). Our three-class example with symmetric initialization and square-root steps has vanishing loss and a limiting spectral-margin gap of at least 85/(11642)85/(1164\sqrt2) at any fixed q≥1q≥1.
  • Jordan quintic oscillations show that increasing depth alone cannot ensure Thm. 4.3’s recovery guarantee. We construct a unit-Frobenius rank-two input whose singular-value responses oscillate, giving δqM≥5/97δ_q^M≥5/97 for all q≥1q≥1. Thus, depth alone cannot ensure a defect below 5/975/97 uniformly over inputs of rank at most pp (p≥2p≥2), so Thm. 4.3(i)'s premise may fail. For Thm. 4.3(ii), depth alone cannot ensure DT→0\mathcal D_T→0, which remains sufficient for recovery.

2. How does practical normalization affect our results? (W1)

We can extend Thm. 4.1's pathwise bound and Thm. 4.3's (calibrated) recovery guarantee for cubic NS to the additive stabilizer Xt,0=Mt/(∣Mt∣F+ϵ)X_{t,0}=M_t/(\\|M_t\\|_F+\epsilon) (ϵ>0\epsilon > 0, set to 10−710^{-7} in [1]) under the conditions below. At fixed depth (cf. Prop. 4.2), small momentum makes the stabilized update nearly linear and shifts selection to the Frobenius max-margin direction.

  • With additive stabilization, Thm. 4.1's pathwise bound and Thm. 4.3's (calibrated) recovery guarantee hold. Let ηt=η0(t+1)−a\eta_t=\eta_0(t+1)^{-a} (η0>0\eta_0 > 0, 0<a≤10 < a≤1). For fixed odd polynomials with ms≤ψ(s)≤Lψsms≤ψ(s)≤L_ψs on [0,1][0,1] (m>0m > 0, Lψ<∞L_ψ < ∞), the spectral-margin gap is bounded by γ\*DT+O(1/log⁡HT)\gamma_\*\mathcal D_T+O(1/\log H_T), with HT=1+∑t<TηtH_T=1+\sum_{t < T}\eta_t. For cubic NS with ht=∣Xt,0∣Fh_t=\\|X_{t,0}\\|_F and rank at most pp, depth O(log⁡(p/(htρt)))O(\log(p/(h_tρ_t))) ensures δtM≤ρtδ_t^M≤ρ_t when 0<ρt≤1/20 < ρ_t≤1/2. With square-root steps, horizon calibration at ρT≍(1+log⁡T)/Tρ_T\asymp(1+\log T)/\sqrt T and adaptive calibration at ρt=1/(t+2)ρ_t=1/(t+2) give a spectral-margin gap of O((1+log⁡T)/T)O((1+\log T)/\sqrt T). At fixed rank, our pre-run bound on hth_t yields sufficient depth O(T+log⁡T)O(\sqrt T+\log T).
  • At fixed depth (cf. Prop. 4.2), additive stabilization can shift selection to the Frobenius max-margin direction. As rt=∣Mt∣F→0r_t=\\|M_t\\|_F→0, Dt=(Λq/ϵ)Mt+o(rt)D_t=(\Lambda_q/\epsilon)M_t+o(r_t), with Λq=(3/2)q\Lambda_q=(3/2)^q for cubic NS and (3.4445)q(3.4445)^q for Jordan's quintic. With this stepsize, fixed odd polynomials obeying these bounds select the Frobenius max-margin direction, rather than ideal Muon's spectral max-margin direction. In another three-class example, the limiting spectral-margin gap is 2/(92)2/(9\sqrt2) at any fixed cubic or positive Jordan depth.

3. Do our results still hold under lower precision? (W1)

BF16 rounding can change alignment and path weights (cf. Thm. 4.1) and BF16 defects can exceed Thm. 4.3(i)'s tolerance. Motivated by [2, 3], we compare BF16 and FP64 NS on identical inputs with FP64 outer states.

  • BF16 diagnostics. For relative spectral-norm error ν<1\nu < 1, the defect relative to the NS input is at most (δ+2ν)/(1−ν)(δ+2\nu)/(1-\nu), where δδ is its exact-arithmetic value. In paired five-round Jordan runs with casting before additive normalization, BF16 yields spectral-margin gaps of 0.02311–0.02772 versus 0.01675–0.01776 for FP64. Only 12.2%–12.5% of the last 2,500 BF16 steps meet the sufficient relative Frobenius error threshold. In matched-input tests with homogeneous normalization and 24 cubic NS rounds, the defect exceeds 10−310^{-3} in 8/32 BF16 and 0/32 FP64 cases. Thus, asymptotic spectral-margin recovery under BF16 remains open.

References

[1] Jordan et al. (2024). Muon: An optimizer for hidden layers in neural networks.

[2] Nakatsukasa & Higham (2012). Backward Stability of Iterations for Computing the Polar Decomposition. SIMAX.

[3] Amsel et al. (2026). The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm. ICLR.

Reviewer_XwoFOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies what is the spectral margin of Muon, when using a finite number of Newton-Schulz steps for computing the polar, and applied to multi-class linear classifier. They identify a quantity they call "path weighted defect" that depends on the error of Newton-schulz and vanishes to zero for exact Muon. But their construction relies on the classical Newton-Schulz method (or a generalization to Taylor approximations) that does not hold for either of the methods used for approximating the polar factor in Muon. As such, either the story of the paper needs to change, or the theory needs to accommodate one of the practical polynomials used by Muon.

Furthermore, I don't see how this theory could be actionable in any sense. Though the experiment section apparently leverages the theory to choose the number of polynomial steps used in approximating the polar factor, I could not understand how exactly they leverage this theory.

Critical strengths

The analysis of characterizing the additional gap to the optimal spectral margin seems interesting, in particular how it is controlled only by the Newton-Schulz error, and how this error can be amortized over iterations.

Bibliography is reasonable, including even very recent related work.

Critical weaknesses

You focus on the actual Newton-Shulz iteration (4), but this has never been used with Muon. Instead, Jordan et all developed a "zeropower_via_newtonschulz5" method which has a different polynomial (it is not a Taylor-NS polynomial). This was the original method used in the nanogpt speed runs. Later Amsel et al develop a sequence of optimal polynomials (polar express), which is now used in modded nanogpt speeds runs. Neither the zeropower_via_newtonschulz5 method nor polar express are admissible according to your definition. In particular, both polynomials are no strictly increasing, and it is important they are not. Because a polynomial that approximates the constant 1 function has to oscillate around 1 at equal distances (the equioscillatoin theorem). Furthermore these polynomials all exceed 1 at their max (again by design). One advantage you may be able to leverage if you switch to analysing polar express, is that the resulting composition of polynomials converges at least quadratically to the polar factor. 

Another weakness is the paper is hard to read, and the writing degrades further into the paper. There are many sentences that could be cut (that don't say anything), or lost objects, statements lacking context and more.

Questions for authors

Questions: 

Q1. Doesn't the loss convergence in 202 depend on the learning rate? 

Q2. Line 229 "Early coarse updates becomes less important as training continues" why is that ? According to the definition in (5), early coarse updates are equally important as late coarse updates. I understand the denominator shrinks as T grows, but your wordings makes it seem there is a difference between earlier and later polar approximations. If anything, earlier in training the learning rates tend to be larger (excluding the warm-up), so earlier coarse updates will weigh more heavily on DTD_T as compared to later coarse updates.

Q3. I'm confused by the lower bound on qTq_T on line 285-286. This lower bound depends on ρT\rho_T, which in turn depends on qTq_T. So both sides of this lower bound depend on qTq_T, is that right? Or am I missing something here?

Now some more minor points: 

0. I read the abstract, and couldn't understand the contribution. What type of implicit bias? This term is thrown around as if it had some explicit meaning. I think part of the issue is that the contribution itself is not that clear. 

  1. Line 067 "linear experiments" I think you mean experiments with a linear classifier. Also the sentence "while fully connected neural netwoks and Transformer diagnostics examine depth effects beyond linear classification" does not say anything. It's as informative as saying "We did experiments with deep learning".

  2. Eq (5), minor notation issue, you already used eie_i to denote a unit coordinate vector, so a little confusing using ete_t here to denote an error.

  3. Line 306: "The amplitude satisfies " don't you mean spectral norm, or at least, magnitude? Amplitude is more for signals and time series. Also "The defect obeys" here you mean "dual alignment" which is what you called this earlier. For mathematical writing, you should repeat the exact same name you gave these quantites before.

  4. Lines 311 to 317: This paragraph is too hard to follow, there are lost object, and unclear statements. For instacne "The NS map does not decrease these values", is confusing considering the previous sentence mentions an inequality max⁡jsj≥p−1/2\max_j s_j \geq p^{-1/2}, do you mean this inequality still holds have the NS map? Furthermore "the defect denominator is at least one", this "defect" is not a simple fraction, so there is no clear denominator. Do you mean after completing the fraction, or do you mean just the fraction term? "The envelope" what do you mean by envelope here? This is just an upper bound for all s∈[p−1/2,1]s \in [p^{-1/2}, 1]. Finally "There are at most p−1p-1 such terms" this I couldn't guess what you mean. 

  5. I wanted to check the proofs of the statements in the main text, but it was too hard to navigate the appendix. In particular, the propositions and lemmas are not restated in the appendix, and it's not clear where to find their proofs. For instance Lemma 4.4, the text indicates the proof is in Appendix B.4. In this appendix I see different statements, notation, and results, and couldn't figure right away where was the proofs of Lemma 4.4? This needs to be re-organized and made clearer. Though I should say the appendix appears to be sufficiently well written.

6 comments
AuthorsOpenReview ↗

Response to Reviewer XwoF (Part 1/2)

Thank you for your careful assessment of our theory and presentation. Your comments helped us clarify the scope of the polynomial analysis, the depth-selection rule, and the links between the main results and their proofs.

1. Do our results hold for practical non-monotonic polynomials (Jordan's quintic and Polar Express)? (W1)

We can extend Thm. 4.1's pathwise bound to Jordan's quintic [1] (e.g., as used in zeropower_via_newtonschulz5) and the frozen five-stage Polar Express sequence [2] under the weaker assumptions below, which allow nonmonotonicity and overshoot above 11. Both can still exhibit a spectral-margin gap at fixed depth (cf. Prop. 4.2). While increasing qq alone cannot ensure Thm. 4.3’s recovery guarantee for Jordan's quintic, ideal Polar Express retains this guarantee through joint calibration, taking advantage of the fast convergence you highlighted.

First, we establish two positive extensions covering Thm. 4.1's pathwise bound for Jordan's quintic and frozen five-stage Polar Express, and Thm. 4.3's (calibrated) recovery guarantee for ideal Polar Express.

  • Even without monotonicity or a unit upper bound, Thm. 4.1 holds under weaker assumptions. In the original training setup with homogeneous normalization and exact arithmetic, it suffices that the full singular-value response ψt\psi_t satisfy ψt(0)=0\psi_t(0)=0 and ms≤ψt(s)≤Kms\le\psi_t(s)\le K for 0≤s≤10\le s\le1, with fixed 0<m≤K<∞0 < m\le K < \infty across updates. These bounds control the update norm and momentum alignment. Jordan's tuned quintic P(s)=3.4445s−4.7750s3+2.0315s5P(s)=3.4445s-4.7750s^3+2.0315s^5 [1] satisfies them with (m,K)=(3/5,5/4)(m,K)=(3/5,5/4) at every positive composition depth. The frozen five-stage Polar Express sequence satisfies them with (m,K)=(0.000324,1.131)(m,K)=(0.000324,1.131). The loss still vanishes, and Thm. 4.1's pathwise bound holds with modified constants.

  • Ideal Polar Express retains Thm. 4.3's recovery guarantee when its design interval and stage count are calibrated jointly. As you noted, ideal Polar Express converges at least quadratically on its design interval [ℓ,1][\ell,1], with cubic convergence for quintic stages [2]. Let εq\varepsilon_q be the maximum error from 11 on this interval after qq stages. For unit-Frobenius inputs of rank at most pp and 0<ℓ≤1/p0 < \ell\le1/\sqrt p with ℓ<1\ell < 1, the momentum-relative defect satisfies δqM≤2εq/(1+εq)+(p−1)ℓ\delta_q^M\le 2\varepsilon_q/(1+\varepsilon_q)+(p-1)\ell. The second term bounds the contribution of singular values below ℓ\ell. In exact arithmetic, choosing ℓ\ell and qq together makes the defect as small as needed for Thm. 4.3(i). Choosing them during training so that the path-weighted defect DT\mathcal D_T vanishes gives recovery in Thm. 4.3(ii).

Next, we show that both maps can still exhibit a spectral-margin gap at fixed depth (cf. Prop. 4.2) and explain how oscillations in Jordan's quintic can limit the applicability of Thm. 4.3's recovery guarantee.

  • Jordan's quintic and Polar Express can still exhibit a spectral-margin gap at fixed depth (cf. Prop. 4.2). For Jordan, a separable three-class construction with prescribed symmetric initialization has vanishing loss and a limiting spectral-margin gap of at least 85/(11642)85/(1164\sqrt2) at every fixed positive depth. For ideal Polar Express, each fixed positive stage count and admissible rank-two design interval admits a separable three-class example with prescribed symmetric initialization, vanishing loss, and a positive limiting spectral-margin gap. A separate construction for the frozen five-stage sequence gives a spectral-margin gap of at least 1/(52)1/(5\sqrt2).

  • Increasing depth alone may not yield Thm. 4.3's recovery guarantee for Jordan's quintic. We construct a unit-Frobenius rank-two input whose singular-value responses oscillate, giving δqM≥5/97\delta_q^M\ge5/97 for all q≥1q\ge1. Thus, for p≥2p\ge2, increasing qq alone cannot guarantee the uniform tolerance in Thm. 4.3(i) when ρT<5/97\rho_T < 5/97. For Thm. 4.3(ii), increasing depth alone does not ensure DT→0\mathcal D_T\to0, but the recovery guarantee still holds if DT→0\mathcal D_T\to0.

References

[1] Jordan et al. (2024). Muon: An optimizer for hidden layers in neural networks.

[2] Amsel et al. (2026). The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm. ICLR.

AuthorsOpenReview ↗

Response to Reviewer XwoF (Part 2/2)

Thank you for highlighting the clarity and readability issues in the latter part of our paper. We address your comments in two sections below: learning rates, path weights, and depth calibration (Q1–Q3), followed by exposition, notation, and proof navigation (W2; Minor 0–5). We will incorporate these changes into the revised manuscript.

2. Clarifying learning rates, path weights, and depth calibration (Q1–Q3)

  • The learning rate affects the loss convergence rate on Line 202(Q1). For ηt=η0/t+1η_t=η_0/\sqrt{t+1} with η0>0η_0 > 0, Appendix B.4 gives L(WT)≤C0exp⁡(−C1T+1)\mathcal L(W_T)≤C_0\exp(-C_1\sqrt{T+1}) for sufficiently large TT. Here C1=γ\*η0/(2p)C_1=\gamma_\*η_0/(2\sqrt p), with optimal spectral margin γ\*\gamma_\* and rank bound pp. In the proof, the cumulative stepsize Θ(η0T)\Theta(η_0\sqrt T) eventually dominates the squared-step remainder O(η02log⁡(T+1))O(η_0^2\log(T+1)), with timing depending on η0η_0. We will clarify this in Section 4.1.

  • How step weights and fixed prefixes affect the bound (Q2). Eq. (5) weights step tt by at/ATa_t/A_T, where at=ηt∣Dt∣opa_t=η_t\\|D_t\\|_{\mathrm{op}} and AT=∑t<TatA_T=\sum_{t < T}a_t. Early steps can weigh more if their realized step lengths are larger. We will clarify the fixed-prefix limit of Eq. (6) on Line 229.

    γ\*∑t<matδtM∣W0∣op+AT→0.\frac{\gamma_\*\sum_{t < m}a_tδ_t^M}{\\|W_0\\|_{\mathrm{op}}+A_T}→0.

    Here δtMδ_t^M is the momentum-relative defect (formerly ete_t). Along one run with fixed mm, the numerator is constant and AT=Ω(T)A_T=\Omega(\sqrt T) grows.

  • NS depth can be chosen before training (Q3). We first specify the tolerance ρTρ_T independently of qTq_T, and then choose the NS depth qTq_T from the resulting bound. We will clarify this in Section 4.2 as follows:

    1. Fix the horizon TT, rank bound pp, and tolerance ρT>0ρ_T > 0.
    2. For cubic NS and p>1p > 1, choose q≥max⁡0,⌈log⁡((p−1)/(eρT))/log⁡(3/2)⌉q≥\max\\{0,\lceil\log((p-1)/(\mathrm eρ_T))/\log(3/2)\rceil\\}. For p=1p=1, q=0q=0 suffices under homogeneous normalization.
    3. Train at the chosen fixed depth, which ensures DT≤ρT\mathcal D_T≤ρ_T by Eq. (5).

    At T=20000T=20000, ρT=(1+log⁡T)/T≈0.0771ρ_T=(1+\log T)/\sqrt T≈0.0771 gives q=4q=4 for p=2p=2 and q=7q=7 for p=4p=4 (Appendix A.1.3), matching our fixed-q=4q=4 run for p=2p=2.


3. Exposition, clarity, and proof navigation (W2; Minor 0–5)

  • Clarifying implicit bias in the abstract (Minor 0). We will define Muon’s implicit bias through the normalized spectral margin selected as loss vanishes in separable multiclass linear classification and summarize the pathwise bound, fixed-depth bias, and precision-based recovery in those terms.

  • Clarifying the neural experiments (Minor 1). On Line 67 we will use "experiments with linear classifiers" and describe the norm-matched control in Appendix A.2, which rescales each nonzero NS update to unit spectral norm. From one to five cubic rounds, mean training cross-entropy after 120 minibatch updates drops from 0.0322 to 0.000189 (fully connected) and 1.777 to 1.039 (patch Transformer) over five paired seeds (Table 4). The Transformer's lower-tail training margin also falls from one to five rounds. These diagnostics complement Fan et al.'s linear analysis [3].

  • Clarity of symbols in Eq. (5) (Minor 2). We distinguish ec\mathbf e_c for the coordinate vector from δtMδ_t^M for the momentum-relative defect.

  • Clarity of spectral terminology (Minor 3). We will replace "amplitude" with spectral norm ∣D∣op\\|D\\|_{\mathrm{op}} and distinguish momentum alignment ⟨M,D⟩/∣M∣\*\langle M,D\rangle/\\|M\\|_\* from the relative dual-alignment defect 1−⟨M,D⟩/(∣M∣\*∣D∣op)1-\langle M,D\rangle/(\\|M\\|_\*\\|D\\|_{\mathrm{op}}).

  • Clarifying the spectral proof in Lemma 4.4 (Minor 4). We will clarify the denominator and term count. Let sj=σj(M)/∣M∣Fs_j=\sigma_j(M)/\\|M\\|_F be the positive normalized singular values, ψq=ϕ∘qψ_q=\phi^{\circ q}, and c=max⁡jψq(sj)c=\max_jψ_q(s_j). Since ψq(sj)≥sjψ_q(s_j)≥s_j, we have c≥max⁡jsj≥p−1/2c≥\max_j s_j≥p^{-1/2}. The defect is

    \qquad c\sum_j s_j≥(\max_j s_j)\sum_j s_j≥\sum_j s_j^2=1.$$ An index attaining $c$ contributes zero, leaving at most $p-1$ terms. Since $c≤1$, each satisfies $s_j(c-ψ_q(s_j))≤s_j(1-ψ_q(s_j))≤s_j(1-s_j)^\Lambda≤1/(\mathrm e\Lambda)$, with $\Lambda=(3/2)^q$.
  • Proof navigation for Lemma 4.4 (Minor 5). Appendix B.1 points to supporting results and proofs. Lemma 4.4 now links to Prop. B.11's proof in B.4, with S=MS=M, r=rank⁡(M)≤pr=\operatorname{rank}(M)≤p, ψ=ϕ∘qψ=\phi^{\circ q}, and Λ=(3/2)q\Lambda=(3/2)^q.

Presentation and readability improvements. Thank you for the readability feedback. We will clarify the setup and proof sketches and make the appendix proofs easier to find.

References

[3] Fan, Schmidt, and Thrampoulidis (2025). Implicit Bias of Spectral Descent and Muon on Multiclass Separable Data. NeurIPS.

Reviewer_XwoFOpenReview ↗

This response requires a new revision

Dear Authors,

I appreciate the efforts, but your new section D in response to my and reviewer gQrf question on using practical and non-monotonic polynomials is an additional 11 pages of proof, which are very hard to read (I sampled pieces and couldn't understand). This is longer than a ICLR paper. If you need to add 11 pages, then your paper needs a new round of reviews.

Reviewer_gQrfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates the effect of Newton-Schulz iteration on Muon dynamics. The authors find that using finite Newton-Schulz iteration for orthogonalization can have an effect on the implicit bias of Muon compared to its exact version. Several experiments provide supporting evidence for the theoretical results.

Critical strengths

This paper is generally well-written and easy to read. I have checked the proofs of the theoretical results, and they appear to be mostly correct. The results are interesting and novel to the best of my knowledge.

Critical weaknesses

I have some concerns regarding the scope and significance of the paper's results, given that it is motivated by the gap between practical Muon and exact orthogonalization methods. In particular,

  • The main theory uses ϕ(s)=32s−12s3\phi(s)=\frac32s-\frac12s^3, or more generally odd polynomials ψ\psi that are strictly increasing on [0,1][0,1] and satisfy s≤ψ(s)≤1s\leq\psi(s)\leq1 for all s∈[0,1]s\in[0,1]. This is in contrast to the NS polynomials used in practice, which typically are not monotone. This property is used in the proof of Proposition B.11 in Lines 1590, so the results based on Proposition B.11 would not directly transfer to the fully practical setting.
  • Normalization by ∥Mt∥F\lVert M_t\rVert_\text{F} without clipping causes DtD_t to behave as approximately (32)qϵ−1Mt(\frac32)^q\epsilon^{-1}M_t once ∥Mt∥F≪ϵ\lVert M_t\rVert_\text{F}\ll\epsilon in the limit as gradients become very small, which is the regime that the theoretical results are based on. This is problematic because the dynamics become more similar to SGD with scaled momentum, losing the orthogonalization structure of Muon.
  • The theoretical results rely on strong assumptions such as deterministic gradients, strict separability, and no decoupled weight decay, and they do not readily extend to more general settings. In particular, the attention surrounding Muon has largely come from its success in training deep neural networks, where features are jointly learned, data is nonseparable and noisy, weight decay is used to prevent overfitting, etc. This makes it difficult to argue that the currently presented results say anything meaningful about the behavior of Muon in practice.
  • There are some experimental weaknesses, such as using small architectures, five seeds, only 120 minibatch updates, and shared hyperparameters (particularly the learning rate) across methods instead of providing a proper hyperparameter grid.
Questions for authors
  1. How are the results affected by considering implementations of Muon that are actually used in practice? In particular, what happens if we use the quintic polynomial of [1] (or other variants [2]), practical normalization [3], or decoupled weight decay as in [4, 5]?
  2. Can you extend the theory to more realistic training settings?

References

[1] Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks. 2024.

[2] Noah Amsel, David Persson, Christopher Musco, Robert M. Gower. The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm. ICLR 2026.

[3] Gyu Yeol Kim, Min-hwan Oh. Convergence of Muon with Newton-Schulz. ICLR 2026.

[4] Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianzhou Wang, Mengnan Dong, Zheng Zhang, Yongsheng Kang, Hao Zhang, Xinran Xu, Yutao Zhang, Yuxin Wu, Xinyu Zhou, Zhilin Yang. Muon is Scalable for LLM Training. 2025.

[5] Lizhang Chen, Jonathan Li, Qiang Liu. Muon Optimizes Under Spectral Norm Constraints. TMLR 2026.

7 comments
AuthorsOpenReview ↗

Response to Reviewer gQrf (Part 1/3)

Thank you for your careful assessment. Your comments helped us clarify which conclusions carry over to practical Muon and improve our experimental evaluation.

1. Do our results hold for practical non-monotonic polynomials (Jordan's quintic and Polar Express)? (W1 & Q1)

For W1, we generalize the update-norm and momentum-alignment bounds from Prop. B.11 under weaker assumptions, thereby extending Thm. 4.1's pathwise bound to nonmonotone maps, including Jordan's quintic [1] and the frozen five-stage Polar Express sequence [2]. Both maps can exhibit a spectral-margin gap at fixed depth (cf. Prop. 4.2). Increasing qq alone cannot ensure Thm. 4.3’s recovery guarantee for Jordan's quintic, while ideal Polar Express recovers the optimal margin via joint calibration.

First, we establish two positive extensions covering Thm. 4.1's pathwise bound for Jordan's quintic and frozen five-stage Polar Express, and Thm. 4.3's (calibrated) recovery guarantee for ideal Polar Express.

  • Even without monotonicity or a unit upper bound, Thm. 4.1 holds under weaker assumptions. Prop. B.11 uses strict monotonicity to infer equal positive singular values from zero defect. For the pathwise bound, the full qq-round response ψq\psi_q need only satisfy ψq(0)=0\psi_q(0)=0 and ms≤ψq(s)≤Kms≤\psi_q(s)≤K on [0,1][0,1] for fixed 0<m≤K<∞0 < m≤K < ∞. Valid (m,K)(m,K) are (3/5,5/4)(3/5,5/4) for Jordan [1] at all q≥1q≥1 and (0.000324,1.131)(0.000324,1.131) for frozen five-stage Polar Express [2]. In both cases, loss vanishes and the pathwise bound holds with modified constants.
  • Ideal Polar Express retains Thm. 4.3's recovery guarantee when its design interval and stage count are calibrated jointly. For unit-Frobenius inputs of rank at most p>1p > 1, choose a design interval [ℓ,1][\ell,1] with 0<ℓ≤1/p0 < \ell≤1/\sqrt p. Let εq\varepsilon_q be the maximum deviation from 11 on this interval after qq ideal stages [2]. Then δqM≤2εq/(1+εq)+(p−1)ℓδ_q^M≤2\varepsilon_q/(1+\varepsilon_q)+(p-1)\ell, where the last term covers singular values below ℓ\ell. Recovery follows as ℓ→0\ell→0 and εq→0\varepsilon_q→0.

Next, we show that both maps can still exhibit a spectral-margin gap at fixed depth (cf. Prop. 4.2) and explain how oscillations in Jordan's quintic limit the applicability of Thm. 4.3's recovery guarantee.

  • Jordan's quintic and Polar Express can still exhibit a spectral-margin gap at fixed depth (cf. Prop. 4.2). Our three-class Jordan example with symmetric initialization has a limiting spectral-margin gap ≥85/(11642)≥85/(1164\sqrt2) at any fixed q≥1q≥1. Ideal Polar Express admits fixed-stage counterexamples. The frozen sequence gives a spectral-margin gap ≥1/(52)≥1/(5\sqrt2). These gaps persist as loss vanishes.
  • Increasing depth alone may not yield Thm. 4.3's recovery guarantee for Jordan's quintic. We construct a unit-Frobenius rank-two input whose singular-value responses oscillate, giving δqM≥5/97δ_q^M≥5/97 for all q≥1q≥1. Thus, for p≥2p≥2, increasing qq alone cannot guarantee the uniform tolerance in Thm. 4.3(i) when ρT<5/97ρ_T < 5/97. For Thm. 4.3(ii), increasing depth alone does not ensure DT→0\mathcal D_T→0, but recovery still follows if DT→0\mathcal D_T→0.

2. How does practical normalization affect our results? (W2 & Q1)

We first clarify that our homogeneous normalization Mt/rtM_t/r_t, where rt=∣Mt∣Fr_t=\\|M_t\\|_F, keeps every nonzero NS input at unit Frobenius norm. As you point out, Jordan's additive stabilizer (dt=rt+ϵd_t=r_t+\epsilon [1]) and Kim–Oh's floor (dt=max⁡1−β,rtd_t=\max\\{1-\beta,r_t\\} in our EMA convention [3]) shrink the NS input as rt→0r_t→0.

  • With practical stabilization, Thm. 4.1's pathwise bound and Thm. 4.3's (calibrated) recovery guarantee hold under the conditions below. For fixed odd ψ\psi with 0<ms≤ψ(s)≤Ls0 < ms≤\psi(s)≤Ls on (0,1](0,1], the spectral-margin gap is at most γ\*DT+O(1/log⁡HT)\gamma_\*\mathcal D_T+O(1/\log H_T), with HT=1+∑t<TηtH_T=1+\sum_{t < T}\eta_t. Choosing cubic depth for input scale ht=rt/dth_t=r_t/d_t and tolerance ρtρ_t gives δtM≤ρtδ_t^M≤ρ_t and ∣Dt∣op≥1−ρt\\|D_t\\|_{\mathrm{op}}≥1-ρ_t for 0<ρt≤1/20 < ρ_t≤1/2. Taking ρt→0ρ_t→0 gives recovery.
  • At fixed depth (cf. Prop. 4.2), both stabilizers can shift selection to the Frobenius max-margin direction. As you noted, small-signal updates resemble scaled momentum SGD. At fixed cubic depth, Dt=(3/2)qMt/d0+o(rt)D_t=(3/2)^qM_t/d_0+o(r_t) as rt→0r_t→0, with d0=ϵd_0=\epsilon (additive) or 1−β1-\beta (Kim–Oh [3]). Under these linear bounds, both select the Frobenius max-margin direction, rather than ideal Muon's spectral max-margin direction. Our additive three-class example has a limiting spectral-margin gap 2/(92)2/(9\sqrt2) at any fixed cubic or positive Jordan depth.

References

[1] Jordan et al. (2024). Muon: An optimizer for hidden layers in neural networks.

[2] Amsel et al. (2026). The Polar Express. ICLR.

[3] Kim and Oh (2026). Convergence of Muon with Newton–Schulz. ICLR.

AuthorsOpenReview ↗

Response to Reviewer gQrf (Part 2/3)

3. Can we relax strict separability to nonseparable data? (W3 & Q2)

Thm. 4.1's pathwise bound and Thm. 4.3's recovery guarantee can extend to separable subsets under additional conditions, although Thms. 4.1–4.3 do not directly cover general nonseparable data.

Thm. 4.1's pathwise bound and Thm. 4.3's recovery guarantee extend to separable subsets under additional conditions. Motivated by Ji and Telgarsky [6], our projected bound for a separable subset SS assumes vanishing step lengths with divergent sum and eventual uniform positive alignment of nonzero updates with the subset gradient. The bound tracks alignment defects, squared steps, and a finite prefix. Projected recovery follows if the path-weighted defect relative to the subset gradient vanishes. In a four-class example with conflicting labels and orthogonal singular channels, cubic NS with prescribed initialization, fixed full-gradient EMA, and square-root stepsizes gives 0≤γS∗−γ^S(WTPS)=O(T−1/2)0≤\gamma_S^*-\widehat\gamma_S(W_TP_S)=O(T^{-1/2}) when both separable responses are at least 1−1/(t+2)1-1/(t+2). Here PSP_S projects onto the subset's feature span. Recovery with overlapping singular channels remains open.

The spectral-margin guarantees of Thms. 4.1–4.3 are not immediately applicable to nonseparable data. When conflicting labels prevent global separation, the original positive spectral max-margin target is unavailable. Specifically, Prop. 4.2 does not show persistent fixed-depth bias for general nonseparable data. Nonseparable logistic regression requires distinguishing separating directions from risk-minimizing components.

4. Can we extend beyond deterministic gradients to stochastic training? (W3 & Q2)

Thm. 4.1's pathwise bound and Thm. 4.3's recovery guarantee can extend under controlled gradient-signal errors and additional conditions, while unconditional guarantees for stochastic gradients are unavailable.

Prop. B.12 relates input and full-gradient defects. For nonzero NS input NtN_t, full gradient GtG_t, and update DtD_t, it gives δtG≤min⁡2,δtN+(2−δtN)∣Nt−Gt∣\*/∣Gt∣\*δ_t^G≤\min\\{2,δ_t^N+(2-δ_t^N)\\|N_t-G_t\\|_\*/\\|G_t\\|_\*\\} when δtN≤1δ_t^N≤1, where δtXδ_t^X is the defect relative to signal XX. In the separable linear setting with bounded step lengths, recovery follows if total path length diverges and the path-weighted full-gradient defect and normalized squared-step remainder vanish. Verifying these conditions for general fixed-batch EMA is open.

Thms. 4.1–4.3 do not provide unconditional guarantees under stochastic gradients. Even unbiased mini-batch gradients need not yield unbiased normalized NS updates. Consequently, increasing NS depth alone does not establish Thm. 4.3's spectral-margin recovery under fixed-batch noise. As Li et al. [7] show, stochastic steepest descent also depends on batch size, momentum, and variance reduction.

5. Can we extend beyond fixed features to joint feature learning? (W3 & Q2)

Extending Thms. 4.1–4.3's max-margin guarantees to joint feature learning requires further analysis. Gronich and Vardi [8] establish conditional margin-KKT results for idealized Muon flow, but recovery for discrete finite-NS training is open. Our spectral-norm-matched experiments show finite-time depth effects but do not prove asymptotic bias. With 20% corrupted CIFAR-10 labels and unchanged clean-data hyperparameters, Jordan-5 minus Jordan-1 shifts mean test accuracy by −2.11 pp on ResNet-18 and +3.80 pp on the patch Transformer at 6,000 updates across six paired seeds.

6. Can we extend our analysis to decoupled weight decay? (W3 & Q1)

Positive decoupled weight decay changes the setting of Thms. 4.1–4.3's spectral-margin guarantees, as bounded iterates preclude the norm divergence required by our max-margin analysis. For λ>0λ > 0, ∣Dt∣op≤K\\|D_t\\|_{\mathrm{op}}≤K, and αt=ληt∈(0,1]\alpha_t=λη_t∈(0,1], Wt+1=(1−αt)Wt−ηtDtW_{t+1}=(1-\alpha_t)W_t-η_tD_t preserves C=W:∣W∣op≤K/λ\mathcal C=\\{W:\\|W\\|_{\mathrm{op}}≤K/λ\\} if W0∈CW_0∈\mathcal C. Unit exact-polar updates give K=1K=1 [5], and the fixed multiplier in [4] rescales KK. If the gradient is Lipschitz on C\mathcal C, αt→0\alpha_t→0, and ∑tαt=∞\sum_t\alpha_t=∞, the αt\alpha_t-weighted average Frank–Wolfe gap vanishes if the oracle error vanishes in the same average.

We thank the reviewer for pointing us to Chen, Li, and Liu's exact-polar analysis [5], which we will cite it in the revised manuscript.

References

[4] Liu et al. (2025). Muon is Scalable for LLM Training.

[5] Chen, Li, and Liu (2026). Muon Optimizes Under Spectral Norm Constraints. TMLR.

[6] Ji and Telgarsky (2019). The Implicit Bias of Gradient Descent on Nonseparable Data. COLT.

[7] Li, Tang, and Zou (2026). The Implicit Bias of Steepest Descent with Mini-batch Stochastic Gradient. ICML.

[8] Gronich and Vardi (2026). The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networks. ICML.

AuthorsOpenReview ↗

Response to Reviewer gQrf (Part 3/3)

Thank you for pointing out the limits of the original evaluation in model size, update count, seed count, and shared learning rates.

7. Larger networks, longer training, and independent tuning (W4)

We compare the methods after independent tuning on larger models and longer runs.

Independent tuning and paired evaluation. On CIFAR-10 we test a 6.29M-parameter ResNet-18 and a 3.19M-parameter patch Transformer at 2,000 and 6,000 updates. We tune each method separately by model and horizon: 3 tuning seeds per cell, 4 decay values, and 5–8 learning rates at 2,000 updates or 3 at 6,000. We use 12 paired evaluation seeds per setting. The AdamW, Jordan-1, and Jordan-5 comparisons contain 504 and 288 records, with 180 reused at 2,000 steps. Cubic-6 and Taylor-4 add 360 and 192 runs. Jordan-4 adds 180 and 96 runs. Jordan-4, main-text Cubic-6, and Appendix A's Taylor-4 each use 12 NS matrix products per matrix update.

CIFAR-10 results. Jordan-4, Cubic-6, and Taylor-4 are follow-ups to Appendix I. Acc and CE denote mean test accuracy and cross-entropy.

UpdatesModelAdamW (Acc / CE)Jordan-1 (Acc / CE)Jordan-4 (Acc / CE)Jordan-5 (Acc / CE)Cubic-6 (Acc / CE)Taylor-4 (Acc / CE)
2,000ResNet-1889.84% / 0.301590.72% / 0.273392.79% / 0.218292.76% / 0.218492.01% / 0.240992.00% / 0.2383
2,000Patch Transformer74.64% / 0.718676.04% / 0.686383.77% / 0.470184.16% / 0.461479.59% / 0.587779.82% / 0.5812
6,000ResNet-1892.37% / 0.251593.48% / 0.220994.03% / 0.208394.20% / 0.209693.49% / 0.211793.80% / 0.2093
6,000Patch Transformer82.63% / 0.515183.46% / 0.487587.11% / 0.410586.48% / 0.410485.32% / 0.448985.15% / 0.4407

Paired differences (95% CIs). The contrasts are Jordan-5 minus Jordan-1 and Taylor-4 minus Cubic-6. Accuracy differences are in percentage points.

UpdatesModelJordan accuracy (pp)Jordan CETaylor vs. cubic accuracy (pp)Taylor vs. cubic CE
2,000ResNet-18+2.04 pp [1.86, 2.23]−0.0550-0.0550 [−0.0589,−0.0511-0.0589,-0.0511]-0.01 [-0.15, 0.12]-0.0026 [-0.0045, -0.0008]
2,000Patch Transformer+8.12 pp [7.81, 8.44]−0.2249-0.2249 [−0.2318,−0.2180-0.2318,-0.2180]+0.23 [0.09, 0.36]-0.0064 [-0.0080, -0.0049]
6,000ResNet-18+0.72 pp [0.60, 0.84]−0.0113-0.0113 [−0.0166,−0.0060-0.0166,-0.0060]+0.31 [0.14, 0.48]-0.0024 [-0.0076, 0.0028]
6,000Patch Transformer+3.02 pp [2.84, 3.20]−0.0771-0.0771 [−0.0812,−0.0730-0.0812,-0.0730]-0.17 [-0.35, 0.01]-0.0081 [-0.0127, -0.0036]

At the 12-product budget, Jordan-4 has higher mean accuracy and lower mean CE than Cubic-6 and Taylor-4 in all four settings. Across the eight paired contrasts, accuracy gains are 0.24–4.18 pp (all 95% CI lower limits ≥0.11 pp), and 6/8 CE intervals lie below zero. Jordan-4 and Jordan-5 each lead in mean accuracy in two settings. Horizons use separate tuning and cosine schedules. Several selections lie on grid boundaries.

8. Testing fixed-depth spectral-margin gaps with practical polynomials (W4)

Motivated by Prop. 4.2, we measure the final spectral-margin gap and accumulated directional defect in matched three-class runs.

Controlled three-class comparisons. We ran 48 FP64 three-class trajectories for 10,000 updates each, with 3 matched initializations per setting. The table matches the NS budget per matrix update under homogeneous normalization. Each cubic round uses two matrix products and each quintic round uses three.

MethodNS rounds per updateNS matrix products per updateMedian spectral-margin gapMedian path-weighted defect
Cubic-666×2=126\times2=125.82×10−45.82\times10^{-4}7.55×10−67.55\times10^{-6}
Taylor-444×3=124\times3=125.98×10−45.98\times10^{-4}1.69×10−61.69\times10^{-6}
Jordan-444×3=124\times3=128.50×10−28.50\times10^{-2}9.36×10−29.36\times10^{-2}

The original Jordan quintic has the largest median spectral-margin gap and path-weighted defect in this comparison. At the same matrix-product budget, Cubic-6 has a smaller final spectral-margin gap than Jordan-4 at all three matched seeds.

Adaptive precision under additive stabilization. With additive stabilization at ϵ=10−7\epsilon=10^{-7}, adaptive depth reduces the median spectral-margin gap from 3.24×10−23.24\times10^{-2} to 2.12×10−32.12\times10^{-3} for cubic NS and from 3.20×10−23.20\times10^{-2} to 2.20×10−32.20\times10^{-3} for the Taylor quintic. Each meets the tolerance in 6/6 runs across homogeneous and additive normalization. At the fixed 12-product budget with additive stabilization, the original Jordan quintic has a median spectral-margin gap of 5.40×10−25.40\times10^{-2}.

Comments on the paper

2 comments
Area_Chair_VQVaOpenReview ↗

Reviewers: please respond to the author rebuttal

Dear reviewers,

The authors have posted responses to your reviews. The author-reviewer discussion closes on November 18 (AoE).

Please read the responses and reply here before then: let the authors know whether your concerns have been addressed, what remains open, and whether you are updating your score. If your assessment is unchanged, please say so explicitly. Engaging with the other reviews is also welcome.

Thank you, Area Chair

Area_Chair_VQVaOpenReview ↗

Final reminder: discussion closes today (Nov 18 AoE)

The author-reviewer discussion closes today, November 18 (AoE).

Dear reviewers,

Thank you to those who have already responded. If you have not yet replied: Please read the authors' response and post a brief reply here before the deadline: whether your concerns have been addressed, what remains open, and whether you are updating your score. If your assessment is unchanged, please say so explicitly. Your response is important for the final recommendation.

Thank you, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.