- Rating
- 3(−1)
- Confidence
- 3(−1)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies the classifier selected by Muon with finitely many Newton–Schulz iterations on separable multiclass linear data. Its main bound connects the spectral-margin gap to directional error accumulated along the update path. A fixed-map norm construction yields datasets with persistent fixed-depth bias, and increasing-precision schedules recover the spectral max margin. Experiments compare exact SVD, fixed depths, and calibrated or adaptive depth using certified margin references. Additional neural experiments examine finite-time depth sensitivity.
Critical strengths
-
The solution-selection result is meaningful and technically convincing. Lemma 4.6 identifies a norm whose supporting direction matches the finite spectral map, and Proposition 4.2 constructs a persistent margin gap of order . This gives a concrete account of how finite depth changes implicit bias. Theorem 4.1 and the precision corollaries complement the lower bound with quantitative recovery guarantees.
-
The validation is closely matched to the theory. The 24 paired linear datasets use certified spectral-margin references, with interval widths below . Table 1 reports median margin ratios of 0.9216 at depth 1, 0.9575 at depth 4, and 0.9680 for exact SVD. Directional-error measurements, horizon sweeps, equal-polynomial-computation comparisons, and norm-matched controls make the observed depth effect convincing. The proof audit supports the stated asymptotic claims.
Critical weaknesses
The main limitation is practical scope. The theory covers cubic and admissible Taylor maps after exact Frobenius normalization. Common implementations use a tuned quintic, a normalization stabilizer, and lower precision. Appendix A acknowledges these differences, and its quintic experiment uses Taylor coefficients. A fixed stabilizer can matter when momentum becomes small, which is relevant to asymptotic solution selection. This limits direct transfer to a particular training implementation. The stated theoretical contribution remains sufficient for clear acceptance.
Questions for authors
I have no unresolved questions that would materially affect my recommendation on the current evidence. The central claims are adequately supported, and the manuscript discloses its main implementation restrictions. Further discussion of the tuned polynomial or a normalization stabilizer is optional.
Response to Reviewer A4Q8
Thank you for your positive evaluation and feedback. Your comments helped connect our results to practical Muon through the tuned quintic, additive stabilization, and lower precision.
1. Do our results hold for the tuned quintic polynomial? (W1)
Under the response bounds below, we can extend Thm. 4.1's pathwise bound to Jordan's tuned quintic P(s)=3.4445s−4.7750s3+2.0315s5 [1]. Yet it leaves a spectral-margin gap at fixed depth (cf. Prop. 4.2) in our three-class construction, and increasing q alone cannot ensure Thm. 4.3's recovery guarantee.
2. How does practical normalization affect our results? (W1)
We can extend Thm. 4.1's pathwise bound and Thm. 4.3's (calibrated) recovery guarantee for cubic NS to the additive stabilizer Xt,0=Mt/(∣Mt∣F+ϵ) (ϵ>0, set to 10−7 in [1]) under the conditions below. At fixed depth (cf. Prop. 4.2), small momentum makes the stabilized update nearly linear and shifts selection to the Frobenius max-margin direction.
3. Do our results still hold under lower precision? (W1)
BF16 rounding can change alignment and path weights (cf. Thm. 4.1) and BF16 defects can exceed Thm. 4.3(i)'s tolerance. Motivated by [2, 3], we compare BF16 and FP64 NS on identical inputs with FP64 outer states.
References
[1] Jordan et al. (2024). Muon: An optimizer for hidden layers in neural networks.
[2] Nakatsukasa & Higham (2012). Backward Stability of Iterations for Computing the Polar Decomposition. SIMAX.
[3] Amsel et al. (2026). The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm. ICLR.