- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the robustness cost of Muon's spectral equalization, motivated by the fact that flattening singular values accelerates weak modes regardless of whether they correspond to useful signal or noise. It analyzes how spectral equalization affects update rank, signal and noise update energy, mode wise learning rates, and drift near clean optima, and connects asymmetric label corruption to coherent weak spectral modes that are promoted by Muon relative to SGD. Experiments under label corruption are used to support the analysis.
Critical strengths
Label noise analysis. The paper connects spectral equalization to asymmetric label corruption by showing that corrupted subpopulations can form coherent weak spectral modes, and derives how Muon increases their relative update energy compared with SGD.
Experimental support. The experiments compare asymmetric and symmetric label corruption and show a larger robustness gap between Muon and SGD under asymmetric corruption, which is consistent with the analysis.
Critical weaknesses
Motivation. The motivation is not new. Prior work already shows that spectral equalization speeds up weak modes, removes the simplicity bias of SGD, and can promote spurious or tail features [1,2,3]. This paper mainly treats weak modes as noise and studies this under label corruption.
Theory. Several results put the conclusion into the assumptions. Theorem 3.2 assumes , orthogonal signal and noise singular subspaces, and a spectral gap, so the amplification result follows almost directly. The issue is not the decomposition itself, but the assumption that signal and noise are already separate singular modes of the full gradient. Under a clean versus corrupted or conditional expectation decomposition, they do not need to remain separate singular modes, so the result does not generally follow. Lemma 3.3 assumes different mode wise rates for amplitude sensitive dynamics and equal rates for equalized dynamics, then derives the learning time gap. Section 4 derives corrupted label components, but still assumes that they remain singular modes of the full gradient, with additional orthogonality assumptions for deep linear and ReLU networks. Overall, several results mainly restate properties already included in the assumptions.
Method novelty. SUMO is incremental. It applies Muon in a top singular subspace and uses the original gradient outside it. This is close to Subspace Momentum [4], which applies momentum in a low dimensional subspace and SGD to the orthogonal residual, and GaLore [5], which uses leading singular directions to build optimization subspaces. The main difference is applying Muon orthogonalization inside the selected subspace.
Experiments. The experiments show that SGD is more robust than Muon under synthetic label corruption, but do not verify the proposed mechanism. The paper does not show that corrupted examples form the weak singular subspaces assumed by the theory. SGD is also more robust under white label noise, so the results do not isolate the coherent noise explanation. The SUMO comparison also changes step sizes, momentum treatment, and orthogonalization, so it does not isolate the effect of restricting spectral equalization.
References
[1] Sara Dragutinović, Yedi Zhang, and Rajesh Ranganath. To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters. arXiv preprint, 2026.
[2] Bhavya Vasudeva, Puneesh Deora, Yize Zhao, Vatsal Sharan, and Christos Thrampoulidis. How Muon’s Spectral Design Benefits Generalization: A Study on Imbalanced Data. ICLR, 2026.
[3] Shuche Wang, Fengzhuo Zhang, Jiaxiang Li, Cunxiao Du, Chao Du, Tianyu Pang, Zhuoran Yang, Mingyi Hong, and Vincent Y. F. Tan. Muon Outperforms Adam in Tail-End Associative Memory Learning. ICLR, 2026.
[4] Thien Hang Nguyen and Huy Nguyen. Lean and Mean Adaptive Optimization via Subset-Norm and Subspace-Momentum with Convergence Guarantees. ICML, 2025.
[5] Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection. ICML, 2024.
Questions for authors
- Section 4 assumes that corrupted examples come from a different feature subpopulation, but the CIFAR and MNIST experiments randomly flip labels within each class. The corrupted and clean examples therefore have the same feature distribution in expectation. Why should this setting produce the separated weak mode assumed in Section 4?
- The theory assumes exact spectral equalization, while the experiments use finite Newton Schulz iterations on the momentum matrix. Does the amplification result still hold for the actual Muon update? If not, what amplification should we expect?
- Theorem 3.1 concerns a single update at fixed Frobenius norm, but the experiment uses final weight stable rank to support a higher rank bias. Muon moves about 9 times farther from initialization than SGD. Does the stable rank gap remain when displacement or path length is matched?
- The author list for Muon Optimizes Under Spectral Norm Constraints seems to be incorrect; it should be cited as Chen et al. (Lizhang Chen, Jonathan Li, and Qiang Liu).
- The submission seems to be missing the required AI use statement.
4 comments
Response to Reviewer BufW continue
Q1. Feature distribution under label noise. This is a valid distinction. Section 4 assumes subpopulation-dependent corruption, whereas the CIFAR-100 and MNIST experiments use class-conditional random flips. Before training, the clean and corrupted subsets of a source class have the same feature distribution in expectation. Comparing corrupted examples with clean examples of the destination class does not establish the separation assumed in Section 4. Training may separate their representations, but we have not measured this. We will distinguish the two corruption models and qualify their connection.
Q2. Newton–Schulz vs. exact equalization. The α² lower bound applies to exact equalization. For finite NS, the output noise-to-signal energy ratio is the sum of squared transformed noise singular values divided by the corresponding sum for signal. At a gap of 4, exact SVD amplified this ratio by 16×, five-step NS by 40.60×, and twenty-step NS by 6.29×. The quintic iterates oscillate: the weak modes have larger output singular values than the strong modes after five steps and smaller ones after twenty. On actual MNIST momentum inputs, twenty steps still left 22–23% relative Frobenius error to the SVD polar factor in the second hidden layer. The implemented polynomial maps 1 to 0.701, so it cannot converge to the exact polar factor for a nonzero input.
Controlled amplification of the noise-to-signal energy ratio, with 64 orthogonal signal and 64 noise modes (App. C.7, Table 8, Fig. 9):
| Gap α | Exact SVD | NS5 | NS20 |
|---|---|---|---|
| 2 | 4.00× | 5.40× | 1.76× |
| 4 | 16.00× | 40.60× | 6.29× |
| 8 | 64.00× | 110.38× | 33.12× |
| 16 | 256.00× | 312.45× | 300.49× |
| 128 | 16,384× | 7,325× | 19,231× |
For a K-step iteration the amplification is A_K = α² [f_K(1/α) / f_K(1)]², where f_K is the iteration's scalar map. This matches the measured values to within 0.013%.
The ViT-B/16 momentum matrices show the same picture (App. C.6, Fig. 8):
| Transform | Rel. error to polar factor | Corrupted-label share of update energy |
|---|---|---|
| Raw | — | 0.397 |
| NS5 | 0.37 | 0.486 |
| NS20 | 0.22 | 0.471 |
| Exact SVD | 0 | 0.480 |
Despite their numerical error, the finite iterations raised the corrupted-label share almost exactly as much as exact SVD did. The amplification is therefore not an artifact of exact equalization.
We will correct the numerical-exactness wording in Appendix B.2.
Q3. Displacement vs. rank. We matched the norm of each matrix update, and five-step NS still fit corrupted labels sooner (W3). Table 2 measures final weight stable rank. Muon retained 72% of its initial stable rank, and SGD retained 17%, with a 9.3-fold difference in displacement. We do not yet know whether the rank difference persists at matched displacement. We will present Table 2 as an empirical observation. Theorem 3.1 concerns the spectrum of one update.
Q4. Citation fix. We will correct the citation to Chen, Li, and Liu, "Muon Optimizes Under Spectral Norm Constraints." Thank you for catching this.
Q5. AI use statement. Thank you for flagging this. The page-one footnote pointed reviewers to ICLR's Reviewer Guidelines and AI Policy for Reviewers, and we will remove it from the revised manuscript. We reported our AI use in the submission form and omitted the dedicated author disclosure from the manuscript. We will add it and explain how we checked the work.
Thank you for your rebuttal
Weaknesses
1. Theory: Overly Strong Orthogonality Assumption
Partially resolved. C.1–C.3 decompose clean and corruption gradients and show that corruption energy is more concentrated in weak spectral directions. However, they do not establish that the two singular subspaces are orthogonal during actual training, nor do they relax the assumptions of Theorem 3.2.
2. Experiments: Insufficient Mechanistic Validation
Partially resolved. The authors add C.1–C.8, directly measuring corruption energy, noisy-label memorization speed, and learning order. However, causal intervention experiments that link spectral amplification to the observed generalization gap are still missing. Moreover, why the robustness gap between SGD and Muon is larger under asymmetric than under symmetric noise remains insufficiently explained.
Questions
Q1: Why Do Random Label Flips Produce Weak Spectral Directions?
Partially addressed. C.3 shows that corruption energy is more concentrated in weak spectral directions, but does not establish the orthogonality and spectral gap required by the theory.
Q4: Incorrect Reference Author List
Unresolved. I could not find any source, including Google Scholar, that lists the authors in the incorrect order. The authors did not explain how this error occurred.
Q5: Missing AI Use Statement
Unresolved. The AI use statement is still missing from the revised manuscript.
Related ICLR Papers
For the authors' reference, I list some related papers accepted at ICLR:
- Why Is SAM Robust to Label Noise? ICLR 2024.
- Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks. ICLR 2024.
- Does SGD Really Happen in Tiny Subspaces? ICLR 2025.
- How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Data. ICLR 2026.
- Muon Outperforms Adam in Tail-End Associative Memory Learning. ICLR 2026.
Current Assessment
Compared with papers accepted at previous ICLR conferences in closely related areas, the experimental scale and quality of this paper fall far short of the level demonstrated by comparable accepted papers.
Therefore, I maintain my rating. In my view, acceptance would require substantial revision. The paper is not ready for acceptance in its current form.
Revised Manuscript Uploaded (with Rebuttal Edits & New Appendix Experiments)
Thank you for following up. We’ve just uploaded the revised manuscript, which incorporates the main-paper edits detailed in our rebuttal along with additional experiments in the appendix. We hope these updates address your comments and that you’ll take them into account in your final assessment.
Response to Reviewer BufW
Thank you for the detailed review, and for recognizing that the paper connects spectral equalization to asymmetric label corruption, showing that corrupted subpopulations can form coherent weak spectral modes whose relative update energy Muon increases compared with SGD, and that our experiments show a larger Muon–SGD robustness gap under asymmetric than symmetric corruption, consistent with this analysis. We quantify how spectral equalization redistributes update energy under label corruption and test how it affects memorization speed. The paper does not propose an optimizer. SUMO (NeurIPS 2025) is an existing method, not ours,and not a contribution of this paper, so its novelty relative to subspace methods does not bear on our contributions. We will add the citation, together with Subspace Momentum and GaLore.
New evidence. We added three controls. On matrices with known orthogonal signal and noise modes, both finite NS variants increased the noise-to-signal energy ratio in all 80 configurations (Q2). In 18 ViT-B/16 runs on CIFAR-100, five-step NS fit corrupted labels sooner in every noisy seed pair at matched update norms (W3). On the actual momentum matrices of ViT-B/16 training runs, five-step NS raised the corrupted-label share of update energy early in training in all 8 run/noise/source combinations we measured (W1, Q2).
W1. Theory and assumptions. We agree that a coherent label-dependent component need not be a separate singular mode of the full gradient. Theorem 3.2 assumes orthogonal signal and noise singular subspaces and a spectral gap. Section 4 gives sufficient conditions for this setting but retains assumptions on representation geometry and mode preservation in the full gradient (Corollaries 4.7 and 4.9; Remark 4.10). It does not prove that arbitrary deep networks satisfy these conditions. We will narrow the statements that suggest otherwise.
To measure the decomposition directly, we computed G_obs, G_clean, and their difference for all 48 hidden matrices of ViT-B/16 (CIFAR-100, 30% asymmetric and symmetric noise). We also tracked the corrupted-label part of the momentum along the training trajectory. Early in training (5% of steps, near the validation-selected checkpoint), the results were as follows.
Five-step NS raised the corrupted-label share of update energy in all 8 run/noise/source combinations (App. C.2, Fig. 4):
The corrupted-label energy was 2.5–30× less concentrated than the clean energy in the leading singular directions, which hold half of the observed gradient's energy (App. C.3, Fig. 5):
This supports the energy-redistribution mechanism without relying on exact subspace separation.
W2. Motivation. The manuscript already cites Dragutinović et al. (2026) (lines 71, 130, 215). We now discuss explicitly in Section 2 their analysis of Muon’s loss of simplicity bias and its behavior with spurious features, and how our energy-ratio analysis complements it. Our analysis quantifies how much spectral equalization changes the noise-to-signal energy ratio under an explicit decomposition. Theorem 3.2 gives an α² lower bound for exact equalization under its assumptions. The ViT control tests how practical NS changes memorization speed.
W3. Experiments and SUMO. SUMO is not our contribution (so its limited novelty shouldn’t affect the assessment of our paper), and neither the abstract nor the contributions (lines 95–107) claim it. It is an existing optimizer that orthogonalizes the momentum only within a top-k gradient subspace and applies the remaining directions at their gradient amplitudes. We include it because it varies the quantity our analysis studies, namely how much of the spectrum is equalized. SGD equalizes no directions, SUMO equalizes the top k, and Muon equalizes all of them. SUMO also illustrates the direction our conclusion motivates: keeping equalization while preserving amplitude information. The comparison is exploratory. Appendix B.5 details how tuning, step size, momentum, and orthogonalization also differ between SUMO and Muon. We will state in Section 5 that SUMO is an existing baseline.
To test the spectral transform itself, we ran a controlled ViT-B/16 comparison at matched update norms. Five-step NS reached 75% accuracy on the assigned labels of corrupted examples at epoch 6 vs. 9 under asymmetric noise and at epoch 9 vs. 11 under symmetric noise, in every seed pair.