- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the conditions under which Muon's spectral normalization improves the optimization process. Using quadratic networks, the authors derive parameter curvature based on the input distribution and identify specific conditions under which Muon—in its exact-polar form—achieves greater relative progress compared to gradient descent (GD) and, under specific coordinate constructions, SignGD.
Starting from a quadratic approximation, the analysis extends this advantage to finite-step optimization of non-linear empirical loss functions, assuming appropriate local initialization. Experiments with a 124-million-parameter NanoGPT model show that, across three paired runs, Muon ultimately achieves a lower validation loss than AdamW.
However, Muon does not consistently maintain a greater relative reduction in loss compared to AdamW throughout the entire training process.
Critical strengths
-
This paper derives the relevant curvature based on input statistics, rather than simply assuming favorable properties for the Hessian matrix. This provides a concrete explanation of how the geometric structure of the data imparts practical utility to spectral normalization.
-
The paper is clearly structured: before presenting the main theorem, it explains the underlying mechanisms using a quadratic form example and explicitly outlines the assumptions and experimental limitations.
Critical weaknesses
-
The theoretical comparison is against GD and SignGD rather than AdamW, establishing a finite-step advantage in loss reduction under specific conditions. Furthermore, the theoretical setting is overly restrictive and fails to establish a sufficient connection to the actual training of NanoGPT.
-
The parameter-distance experiments do not support the local regime required by the theorem. The initialization condition implies: Using the terminal checkpoint as a reference, the minimum ratio observed across 240 nonterminal combinations was approximately 3.57. Consequently, none of the tested checkpoints satisfied this necessary proximity condition. While this result does not falsify the theorem, it weakens the proposed connection to the actual training process.
-
The mean-alignment experiments measured alignment but failed to demonstrate that the aligned directions contribute substantially to the fitted model’s response. In the quadratic model, a high cosine value must be considered together with the corresponding response strength. Although the reported peak cosines approached 1, the corresponding relative singular values were approximately . The paper also reports that most of the mean’s projection energy lies in the fitted model’s numerical nullspace, while the largest resolved-direction cosines were only 0.000487, 0.01030, and 0.08473. These results do not establish the active-feature alignment required by the theory.
Questions for authors
No