- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates the geometric and numerical factors underlying the loss reduction achieved by Muon's spectral flattening. It makes three main contributions.
-
The authors introduce a fixed-polar construction that preserves the candidate's singular values, update norm, gradient alignment, and ideal polar update while varying the orientation of its singular vectors. Through a constructed quadratic example and a local theoretical result, they demonstrate that these quantities alone are insufficient to determine whether spectral flattening improves the loss, highlighting the role of curvature and singular-vector orientation.
-
The paper proposes using endpoint gradients to better predict the loss response to small orientation perturbations at finite step sizes. Experiments show improved predictive accuracy in early layers, although much of the improvement can be recovered through simple scalar calibration, and generalization across evaluation data is limited in deeper layers.
-
The authors investigate numerical distortions introduced by finite-round Newton–Schulz iterations and floating-point arithmetic. They show that FP32 substantially improves the fidelity of the measured loss responses compared with BF16, while the remaining discrepancies are largely attributable to the finite-round approximation itself.
The authors validate these findings through analytical constructions, controlled quadratic experiments, and one-step loss measurements on 30 frozen state–block combinations from three training trajectories of a 124M-parameter Transformer.
Critical strengths
-
Insightful problem formulation. The fixed-polar construction offers an interesting perspective on Muon's many-to-one mapping, raising a meaningful question about whether spectral information alone can explain the benefits of spectral flattening.
-
Rigorous theoretical analysis. The authors provide a nontrivial mathematical construction demonstrating that identical spectral information and gradient alignment can lead to opposite preferences between Muon and its comparator, highlighting the role of curvature and orientation.
Critical weaknesses
-
Limited generality and practical relevance. The preference reversal is demonstrated on a carefully constructed quadratic objective. Although the authors provide a local theoretical generalization, its prevalence and significance in realistic LLM training remain unclear. The experiments focus on one-step loss responses rather than long-term training dynamics.
-
Limited significance of the endpoint-gradient analysis. The improved local prediction follows naturally from a Taylor expansion at the update endpoint. Simple scalar calibration recovers most of the reported gains, and the analysis provides limited actionable insights for understanding or improving Muon.
-
Unclear optimization implications of numerical fidelity. The authors demonstrate that low-precision Newton–Schulz computations can distort the ideal response, but do not establish whether these deviations improve or degrade actual training performance. In particular, it remains unclear why closer approximation to the exact polar factor should necessarily lead to better optimization.
Questions for authors
- How prevalent and significant are the observed effects in realistic LLM training? Specifically, how frequently does the norm-matched comparator outperform Muon, i.e., ? How large are the orientation-induced changes relative to the baseline advantage ?
- How sensitive are the conclusions to the choice of Frobenius-norm matching? Given that Muon's polar update is associated with steepest descent under a spectral-norm constraint, would the conclusions change under spectral-norm matching?
- To what extent does one-step loss advantage predict long-term training behavior? Since the experiments focus on frozen model states, i wonder whether the observed orientation-dependent effects translate into meaningful differences in convergence. Could the authors provide evidence connecting these local observations to actual training dynamics?
- Why should a closer approximation to the exact polar factor necessarily lead to better optimization? The spectral distortions introduced by finite round NS iter may not always be detrimental. Have the authors investigated whether these deviations improve or degrade actual loss reduction or training convergence?