- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper interprets Muon as an accumulation-free Kronecker-factored curvature approximation, using the momentum Gram as the input curvature factor and the identity as the output factor. The authors provide a signal-noise analysis, evaluate curvature approximation accuracy against GGN estimates in neural networks, and conduct controlled experiments investigating estimation noise, output-scale heterogeneity, and output coupling.
Critical strengths
-
Systematic empirical analysis. The paper evaluates Muon's implicit curvature approximation against a GGN reference, distinguishing subspace alignment, spectral accuracy, and output-side curvature structure. These measurements provide useful insights beyond existing algebraic interpretations.
-
Informative controlled experiments. The experiments separately investigate estimation noise, output-scale heterogeneity, and output coupling, helping identify different limitations of Muon's implicit curvature approximation.
Critical weaknesses
-
Insufficient differentiation from prior work. The structured-Fisher interpretation of Muon has already been investigated by [1]. The present paper provides more extensive empirical diagnostics, but its conceptual advances over existing curvature-based interpretations are not sufficiently clear.
-
Limited theoretical justification. The signal-noise analysis assumes stationary, independent gradients and does not account for momentum accumulated along the optimization trajectory. Moreover, the identity does not necessarily imply alignment with leading curvature directions, as this depends on the structure of . Consequently, the theoretical analysis provides only a conditional explanation for the curvature alignment observed in experiments.
-
Unclear connection between curvature approximation and optimization performance. Although the paper finds that lower spectral mismatch often corresponds to better training loss, changing the curvature estimation window also affects the update direction and optimization dynamics. It is therefore unclear whether the performance differences are actually caused by more accurate curvature estimation. While the controlled experiments demonstrate the effects of different curvature structures, the extent to which Muon's curvature approximation errors affect its performance on real language models remains unclear.
[1] Wenbo Gong, Meyer Scetbon, Chao Ma, and Ted Meeds. Towards efficient optimizer design for LLM via structured Fisher approximation with a low-rank extension. In International Conference on Learning Representations (ICLR), 2026.
Questions for authors
- Could the authors clarify the main conceptual advances over the structured-Fisher interpretation of Muon in Gong et al. (2025), particularly beyond the empirical characterization of curvature approximation accuracy?