- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The work investigates the interplay between sharpness aware minimization (SAM) and gradient orthogonalization method (MUON). A new optimization scheme is proposed combining both, each step uses Newton-Schulz to orthogonalize gradients and the lookahead step from the original SAM is kept. The resulting geometry is theoretically studied by deriving sensitivity bounds involving the nuclear norm instead of the Frobenius norm. The work evaluates on both language and vision tasks. On top of that, a robustness and a brief compute cost analysis is provided.
Critical strengths
The combination of Muon and SAM is a natural extension and a worthwhile exploration for understanding their interaction and can lead to improved generalization performance.
Critical weaknesses
-
The experimental validation is inadequate. The optimization method comparison is not compute matched. Muon uses roughly half the compute per epoch a more fair comparison would be doubling its epochs or halving the epochs of MuonSAM. Moreover, the experimental scale is limited which makes it hard to evaluate the significance of the resulting combination, especially for the language models which are only evaluated in a finetuning setting. Furthermore standard ViT training protocol is 300 epochs, and uses several data augmentations such as Mixup and Cutmix to improve generalization performance. For the robustness experiments there is also no theoretical guidance provided for why we would expect MuonSam to work better. Finally, the theory seems disconnected from the empirical evaluation which mainly focusses on performance improvements.
-
Novelty of the proposed method is not clear as previous work composed general spectral norm methods with SAM in [1].
[1] Zhong, Wenzhi et al. “Sharpness-Aware Minimization and Muon: Robustness under the Spectral Norm.” ArXiv abs/2607.26001 (2026): n. pag.
Questions for authors
Can the authors provide training runs that are compute matches for the direct compared methods?
Can the authors evaluate language models in a compute matched pretraining setting?
Can the authors discuss the relative novelty compared to previous work?