- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates the interaction between inner perturbation geometry and outer optimizers in Sharpness-Aware Minimization (SAM). The authors introduce SpecSAM, which uses layerwise spectral-norm perturbations for the inner step, and uses other optimizers like AdamW, SGDW or Muon for the outer step. Experiments show that SpecSAM-Muon achieves the best validation accuracy among the evaluated methods.
Critical strengths
- How SAM's inner perturbation geometry and the outer optimizer interact is an interesting question.
- The paper compares different combinations of inner perturbations and outer optimizers. Ablation studies test the effects of layerwise normalization and spectral geometry. Additional effective-rank and sharpness analyses provide further empirical observations.
Critical weaknesses
- The paper explicitly claims that it's an empirical investigation paper and aims to understand how different geometric choices interact in SAM's inner and outer steps. However, the main findings are largely descriptive observations of performance differences, and no general principles are found. The final conclusion, explanation of results and insights are limited. The effective-rank analysis offers a possible hypothesis but does not establish a convincing connection to the observed performance differences.
- The discussed optimization geometries in the paper are Euclidean and Spectral norm, which are limited. [1] has discussed multiple norm choices for matrices. A broad discussion and empirical results across more norm choices may give a clearer conclusion.
- Prior works have already explored alternative perturbation geometries in SAM, including nuclear-norm-constrained perturbations [2] and preconditioned perturbations [3]. Although their motivations and formulations are a bit different, discussing the differences between these works and what new insights are obtained from this paper would be useful.
- The main experiments are restricted to vision tasks. Considering the impact of Muon on LLM training, an additional experiment on a language task would be helpful.
- The experiments compare methods under the same epoch setting. However, SAM requires approximately double the computational cost compared with base optimizers. The same wall-clock comparison would be a fair and necessary setting.
[1] Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, and Volkan Cevher. "Training deep learning models with norm-constrained lmos." Advances in Neural Information Processing Systems, 2025.
[2] Thomas Pethick, Parameswaran Raman, Lenon Minorics, Mingyi Hong, Shoham Sabach, and Volkan Cevher. "νSAM: Memory-efficient sharpness-aware minimization via nuclear norm constraints." Transactions on Machine Learning Research, 2025.
[3] Yilang Zhang, Bingcong Li, and Georgios B. Giannakis. "Preconditioned sharpness-aware minimization: Unifying analysis and a novel learning algorithm." ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025.
Questions for authors
- Can the authors provide further explanations or general insights into why spectral perturbations benefit Muon but not AdamW/SGDW? And why can $ \ ell_2$ perturbation benefit both Muon and AdamW/SGDW? How do inner and outer steps in SAM interact?
- Why do the authors consider only Euclidean and spectral norms? Would exploring additional matrix norms, as discussed in [1], provide a more comprehensive understanding?
- Could the authors provide language-modeling experiments to validate whether the findings generalize to LLM training?
- Could the authors provide a comparison under the same wall-clock time?
3 comments
Rebuttal ZhTJ (2/2)
Could the authors provide language-modeling experiments to validate whether the findings generalize to LLM training?
We have added an experiment (Sec. 4.3) where we perform full fine-tuning of Qwen2.5-1.5B on Commonsense170K for two epochs, using the same rigorous two-stage hyperparameter tuning protocol as in the ImageNet experiments. We report the results on eight commonsense-reasoning benchmarks over five seeds.
Though the differences between the methods are smaller than on ImageNet-1K, we observe a similar interaction between inner and outer geometry. As mentioned earlier, the interaction between inner geometry and outer optimizer is statistically significant in this experiment, with SpecSAM-Muon achieving the highest average accuracy (81.30%).
Could the authors provide a comparison under the same wall-clock time?
We have added a compute-matched comparison in Appendix C. We first measure the throughput of all six methods on both ImageNet-1K architectures, finding that SAM and SpecSAM have approximately half the throughput of their corresponding base optimizers, and very similar throughput to one another. Therefore, running SAM and SpecSAM methods for half as many epochs as non-SAM methods (i.e. matching the number of gradient evaluations) provides a simple way to approximately match wall-clock times. The results are given in the table below:
| Method | ViT-Small/16 | ResNet-50 |
|---|---|---|
| AdamW/SGDW | 74.36 | 77.28 |
| Muon | 74.92 | 77.01 |
| SAM-AdamW/SGDW | 78.26 | 76.95 |
| SAM-Muon | 79.25 | 77.30 |
| SpecSAM-AdamW/SGDW | 78.12 | 76.99 |
| SpecSAM-Muon | 79.65 | 78.03 |
SpecSAM-Muon achieves the highest validation accuracy under this budget on both architectures. In particular, while neither SAM-SGDW nor SAM-Muon improves upon SGDW at a matched number of gradient evaluations on ResNet-50, SpecSAM-Muon improves upon it by 0.69 percentage points.
We hope that these additional experiments, particularly the statistical analysis of the interaction and the language-model and compute-matched experiments, address your concerns regarding the empirical findings and their generality. We would be grateful if you would reconsider your assessment of our paper, for the individual categories and overall.
References:
Pierfrancesco Beneventano, Mahmoud Abdelmoneum, and Tomaso Poggio. The spectral dynamics and noise geometry of muon, 2026. URL https://arxiv.org/abs/2606.08388.
Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts, particularly the language-modeling experiments. I have therefore increased my score to weak reject (2).
However, my main concern remains. The paper explicitly aims to provide insights into the interaction between inner and outer optimization geometries, but the results remain largely empirical observations without a convincing explanation of the underlying mechanisms or broader generalizable principles. Therefore, I remain inclined toward rejection.
Rebuttal ZhTJ (1/2)
Apologies for the delayed response. Thank you for taking the time to review our paper, and for your suggestions. In response to your comments, we have conducted additional experiments on language-model fine-tuning, transfer learning and compute-matched training, as well as expanding our statistical analysis of the interaction, and our discussion of the method's motivation, related work, and results. Changes are highlighted in blue.
We find that the relative effectiveness of spectral perturbations depends on the outer optimizer; naively inserting Muon into regular Euclidean SAM is suboptimal, and we demonstrate an efficient practical prescription for performing spectral perturbations instead, which maximize the benefit of Muon in this setting.
To more rigorously establish the interaction between inner and outer geometry, we have added a statistical test of whether replacing Euclidean perturbations with spectral perturbations benefits Muon more than AdamW/SGDW (Appendix D). The estimated interaction is positive for all three models studied, and statistically significant for ResNet-50 (p=0.010) and the newly-added Qwen2.5-1.5B task (p=0.031), though not for ViT-Small/16 (p=0.079, though this is likely because the standard error across seeds here is much higher than the other 2 settings).
Furthermore, the LayerSAM ablation demonstrates that retaining the layerwise perturbation structure but replacing the spectral norm with the Frobenius norm removes the advantage of SpecSAM-Muon, strengthening the importance of the spectral norm.
We believe these results provide solid evidence on the choice of perturbation geometry when using SAM in conjunction with Muon, as well as providing an immediately practical optimizer.
We have provided further explanation in our updated discussion. From Eq. 3, standard SAM may be interpreted as penalizing the overall magnitude of the gradient. SpecSAM, on the other hand, penalizes the nuclear norms of the layerwise gradient matrices. The latter form of regularization specifically depends on the spectral structure of the gradients, and our intuition is that its effectiveness depends on the implicit bias of the outer optimizer.
In particular, recent theory on Muon's implicit bias (Beneventano et al. 2026) suggests that Muon finds solutions with flatter spectra, which agrees with our observation of Muon finding high effective rank solutions. One hypothesis is that Muon's implicit bias allows it to benefit from spectral sharpness regularization (penalise sum of singular values) while retaining sufficient representational capacity (spread information across singular values, instead of collapsing to a couple of directions).
In order to retain practical relevance, the choice of geometries was motivated by the optimizers commonly used in practice (SGD, Adam, and Muon). While works like Bernstein & Newhouse (2025) neatly formulated the difference between modern optimisers like SGD and Muon as steepest descent under different step-size norms, we are not aware of any previous work that then uses this insight to propose a more principled way to use Muon in sharpness aware minimization. Given compute constraints, we therefore focused on performing a rigorous evaluation on these 3 optimizers. We agree that exploring a wider range of norms and their interactions would be a very interesting study, as well as diving deeper into the theoretical mechanisms underpinning the interactions, and the norm-agnostic setup we use would allow such a study, but we leave this to future work, and point out that our work demonstrates that such a direction is has actual practical relevance.
We have expanded our related work section (Sec. 5) to discuss these works and better distinguish our contribution.
In particular, νSAM constrains the perturbation under the nuclear norm, yielding rank-one perturbations with the aim of reducing the memory overhead of SAM. In contrast, we constrain the perturbation under the spectral norm, yielding potentially full-rank perturbations While prior work has studied the choice of geometry used to define sharpness, our focus is on how the effectiveness of this choice depends on the outer optimizer.