acceptodds
Under review as a conference paper at ICLR 2027

Sharpness-Aware Minimization and Muon: Robustness Under the Spectral Norm

Abstract

Sharpness-Aware Minimization (SAM) aims to improve generalization by encouraging insensitivity to small, worst-case parameter perturbations. However, the notion of a “small” perturbation is inherently geometry-dependent: while existing SAM variants have explored a wide range of choices, a clear perspective on which geometries are most effective in practice remains elusive. Recent work on matrix-aware optimization, particularly the Muon optimizer, suggests that respecting the matrix structure of hidden-layer weights can lead to strong empirical performance. Motivated by this, we study matrix-aware geometry in both stages of SAM: we introduce a layerwise spectral inner perturbation for matrix-valued hidden-layer parameters and combine it with either AdamW/SGDW or Muon in the outer update. Across ImageNet-1K experiments on ViT-Small/16 and ResNet-50, we find that the combination of a spectral inner step with a Muon outer step performs consistently strongly, achieving the best validation accuracy on both models among the evaluated methods.

Primary area:
optimization
Keywords:
Sharpness-Aware Minimization, Muon, Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZhTJOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the interaction between inner perturbation geometry and outer optimizers in Sharpness-Aware Minimization (SAM). The authors introduce SpecSAM, which uses layerwise spectral-norm perturbations for the inner step, and uses other optimizers like AdamW, SGDW or Muon for the outer step. Experiments show that SpecSAM-Muon achieves the best validation accuracy among the evaluated methods.

Critical strengths
  1. How SAM's inner perturbation geometry and the outer optimizer interact is an interesting question.
  2. The paper compares different combinations of inner perturbations and outer optimizers. Ablation studies test the effects of layerwise normalization and spectral geometry. Additional effective-rank and sharpness analyses provide further empirical observations.
Critical weaknesses
  1. The paper explicitly claims that it's an empirical investigation paper and aims to understand how different geometric choices interact in SAM's inner and outer steps. However, the main findings are largely descriptive observations of performance differences, and no general principles are found. The final conclusion, explanation of results and insights are limited. The effective-rank analysis offers a possible hypothesis but does not establish a convincing connection to the observed performance differences.
  2. The discussed optimization geometries in the paper are Euclidean and Spectral norm, which are limited. [1] has discussed multiple norm choices for matrices. A broad discussion and empirical results across more norm choices may give a clearer conclusion.
  3. Prior works have already explored alternative perturbation geometries in SAM, including nuclear-norm-constrained perturbations [2] and preconditioned perturbations [3]. Although their motivations and formulations are a bit different, discussing the differences between these works and what new insights are obtained from this paper would be useful.
  4. The main experiments are restricted to vision tasks. Considering the impact of Muon on LLM training, an additional experiment on a language task would be helpful.
  5. The experiments compare methods under the same epoch setting. However, SAM requires approximately double the computational cost compared with base optimizers. The same wall-clock comparison would be a fair and necessary setting.

[1] Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, and Volkan Cevher. "Training deep learning models with norm-constrained lmos." Advances in Neural Information Processing Systems, 2025.

[2] Thomas Pethick, Parameswaran Raman, Lenon Minorics, Mingyi Hong, Shoham Sabach, and Volkan Cevher. "νSAM: Memory-efficient sharpness-aware minimization via nuclear norm constraints." Transactions on Machine Learning Research, 2025.

[3] Yilang Zhang, Bingcong Li, and Georgios B. Giannakis. "Preconditioned sharpness-aware minimization: Unifying analysis and a novel learning algorithm." ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025.

Questions for authors
  1. Can the authors provide further explanations or general insights into why spectral perturbations benefit Muon but not AdamW/SGDW? And why can $ \ ell_2$ perturbation benefit both Muon and AdamW/SGDW? How do inner and outer steps in SAM interact?
  2. Why do the authors consider only Euclidean and spectral norms? Would exploring additional matrix norms, as discussed in [1], provide a more comprehensive understanding?
  3. Could the authors provide language-modeling experiments to validate whether the findings generalize to LLM training?
  4. Could the authors provide a comparison under the same wall-clock time?
3 comments
AuthorsOpenReview ↗

Rebuttal ZhTJ (1/2)

Apologies for the delayed response. Thank you for taking the time to review our paper, and for your suggestions. In response to your comments, we have conducted additional experiments on language-model fine-tuning, transfer learning and compute-matched training, as well as expanding our statistical analysis of the interaction, and our discussion of the method's motivation, related work, and results. Changes are highlighted in blue.

the main findings are largely descriptive observations of performance differences, and no general principles are found.

We find that the relative effectiveness of spectral perturbations depends on the outer optimizer; naively inserting Muon into regular Euclidean SAM is suboptimal, and we demonstrate an efficient practical prescription for performing spectral perturbations instead, which maximize the benefit of Muon in this setting.

To more rigorously establish the interaction between inner and outer geometry, we have added a statistical test of whether replacing Euclidean perturbations with spectral perturbations benefits Muon more than AdamW/SGDW (Appendix D). The estimated interaction is positive for all three models studied, and statistically significant for ResNet-50 (p=0.010p=0.010) and the newly-added Qwen2.5-1.5B task (p=0.031p=0.031), though not for ViT-Small/16 (p=0.079p=0.079, though this is likely because the standard error across seeds here is much higher than the other 2 settings).

Furthermore, the LayerSAM ablation demonstrates that retaining the layerwise perturbation structure but replacing the spectral norm with the Frobenius norm removes the advantage of SpecSAM-Muon, strengthening the importance of the spectral norm.

We believe these results provide solid evidence on the choice of perturbation geometry when using SAM in conjunction with Muon, as well as providing an immediately practical optimizer.

Can the authors provide further explanations or general insights into why spectral perturbations benefit Muon but not AdamW/SGDW? And why can [Euclidean] perturbation benefit both Muon and AdamW/SGDW?

We have provided further explanation in our updated discussion. From Eq. 3, standard SAM may be interpreted as penalizing the overall magnitude of the gradient. SpecSAM, on the other hand, penalizes the nuclear norms of the layerwise gradient matrices. The latter form of regularization specifically depends on the spectral structure of the gradients, and our intuition is that its effectiveness depends on the implicit bias of the outer optimizer.

In particular, recent theory on Muon's implicit bias (Beneventano et al. 2026) suggests that Muon finds solutions with flatter spectra, which agrees with our observation of Muon finding high effective rank solutions. One hypothesis is that Muon's implicit bias allows it to benefit from spectral sharpness regularization (penalise sum of singular values) while retaining sufficient representational capacity (spread information across singular values, instead of collapsing to a couple of directions).

Why do the authors consider only Euclidean and spectral norms? Would exploring additional matrix norms, as discussed in [1], provide a more comprehensive understanding?

In order to retain practical relevance, the choice of geometries was motivated by the optimizers commonly used in practice (SGD, Adam, and Muon). While works like Bernstein & Newhouse (2025) neatly formulated the difference between modern optimisers like SGD and Muon as steepest descent under different step-size norms, we are not aware of any previous work that then uses this insight to propose a more principled way to use Muon in sharpness aware minimization. Given compute constraints, we therefore focused on performing a rigorous evaluation on these 3 optimizers. We agree that exploring a wider range of norms and their interactions would be a very interesting study, as well as diving deeper into the theoretical mechanisms underpinning the interactions, and the norm-agnostic setup we use would allow such a study, but we leave this to future work, and point out that our work demonstrates that such a direction is has actual practical relevance.

Prior works have already explored alternative perturbation geometries in SAM, including nuclear-norm-constrained perturbations [2] and preconditioned perturbations [3].

We have expanded our related work section (Sec. 5) to discuss these works and better distinguish our contribution.

In particular, νSAM constrains the perturbation under the nuclear norm, yielding rank-one perturbations with the aim of reducing the memory overhead of SAM. In contrast, we constrain the perturbation under the spectral norm, yielding potentially full-rank perturbations While prior work has studied the choice of geometry used to define sharpness, our focus is on how the effectiveness of this choice depends on the outer optimizer.

AuthorsOpenReview ↗

Rebuttal ZhTJ (2/2)

Could the authors provide language-modeling experiments to validate whether the findings generalize to LLM training?

We have added an experiment (Sec. 4.3) where we perform full fine-tuning of Qwen2.5-1.5B on Commonsense170K for two epochs, using the same rigorous two-stage hyperparameter tuning protocol as in the ImageNet experiments. We report the results on eight commonsense-reasoning benchmarks over five seeds.

Though the differences between the methods are smaller than on ImageNet-1K, we observe a similar interaction between inner and outer geometry. As mentioned earlier, the interaction between inner geometry and outer optimizer is statistically significant in this experiment, with SpecSAM-Muon achieving the highest average accuracy (81.30%).

Could the authors provide a comparison under the same wall-clock time?

We have added a compute-matched comparison in Appendix C. We first measure the throughput of all six methods on both ImageNet-1K architectures, finding that SAM and SpecSAM have approximately half the throughput of their corresponding base optimizers, and very similar throughput to one another. Therefore, running SAM and SpecSAM methods for half as many epochs as non-SAM methods (i.e. matching the number of gradient evaluations) provides a simple way to approximately match wall-clock times. The results are given in the table below:

MethodViT-Small/16ResNet-50
AdamW/SGDW74.3677.28
Muon74.9277.01
SAM-AdamW/SGDW78.2676.95
SAM-Muon79.2577.30
SpecSAM-AdamW/SGDW78.1276.99
SpecSAM-Muon79.6578.03

SpecSAM-Muon achieves the highest validation accuracy under this budget on both architectures. In particular, while neither SAM-SGDW nor SAM-Muon improves upon SGDW at a matched number of gradient evaluations on ResNet-50, SpecSAM-Muon improves upon it by 0.69 percentage points.

We hope that these additional experiments, particularly the statistical analysis of the interaction and the language-model and compute-matched experiments, address your concerns regarding the empirical findings and their generality. We would be grateful if you would reconsider your assessment of our paper, for the individual categories and overall.

References:

Pierfrancesco Beneventano, Mahmoud Abdelmoneum, and Tomaso Poggio. The spectral dynamics and noise geometry of muon, 2026. URL https://arxiv.org/abs/2606.08388.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts, particularly the language-modeling experiments. I have therefore increased my score to weak reject (2).

However, my main concern remains. The paper explicitly aims to provide insights into the interaction between inner and outer optimization geometries, but the results remain largely empirical observations without a convincing explanation of the underlying mechanisms or broader generalizable principles. Therefore, I remain inclined toward rejection.

Reviewer_sajSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigate the problem that in Sharpness-Aware Minimization (SAM), what is the right notion of a "small" weight perturbation? The authors rewrite SAM in a norm-agnostic framework that cleanly separates two design choices: the geometry of the inner adversarial perturbation and the optimizer used for the outer update. Then they study the combination systematically. Their main proposal, SpecSAM, applies a layerwise spectral-norm perturbation to matrix-valued hidden-layer parameters (the steepest-ascent direction under a spectral budget, computed by orthogonalizing each layer's gradient with a few Newton–Schulz iterations), while non-matrix parameters keep the standard Euclidean perturbation with a separately tuned radius. The outer step is instantiated with either AdamW/SGDW or Muon, giving a grid of six methods. The central finding is an interaction between the two stages. A Euclidean inner perturbation helps AdamW and Muon by roughly the same amount, but the spectral inner perturbation selectively amplifies Muon: SpecSAM with a Muon outer step achieves the best validation accuracy on both architectures tested (80.23 top-1 on ViT-Small/16, 78.55 on ResNet-50, ImageNet-1K), whereas the same spectral perturbation paired with AdamW/SGDW is markedly weaker. An ablation with layerwise Frobenius-normalized perturbations (LayerSAM) shows the gains come from the spectral geometry itself, not merely from per-layer budget allocation, and only materialize when the outer optimizer is also matrix-aware. Validation consists of full ImageNet-1K training runs on a ViT and a ResNet, with three seeds, significance testing, and tuned hyperparameters shared fairly between baselines and SAM variants. Supporting diagnostics on CIFAR-100 sweep the perturbation radius and track the train/validation gap, effective rank, and post-training sharpness. The authors are candid about limitations: only two vision architectures, three seeds, equal-epoch rather than equal-wall-clock comparison, and no settled explanation for why the two geometries interact.

Critical strengths
  1. The inner/outer interaction finding is a real, consequential empirical contribution. The paper's central result is a non-trivial, well-supported discovery rather than a single-model anecdote. It replicates across a ViT and a ResNet with opposite base-optimizer preferences (Muon ahead of AdamW on ViT, behind on ResNet), and it is immediately actionable: "pair spectral sharpness with a matrix-aware outer step" is practical guidance the community can adopt. This finding, not the algorithm itself (a momentum-free Muon ascent step is technically simple), is exactly the kind of systematic, control-driven empirical knowledge that the fast-moving Muon/SAM literature currently lacks.

  2. The experimental hygiene makes the causal claim defensible. The two-stage tuning protocol removes the most common confound in SAM comparisons, where a method wins through better base hyperparameters. Critically, the LayerSAM ablation cleanly separates "per-layer budget allocation" from "spectral norm choice," showing the gain comes specifically from spectral geometry and only in combination with a matrix-aware outer step. Without this control, the headline result would be attributable to the mere act of perturbing layers independently. Together with 3 seeds, Welch t-tests, fully disclosed grids, and released code, this rigor is what lifts the paper above a "we tried SAM with Muon and it worked" report and anchors my positive assessment despite the modest conceptual novelty.

Critical weaknesses
  1. The paper asserts that "no work has studied the effects of spectral geometry in either the inner or outer step of the SAM setting," yet νSAM (Pethick et al., TMLR 2025) constrains the SAM perturbation via the nuclear norm, which is precisely the dual of the spectral norm. The norm-agnostic template is also largely a repackaging of the unified steepest-descent view of SAM/Muon already present in Bernstein & Newhouse (2025), which the paper cites but does not position itself against. This matters beyond a missing citation: the paper's contribution is an empirical interaction study, and its value hinges on the community reading it as such. As written, the framing overstates conceptual novelty and obscures how much of the method (momentum-free Muon ascent, orthogonalized perturbation) is recombination. If the authors cannot clearly articulate what is learned here that νSAM and the steepest-descent framing do not already imply, the paper risks reading as a incremental grid evaluation — a borderline contribution for ICLR even with good execution.

  2. The headline claim is an interaction, but no interaction is ever tested, and the evidence is thin exactly where it matters most. The paper's thesis is a difference-in-differences: spectral inner perturbation gains +5.28 with Muon vs +4.45 with AdamW (ViT), +1.50 vs +0.42 (ResNet). Yet the validation consists solely of pairwise Welch t-tests within each architecture; there is no formal test (or even a confidence interval) on the contrast itself, which with 3 seeds is substantially noisier than the pairwise comparisons reported. The decisive margins are small: 0.26 points separate SpecSAM-Muon from SAM-Muon on ViT, and the "ranking flip" on ResNet rests on a +1.50-vs-+0.42 contrast. Compounding this, the two geometries are compared after per-method radius tuning on different scales (spectral ρ around 5×10⁻⁴ vs SAM's ρ around 10⁻¹), so the "Euclidean is additive, spectral is selective" pattern could partly reflect differences in tuning difficulty or loss-landscape flatness across geometries rather than a true interaction. Since the interaction is the paper's only claim to conceptual significance, everything else (SpecSAM beats baselines) is a standard performance result. An unproven interaction on 3 seeds leaves the central scientific conclusion resting on suggestive rather than established evidence.

Questions for authors
  1. How does SpecSAM differ, empirically and conceptually, from νSAM (Pethick et al., TMLR 2025), which constrains the SAM perturbation by nuclear norm?

  2. Can you provide a formal statistical test of the interaction itself. For example, a two-way analysis (inner geometry × outer optimizer) or a confidence interval on the difference-in-differences (e.g., the +1.50 vs +0.42 contrast on ResNet-50) . Your conclusions rest on the interaction pattern, not on pairwise wins, and pairwise Welch t-tests do not establish that the interaction is nonzero.

  3. Could you disentangle the spectral geometry from per-geometry tuning difficulty by reporting, on ImageNet or a substantial subset, the accuracy of each SAM variant as a function of its perturbation radius alongside the selected (tuned) operating points? If the "Euclidean is additive, spectral is selective" pattern holds across radii rather than only at each method's individually tuned optimum, the interaction claim would be much stronger and the tuning confound largely eliminated.

2 comments
AuthorsOpenReview ↗

Rebuttal sajS (1/2)

Apologies for the delayed response. Thank you for taking the time to review our paper, and for your constructive comments. We are glad that you found the inner/outer interaction to be an interesting empirical finding and appreciated the experimental methodology. We agree with your characterization of the intended contribution of our work. We have conducted several additional experiments and revised the exposition and discussion in response to your comments. Changes in our new draft are highlighted in blue.

The paper asserts that "no work has studied the effects of spectral geometry in either the inner or outer step of the SAM setting," yet νSAM (Pethick et al., TMLR 2025) constrains the SAM perturbation via the nuclear norm, which is precisely the dual of the spectral norm.

How does SpecSAM differ, empirically and conceptually, from νSAM (Pethick et al., TMLR 2025), which constrains the SAM perturbation by nuclear norm?

Thank you for pointing out this omission. We have expanded our related work section (Sec. 5) to discuss further works, including νSAM, and have made sure our wording is precise in other areas such as the introduction.

Although the two methods are closely related mathematically, the perturbations they produce are quite different: constraining perturbations under the nuclear norm yields rank-one perturbations, whereas constraining them under the spectral norm yields potentially full-rank perturbations. The motivation of νSAM is to exploit the rank-one structure of the perturbations to reduce the memory overhead of SAM, while we are interested in the performance benefits of introducing matrix-aware geometry into both steps of SAM. The contributions of our paper are neither investigated by νSAM nor implied by its results.

The norm-agnostic template is also largely a repackaging of the unified steepest-descent view of SAM/Muon already present in Bernstein & Newhouse (2025), which the paper cites but does not position itself against.

The norm-agnostic template should primarily be viewed as background rather than a contribution, and we have updated our writing to make this clear. Its purpose is to provide a common notation for the methods studied in the paper, and to clearly distinguish the geometry used to define the robust loss from the geometry used to minimize it. Nevertheless, Bernstein & Newhouse (2025) do not consider Sharpness Aware Minimization at all.

Can you provide a formal statistical test of the interaction itself. For example, a two-way analysis (inner geometry × outer optimizer) or a confidence interval on the difference-in-differences [...]?

Thank you for this suggestion! We have added a direct test of the interaction in Appendix D. Specifically, we compute the following difference-in-differences for each seed:

Δ=(ASpecSAM-Muon−ASAM-Muon)−(ASpecSAM-Base−ASAM-Base),\Delta = (A_{\text{SpecSAM-Muon}} - A_{\text{SAM-Muon}}) - (A_{\text{SpecSAM-Base}} - A_{\text{SAM-Base}}),

where Base denotes AdamW or SGDW, and AA denotes accuracy. This measures whether switching from Euclidean to spectral perturbations is more beneficial when using Muon. We perform a two-sided one-sample tt-test over the seed-wise differences.

The results of the interaction tests are as follows, and are given in Table 16 (including an additional experiment, performing full fine-tuning of Qwen2.5-1.5B on Commonsense170K over five seeds (Sec. 4.3)) (all differences are in percentage points):

ModelInteraction (Δ\Delta)Std. error95% CIpp-value
ViT-Small/16+0.740.22[-0.21, 1.69]0.079
ResNet-50+0.930.09[0.54, 1.33]0.010
Qwen2.5-1.5B+0.200.06[0.03, 0.37]0.031

The estimated interaction is positive in all three settings, and statistically significant for ResNet-50 and Qwen2.5-1.5B (we note the standard error across seeds for ViT is quite large, which explains its large p-value).

We have additionally evaluated the ImageNet-pretrained models on seven downstream classification datasets. SpecSAM-Muon achieves the highest average transfer accuracy for both architectures.

AuthorsOpenReview ↗

Rebuttal sajS (2/2)

Could you disentangle the spectral geometry from per-geometry tuning difficulty by reporting, on ImageNet or a substantial subset, the accuracy of each SAM variant as a function of its perturbation radius alongside the selected (tuned) operating points?

We would like to highlight that Sec. 4.2 (Fig. 2) investigates precisely this question, albeit on CIFAR-100 rather than ImageNet-1K. We sweep the perturbation radius for all four SAM variants, using the same learning rate and weight decay across methods. The results are consistent with our ImageNet observations: SpecSAM-AdamW performs worst, while SpecSAM-Muon handles the trade-off between training accuracy and the generalization gap slightly better than SAM-Muon.

We also note that the difference in the scale of the radii is to be expected, since the radii measure different quantities (a global Euclidean norm versus a layerwise spectral norm). We therefore tune the radii separately for each method, with the learning rate and weight decay fixed to those of the corresponding base optimizer.

Unfortunately, due to the computational cost, we have not conducted an analogous full-budget radius sweep on ImageNet-1K.

We hope that the additional statistical analysis, experiments and revisions to the framing of the paper address your concerns. Given your positive assessment of the empirical contribution and experimental methodology, we'd appreciate if you could re-evaluate your assessment of the paper.

Reviewer_o53xOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the interaction between the inner perturbation geometry of Sharpness-Aware Minimization (SAM) and the outer optimizer. The authors introduce SpecSAM, which uses layerwise spectral-norm perturbations for matrix-valued parameters, and combine it with AdamW/SGDW or Muon. Experiments on ImageNet-1K with ViT-Small/16 and ResNet-50 show that SpecSAM performs best when paired with Muon. The paper also analyzes sharpness, effective rank, and generalization behavior.

Critical strengths

The paper studies the interaction between SAM perturbation geometry and outer optimization, providing an interesting perspective on matrix-aware sharpness-aware training.

The proposed spectral perturbation has a clear mathematical formulation and can be integrated with existing optimizers.

The experimental design systematically compares different inner perturbations and outer optimizers, with additional layerwise ablations to isolate the effect of spectral geometry.

SpecSAM-Muon achieves the best validation accuracy on both ImageNet-1K architectures, and the paper provides additional sharpness and effective-rank analyses to investigate the observed behavior.

Critical weaknesses
  1. The spectral norm is used to measure the maximum impact of weight matrix perturbation on the input features. This motivation is relatively intuitive, but it is not a new idea. Muon itself is also an existing matrix-aware optimizer that processes the matrix update direction through orthogonalization. This paper introduces spectral geometry into the inner perturbation of SAM and combines it with the outer update of Muon. Overall, this is a relatively natural extension. Existing works such as ASAM, Fisher SAM, and Riemannian SAM have also studied different perturbation geometries. The authors need to clearly explain the technical innovation of SpecSAM compared to these methods and whether combining it with Muon has unique contributions beyond the simple combination of methods.

  2. One of the core findings of the paper is that the effect of Spectral SAM depends on the outer optimizer, but the explanation for this phenomenon is not sufficient. Table 2 shows that on ViT-Small/16, SpecSAM-AdamW is 0.46 percentage points lower than SAM-AdamW, while SpecSAM-Muon is 0.26 percentage points higher than SAM-Muon. On ResNet-50, there is a similar phenomenon: SpecSAM-SGDW is lower than SAM-SGDW, but SpecSAM-Muon is 0.71 percentage points higher than SAM-Muon. This indicates that spectral perturbation itself is not necessarily superior to Euclidean perturbation. The authors need to explain why spectral geometry combined with Muon generates benefits and what factors determine the compatibility relationship between different inner/outer geometries, rather than just remaining at the level of experimental observation.

  3. The inner perturbation and outer optimizer adopt similar spectral geometry, but this does not directly indicate that the two have better optimization compatibility. SpecSAM performs orthogonalization on the current gradient matrix in the inner step, while Muon performs orthogonalization on the momentum matrix in the outer step. Their optimization goals and actual update directions are not the same. The paper lacks direct theoretical analysis or mechanism experiments to explain why this combination can improve optimization dynamics. It is recommended to further analyze how spectral perturbation changes the perturbed gradient and how Muon handles these changes, and compare them with the update behaviors of other outer optimizers.

  4. The author explains the performance advantage of SpecSAM-Muon using a relatively high effective rank, but currently lacks direct evidence to support this mechanism. Appendix C.3.1 shows that Muon-based methods usually maintain high weights, gradients, and feature effective ranks. However, a higher effective rank does not necessarily mean better generalization, nor can it directly prove that the model is more suitable for spectral perturbation. The author has not established a clear relationship between effective rank and geometric compatibility. It is recommended to further analyze the connection between effective rank, perturbation strength, and validation performance, and verify whether a higher rank is indeed the main reason for SpecSAM-Muon's benefit.

  5. The experimental results of SpecSAM-Muon show some improvement, but the performance gain is relatively limited. Moreover, the sharpness analysis does not fully support its generalization mechanism. In Table 2, SpecSAM-Muon improves by 0.26 and 0.71 percentage points compared to SAM-Muon on ViT-Small/16 and ResNet-50, respectively. The advantage on ImageNet-R is also not supported by clear statistical significance. Additionally, Appendix C.3.2 indicates that, under a fixed generalization gap, SpecSAM variants often have greater sharpness in Euclidean and spectral sharpness. This suggests that better validation performance cannot simply be attributed to finding flatter solutions. The author needs to further explain this phenomenon and distinguish whether the performance improvement comes from flatness, optimization dynamics, or changes in representation capacity.

  6. The overall writing of the paper lacks intuitive motivation and mechanism explanations for machine learning researchers. The main text introduces the method through norm-agnostic formulation, dual norm, and matrix orthogonalization, but does not fully explain why spectral geometry was chosen and why this inner perturbation should be particularly suitable for Muon. The article also lacks framework figures, toy examples, and mechanism experiments that directly show the interaction between inner and outer geometries. It is suggested to organize the main text in accordance with the logical sequence of "research problem", "method motivation", "optimization mechanism" and "empirical evidence", and explain the key insights through simple machine learning examples and diagrams, so that readers can more easily understand the method design and its practical effects.

Questions for authors

Please see the weakness.

2 comments
AuthorsOpenReview ↗

Rebuttal o53x (1/2)

Apologies for the delayed response. Thank you for taking the time to review our paper, and for your constructive comments. We are glad that you found the research question interesting and appreciated the experimental design. We have conducted several additional experiments and revised the exposition and discussion in response to your comments. Changes in our new draft are highlighted in blue.

The authors need to clearly explain the technical innovation of SpecSAM compared to these methods and whether combining it with Muon has unique contributions beyond the simple combination of methods.

We have expanded the related work section (Sec. 5) to better distinguish our work from prior work on alternative SAM geometries, including νSAM and PreSAM as suggested by another reviewer. No previous work uses the spectral norm as a perturbation constraint (the closest is νSAM, who use its dual, the nuclear norm) or provides a practical prescription for introducing modern matrix-aware optimizers into sharpness aware minimization. It is not a priori clear that geometry which works well for minimizing the training loss (as in Muon) should also work well for defining the robust loss in SAM. Hence, our work provides a systematic study of this interaction.

The authors need to explain why spectral geometry combined with Muon generates benefits and what factors determine the compatibility relationship between different inner/outer geometries, rather than just remaining at the level of experimental observation.

We have updated our discussion with some clearer theoretical motivation. As discussed in Eq. 3, different choices of inner geometry amount to different forms of gradient-norm regularization. While standard SAM penalizes the overall magnitude of the gradient, SpecSAM penalizes the sum of the singular values of the layerwise gradient matrices. The latter is a form of regularization that specifically depends on the matrix structure of the gradients, and it is conceivable that its effectiveness depends more strongly on the implicit bias of the outer optimizer. Indeed, recent work (Beneventano et al. 2026) suggests that Muon finds solutions with flatter spectra, which lines up with our high effective rank observations, and one might hypothesize that this implicit bias might allow Muon to benefit from spectral sharpness regularization (penalize sum of singular values) while retaining sufficient representational capacity (spread across all singular values instead of collapsing on a few).

To more rigorously establish the interaction empirically, we have added a statistical test of whether switching from Euclidean to spectral perturbations benefits Muon more than AdamW/SGDW (Appendix D). The interaction is positive in all three experimental settings, and statistically significant for ResNet-50 (p=0.010p=0.010) and our new Qwen2.5-1.5B fine-tuning experiment (p=0.031p=0.031), though not quite for ViT-Small/16 (p=0.079p=0.079, likely due to the higher standard error across seeds in this setting).

SpecSAM performs orthogonalization on the current gradient matrix in the inner step, while Muon performs orthogonalization on the momentum matrix in the outer step. Their optimization goals and actual update directions are not the same.

While an idealised version would study SpectralGD in the outer step (orthogonalize gradient instead of momentum), we wished to keep the paper as practically relevant as possible, and momentum is a key component of practical Muon recipes. We do not believe the use of momentum in the outer step significantly alters the motivation of the method, and our interaction experiments show that a spectral inner step is specifically beneficial for Muon rather than AdamW / SGDW (Appendix D).

The author has not established a clear relationship between effective rank and geometric compatibility.

As mentioned above, we have updated our discussion to explain how the high effective rank bias we observe lines up with current Muon implicit bias theory, and why we might expect this to lead to combine favourably with a spectral inner step.

The experimental results of SpecSAM-Muon show some improvement, but the performance gain is relatively limited.

The performance gains of Muon vs AdamW is general are relatively small (yet Muon is now widely adopted) so we cannot reasonably expect dramatic differences here either, but the headline improvements are still statistically significant. The main point is to demonstrate that using a Euclidean perturbation step with a Muon outer optimiser is suboptimal, with a spectral perturbation step being a better approach (as demonstrated by our interaction experiments) yet efficient to implement.

We also take the opportunity to point out our other new favourable results in language modelling, transfer learning, and a compute-matched comparison.

AuthorsOpenReview ↗

Rebuttal o53x (2/2)

This suggests that better validation performance cannot simply be attributed to finding flatter solutions.

This is indeed a surprising result. As discussed in Sec. 4.2, the effectiveness of SAM depends on the trade-off between fitting the training data and reducing the generalization gap. In particular, SpecSAM-AdamW appears to sacrifice too much training accuracy for the reduction in generalization gap that it achieves. Future work could investigate this more thoroughly with tractable toy models to isolate the effects properly, but this is out of the scope of this work, which seeks to retain immediate practical relevance.

The overall writing of the paper lacks intuitive motivation and mechanism explanations for machine learning researchers.

We have added explanatory comments and signposting throughout Secs. 2 and 3. In particular, we now more explicitly connect the matrix-aware motivation for spectral geometry in neural network optimization to its use in the inner step of SAM, explain the regularizing effect of the different inner geometries, and distinguish the effects of layerwise budgeting from those of the matrix norm itself. As mentioned previously, we have added discussion of the reason for the inner-outer geometry interaction to our revision, though a true theory of the mechanism behind the interaction studied here would first require a solid explanation of Muon's superior performance more generally, which has not yet been settled in the literature, and so is out of scope for this work.

We hope that we have addressed your concerns, and would be grateful if you would reconsider your assessment of the paper.

References:

Pierfrancesco Beneventano, Mahmoud Abdelmoneum, and Tomaso Poggio. The spectral dynamics and noise geometry of muon, 2026. URL https://arxiv.org/abs/2606.08388.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.