acceptodds
Under review as a conference paper at ICLR 2027

Orthogonalization Separates Sharpness Exploration from Perturbed Gradient Updates

Abstract

In Sharpness-Aware Minimization (SAM), the perturbed gradient updates the model parameters, while the clean gradient determines the parameter perturbation. This raises a natural question: does matrix orthogonalization have different effects when applied to the parameter perturbation versus the subsequent parameter update? We thus investigate how SAM changes when matrix orthogonalization is introduced separately into these two stages: 1) for clean gradient, orthogonalization changes the first order sharpness term from a Frobenius norm quantity to a normalized nuclear norm quantity, providing a nuclear norm view in sharpness exploration; 2) for perturbed gradient, orthogonalization changes how gradient variation caused by the perturbation enters the final update. We further analyze the effects of finite step Newton–Schulz approximation and global normalization on these two mechanisms. With the above analysis, MuonSAM instantiates this construction by orthogonalizing both the clean and perturbed gradients. Experiments on vision and language tasks verify the theoretical predictions and provide evidence for the roles of orthogonalization in sharpness exploration and model updates.

Primary area:
optimization
Keywords:
Sharpness-aware optimization.Muon.Orthogonalization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_hpcrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The work investigates the interplay between sharpness aware minimization (SAM) and gradient orthogonalization method (MUON). A new optimization scheme is proposed combining both, each step uses Newton-Schulz to orthogonalize gradients and the lookahead step from the original SAM is kept. The resulting geometry is theoretically studied by deriving sensitivity bounds involving the nuclear norm instead of the Frobenius norm. The work evaluates on both language and vision tasks. On top of that, a robustness and a brief compute cost analysis is provided.

Critical strengths

The combination of Muon and SAM is a natural extension and a worthwhile exploration for understanding their interaction and can lead to improved generalization performance.

Critical weaknesses
  1. The experimental validation is inadequate. The optimization method comparison is not compute matched. Muon uses roughly half the compute per epoch a more fair comparison would be doubling its epochs or halving the epochs of MuonSAM. Moreover, the experimental scale is limited which makes it hard to evaluate the significance of the resulting combination, especially for the language models which are only evaluated in a finetuning setting. Furthermore standard ViT training protocol is 300 epochs, and uses several data augmentations such as Mixup and Cutmix to improve generalization performance. For the robustness experiments there is also no theoretical guidance provided for why we would expect MuonSam to work better. Finally, the theory seems disconnected from the empirical evaluation which mainly focusses on performance improvements.

  2. Novelty of the proposed method is not clear as previous work composed general spectral norm methods with SAM in [1].

[1] Zhong, Wenzhi et al. “Sharpness-Aware Minimization and Muon: Robustness under the Spectral Norm.” ArXiv abs/2607.26001 (2026): n. pag.

Questions for authors

Can the authors provide training runs that are compute matches for the direct compared methods?

Can the authors evaluate language models in a compute matched pretraining setting?

Can the authors discuss the relative novelty compared to previous work?

Reviewer_54xSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the roles of matrix orthogonalization in the two stages of SAM. The authors propose MuonSAM, which applies Muon-style approximate orthogonalization to both stages for matrix parameters, while using separate treatment for other parameters. The theoretical analysis examines the first-order sensitivity of polar perturbations, Newton–Schulz approximation, effective perturbation magnitudes under global normalization, and the relation between inner and outer momentum directions. Experiments on vision and language tasks report improvements over the corresponding Muon baselines, alongside robustness, efficiency, and numerical diagnostic evaluations.

Critical strengths

S1. The paper distinguishes the role of orthogonalization in selecting a perturbation from its role in forming the final update. This is a useful way to study the interaction between Muon and SAM.

S2. Beyond introducing MuonSAM, the authors examine polar-direction sensitivity, finite-step Newton–Schulz approximation, realized perturbation magnitudes, and inner–outer momentum relations. The accompanying diagnostics help substantiate these mathematical observations.

S3. The method improves upon matched Muon baselines on the evaluated vision and language tasks. The robustness evaluations and runtime measurements also provide useful practical context.

Critical weaknesses

W1. The paper focuses on orthogonalization in both perturbation construction and parameter updates, but the current performance comparisons do not fully disentangle their individual effects. A controlled ablation comparing orthogonalization in the perturbation stage only, the update stage only, and both stages together would make the main contribution more convincing.

W2. The authors provide a comprehensive theoretical analysis covering aspects such as the nuclear norm, the Newton–Schulz approximation, the perturbation radius, and the momentum direction, and they also verify the relevant mathematical properties through numerical experiments. However, while these analyses primarily explain how orthogonalization alters the optimization process, they do not sufficiently explain why these changes lead to improved final model accuracy.

W3. The paper notes that the actual perturbation radius of MuonSAM is influenced by factors such as the global gradient norm and matrix rank, and thus does not necessarily equal the specified value ρ\rho. While the authors have conducted theoretical and numerical analyses of this variation, it remains unclear from the primary performance comparisons whether the actual perturbation magnitudes across different optimizers are truly comparable. It is suggested that the authors provide a brief analysis or compare performance while matching the actual perturbation norms, in order to better demonstrate the specific impact of the orthogonalization direction itself.

Questions for authors
  • Could the authors provide an ablation that separates the effect of orthogonalization in the perturbation stage from its effect in the update stage? Does using both stages provide additional benefit beyond either one alone?
  • Among the analyzed geometric properties, which do the authors consider most important for explaining the accuracy improvements? Is there empirical evidence directly linking these properties to final performance? That is, can the authors justify why it is necessary to apply matrix orthogonalization? The current work only combines Muon and SAM in a simple way.
  • How do the conclusions change when methods are compared using similar realized perturbation magnitudes rather than only nominal perturbation scales? Moreover, results on ImageNet-1K are also missing. The performance of MuonSAM may differ on the small-scale and large-scale datasets.
  • Symbol inconsistency in Algorithm 1. And the paper uses "MuonSAM" in some places and "MUONSAM" in others, while Figure 1 uses "MUON-SAM".
Reviewer_5o27OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks a question that is how MUON + SAM will affect the model performance. The paper provides theoretical analysis and experimental findings. More particular, Theorem 1 obtained a Taylor approximation of the first order and bound for the second order terms under Muon + SAM perturbation. Proposition 1 gave a bound for the difference between matrices input and output using Nesterov interpolation, under some Lipschitz-conditions. Theorem 2 showed a formula after KK iterations. Experimental results show that Muon+SAM is better from 1-2.7% in 5 settings. However, in the first setting, Muon + SAM is even worse than SAM alone.

Critical strengths

The paper is easy to follow. It provides some bounds and formulas for the case of Muon + SAM. The empirical results show a gain in 5 presented settings.

Critical weaknesses

The maths in this work are quite simple and straightforward from the theoretical results of SAM and Muon. Under some certain conditions involving Lipschitz conditions, we expand the Taylor expansion to obtain some certain bounds.

Mathematically, we knew that Muon sets all singular values to 1, then it will change the sharpness from Frobenius norm to nuclear norm. But it does not guarantee a better generalization. By a standard of theoretical paper, I would give it a rejection.

There is no explanation for the failed case of CIFAR-10.

The empirical evaluations relies on outdated answers small-scale architecture.

Questions for authors

In Table 2, what are possible reasons why SAM has a better average than that of Muon + SAM?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.