acceptodds
Under review as a conference paper at ICLR 2027

OrthoSAM: Orthogonal Perturbations for Sharpness-Aware Minimization

15 views

Abstract

Sharpness-Aware Minimization (SAM) improves generalization by minimizing the worst-case loss within a local neighborhood, encouraging the optimizer to find flatter minima. However, standard SAM treats neural network weights as flattened vectors and applies unconstrained perturbations in the Euclidean space, ignoring the intrinsic matrix structure of modern neural networks. As a result, the adversarial perturbation can become dominated by a few large singular directions, while other important directions receive much less attention. In this work, we propose **OrthoSAM**, a matrix-aware optimization method based on the idea that flatness should be explored more evenly across different directions. Instead of using unrestricted perturbations, OrthoSAM constrains perturbations to a structured manifold with orthogonality properties, reducing the imbalance caused by dominant singular values. This encourages a more uniform exploration of the local loss landscape across matrix directions. To make the method efficient in practice, we use the Newton–Schulz iteration to obtain approximately orthogonal perturbations without requiring expensive singular value decomposition (SVD). Extensive experiments on multiple architectures and datasets show that OrthoSAM consistently improves generalization and provides stronger robustness to label noise.

Primary area:
optimization
Keywords:
sharpness-aware minimization, generalization, matrix-aware optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RUEnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OrthoSAM, a SAM variant that leverges Muon-style matrix orthogonalization technique. The method applies orthogonalization in both stage of SAM and aim to reduce the leading singular direction dominant. This paper evaluate the effectiveness of proposed method on image classification, language tasks, and label-noise robustness benchmarks, reporting improvements over SAM and its variants.

Critical strengths
  1. the paper is easy to follow, the proposed method is straightforward. OrthoSAM applies matrix orthogonalization to both ascent and descent stage of SAM, which is clear and simple.

  2. the empirical evaluation covers diverse settings and shows meaningful improvements on some benchmarks. Experiments include CNN and Transformer architectures, image classification, language tasks, and label-noise settings.

Critical weaknesses

1.the core idea of the algorithm is to introduce Muon-style orthogonalization into the ascent and descent stages of SAM, but the substantive contribution beyond combining these components remains unclear. Although the method shows promising emprical results on various settings, this paper's contribution might be still limited.

2.the additional computational overhead is substantial. Beyond the two forward backward process like SAM, OrthoSAM performs multiple Newton–Schulz iterations in both ascent and descent stages, need approximately 1.6 times SAM’s per-epoch training time.

3 (minor).the implementation differences between the ablation baselines require clarification.

Questions for authors

See weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer RUEn: W1 — Novelty and Contributions

Thank you for raising this concern. Our contribution is to formulate and study an orthogonal perturbation geometry for sharpness-aware training, together with its theoretical characterization and empirical evaluation under controlled descent rules. First, we introduce the orthogonal perturbation neighborhood Nρ(θ)=Bρ(θ)∩(θ+R⊥m×n)N_{\rho}(\theta)=B_{\rho}(\theta)\cap\left(\theta+\mathbb{R}_{\perp}^{m\times n}\right) (Eq. (5)),

where R⊥m×n={Δ:Δ⊤Δ=αIn, α≥0}\mathbb{R}_{\perp}^{m\times n}=\left\lbrace \Delta:\Delta^{\top}\Delta=\alpha I_n,\ \alpha\geq 0 \right\rbrace. This provides a geometric alternative to SAM's Euclidean ball. The motivation is that SAM's linearized ascent weights each singular component in proportion to its singular value (Eq. (4)), concentrating the perturbation budget when the gradient spectrum is uneven.

Second, the perturbation follows from the constrained ascent problem. For G=∇LS(θ)=UΣV⊤G=\nabla L_S(\theta)=U\Sigma V^{\top}, Theorem 4.2 solves max⁡∥Δ∥F≤ρ{⟨G,Δ⟩+Iortho(Δ)}\displaystyle \max_{\lVert \Delta \rVert_F\leq\rho}\left\lbrace \langle G,\Delta\rangle+I_{\mathrm{ortho}}(\Delta)\right\rbrace and gives the ideal solution Δ⋆=ρnUV⊤\displaystyle \Delta_{\star}=\frac{\rho}{\sqrt{n}}UV^{\top}. Newton–Schulz supplies an efficient numerical approximation to the polar factor, avoiding an explicit SVD. The distinction between the two ascent objectives is analyzed in Appendix C.3, Eqs. (53)–(54), already included in the submission. For the same gradient matrix and Frobenius radius, their linearized sharpness values are SF=ρ∥G∥FS_F=\rho\lVert G\rVert_F for SAM and S⊥=ρ∥G∥∗n\displaystyle S_{\perp}=\frac{\rho\lVert G\rVert_{\ast}}{\sqrt{n}} for the orthogonal neighborhood. Their ratio, S⊥SF=∥G∥∗n∥G∥F\displaystyle \frac{S_{\perp}}{S_F}=\frac{\lVert G\rVert_{\ast}}{\sqrt{n}\lVert G\rVert_F}, depends on the gradient spectrum. Thus, the proposed geometry changes the first-order sharpness objective itself. We also provide theoretical guarantees with explicit scope. Theorem 4.1 gives a PAC-Bayes generalization bound involving the worst-case empirical loss over Nρ(θ)N_{\rho}(\theta). Theorem 4.3 establishes a stationarity guarantee for the idealized polar descent update with norm-bounded perturbations, using ∥Ol∥F2≤n\lVert O_l\rVert_F^2\leq n and ⟨gl,Ol⟩=∥gl∥∗\langle g_l,O_l\rangle=\lVert g_l\rVert_{\ast}. These results characterize the structured neighborhood and idealized matrix update.

The contribution of the perturbation is further examined by holding the descent rule fixed. The following results are already available in Appendix C.5, Table 16. We represent the best accuracy over the evaluated radius grid for each variant.

MethodDescent ruleCIFAR-100 / Tiny-ViTTiny-ImageNet / ResNet-50
SAM + orthogonalized descentOrthoSAM's orthogonalized descent62.6563.18
OrthoSAMOrthoSAM's orthogonalized descent64.1767.55
Improvement from changing the perturbationSame descent rule+1.52 points+4.37 points

With the same orthogonalized descent, the proposed perturbation improves the best reported accuracy on both architectures.

Appendix C.8 already includes a further comparison with AdamW descent held fixed, on CIFAR-100/Tiny-ViT. We represent those results below. Here, “AdamW + orthogonal perturbation” denotes the variant called “AdamW + OrthoSAM” in the appendix.

MethodPerturbationDescent ruleAccuracy (%)
AdamWNoneAdamW64.15
AdamW + SAMSAMAdamW64.20
AdamW + orthogonal perturbationOrthogonalAdamW65.38

The 1.18-point improvement over AdamW + SAM provides evidence for the benefit of the proposed perturbation under a fixed adaptive descent rule, extending its empirical contribution beyond its pairing with Muon-style updates.

AuthorsOpenReview ↗

Response to Reviewer RUEn: W2 — Computational Overhead and Accuracy–Time Trade-off

Regarding computational overhead, we agree that the additional Newton–Schulz operations incur a substantial per-epoch cost. On CIFAR-100/Tiny-ViT, OrthoSAM takes 0.445 minutes per epoch, compared with 0.278 minutes for SAM, corresponding to approximately 1.6× the per-epoch training time. To evaluate the practical accuracy–time trade-off, Appendix C.6, Table 17 already includes a comparison under a common budget of up to 50 minutes, with a maximum of 200 epochs, on a single NVIDIA A100 (80 GB). Each method is allocated min⁡(200,⌊50/tepoch⌋)\min(200,\lfloor 50/t_{\mathrm{epoch}}\rfloor) epochs. The reported test accuracy comes from the checkpoint selected by validation accuracy within this budget. We represent the results below.

MethodTime per epoch (minutes)Epochs completedAccuracy (%)
OrthoSAM0.44511265.28
Muon0.23420063.07
Muon + SAM0.33614862.70
MSAM0.16520053.54
ASAM0.28317652.75
MongeSAM0.27817952.56
FriendlySAM0.28317651.29
SAM0.27817951.23
FisherSAM0.30416450.87
GSAM0.31915650.31
ESAM0.26219043.12

OrthoSAM achieves the highest accuracy among the evaluated methods under this capped training-budget protocol, despite completing fewer epochs. It exceeds Muon by 2.21 percentage points and Muon + SAM by 2.58 points. The 200-epoch cap means that some faster methods finish before the 50-minute limit; this is the precise scope of the comparison.

AuthorsOpenReview ↗

Response to Reviewer RUEn: W3 — Implementation Differences Across Ablation Baselines

Regarding the implementation differences between the ablation baselines, we appreciate the request for clarification. Appendix D, Algorithms 1–2 already specify the Newton–Schulz routine, the ascent and descent steps, and the fallback branches, while Appendix C.5 already explains the perturbation normalization. We summarize the ablation variants below.

MethodPerturbation ruleDescent rule
SGDNoneSGD with momentum
SAMStandard SAM perturbationSGD with momentum
Orthogonal perturbation onlyOrthogonal perturbationSGD with momentum
Muon-styleNoneReference Muon update
SAM + Muon-styleStandard SAM perturbationReference Muon update
SAM + orthogonalized descentStandard SAM perturbationOrthoSAM's Newton–Schulz update
OrthoSAMOrthogonal perturbationOrthoSAM's Newton–Schulz update

For a layer gradient GWG_W, standard SAM uses global normalization, giving ΔWSAM=ρGW/∥Gall∥F\Delta_W^{\mathrm{SAM}}=\rho G_W/\|G_{\mathrm{all}}\|_F,

where GallG_{\mathrm{all}} denotes the gradient over all parameters. In contrast, the orthogonal perturbation is normalized separately for each eligible matrix, giving ΔWortho=ρ NSK(GW)/∥NSK(GW)∥F\Delta_W^{\mathrm{ortho}}=\rho\,\mathrm{NS}_K(G_W)/\|\mathrm{NS}_K(G_W)\|_F. Thus, ρ\rho represents a global perturbation budget for SAM and a per-matrix radius for the orthogonal branch. This distinction is explicitly discussed in Appendix C.5. After evaluating the gradient G~W\widetilde G_W at the perturbed parameters, OrthoSAM's matrix descent applies W←W−η NSK(G~W)W\leftarrow W-\eta\,\mathrm{NS}_K(\widetilde G_W), with K=5K=5 in the reported experiments. The descent step directly uses the Newton–Schulz output, whereas the ascent additionally normalizes that output to set the perturbation radius.

The distinction between “Muon-style” and “orthogonalized descent” is also documented in Appendix D. “Muon-style” denotes the reference Muon update, including its matrix-shape-dependent scaling. OrthoSAM's orthogonalized descent uses the direct Newton–Schulz output and omits that additional scaling. Consequently, “SAM + orthogonalized descent” uses the same matrix descent rule as full OrthoSAM, allowing their comparison to examine the change in perturbation rule. The Newton–Schulz branches apply to eligible matrix-shaped hidden weights, with convolutional kernels reshaped into matrices. Embeddings, classifier heads, and one-dimensional parameters are excluded from these branches. For these fallback parameters, Algorithm 2 specifies the globally normalized SAM perturbation and a standard gradient descent update.

We will consolidate these definitions in the main ablation discussion so that the distinctions can be checked directly.

Reviewer_3Xz8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OrthoSAM, a matrix-aware variant of Sharpness-Aware Minimization (SAM). The authors identify a limitation of standard SAM: it flattens weight matrices into vectors, so the adversarial perturbation is weighted proportionally to the singular values of the per-layer gradient. Since gradient singular spectra in deep networks are typically skewed, most of the perturbation budget concentrates on a few dominant directions, leading to anisotropic exploration of the local loss landscape and potentially noisier late-stage training. To address this, OrthoSAM constrains the perturbation of each weight matrix to lie in an orthogonal neighborhood , so that all singular directions are weighted equally. The authors extend SAM's PAC–Bayes bound to this structured neighborhood (Theorem 4.1) and prove the convergence rate for the idealized update . Experiments on CIFAR-10/100, Tiny-ImageNet, and GLUE across convolutional and transformer backbones show consistent improvements over SAM and recent variants, particularly on Tiny-ViT and Tiny-ImageNet. Ablations indicate the orthogonal perturbation and orthogonalized descent are complementary rather than additive, and OrthoSAM demonstrates stronger robustness to high label-noise rates.

Critical strengths
  1. A well-diagnosed and precisely formalized limitation of SAM, leading to a simple, principled fix. The paper identifies that standard SAM's flattened-vector perturbation weights each singular direction of the gradient proportionally to its singular value, so the perturbation budget is dominated by a few sharp directions. This observation is not merely heuristic: the authors rigorously derive the closed-form optimal orthogonal perturbation, extend SAM's PAC–Bayes generalization bound to the orthogonal neighborhood, and prove an convergence rate. The Newton–Schulz implementation makes the method essentially as cheap as standard SAM, giving a clean, theory-backed, and practical answer to a real geometric shortcoming of the SAM framework. This combination of diagnosis, theory, and practicality is what makes the contribution consequential rather than incremental.
Critical weaknesses
  1. The NLP fine-tuning experiments are unreliable and undermine confidence in the empirical claims beyond vision benchmarks. In Table 6, the SGD and SAM baselines on GLUE are implausibly poor: SGD achieves 54.13% on SST-2 (a trivially easy binary task where even plain SGD fine-tuning typically exceeds 90%), and SAM reports a negative MCC on CoLA (−2.07) and worse SST-2 than SGD. These numbers suggest the baselines were run with an inappropriate recipe (e.g., SGD without proper learning-rate tuning on Adam-tuned architectures), and OrthoSAM's large gaps on this table therefore cannot be taken as evidence of a real advantage in NLP. Since the abstract and contributions explicitly claim validation "across vision and language benchmarks," a broken comparison in one of the two claimed domains is consequential: it calls the reliability of the experimental protocol into question more broadly, and weakens one of the headline claims of the paper.

  2. The source of the gain is confounded with Muon-style orthogonalized descent, and the convergence theory covers an idealization that does not match the actual algorithm. OrthoSAM changes two things relative to SAM, and Section 5.4 shows the orthogonalized descent is what recovers most of the gain on Tiny-ViT. Meanwhile, Theorem 4.3 and Corollary 4.4 analyze the idealized polar update, not the Newton–Schulz approximation or the perturbation–descent interaction the paper argues is essential. The theory therefore neither justifies the two-step coupling empirically claimed in Section 5.4 nor analyzes the algorithm as implemented. It largely reuses existing smooth non-convex convergence machinery. As a result, the paper's central narrative is not fully substantiated by either its theory or its attribution analysis.

Questions for authors

In Table 6, SGD achieves 54.13% on SST-2 and SAM a negative MCC on CoLA and performs worse than SGD on SST-2, which is far outside any reasonable fine-tuning baseline for BERT-base. Can you clarify the exact training recipe used for the SGD/SAM baselines (learning rate schedule, warmup, number of epochs) and explain these numbers?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 3Xz8: W1+Q1 - About the NLP fine-tuning experiments

Thank you for highlighting this concern. We agree that the shared learning rate used in Table 6 was insufficient to establish a well-tuned SGD/SAM comparison. We clarify the complete recipe below, provide additional learning-rate sweeps, and highlight the controlled AdamW comparisons already included in the appendix.

a) Training recipe and interpretation of Table 6

Complementing Appendix B.1, all Table 6 runs fine-tuned bert-base for 3 epochs, using a linear learning-rate schedule decaying from 2×10−52\times10^{-5} to zero, with 0 warmup steps, batch size 3232, momentum 0.90.9, weight decay 0.010.01, and global gradient clipping at 1.01.0. The perturbation radius were ρ=0.005\rho=0.005 for SST-2 and ρ=0.002\rho=0.002 for CoLA.

A shared numerical learning rate does not imply comparable update magnitudes across these optimizers. OrthoSAM normalizes the matrix gradient before applying its NS transformation, giving its matrix updates a different dependence on gradient scale from SGD/SAM. The additional sweeps below show that increasing the learning rate substantially improves SGD/SAM, indicating that the original learning-rate choice was poorly suited to these baselines within the three-epoch budget. Accordingly, the large gaps in Table 6 should be interpreted in the context of that shared learning-rate setting.

b) Additional learning-rate sweeps

We extended the peak learning-rate grid to {7×10−5,10−4,2×10−4,5×10−4,7×10−4,10−3}\{7\times10^{-5},10^{-4},2\times10^{-4},5\times10^{-4},7\times10^{-4},10^{-3}\}, keeping the remaining training recipe and the perturbation radius for each task unchanged. These additional experiments have been incorporated into Table 7 in Section 5.4 of the revised manuscript. The first column reproduces the original Table 6 results.

SST-2 accuracy (%)

Method2×10−52\times10^{-5}7×10−57\times10^{-5}1×10−41\times10^{-4}2×10−42\times10^{-4}5×10−45\times10^{-4}7×10−47\times10^{-4}1×10−31\times10^{-3}
SGD54.1384.2985.7887.2787.6287.6286.81
SAM54.7084.1785.5586.7087.8487.1686.93
OrthoSAM90.2590.9491.0690.9489.5689.2289.56

CoLA MCC, scaled by 100

Method2×10−52\times10^{-5}7×10−57\times10^{-5}1×10−41\times10^{-4}2×10−42\times10^{-4}5×10−45\times10^{-4}7×10−47\times10^{-4}1×10−31\times10^{-3}
SGD1.82-2.071.821.8234.2640.4643.07
SAM-2.07-2.071.824.6433.0341.1243.93
OrthoSAM0.0044.9647.7552.3252.3752.5150.81

The best observed SGD/SAM results improve substantially, reaching 87.84% accuracy on SST-2 and 43.93 MCC points on CoLA. Across the six additional learning rates, OrthoSAM achieves higher scores than both baselines on both tasks. Comparing each method's best result within the explored grid, OrthoSAM reaches 91.06% on SST-2 and 52.51 on CoLA, exceeding the strongest SGD/SAM result by 3.22 percentage points and 8.58 MCC points, respectively.

c) Controlled AdamW comparisons already included in the appendix

To evaluate orthogonal perturbations with an adaptive optimizer, Appendix C.8, Table 20 already includes a comparison of AdamW, AdamW + SAM, and AdamW + orthogonal perturbation, with the descent rule held fixed as AdamW. We represent the results below. The last variant is called “AdamW + OrthoSAM” in the appendix; here we name it explicitly to clarify that its descent remains AdamW.

MethodCoLASST-2MRPCQQPMNLIQNLIRTEWNLISTS-B
AdamW54.9692.2083.09/88.0090.54/87.3384.22/84.3390.9265.3438.0388.05/87.78
AdamW + SAM58.3092.2083.33/88.2890.61/87.4384.45/84.3691.1062.4549.3088.03/87.79
AdamW + orthogonal perturbation59.5792.4384.31/88.8190.68/87.5284.31/84.4091.3664.6256.3488.17/87.89

These baselines achieve 92.20% on SST-2, while AdamW + SAM reaches 58.30 MCC points on CoLA. With the same AdamW descent, orthogonal perturbation achieves the best scores on seven of the nine GLUE tasks, including gains of 0.23 percentage points on SST-2 and 1.27 MCC points on CoLA over AdamW + SAM.

Appendix C.9, Table 21 already provides a supplementary language-modeling comparison on NanoGPT/FineWeb:

MethodValidation cross-entropy ↓
Muon3.851
AdamW + orthogonal perturbation, ρ=0.05\rho=0.053.947
AdamW + SAM, ρ=0.05\rho=0.054.019
AdamW4.494

Muon achieves the lowest cross-entropy overall. Within the controlled AdamW comparison, orthogonal perturbation reduces cross-entropy from 4.019 to 3.947 relative to SAM perturbation.

We will revise the NLP experiments to include the additional learning-rate sweeps, and clarify the limitations of the original shared-learning-rate comparison.

AuthorsOpenReview ↗

Response to Reviewer 3Xz8: W2 - Attribution of Gains and Scope of Theoretical Guarantees

Thank you for this detailed concern. We address the attribution of the empirical gains, the scope of the convergence analysis, and the additional evidence on finite Newton–Schulz iterations below.

a) Contribution of the perturbation under a fixed descent rule

We agree that orthogonalized descent accounts for most of the improvement over SAM on CIFAR-100/Tiny-ViT. The additional contribution of the perturbation can be assessed by holding the descent rule fixed. The following results are already available in Appendix C.5, Table 16. Each entry summarizes the best accuracy over the evaluated perturbation-radius grid.

MethodDescent ruleCIFAR-100 / Tiny-ViTTiny-ImageNet / ResNet-50
SAMSGD with momentum52.7263.92
Orthogonal perturbation onlySGD with momentum52.5560.91
SAM + orthogonalized descentOrthoSAM's orthogonalized descent62.6563.18
OrthoSAMOrthoSAM's orthogonalized descent64.1767.55

On Tiny-ViT, changing the descent rule while retaining SAM perturbation raises accuracy from 52.72% to 62.65%. With that descent rule fixed, changing to the orthogonal perturbation provides a further 1.52-point improvement, reaching 64.17%. On Tiny-ImageNet/ResNet-50, the corresponding improvement is 4.37 points, from 63.18% to 67.55%. Thus, the relative contributions vary across architectures, and the perturbation provides an additional benefit in these comparisons.

Appendix C.8, Table 20 already includes a complementary comparison with AdamW descent held fixed. We reproduce the GLUE results below. “AdamW + orthogonal perturbation” is the variant called “AdamW + OrthoSAM” in the appendix; its descent rule remains AdamW.

MethodCoLASST-2MRPCQQPMNLIQNLIRTEWNLISTS-B
AdamW54.9692.2083.09/88.0090.54/87.3384.22/84.3390.9265.3438.0388.05/87.78
AdamW + SAM58.3092.2083.33/88.2890.61/87.4384.45/84.3691.1062.4549.3088.03/87.79
AdamW + orthogonal perturbation59.5792.4384.31/88.8190.68/87.5284.31/84.4091.3664.6256.3488.17/87.89

Orthogonal perturbation achieves the best scores on seven of the nine GLUE tasks, including improvements of 1.27 MCC points on CoLA and 0.23 percentage points on SST-2 over AdamW + SAM. These comparisons provide evidence for the contribution of the perturbation with an adaptive descent rule that does not use Muon-style orthogonalization.

b) Theoretical contribution and the interpretation of coupling

We will clarify the distinction between the convergence guarantee and the empirical benefit of the two-step design. Theorem 4.3 requires only ∥Δt∥F≤ρ\|\Delta_t\|_F\le\rho and does not establish an advantage specific to the pairing. The controlled ablations in part (a) demonstrate an additional benefit from orthogonal perturbations under a fixed descent rule in the evaluated settings, we will revise statements suggesting that the pairing is universally essential.

Regarding the use of standard convergence machinery, we agree that the smoothness and telescoping arguments are established techniques. Our theoretical development also includes the PAC-Bayes generalization bound for the structured neighborhood in Theorem 4.1. Under its stated bounded-loss and Lipschitz assumptions, it yields:

LD(θ)≤max⁡θ′∈Nρ(θ)LS(θ′)+C(θ,N,δ,ρ)+γρ,L_{\mathcal D}(\theta)\le \max_{\theta'\in N_\rho(\theta)}L_S(\theta')+\mathcal C(\theta,N,\delta,\rho)+\gamma\rho,

where C\mathcal C is the explicit complexity term in Eq. (6). Theorem 4.2 then derives the closed-form optimizer of the corresponding linearized ascent problem. These results characterize the proposed perturbation objective and its relationship to the generalization bound.

Finally, Theorem 4.3 and Corollary 4.4 provide stationarity guarantees for the idealized polar matrix update. The extended experiment, now Table 9 in Section 5.6 of the revised manuscript, directly quantifies the finite-NS perturbation error and its observed accuracy–runtime trade-off. We will make this theoretical scope explicit: the finite-NS implementation is evaluated empirically, while extending the stationarity guarantee to that implementation requires additional approximation analysis.

AuthorsOpenReview ↗

Response to Reviewer 3Xz8: W2 - Attribution of Gains and Scope of Theoretical Guarantees

c) Idealized polar analysis and finite Newton–Schulz implementation

Theorem 4.3 and Corollary 4.4 analyze the idealized polar update, as stated in Section 4.2. This modeling choice has precedent in earlier Muon analyses: for example, [1] studies the SVD-based update while evaluating Newton–Schulz implementations empirically. Our use of Newton–Schulz is motivated by computational efficiency. The original Muon description [3] explicitly chooses NS over SVD because of the latter's computational cost, and highlights that NS can run in bfloat16. Its matrix-multiplication-based implementation provides an efficient way to approximate the polar update on GPUs. Following this practical design, OrthoSAM uses a small, fixed number of NS iterations to avoid repeated SVD computations in both the ascent and descent stages. This computational motivation does not, by itself, extend an idealized convergence guarantee to the finite-NS implementation. More recent work analyzes finite Newton–Schulz convergence directly [2]. We will therefore make the scope of our guarantee clearer and distinguish the idealized analysis from the empirical evaluation and approximation-error measurements of our finite-iteration implementation.

In the original submission, Appendix C.4, Table 12 reported accuracy, runtime, and orthogonality residuals for K∈{1,2,3,5}K\in\left\lbrace 1,2,3,5\right\rbrace. We have extended this experiment to K∈{10,20,40,80}K\in\left\lbrace 10,20,40,80\right\rbrace and added the relative error between the realized perturbation and the ideal perturbation. The extended study has been moved to the main text and is now presented as Table 9 in Section 5.6 of the revised manuscript.

For a gradient matrix G=UΣV⊤G=U\Sigma V^{\top}, orient the matrix so that m≥nm\geq n. Let O=NSK(G)O=\mathrm{NS}_K(G),

ΔK=ρO∥O∥F\displaystyle \Delta_K=\frac{\rho O}{\lVert O\rVert_F}, and Δ⋆=ρnUV⊤\displaystyle \Delta_{\star}=\frac{\rho}{\sqrt{n}}UV^{\top}. We report the orthogonality residual r=∥O⊤O−In∥Fn\displaystyle r=\frac{\lVert O^{\top}O-I_n\rVert_F}{\sqrt{n}} and the relative perturbation error efull=∥ΔK−Δ⋆∥Fρ\displaystyle e^{\mathrm{full}}=\frac{\lVert\Delta_K-\Delta_{\star}\rVert_F}{\rho}, averaged over the two-dimensional hidden layers.

Table 9, Section 5.6 of the revised manuscript — Extended Newton–Schulz iteration study on CIFAR-100

BackboneMetricK=1K=1K=2K=2K=3K=3K=5K=5K=10K=10K=20K=20K=40K=40K=80K=80
ResNet-50Accuracy (%)82.4083.3283.7084.4882.6482.1482.5982.51
ResNet-50Time (minutes/epoch)1.261.441.521.551.912.644.096.99
ResNet-50Residual rr0.960.850.640.440.510.520.510.51
ResNet-50efulle^{\mathrm{full}}0.9290.7570.5600.3920.4260.4320.4330.436
WRN-16-4Accuracy (%)78.6178.7079.0479.4778.6177.8178.1078.12
WRN-16-4Time (minutes/epoch)0.360.430.440.450.560.791.242.14
WRN-16-4Residual rr0.910.740.490.390.430.470.430.43
WRN-16-4efulle^{\mathrm{full}}0.6930.5620.3320.2800.3010.3660.3060.313

From K=1K=1 to K=5K=5, the relative perturbation error decreases by approximately 58% on ResNet-50 and 60% on WRN-16-4, while accuracy increases by 2.08 and 0.86 percentage points, respectively. The per-epoch runtime increases by approximately 23%–25% relative to K=1K=1. Across both backbones, K=5K=5 achieves the highest reported accuracy and the lowest measured residual and perturbation error among the tested values. Increasing KK to 40 or 80 raises runtime to approximately 2.6–4.8 times the K=5K=5 runtime, without further reducing either error measure. This behavior is consistent with the tuned polynomial in Algorithm 1 being designed for a small number of iterations rather than monotonic convergence to exact orthogonality. These measurements support K=5K=5 as a practical choice in the evaluated configurations and explicitly quantify the remaining approximation gap. The convergence theorem retains its stated scope of an idealized polar update.

References

[1] On the Convergence Analysis of Muon (TMLR).

[2] Convergence of Muon with Newton–Schulz (ICLR 2026).

[3] Muon: An optimizer for hidden layers in neural networks. (2024)

Reviewer_WFbQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  1. This paper studies the geometry of SAM perturbations. It argues that standard SAM perturbations tend to concentrate in directions associated with large singular values of the gradient matrix and proposes orthogonalization to obtain more balanced matrix perturbations.
  2. The method uses the orthogonal factor of the gradient in both the ascent perturbation and descent update stages, approximated using Newton–Schulz iterations. The paper also provides a generalization bound and a convergence analysis of the idealized updates.
  3. The paper reports performance improvements on image classification and NLP tasks, with particularly pronounced gains on TinyViT.
Critical strengths
  1. Redesigning the perturbation based on the structure of matrices as linear operators is reasonable. It introduces a regularization preference different from that induced by Euclidean geometry and aligns with current developments in matrix optimization methods. The idealized linear inner problem admits an explicit closed-form solution. The algorithm is relatively simple, and its implementation using Newton–Schulz iterations is feasible.
  2. The experiments include component ablations, controls that share the same AdamW descent rule, perturbation radius sweeps, and comparisons under equal time budgets.
Critical weaknesses
  1. Large singular values of the gradient matrix indicate larger first-order changes along certain matrix directions. They do not imply larger Hessian curvature in those directions. Equation 11 only describes a first-order decrease and therefore cannot establish that these directions are detrimental to training. The matrix polynomial operations used in Newton–Schulz iterations cannot fill in directions corresponding to zero singular values of the gradient matrix. For example, for G = diag(1, 0), the normalized perturbation remains diag(rho, 0), which does not satisfy the required condition Delta^T Delta = (rho^2 / 2) I. Even when the gradient is full rank, a finite number of iterations cannot guarantee exactly equal singular values. The orthogonality residuals in Table 12 are still substantial. The approximation error therefore needs to be analyzed, showing only that the perturbation norm equals rho is insufficient to establish that the implemented algorithm realizes the theoretical neighborhood.
  2. Some baseline results may not have been sufficiently optimized. For example, in the SST-2 results in Table 6, random guessing corresponds to 50% accuracy in this binary classification setting, whereas SAM achieves only 54.70%, which clearly indicates a problem.
Questions for authors

Please see weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer WFbQ: W1 - First-Order Interpretation and Newton–Schulz Approximation

Thank you for identifying these issues. We agree that gradient singular values do not determine Hessian curvature, and that preserving the perturbation norm alone does not establish membership in the orthogonal neighborhood. We clarify the first-order motivation, address rank deficiency explicitly, and provide additional analysis and measurements of the approximation error below.

a) What the first-order analysis establishes

Equation (11) describes the first-order loss change along the singular components of the gradient. For a twice differentiable loss, the corresponding second-order expansion is:

LS(θ~−ηA~i)−LS(θ~)=−ησ~i+η22⟨A~i,∇2LS(θ~)[A~i]⟩+o(η2).L_S(\tilde{\theta}-\eta\tilde{A}_i)-L_S(\tilde{\theta})=-\eta\tilde{\sigma}_i+\frac{\eta^2}{2}\langle\tilde{A}_i,\nabla^2L_S(\tilde{\theta})[\tilde{A}_i]\rangle+o(\eta^2).

The gradient singular value σ~i\tilde{\sigma}_i determines the first-order term, whereas directional curvature depends on the separate Hessian term. Equation (11) therefore does not establish that directions with larger singular values have higher curvature or are detrimental to training. We will remove that implication and revise the surrounding discussion to describe spectral reweighting: the ideal polar update replaces unequal singular-value weights with equal weights along the gradient's singular components. The corresponding distinction between the perturbation objectives was already analyzed in Appendix C.3, Eqs. (53)–(54), of the original submission. Their linearized sharpness values are:

SF(G)=ρ∥G∥F,S⊥(G)=ρn∥G∥∗.S_F(G)=\rho\lVert G\rVert_F,\quad S_{\perp}(G)=\frac{\rho}{\sqrt{n}}\lVert G\rVert_{\ast}.

These expressions characterize the different first-order objectives induced by the two geometries. They do not establish a Hessian-curvature ordering or a general guarantee of improved training.

Our use of first-order updates follows the computational approach of original SAM [1]. [1] reports lower test error without the second-order terms in its WideResNet-40x2/CIFAR-10 experiment. [2] also discuss this observation and why more accurate inner maximization need not improve generalization without adjusting the perturbation radius. However, the second-order terms ablated in [1] arise from differentiating through the gradient-dependent perturbation. These references support the choice of a first-order algorithm; they do not justify inferring curvature from gradient singular values.

References

[1] Sharpness-Aware Minimization for Efficiently Improving Generalization (ICLR 2021)

[2] Towards Understanding Sharpness-Aware Minimization (ICML 2022)

AuthorsOpenReview ↗

Response to Reviewer WFbQ: W1 - First-Order Interpretation and Newton–Schulz Approximation

b) Rank deficiency, finite Newton–Schulz approximation, and the scope of the theoretical guarantees

We agree that finite Newton–Schulz iterations do not generally enforce exact orthogonality, particularly for rank-deficient gradients. We will clarify the distinction between the ideal constrained-ascent solution in Theorem 4.2 and its finite-NS implementation. Importantly, the reviewer's example does not violate the ascent assumption of Theorem 4.3 or Corollary 4.4. As stated in Section 4.2, the convergence analysis requires only ∥Δl∥F≤ρ\|\Delta_l\|_F\le\rho for the ascent perturbation, rather than exact orthogonality. The normalized NS perturbation satisfies this norm requirement even in the rank-deficient example. Specifically, Appendix A.3, Eq. (45), controls the effect of the perturbation through smoothness: ∥∇LB(θl+Δl)−∇LB(θl)∥F≤β∥Δl∥F≤βρ.\|\nabla L_B(\theta_l+\Delta_l)-\nabla L_B(\theta_l)\|_F\le\beta\|\Delta_l\|_F\le\beta\rho. Consequently, Theorem 4.3 and Corollary 4.4 remain valid under their stated idealized polar descent assumption, including when the ascent perturbation is computed using finite NS iterations. Exact realization of the orthogonal ascent neighborhood is not needed for this convergence argument. We will make this scope explicit and distinguish it from a convergence guarantee for finite-NS descent.

To assess the practical approximation directly, we have also extended the study already included in Appendix C.4, Table 12 of the original submission, adding larger iteration counts and the relative error between the realized and ideal perturbations. These results now appear in Table 9, Section 5.6 of the revised manuscript. They quantify the approximation gap and support K=5K=5 as a practical accuracy–cost choice in the evaluated configurations. In the original submission, Appendix C.4, Table 12 already reported accuracy, runtime, and orthogonality residuals for K∈{1,2,3,5}K\in\{1,2,3,5\}. We have extended the experiment to K∈{10,20,40,80}K\in\{10,20,40,80\} and added the relative perturbation error for all tested iteration counts. The extended study is now presented as Table 9 in Section 5.6 of the revised manuscript. We report r=∥OK⊤OK−In∥Fn\displaystyle r=\frac{\lVert O_K^{\top}O_K-I_n\rVert_F}{\sqrt{n}} and eKfull=∥ΔK−Δ⋆∥Fρ\displaystyle e_K^{\mathrm{full}}=\frac{\lVert\Delta_K-\Delta_{\star}\rVert_F}{\rho}, averaged over the two-dimensional hidden layers. The first metric measures orthogonality of the raw NS output; the second directly compares the realized, normalized perturbation with the ideal constrained solution.

Table 9, Section 5.6 — Extended Newton–Schulz iteration study on CIFAR-100

BackboneMetricK=1K=1K=2K=2K=3K=3K=5K=5K=10K=10K=20K=20K=40K=40K=80K=80
ResNet-50Accuracy (%)82.4083.3283.7084.4882.6482.1482.5982.51
ResNet-50Time (minutes/epoch)1.261.441.521.551.912.644.096.99
ResNet-50Residual rr0.960.850.640.440.510.520.510.51
ResNet-50eKfulle_K^{\mathrm{full}}0.9290.7570.5600.3920.4260.4320.4330.436
WRN-16-4Accuracy (%)78.6178.7079.0479.4778.6177.8178.1078.12
WRN-16-4Time (minutes/epoch)0.360.430.440.450.560.791.242.14
WRN-16-4Residual rr0.910.740.490.390.430.470.430.43
WRN-16-4eKfulle_K^{\mathrm{full}}0.6930.5620.3320.2800.3010.3660.3060.313

From K=1K=1 to K=5K=5, the reported relative perturbation error decreases by approximately 58% on ResNet-50 and 60% on WRN-16-4, while accuracy increases by 2.08 and 0.86 percentage points, respectively. Per-epoch runtime increases by approximately 23%–25% relative to K=1K=1. Across both backbones, K=5K=5 achieves the highest reported accuracy and the lowest measured residual and perturbation error among the tested values. Increasing KK to 40 or 80 raises runtime to approximately 2.6–4.8 times the K=5K=5 runtime, without further reducing either error measure. The polynomial used in Algorithm 1 has coefficients (a,b,c)=(3.4445,−4.7750,2.0315)(a,b,c)=(3.4445,-4.7750,2.0315) (like in original Muon [1]), for which p(1)=a+b+c=0.701p(1)=a+b+c=0.701. Thus, unit singular values are not fixed points of the raw iteration, and increasing KK should not be presented as guaranteeing convergence to exact orthogonality. These results support K=5K=5 as a practical accuracy–cost choice in the evaluated configurations, while explicitly documenting the remaining approximation gap. The measured relative errors of 0.3920.392 and 0.2800.280 are non-negligible, and the layer-averaged measurements do not provide a uniform error guarantee. We therefore use these results to assess the finite-iteration implementation empirically, without claiming that it exactly realizes the theoretical neighborhood.

References

[1] Muon: An optimizer for hidden layers in neural networks (2024).

AuthorsOpenReview ↗

Response to Reviewer WFbQ : W2 - About SST-2 results

Thank you for pointing this out. We agree that SAM's 54.70% accuracy on SST-2 reflects an inadequately tuned baseline under the original training recipe. We have directly investigated this issue through additional learning-rate sweeps and incorporated the results into Table 7, Section 5.4 of the revised manuscript.

a) Original recipe and the effect of learning-rate tuning

The original Table 6 fine-tuned BERT-base for 3 epochs, using a linear learning-rate schedule decaying from 2×10−52\times10^{-5} to zero, with 0 warmup steps, batch size 3232, momentum 0.90.9, weight decay 0.010.01, and global gradient clipping at 1.01.0. The perturbation radius were ρ=0.005\rho=0.005 for SST-2 and ρ=0.002\rho=0.002 for CoLA.

The additional experiments show that the original learning rate was poorly suited to SGD/SAM within this three-epoch budget. Keeping the remaining recipe unchanged, increasing the peak learning rate to 5×10−45\times10^{-4} improves SAM's SST-2 accuracy from 54.70% to 87.84%. SGD similarly improves from 54.13% to 87.62%. These results substantiate the reviewer's concern about the original baseline tuning.

b) Additional learning-rate sweeps

We evaluated six additional peak learning rates, {7×10−5,10−4,2×10−4,5×10−4,7×10−4,10−3}\{7\times10^{-5},10^{-4},2\times10^{-4},5\times10^{-4},7\times10^{-4},10^{-3}\}, keeping the training duration, remaining hyperparameters, and task-specific perturbation radius unchanged. These new results are reported in Table 7, Section 5.4 of the revised manuscript; the first column reproduces the original Table 6 results.

SST-2 accuracy (%)

Method2×10−52\times10^{-5} (original Table 6)7×10−57\times10^{-5}10−410^{-4}2×10−42\times10^{-4}5×10−45\times10^{-4}7×10−47\times10^{-4}10−310^{-3}
SGD54.1384.2985.7887.2787.6287.6286.81
SAM54.7084.1785.5586.7087.8487.1686.93
OrthoSAM90.2590.9491.0690.9489.5689.2289.56

Comparing each method's best result within the explored grid, OrthoSAM reaches 91.06%, versus 87.84% for SAM and 87.62% for SGD. The improvement over the strongest SGD/SAM result is therefore 3.22 percentage points.

We also repeated the sweep on CoLA to examine whether the learning-rate sensitivity extends to another task.

CoLA MCC, scaled by 100

Method2×10−52\times10^{-5} (original Table 6)7×10−57\times10^{-5}10−410^{-4}2×10−42\times10^{-4}5×10−45\times10^{-4}7×10−47\times10^{-4}10−310^{-3}
SGD1.82-2.071.821.8234.2640.4643.07
SAM-2.07-2.071.824.6433.0341.1243.93
OrthoSAM0.0044.9647.7552.3252.3752.5150.81

SAM's best observed CoLA score increases from the original -2.07 to 43.93. OrthoSAM reaches 52.51, exceeding the strongest SGD/SAM result within this grid by 8.58 MCC points.

We will qualify the interpretation of the original Table 6 gaps and emphasize the additional learning-rate sweeps and the controlled AdamW comparisons when presenting the NLP results.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Common Response to All Reviewers

We thank all reviewers for their constructive feedback and for recognizing the motivation for matrix-aware perturbations, the closed-form ascent solution, and the breadth of our empirical evaluation. In response, we clarify the scope of the theory, report new learning-rate and Newton–Schulz experiments, and explain how the existing ablations and training-budget comparisons inform the interpretation of OrthoSAM's gains.

1. Clarifying the perturbation geometry and theoretical scope (R. WFbQ, R. 3Xz8, R. RUEn).

OrthoSAM introduces a structured neighborhood that changes how SAM's linearized ascent allocates the perturbation budget across the gradient's singular components. We clarify that Eq. (11) describes first-order directional loss changes. Hessian curvature depends on a separate second-order term. Theorem 4.1 gives a PAC-Bayes generalization bound involving the structured neighborhood, and Theorem 4.2 gives the exact solution to its linearized ascent problem. Theorem 4.3 and Corollary 4.4 establish stationarity bounds for idealized polar descent. Their ascent assumption, ∥Δt∥F≤ρ\lVert\Delta_t\rVert_F\leq\rho, also accommodates normalized finite Newton–Schulz perturbations that satisfy the norm bound.

2. Quantifying the finite Newton–Schulz approximation (R. WFbQ, R. 3Xz8).

Finite Newton–Schulz iterations preserve zero singular values and can leave nonzero singular values unequal even for full-rank gradients. Frobenius normalization sets the perturbation radius, so we separately measure the distance to the ideal constrained solution. We expanded the original Appendix C.4 study to include up to 80 iterations and added relative perturbation errors in Table 9, Section 5.6. From K=1K=1 to K=5K=5, these errors decrease from 0.929 to 0.392 on ResNet-50 and from 0.693 to 0.280 on WRN-16-4. On both CIFAR-100 backbones, K=5K=5 gives the highest reported accuracy and lowest measured errors. Larger tested iteration counts increase runtime without improving either error measure. The reported errors quantify the remaining gap to the ideal perturbation.

3. Retuning the NLP baselines (R. WFbQ, R. 3Xz8).

The original shared peak learning rate of 2×10−52\times10^{-5} was poorly suited to SGD and SAM under the three-epoch BERT-base recipe. We evaluated six additional peak learning rates on SST-2 and CoLA while holding the rest of the recipe and the task-specific perturbation radii fixed. In Table 7, Section 5.4, SAM's best observed SST-2 accuracy rises from 54.70% to 87.84%, and its best CoLA score rises from −2.07 to 43.93 (MCC scaled by 100). Comparing each method's best result in the expanded grid, OrthoSAM reaches 91.06% on SST-2 and 52.51 on CoLA, exceeding the best SGD or SAM result by 3.22 percentage points and 8.58 MCC points, respectively.

4. Measuring the perturbation's contribution under fixed descent rules (R. 3Xz8, R. RUEn).

Orthogonalized descent accounts for most of the improvement over SAM on CIFAR-100/Tiny-ViT, raising accuracy from 52.72% to 62.65% while retaining the SAM perturbation. In the comparisons already reported in Appendix C.5, switching to the orthogonal perturbation with that descent rule held fixed improves the best accuracy over the radius grid by 1.52 points on CIFAR-100/Tiny-ViT and 4.37 points on Tiny-ImageNet/ResNet-50. The existing Appendix C.8 comparisons also hold AdamW descent fixed. There, replacing the SAM perturbation raises Tiny-ViT accuracy from 64.20% to 65.38% and gives the highest reported scores among the three AdamW variants on seven of nine GLUE tasks. These ablations identify where the perturbation adds to the gains from the descent rule.

5. Clarifying training cost and ablation implementations (R. RUEn).

On CIFAR-100/Tiny-ViT, OrthoSAM takes 1.6× SAM's time per epoch. In the comparison already reported in Appendix C.6, with a budget of up to 50 minutes and a cap of 200 epochs, OrthoSAM reaches 65.28% accuracy, compared with 63.07% for Muon and 62.70% for Muon + SAM. Our response also specifies the ascent and descent rules used in each ablation. Standard SAM uses global perturbation normalization, while OrthoSAM sets the radius per eligible matrix. For descent, reference Muon applies matrix-shape scaling, while OrthoSAM uses the Newton–Schulz output directly.

The revised manuscript is available on OpenReview. Our individual replies provide the full experimental results, training settings, and derivations. We thank the reviewers again for their time and for considering these updates.

Best regards,

The Authors of Paper #26448

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.