- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes OrthoSAM, a SAM variant that leverges Muon-style matrix orthogonalization technique. The method applies orthogonalization in both stage of SAM and aim to reduce the leading singular direction dominant. This paper evaluate the effectiveness of proposed method on image classification, language tasks, and label-noise robustness benchmarks, reporting improvements over SAM and its variants.
Critical strengths
-
the paper is easy to follow, the proposed method is straightforward. OrthoSAM applies matrix orthogonalization to both ascent and descent stage of SAM, which is clear and simple.
-
the empirical evaluation covers diverse settings and shows meaningful improvements on some benchmarks. Experiments include CNN and Transformer architectures, image classification, language tasks, and label-noise settings.
Critical weaknesses
1.the core idea of the algorithm is to introduce Muon-style orthogonalization into the ascent and descent stages of SAM, but the substantive contribution beyond combining these components remains unclear. Although the method shows promising emprical results on various settings, this paper's contribution might be still limited.
2.the additional computational overhead is substantial. Beyond the two forward backward process like SAM, OrthoSAM performs multiple Newton–Schulz iterations in both ascent and descent stages, need approximately 1.6 times SAM’s per-epoch training time.
3 (minor).the implementation differences between the ablation baselines require clarification.
Questions for authors
See weaknesses.
6 comments
Response to Reviewer RUEn: W2 — Computational Overhead and Accuracy–Time Trade-off
Regarding computational overhead, we agree that the additional Newton–Schulz operations incur a substantial per-epoch cost. On CIFAR-100/Tiny-ViT, OrthoSAM takes 0.445 minutes per epoch, compared with 0.278 minutes for SAM, corresponding to approximately 1.6× the per-epoch training time. To evaluate the practical accuracy–time trade-off, Appendix C.6, Table 17 already includes a comparison under a common budget of up to 50 minutes, with a maximum of 200 epochs, on a single NVIDIA A100 (80 GB). Each method is allocated epochs. The reported test accuracy comes from the checkpoint selected by validation accuracy within this budget. We represent the results below.
| Method | Time per epoch (minutes) | Epochs completed | Accuracy (%) |
|---|---|---|---|
| OrthoSAM | 0.445 | 112 | 65.28 |
| Muon | 0.234 | 200 | 63.07 |
| Muon + SAM | 0.336 | 148 | 62.70 |
| MSAM | 0.165 | 200 | 53.54 |
| ASAM | 0.283 | 176 | 52.75 |
| MongeSAM | 0.278 | 179 | 52.56 |
| FriendlySAM | 0.283 | 176 | 51.29 |
| SAM | 0.278 | 179 | 51.23 |
| FisherSAM | 0.304 | 164 | 50.87 |
| GSAM | 0.319 | 156 | 50.31 |
| ESAM | 0.262 | 190 | 43.12 |
OrthoSAM achieves the highest accuracy among the evaluated methods under this capped training-budget protocol, despite completing fewer epochs. It exceeds Muon by 2.21 percentage points and Muon + SAM by 2.58 points. The 200-epoch cap means that some faster methods finish before the 50-minute limit; this is the precise scope of the comparison.
Response to Reviewer RUEn: W3 — Implementation Differences Across Ablation Baselines
Regarding the implementation differences between the ablation baselines, we appreciate the request for clarification. Appendix D, Algorithms 1–2 already specify the Newton–Schulz routine, the ascent and descent steps, and the fallback branches, while Appendix C.5 already explains the perturbation normalization. We summarize the ablation variants below.
| Method | Perturbation rule | Descent rule |
|---|---|---|
| SGD | None | SGD with momentum |
| SAM | Standard SAM perturbation | SGD with momentum |
| Orthogonal perturbation only | Orthogonal perturbation | SGD with momentum |
| Muon-style | None | Reference Muon update |
| SAM + Muon-style | Standard SAM perturbation | Reference Muon update |
| SAM + orthogonalized descent | Standard SAM perturbation | OrthoSAM's Newton–Schulz update |
| OrthoSAM | Orthogonal perturbation | OrthoSAM's Newton–Schulz update |
For a layer gradient , standard SAM uses global normalization, giving ,
where denotes the gradient over all parameters. In contrast, the orthogonal perturbation is normalized separately for each eligible matrix, giving . Thus, represents a global perturbation budget for SAM and a per-matrix radius for the orthogonal branch. This distinction is explicitly discussed in Appendix C.5. After evaluating the gradient at the perturbed parameters, OrthoSAM's matrix descent applies , with in the reported experiments. The descent step directly uses the Newton–Schulz output, whereas the ascent additionally normalizes that output to set the perturbation radius.
The distinction between “Muon-style” and “orthogonalized descent” is also documented in Appendix D. “Muon-style” denotes the reference Muon update, including its matrix-shape-dependent scaling. OrthoSAM's orthogonalized descent uses the direct Newton–Schulz output and omits that additional scaling. Consequently, “SAM + orthogonalized descent” uses the same matrix descent rule as full OrthoSAM, allowing their comparison to examine the change in perturbation rule. The Newton–Schulz branches apply to eligible matrix-shaped hidden weights, with convolutional kernels reshaped into matrices. Embeddings, classifier heads, and one-dimensional parameters are excluded from these branches. For these fallback parameters, Algorithm 2 specifies the globally normalized SAM perturbation and a standard gradient descent update.
We will consolidate these definitions in the main ablation discussion so that the distinctions can be checked directly.
Response to Reviewer RUEn: W1 — Novelty and Contributions
Thank you for raising this concern. Our contribution is to formulate and study an orthogonal perturbation geometry for sharpness-aware training, together with its theoretical characterization and empirical evaluation under controlled descent rules. First, we introduce the orthogonal perturbation neighborhood Nρ(θ)=Bρ(θ)∩(θ+R⊥m×n) (Eq. (5)),
where R⊥m×n={Δ:Δ⊤Δ=αIn, α≥0}. This provides a geometric alternative to SAM's Euclidean ball. The motivation is that SAM's linearized ascent weights each singular component in proportion to its singular value (Eq. (4)), concentrating the perturbation budget when the gradient spectrum is uneven.
Second, the perturbation follows from the constrained ascent problem. For G=∇LS(θ)=UΣV⊤, Theorem 4.2 solves ∥Δ∥F≤ρmax{⟨G,Δ⟩+Iortho(Δ)} and gives the ideal solution Δ⋆=nρUV⊤. Newton–Schulz supplies an efficient numerical approximation to the polar factor, avoiding an explicit SVD. The distinction between the two ascent objectives is analyzed in Appendix C.3, Eqs. (53)–(54), already included in the submission. For the same gradient matrix and Frobenius radius, their linearized sharpness values are SF=ρ∥G∥F for SAM and S⊥=nρ∥G∥∗ for the orthogonal neighborhood. Their ratio, SFS⊥=n∥G∥F∥G∥∗, depends on the gradient spectrum. Thus, the proposed geometry changes the first-order sharpness objective itself. We also provide theoretical guarantees with explicit scope. Theorem 4.1 gives a PAC-Bayes generalization bound involving the worst-case empirical loss over Nρ(θ). Theorem 4.3 establishes a stationarity guarantee for the idealized polar descent update with norm-bounded perturbations, using ∥Ol∥F2≤n and ⟨gl,Ol⟩=∥gl∥∗. These results characterize the structured neighborhood and idealized matrix update.
The contribution of the perturbation is further examined by holding the descent rule fixed. The following results are already available in Appendix C.5, Table 16. We represent the best accuracy over the evaluated radius grid for each variant.
With the same orthogonalized descent, the proposed perturbation improves the best reported accuracy on both architectures.
Appendix C.8 already includes a further comparison with AdamW descent held fixed, on CIFAR-100/Tiny-ViT. We represent those results below. Here, “AdamW + orthogonal perturbation” denotes the variant called “AdamW + OrthoSAM” in the appendix.
The 1.18-point improvement over AdamW + SAM provides evidence for the benefit of the proposed perturbation under a fixed adaptive descent rule, extending its empirical contribution beyond its pairing with Muon-style updates.