- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper derives a spectral-norm trajectory bound under bounded orthogonalization error and positive weight decay, together with a nuclear-norm interpretation of the update. Experiments compare Muon across several CNN/ViT architectures. Muon often outperforms AdamW but does not consistently outperform SGD or SAM.
Critical strengths
- The evaluation spans five network architectures, multiple ℓp threat models, extra white-box adversarial attacks, and includes experiments on both CIFAR-10 and ImageNet.
- The authors present unfavorable results for ViT models and ImageNet, and explicitly recognize robust overfitting as well as the loose nature of their theoretical bounds.
Critical weaknesses
-
Theorem 3.2 merely links the resulting loose per-layer bound to a first-order proxy for robustness. When λ=5×10−4, the per-layer upper bound can grow very large; multiplying these per-layer bounds across network layers will likely yield little meaningful insight. The analysis fails to demonstrate superior robust optimization, generalization, or convergence properties relative to SGD.
-
The authors note they did not perform comprehensive optimizer-specific hyperparameter searches, yet the empirical ranking of optimizers forms the core empirical inquiry of this paper. AdamW struggles substantially on ViT architectures, whereas Muon underperforms SGD for PreActResNet-18 multi-norm training, most ViT setups, and ImageNet. Several reported performance gains are smaller than one percentage point, with no results on seed-to-seed variance provided.
Questions for authors
-
What learning-rate and weight-decay search spaces and tuning budgets were used for each optimizer? Were the reported tables produced from single runs?
-
Why are checkpoints selected using robustness rather than the reported union robustness? Do the main rankings remain unchanged under union-based selection?
-
What numerical values do the theoretical spectral bounds take under the actual training configuration, and are they sufficiently tight to distinguish Muon from SGD?
6 comments
Revision follow-up
Dear Reviewer Q3bM,
We have submitted a revised manuscript incorporating targeted analyses addressing the questions raised in your review.
The revision includes four additions:
-
Hyperparameter sensitivity: A matched-budget three-trial LR audit for each optimizer, with LR selection based exclusively on validation Union robustness. The resulting full-test ranking remains SGD > Muon > AdamW.
-
Seed variability: A three-seed PreActResNet-18 comparison, reporting mean and standard deviation for all three optimizers.
-
Checkpoint selection: A controlled comparison between validation and Union-based selection on identical training trajectories. The optimizer ranking remains unchanged.
-
Theoretical bound: A numerical audit under the actual training schedule. The finite-horizon bound is 1023.59, compared with an observed maximum Muon weight spectral norm of 49.97. We explicitly clarify that this conservative bound establishes a worst-case trajectory property, not a quantitative robustness certificate or a ranking guarantee over SGD.
We have also clarified the distinction between the original single-run results and the additional controlled audits, rather than combining results obtained under different configurations.
We believe these additions directly address the empirical and theoretical questions raised in the review. The revised manuscript presents the resulting scope and limitations explicitly.
Best Regards!
Submission9034 Authors
Thank you for the detailed response and revision
Thank you for the detailed response and revision. The additional audits improve the transparency of the evaluation and address several of my protocol-related questions. However, they do not change my main concern regarding the overall strength of the contribution.
The numerical audit confirms that the spectral bound is highly conservative and cannot distinguish Muon from SGD or provide a practical robustness guarantee. Moreover, SGD remains stronger in the controlled comparison and on ImageNet, while the selected-checkpoint advantage over the magnitude-matched control disappears at the final checkpoint.
Therefore, the revision appropriately narrows the claims, but the remaining conclusion that optimizer choice affects adversarial-training trajectories does not constitute a sufficiently strong theoretical or empirical advance. I therefore maintain my score of 2.
Dear Reviewer Q3bM,
The review raises four concrete questions about the interpretation of our empirical comparison and theoretical bound. We have now performed targeted audits for each of them. These results refine the scope of our claims, but they do not change the central conclusion that the outer optimizer materially affects adversarial-training dynamics and robustness.
1. Optimizer-specific tuning budget.
Our main empirical claim is not that the reported configurations are globally optimal for each optimizer. The relevant question is whether the observed ordering is merely an artifact of one particular learning-rate choice. To test this directly, we added a matched-budget local LR audit on PreActResNet-18 under ℓ∞+ℓ1 AT.
Each optimizer receives exactly three LR trials: SGD {0.03,0.1,0.3}, AdamW {0.01,0.03,0.1}, and Muon {0.03,0.1,0.3}. Weight decay and the remaining training protocol are fixed. LR is selected exclusively by validation Union robustness on the same fixed subset, and the full test set is evaluated only after selection.
The selected LRs are 0.1 for SGD, 0.01 for AdamW, and 0.1 for Muon. Their corresponding full-test Union accuracies are 40.86%, 33.53%, and 36.50%, respectively. Thus, under the same three-trial tuning budget, the qualitative ordering remains SGD>Muon>AdamW.
The fact that AdamW selects 0.01 rather than 0.1 also confirms that optimizer-specific tuning is consequential. We therefore do not claim hyperparameter-independent or globally optimal rankings. The result instead shows that the qualitative comparison in this controlled setting is not explained by a single unfavorable LR choice.
2. Single runs and seed variability.
The original tables report point estimates from single runs; we will make this explicit. To assess seed sensitivity in the main PreActResNet-18 multi-norm setting, we additionally ran three independent seeds.
At the selected checkpoint, Union robustness is 38.59±0.29 for SGD, 37.16±0.43 for Muon, and 36.14±1.37 for AdamW.
This result does not support a claim that Muon is superior to SGD in this setting; SGD is stronger. It does, however, show that the qualitative ordering observed here is reproducible across seeds. We will also avoid treating sub-1 pp single-run differences in other settings as statistically established improvements.
3. ℓ∞ versus Union checkpoint selection.
We directly tested whether the original checkpoint criterion changes the optimizer ranking. On the same training trajectories, we simultaneously maintain checkpoints selected by validation ℓ∞ robustness and validation Union robustness.
Under ℓ∞ selection, the full-test Union accuracies for SGD/Muon/AdamW are 41.14/36.52/29.67. Under Union selection, they are 40.85/36.54/29.68. The ranking is therefore unchanged: SGD>Muon>AdamW.
Muon and AdamW select the same epoch under both criteria, while SGD changes from epoch 71 to 69. Thus, the qualitative ordering in this experiment is not an artifact of using ℓ∞ rather than Union robustness for checkpoint selection. We will nevertheless use and recommend Union-based validation when the reported target metric is Union robustness, because it is the metric-aligned protocol.
4. Numerical value and role of the spectral bound.
The numerical audit confirms that Theorem 3.1 is conservative. Under the actual configuration with λ=5×10−4 and the stepwise LR schedule, the schedule-aware finite-horizon ceiling is 1023.59, whereas the largest observed Muon weight spectral norm is 49.97. The gap is 20.48-fold. The stationary ceiling is 2000, corresponding to a 40.03-fold gap.
Therefore, the answer to the reviewer's question is direct: the bound is not tight enough to distinguish Muon from SGD. We do not use it for that purpose.
Theorem 3.1 establishes a different statement: bounded operator-norm updates together with positive weight decay imply a finite worst-case spectral-norm trajectory for Muon. Theorem 3.2 then gives a structural connection from such layerwise control to a first-order robustness proxy. These are optimizer-side structural results, not quantitative robustness certificates or comparative guarantees against SGD.
We will revise the manuscript to make this role explicit and to avoid any wording suggesting that the theoretical bound predicts the empirical optimizer ranking.
Overall interpretation.
The added audits support a narrower but stronger conclusion than a universal optimizer ranking. Optimizer choice materially changes AT behavior, but the ordering is architecture- and recipe-dependent. In the controlled PreActResNet-18 setting examined above, SGD remains stronger than Muon, while Muon remains stronger than AdamW after matched-budget LR selection. The theoretical result explains an optimizer-side stability property.