acceptodds
Under review as a conference paper at ICLR 2027

When Muon Optimizer Meets Adversarial Training: A Theoretical and Empirical Study

Abstract

Adversarial training (AT) remains one of the most reliable empirical defenses against adversarial attacks. Its robustness critically depends on how the underlying min-max objective is optimized. In practice, Stochastic Gradient Descent (SGD) optimizer remains the default optimization choice for AT, whereas adaptive optimizers often improve standard training but may yield inferior robustness. Recently, the Muon optimizer, which orthogonalizes matrix-valued updates via an approximate polar decomposition, has achieved notable success in large-scale training, motivating the investigation of its behavior beyond standard training settings. Robustness claims in AT can be highly sensitive to the optimizer used in the outer minimization, yet optimizer choice is often treated as an implementation detail. A defense may appear robust under one training dynamic but degrade under heterogeneous or adaptive threat models. This raises a question: how does orthogonalized outer optimization alter the robustness and training dynamics of AT, and in which regimes is it beneficial? Focusing on this problem, this paper revisits the choice of optimizer as a security-relevant component of robustness. Theoretically, we show that Muon's polar-factor-based update, together with weight decay, induces a finite worst-case bound on matrix-weight trajectories without explicitly projecting the learned weights. Empirically, across five architectures and multiple threat models on CIFAR-scale settings, Muon often outperforms AdamW and remains competitive with SGD, while SGD remains stronger in some ViT and ImageNet settings. Our results show that outer-optimizer choice can materially change the observed robustness profile under the evaluated AT configurations. These results delineate the regimes in which orthogonalized updates are beneficial. Our findings therefore motivate treating the optimizer as a robustness-relevant design and evaluation variable rather than as an implementation detail.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
machine learning theory, adversarial training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Q3bMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper derives a spectral-norm trajectory bound under bounded orthogonalization error and positive weight decay, together with a nuclear-norm interpretation of the update. Experiments compare Muon across several CNN/ViT architectures. Muon often outperforms AdamW but does not consistently outperform SGD or SAM.

Critical strengths
  1. The evaluation spans five network architectures, multiple ℓp​ threat models, extra white-box adversarial attacks, and includes experiments on both CIFAR-10 and ImageNet.
  2. The authors present unfavorable results for ViT models and ImageNet, and explicitly recognize robust overfitting as well as the loose nature of their theoretical bounds.
Critical weaknesses
  1. Theorem 3.2 merely links the resulting loose per-layer bound to a first-order proxy for robustness. When λ=5×10−4, the per-layer upper bound can grow very large; multiplying these per-layer bounds across network layers will likely yield little meaningful insight. The analysis fails to demonstrate superior robust optimization, generalization, or convergence properties relative to SGD.

  2. The authors note they did not perform comprehensive optimizer-specific hyperparameter searches, yet the empirical ranking of optimizers forms the core empirical inquiry of this paper. AdamW struggles substantially on ViT architectures, whereas Muon underperforms SGD for PreActResNet-18 multi-norm training, most ViT setups, and ImageNet. Several reported performance gains are smaller than one percentage point, with no results on seed-to-seed variance provided.

Questions for authors
  1. What learning-rate and weight-decay search spaces and tuning budgets were used for each optimizer? Were the reported tables produced from single runs?

  2. Why are checkpoints selected using ℓ∞\ell_\infty robustness rather than the reported union robustness? Do the main rankings remain unchanged under union-based selection?

  3. What numerical values do the theoretical spectral bounds take under the actual training configuration, and are they sufficiently tight to distinguish Muon from SGD?

6 comments

Dear Reviewer Q3bM,

The review raises four concrete questions about the interpretation of our empirical comparison and theoretical bound. We have now performed targeted audits for each of them. These results refine the scope of our claims, but they do not change the central conclusion that the outer optimizer materially affects adversarial-training dynamics and robustness.

1. Optimizer-specific tuning budget.

Our main empirical claim is not that the reported configurations are globally optimal for each optimizer. The relevant question is whether the observed ordering is merely an artifact of one particular learning-rate choice. To test this directly, we added a matched-budget local LR audit on PreActResNet-18 under ℓ∞+ℓ1\ell_\infty+\ell_1 AT.

Each optimizer receives exactly three LR trials: SGD {0.03,0.1,0.3}\{0.03,0.1,0.3\}, AdamW {0.01,0.03,0.1}\{0.01,0.03,0.1\}, and Muon {0.03,0.1,0.3}\{0.03,0.1,0.3\}. Weight decay and the remaining training protocol are fixed. LR is selected exclusively by validation Union robustness on the same fixed subset, and the full test set is evaluated only after selection.

The selected LRs are 0.1 for SGD, 0.01 for AdamW, and 0.1 for Muon. Their corresponding full-test Union accuracies are 40.86%, 33.53%, and 36.50%, respectively. Thus, under the same three-trial tuning budget, the qualitative ordering remains SGD>Muon>AdamW\mathrm{SGD}>\mathrm{Muon}>\mathrm{AdamW}.

The fact that AdamW selects 0.01 rather than 0.1 also confirms that optimizer-specific tuning is consequential. We therefore do not claim hyperparameter-independent or globally optimal rankings. The result instead shows that the qualitative comparison in this controlled setting is not explained by a single unfavorable LR choice.

2. Single runs and seed variability.

The original tables report point estimates from single runs; we will make this explicit. To assess seed sensitivity in the main PreActResNet-18 multi-norm setting, we additionally ran three independent seeds.

At the selected checkpoint, Union robustness is 38.59±0.2938.59\pm0.29 for SGD, 37.16±0.4337.16\pm0.43 for Muon, and 36.14±1.3736.14\pm1.37 for AdamW.

This result does not support a claim that Muon is superior to SGD in this setting; SGD is stronger. It does, however, show that the qualitative ordering observed here is reproducible across seeds. We will also avoid treating sub-1 pp single-run differences in other settings as statistically established improvements.

3. ℓ∞\ell_\infty versus Union checkpoint selection.

We directly tested whether the original checkpoint criterion changes the optimizer ranking. On the same training trajectories, we simultaneously maintain checkpoints selected by validation ℓ∞\ell_\infty robustness and validation Union robustness.

Under ℓ∞\ell_\infty selection, the full-test Union accuracies for SGD/Muon/AdamW are 41.14/36.52/29.67. Under Union selection, they are 40.85/36.54/29.68. The ranking is therefore unchanged: SGD>Muon>AdamW\mathrm{SGD}>\mathrm{Muon}>\mathrm{AdamW}.

Muon and AdamW select the same epoch under both criteria, while SGD changes from epoch 71 to 69. Thus, the qualitative ordering in this experiment is not an artifact of using ℓ∞\ell_\infty rather than Union robustness for checkpoint selection. We will nevertheless use and recommend Union-based validation when the reported target metric is Union robustness, because it is the metric-aligned protocol.

4. Numerical value and role of the spectral bound.

The numerical audit confirms that Theorem 3.1 is conservative. Under the actual configuration with λ=5×10−4\lambda=5\times10^{-4} and the stepwise LR schedule, the schedule-aware finite-horizon ceiling is 1023.59, whereas the largest observed Muon weight spectral norm is 49.97. The gap is 20.48-fold. The stationary ceiling is 2000, corresponding to a 40.03-fold gap.

Therefore, the answer to the reviewer's question is direct: the bound is not tight enough to distinguish Muon from SGD. We do not use it for that purpose.

Theorem 3.1 establishes a different statement: bounded operator-norm updates together with positive weight decay imply a finite worst-case spectral-norm trajectory for Muon. Theorem 3.2 then gives a structural connection from such layerwise control to a first-order robustness proxy. These are optimizer-side structural results, not quantitative robustness certificates or comparative guarantees against SGD.

We will revise the manuscript to make this role explicit and to avoid any wording suggesting that the theoretical bound predicts the empirical optimizer ranking.

Overall interpretation.

The added audits support a narrower but stronger conclusion than a universal optimizer ranking. Optimizer choice materially changes AT behavior, but the ordering is architecture- and recipe-dependent. In the controlled PreActResNet-18 setting examined above, SGD remains stronger than Muon, while Muon remains stronger than AdamW after matched-budget LR selection. The theoretical result explains an optimizer-side stability property.

AuthorsOpenReview ↗

Revision follow-up

Dear Reviewer Q3bM,

We have submitted a revised manuscript incorporating targeted analyses addressing the questions raised in your review.

The revision includes four additions:

  1. Hyperparameter sensitivity: A matched-budget three-trial LR audit for each optimizer, with LR selection based exclusively on validation Union robustness. The resulting full-test ranking remains SGD > Muon > AdamW.

  2. Seed variability: A three-seed PreActResNet-18 comparison, reporting mean and standard deviation for all three optimizers.

  3. Checkpoint selection: A controlled comparison between validation ℓ∞\ell_\infty and Union-based selection on identical training trajectories. The optimizer ranking remains unchanged.

  4. Theoretical bound: A numerical audit under the actual training schedule. The finite-horizon bound is 1023.59, compared with an observed maximum Muon weight spectral norm of 49.97. We explicitly clarify that this conservative bound establishes a worst-case trajectory property, not a quantitative robustness certificate or a ranking guarantee over SGD.

We have also clarified the distinction between the original single-run results and the additional controlled audits, rather than combining results obtained under different configurations.

We believe these additions directly address the empirical and theoretical questions raised in the review. The revised manuscript presents the resulting scope and limitations explicitly.

Best Regards!

Submission9034 Authors

Reviewer_Q3bMOpenReview ↗

Thank you for the detailed response and revision

Thank you for the detailed response and revision. The additional audits improve the transparency of the evaluation and address several of my protocol-related questions. However, they do not change my main concern regarding the overall strength of the contribution.

The numerical audit confirms that the spectral bound is highly conservative and cannot distinguish Muon from SGD or provide a practical robustness guarantee. Moreover, SGD remains stronger in the controlled comparison and on ImageNet, while the selected-checkpoint advantage over the magnitude-matched control disappears at the final checkpoint.

Therefore, the revision appropriately narrows the claims, but the remaining conclusion that optimizer choice affects adversarial-training trajectories does not constitute a sufficiently strong theoretical or empirical advance. I therefore maintain my score of 2.

Reviewer_8JGZOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates the Muon optimizer in adversarial training, focusing on how the choice of outer optimizer affects robustness. The authors provide a theoretical analysis of Muon's orthogonalized updates, including a bound on the spectral norm of weight trajectories and a nuclear-norm-based interpretation of its descent behavior. They also conduct experiments comparing Muon with SGD and AdamW across several architectures, datasets, and threat models, and include additional comparisons with SAM and TRADES. The main empirical finding is that Muon generally performs better than AdamW and can achieve competitive, or sometimes superior, robustness to SGD, particularly on larger CNNs. However, its advantages are not consistent across all settings, and SGD remains stronger in several ViT and ImageNet experiments.

Critical strengths
  1. The paper studies a relevant problem that deserves more attention. The choice of outer optimizer is an important part of adversarial training, but SGD remains the default. Given the recent interest in Muon, I think it is worth investigating whether its different update geometry can offer any benefits for adversarial robustness. The attempt to connect the optimizer's mathematical properties with adversarial training is also interesting.

  2. The empirical evaluation covers a reasonably broad range of settings. The authors evaluate multiple architectures and threat models, including both CNNs and ViTs, and provide additional results on ImageNet. I also appreciate that they report both best and last checkpoints, as well as cases where Muon does not outperform SGD. Additional evaluations using stronger attacks and different AT objectives are useful for understanding its empirical behavior.

Critical weaknesses
  1. Despite the extensive experiments and theoretical analysis, I struggled to identify a clear scientific takeaway from the paper.
    This is my main concern. The experiments show that Muon performs better than SGD in some settings but worse in others. However, the paper does not really explain why these differences occur or identify the conditions under which Muon is likely to be beneficial. For example, why does Muon perform relatively well on larger CNNs but fall behind SGD on ViTs and ImageNet? Why does it sometimes converge faster but still suffer from severe robust overfitting? These seem like important observations that deserve further investigation.
    The theoretical analysis does not help much in answering these questions. The spectral-norm bound mainly establishes a worst-case stability property, while the nuclear-norm analysis characterizes the update direction. Neither seems to provide a convincing explanation for the observed differences in robustness. The additional spectral analyses are interesting, but they remain largely descriptive rather than establishing a connection between the proposed mechanism and the empirical results.
    As a result, I was left wondering what we actually learned from this study beyond the observation that Muon can be a reasonable alternative to SGD in certain settings. Given that both Muon and the adversarial training framework already exist, I would expect this study to provide either a more consistent practical advantage or a clearer understanding of when and why Muon helps. I think the paper would be more convincing if the authors could draw clearer insights from these mixed results, particularly regarding when and why Muon is beneficial for adversarial training.

  2. The experimental comparisons are not sufficiently convincing to support the conclusions about the relative performance of different optimizers.
    The authors acknowledge that they do not perform exhaustive optimizer-specific hyperparameter searches. However, learning rate, weight decay, and the training schedule can substantially affect adversarial training, particularly when comparing optimizers with very different update rules. This is especially concerning for the ViT experiments, where AdamW performs extremely poorly in some settings. It is difficult to tell whether these results reflect an inherent limitation of AdamW or simply an unfavorable training configuration.
    Another concern is the lack of results across multiple random seeds or any reported standard deviations. Some of Muon's improvements over SGD are relatively small, making it difficult to judge whether they are statistically meaningful. The checkpoint-selection protocol also deserves clarification: the authors select the best checkpoint based on validation $ l_\infty$ robustness, even though union robustness is an important evaluation metric and is later recommended as the checkpoint-selection criterion.
    These issues make it difficult to determine how much of the observed performance difference can actually be attributed to the optimizer itself.

Questions for authors
  1. What would you consider the main insight of this study beyond the empirical comparison of optimizers?
    The results suggest that Muon's effectiveness depends strongly on the architecture and training setting, but I could not find a clear explanation of this dependence. Are there any properties of the model, gradients, or optimization process that could help predict when Muon would outperform SGD? I think understanding these differences would make the study much more valuable.

  2. Could you provide stronger evidence connecting the theoretical analysis to the empirical robustness results?
    For example, do the spectral-norm bounds or the nuclear-norm-aligned updates actually explain why Muon performs better in certain experiments? It would be helpful to see more direct evidence or controlled ablations that isolate the effect of orthogonalization from other factors, such as effective update magnitude. I am also curious how the proposed stability mechanism relates to the severe robust overfitting observed in some experiments.

  3. Could you clarify how the hyperparameters were selected for each optimizer, and whether the reported results are consistent across different random seeds?
    In particular, were comparable tuning efforts made for SGD, AdamW, and Muon? The poor AdamW results on ViTs make this especially important. Reporting standard deviations for the main experiments would also help assess the significance of the relatively small improvements in some settings. Additionally, why was the best checkpoint selected using ℓ∞ robustness rather than union robustness?

  4. Could you clarify what Figure 1 is intended to illustrate?
    I had some difficulty understanding the purpose of this figure. It seems to simply place Muon, adversarial training, and empirical evaluation together, without really explaining how they interact or what the proposed framework adds. Since the connection between the theoretical analysis and the empirical findings is an important part of the paper, I think a more informative figure illustrating this connection would be much more useful.

7 comments

Dear Reviewer 8JGZ,

The review identifies an important distinction that we will make more explicit in the revision: mixed optimizer rankings alone are not the scientific takeaway of this work. Our claim is not that Muon should replace SGD, nor that one optimizer admits a universal robustness advantage. The central finding is that the geometry of the outer update is itself a consequential variable in adversarial training, and that orthogonalization changes the optimization trajectory in ways that cannot be reduced to a change in effective step magnitude.

1. What do we learn beyond an empirical optimizer comparison?

The new controlled ablation directly isolates the effect of orthogonalization. Let MtM_t denote the momentum accumulator. Muon uses UtMuon=Ortho⁡(Mt)U_t^{\mathrm{Muon}}=\operatorname{Ortho}(M_t). We construct a non-orthogonal control that keeps the original momentum direction but exactly matches Muon's Frobenius update magnitude at every matrix block and every step: Utctrl=∥UtMuon∥F∥Mt∥FMtU_t^{\mathrm{ctrl}}=\frac{\|U_t^{\mathrm{Muon}}\|_F}{\|M_t\|_F}M_t.

The two runs use identical initialization and the same 39,100 sampled training-norm choices. The worst relative update-norm mismatch is only 2.39×10−72.39\times10^{-7}. Nevertheless, Muon reaches 39.34% selected-checkpoint Union robustness versus 35.86% for the non-orthogonal control. At epoch 100, the maximum weight spectral norm is 51.04 for Muon versus 65.08 for the control, and the median condition number is 73.85 versus 116.01.

This result establishes a more specific conclusion than the original empirical comparison: orthogonalization has a measurable effect on AT dynamics beyond effective update magnitude. We view this as the main mechanistic contribution of the study.

2. Does this explain when Muon will outperform SGD?

Not universally, and we do not think the current evidence justifies such a predictor. The controlled result above identifies what orthogonalization changes; it does not imply that the resulting trajectory must yield higher robust accuracy than SGD for every architecture or schedule.

This distinction is important for interpreting the mixed results. Update-level spectral control and robust generalization are different properties. In the same magnitude-matched experiment, final-checkpoint Union robustness is 35.96% for Muon and 36.71% for the control. Thus, orthogonalization changes the selected robustness frontier and spectral dynamics, but it does not prevent late-stage robust overfitting. We will revise the paper to separate these two claims explicitly.

This also explains why we do not interpret the ViT and ImageNet results as contradictions. They show that the benefit of a particular update geometry is mediated by architecture and training recipe rather than producing an architecture-independent ordering. SGD remains stronger in several of these regimes, and we retain those negative results as part of the paper's conclusion rather than treating them as exceptions to be explained away.

3. How does the theory connect to the empirical results?

Theorem 3.1 is not intended to predict the ranking between Muon and SGD. Its role is narrower: bounded operator-norm updates, together with positive weight decay, imply a worst-case ceiling on the Muon weight trajectory. We agree that this distinction was not sufficiently sharp in the original presentation.

We therefore instantiated the bound numerically under the actual PreActResNet-18 configuration. The schedule-aware finite-horizon ceiling is 1023.59, whereas the largest observed Muon weight spectral norm is 49.97, a 20.48-fold gap. The stationary ceiling is 2000. The bound is therefore conservative and is not sufficiently tight to predict robust accuracy or distinguish Muon from SGD.

We will state this explicitly. The theoretical result provides an optimizer-side structural property; the magnitude-matched ablation supplies the empirical evidence that the corresponding orthogonalization mechanism materially changes the trajectory. We will not present the bound itself as an explanation of optimizer ranking or as a robustness certificate.

4. Are the empirical comparisons robust to tuning, seeds, and checkpoint selection?

We added three targeted checks.

First, in a matched-budget local LR audit, each optimizer receives exactly three trials: SGD {0.03,0.1,0.3}\{0.03,0.1,0.3\}, AdamW {0.01,0.03,0.1}\{0.01,0.03,0.1\}, and Muon {0.03,0.1,0.3}\{0.03,0.1,0.3\}. LR is selected exclusively by validation Union robustness before full-test evaluation. AdamW selects 0.01, confirming that optimizer-specific tuning matters. After selection, full-test Union robustness is 40.86% for SGD, 36.50% for Muon, and 33.53% for AdamW. Thus, the qualitative ordering remains SGD>Muon>AdamW\mathrm{SGD}>\mathrm{Muon}>\mathrm{AdamW} under the matched tuning budget. We characterize this as a local sensitivity audit, not an exhaustive hyperparameter search.

Second, over three independent PreActResNet-18 runs, selected-checkpoint Union robustness is 38.59±0.2938.59\pm0.29 for SGD, 37.16±0.4337.16\pm0.43 for Muon, and 36.14±1.3736.14\pm1.37 for AdamW. We will therefore avoid interpreting small single-run differences elsewhere as statistically established superiority.

Third, we compared checkpoint selection by validation ℓ∞\ell_\infty robustness and validation Union robustness on identical training trajectories. The ranking remains SGD>Muon>AdamW\mathrm{SGD}>\mathrm{Muon}>\mathrm{AdamW} under both criteria. Muon and AdamW select the same epoch under both rules; SGD changes only from epoch 71 to 69.

5. Revised scientific takeaway.

We will sharpen the paper around the following claim: outer-optimizer geometry is not a neutral implementation detail in AT. Orthogonalization changes the optimization trajectory beyond update magnitude, but this mechanism does not imply a universal optimizer ranking or immunity to robust overfitting. The architecture- and recipe-dependent ranking is therefore part of the finding, not evidence for a universal Muon advantage.

We will revise the abstract and conclusion accordingly and redesign Figure 1 to separate three levels that were insufficiently distinguished in the original version: update geometry, optimization/spectral dynamics, and robust generalization.

AuthorsOpenReview ↗

Revision follow-up

Dear Reviewer 8JGZ,

We have submitted a revised manuscript with additional controlled analyses addressing the main concerns in your review.

Most importantly, we added the magnitude-matched non-orthogonal ablation you suggested. By matching the Frobenius update magnitude at every matrix block and optimization step, the experiment isolates the effect of orthogonalization from effective update scaling. Muon achieves 39.34% selected-checkpoint Union robustness, compared with 35.86% for the control. The corresponding spectral and conditioning diagnostics also differ substantially.

We additionally report the final-checkpoint results, where the control slightly outperforms Muon (36.71% vs. 35.96%). We consider this distinction important: orthogonalization changes spectral optimization dynamics, but does not guarantee immunity to robust overfitting.

The revision also incorporates the matched-budget LR audit, three-seed comparison, checkpoint-selection sensitivity analysis, and a numerical assessment of the theoretical bound. We explicitly acknowledge that the bound is conservative and is not intended to predict the optimizer ranking.

Our revised central claim is that orthogonalized update geometry has an identifiable effect beyond update magnitude, while its translation into robust generalization remains architecture- and training-dependent.

We hope these additions address the central scientific concerns and provide a stronger basis for re-evaluating the revised contribution.

Yours Sincerely,

Submission9034 Authors

Reviewer_J5Q8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies how different optimizers (AdamW, SGD, Muon) affect the performance of adversarial training in different classifiers and datasets. They mainly focus on the Muon optimizer as it becoming more relevant in different foundation model training works. They show different properties of the Muon optimizer and that it can generate better results in some scenarios. They claim that based on some of their results, the outer optimizer choice can make a difference in the robustness and should be tested more on adversarial training works. Empirically they show their results in different classifiers on the CIFAR-10 and the ImageNet dataset.

Critical strengths

Here are some strengths of the paper in my opinion:

  1. They study the Muon optimizer for Adversarial Training (AT) for the first time which is useful in understanding if that optimizer changes the performance of the defended classifier.

  2. They compare their results starting from CIFAR-10 on different classifiers to ImageNet dataset. They study how the optimizer performs on different neural network architectures and report the result

  3. The paper compares different attack types and norms

Critical weaknesses

Here are some weakness of the paper in my opinion:

  1. The paper studies well the effect of the Muon optimizer and analyzes it but overall the contribution of the paper seems small to me.

  2. The results don't show a general pattern and the Muon optimizer still doesn't outperform the current used SGD in different scenarios. So, I doubt the use of Muon as a general optimizer for AT at this point based on the results shown.

  3. The method underperforms on bigger models and datasets like ImageNet. For example, on Table 4, the method consistently underperforms. And also the Muon's performance decreases sharply on the "last checkpoint" you have shown.

Questions for authors

Please look at the weakness section

5 comments

Dear Reviewer J5Q8,

We appreciate the reviewer for recognizing the relevance of studying Muon in adversarial training and the breadth of our evaluation. We do not view universal superiority over SGD as the criterion for the contribution of this study. Our central question is whether outer-optimizer geometry materially changes adversarial-training dynamics and robustness, and the empirical and controlled results show that it does. Muon is particularly informative in this context because its orthogonalized matrix updates provide a qualitatively different update geometry from SGD and AdamW. The mixed rankings across architectures are therefore not a contradiction of the paper's central claim; rather, they show that optimizer geometry interacts with architecture and training regime instead of inducing a universal ordering.

What is the contribution if Muon does not consistently beat SGD? Our intended contribution is not a new optimizer that uniformly dominates SGD. Rather, the study asks whether the outer optimizer is a neutral implementation choice in adversarial training. Our results indicate that it is not. Different optimizer geometries produce materially different robustness trajectories, cross-norm robustness profiles, and checkpoint behavior under the same AT objective. Muon serves as a particularly informative case because its matrix-valued updates are orthogonalized rather than coordinate-wise rescaled.

We added a controlled experiment to determine whether the observed behavior of Muon is merely due to a different effective update magnitude. For the momentum accumulator MtM_t, Muon uses UtMuon=Ortho⁡(Mt)U_t^{\mathrm{Muon}}=\operatorname{Ortho}(M_t), while the control retains the original momentum direction but matches Muon's Frobenius update norm at every matrix block and every step: Utctrl=∥UtMuon∥F∥Mt∥FMtU_t^{\mathrm{ctrl}}=\frac{\|U_t^{\mathrm{Muon}}\|_F}{\|M_t\|_F}M_t. Initialization, training-norm sequence, learning-rate schedule, momentum, and weight decay are identical, and the mean relative norm-matching error is 2.78×10−82.78\times10^{-8}. Muon obtains 39.34% selected-checkpoint union robustness versus 35.86% for this control. At epoch 100, its maximum weight spectral norm is 51.04 versus 65.08 for the control, and the median condition number is 73.85 versus 116.01. This provides direct evidence that orthogonalization changes the optimization trajectory beyond merely changing the step magnitude.

At the same time, we agree that this mechanism does not imply better late-stage generalization. The final union robustness is 35.96% for Muon and 36.71% for the matched control. This result reinforces, rather than hides, the robust-overfitting limitation: controlled spectral dynamics and robust-generalization stability are distinct properties.

Regarding SGD and ImageNet, we agree with the reviewer. SGD remains stronger in our ImageNet experiment, and we will continue to report this as a negative result and a scope boundary. We will remove wording that could suggest a general superiority of Muon. A new three-seed PreActResNet-18 experiment also gives selected-checkpoint union robustness of 38.59±0.2938.59\pm0.29 for SGD, 37.16±0.4337.16\pm0.43 for Muon, and 36.14±1.3736.14\pm1.37 for AdamW. Thus, this controlled setting again supports SGD over Muon rather than the reverse.

Accordingly, we will revise the abstract, introduction, and conclusion around a narrower takeaway: optimizer geometry is a consequential component of AT, but it does not induce a universal robustness ordering. Muon's value in this study is to expose and analyze that dependence, not to establish it as a general-purpose replacement for SGD. We will also explicitly distinguish selected-checkpoint optimization behavior from last-checkpoint robust generalization.

Yours Sincerely,

Submission 9034 Team

AuthorsOpenReview ↗

Revision follow-up

Dear Reviewer J5Q8,

We have submitted a revised manuscript that sharpens the central contribution of this work. In particular, we clarify that the objective is not to establish Muon as a universal replacement for SGD, but to investigate whether outer-optimizer geometry materially affects adversarial-training dynamics.

The revision incorporates a magnitude-matched non-orthogonal control, showing a 3.48 percentage-point difference in selected-checkpoint Union robustness even when per-block update magnitudes are matched. We also add three-seed results for PreActResNet-18 and retain the ImageNet results, where SGD remains stronger, as an explicit scope boundary.

We believe these results support a more specific contribution: optimizer geometry is an independent and consequential dimension of adversarial-training design, rather than merely another implementation choice. We hope the revised manuscript will be considered in light of this clarified contribution.

Yours Sincerely,

Submission 9034 Authors

Reviewer_J5Q8OpenReview ↗

Reply to Authors

Dear Authors,

Thank you for your rebuttal. My concern still remain as the main story of the paper is still weak. The results of using Muon optimizer do not show a specific pattern and a useful takeaway. Because of that, I will keep my current score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

We appreciate the reviewers' continued engagement. At this stage, we believe the remaining disagreement is primarily about the scientific interpretation of the evidence rather than missing experimental controls.

The revised manuscript establishes a consistent finding: orthogonalization can improve the optimization-side robustness trajectory without improving robust-generalization stability. The magnitude-matched control isolates this effect beyond update scale, while the selected-to-final checkpoint reversal shows that favorable spectral/update dynamics are not sufficient for superior late-stage robustness. The multi-seed, matched-budget hyperparameter, checkpoint-selection, and numerical-bound audits further show that this conclusion is not an artifact of a particular seed, learning rate, selection rule, or an overly literal interpretation of the theory.

Accordingly, the contribution is not a universal ranking of Muon against SGD, but a separation between two phenomena that should not be conflated in adversarial training: optimizer-side trajectory control and robust generalization. We have reorganized the manuscript and Figure 1 around this distinction and clarified the scope of both the theoretical and empirical claims. We respectfully ask that the revised submission be assessed on this clarified scientific conclusion and the controlled evidence supporting it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.