acceptodds
Under review as a conference paper at ICLR 2027

To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters

Abstract

While Adam has long been the ubiquitous default optimizer for deep neural networks, Muon has recently seen rapid adoption due to its superior training speed. Although much of the literature focuses on validating the benefits of Muon, our work investigates the potential downsides of the mechanism driving this speedup. On the theoretical front, we analyze the learning dynamics of simplified Muon on deep linear networks and multi-head linear attention. Our analysis reveals that Muon gains speed by avoiding saddle points, but at the expense of the simplicity bias exhibited by gradient descent, under which the complexity of the learned functional solution increases progressively. On the empirical front, we show experiments that demonstrate the consequences of losing the simplicity bias. Muon struggles to uncover common underlying structure across tasks and may be prone to fitting spurious features. More broadly, this paper serves as a reminder that faster optimization is rarely a free lunch; improvements in optimization can come at the cost of changes in the inductive biases that shape generalization

Primary area:
optimization
Keywords:
Muon, Deep Learning Optimization, Implicit Bias

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_mLdrOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3(+1)
Presentation
3
Soundness
3
Read review
Paper summary

This work investigates the simplicity bias of the Muon optimizer. The theory is concerned with the spectral flow proxy for Muon and it shows that singular directions are learned together instead of in order which contrasts with gradient flow. The consequences are demonstrated in controlled experiments. It is shown that Muon picks up spurious features easier than SGD. Moreover, in a cyclic task spectral GD is not able to find the generalizable solution and instead memorizes in contrast to GD.

Critical strengths

The work makes an important point regarding optimizer choice affecting the evolution of the singular values and therefore the solution reached. This is well substantiated by a theoretical toy setting and clear controlled experimental design. It furthermore highlights that optimization speed should not be the sole criterion for using an optimization procedure.

Critical weaknesses

Empirical evaluation and guidance is limited. As pointed out in the limitations the work is focused more on tractable settings, however, while the theory describes the corresponding settings well it provides limited predictive power for large scale settings or the considered control experiments for the cyclic task. Concretely, the theory is concerned with singular value growth which eventually would reach a similar solution but for example Section 4 relies on the final solution reached which is of higher rank in case of spectral GD.

Moreover, the work highlights other empirical works to substantiate the relevance to more practical settings. These settings compare mainly with Adam introducing an important confounder as Adam may also not follow a saddle-to-saddle dynamics.

Questions for authors

Can the authors provide an ablation adding explicit regularization to promote low rank (not weight decay) in the task considered in Section 4, to isolate the incremental learning from the rank bias?

2 comments

We thank the reviewer for their feedback, and are glad they recognize that optimization speed alone doesn’t tell us the full story, as shown in our paper! We’re happy to hear they found the paper well substantiated by theory and clear controlled experiments!

Concretely, the theory is concerned with singular value growth which eventually would reach a similar solution but for example Section 4 relies on the final solution reached which is of higher rank in case of spectral GD

There might have been a misunderstanding in what piece of theory we use for our experiments:

  • In our simplified theoretical settings we show Spectral GD and GD have different learning dynamics, but both of them indeed converged to the same solution
  • The piece of theory we take away, the piece that generalizes to more complex non-linear settings, is the difference in dynamics
  • In non-linear settings, as we showed empirically, this difference in dynamics leads to learning different solutions (which wasn't the case in theory settings)
  • With our empirical setups, we demonstrated properties of these different solutions, namely Muon one generalizes worse than SGD one

If this makes it clearer, we will make sure to add it to the main text.

These settings compare mainly with Adam introducing an important confounder as Adam may also not follow a saddle-to-saddle dynamics.

The reviewer’s concern is justified, and we indeed show (empirically) in Appendix E that Adam and AdamW also lose the simplicity bias. However, the work we referenced [1] does contain a memorization study including SGD in Figure 7. They empirically show that SGD memorizes less, Muon more, and AdamW is somewhere in the middle. We now included this additional explanation in the conclusion.

Can the authors provide an ablation adding explicit regularization to promote low rank (not weight decay) in the task considered in Section 4, to isolate the incremental learning from the rank bias?

This is an interesting question! We showed that {the simplicity bias of SGD => low rank bias} and {simplicity bias of SGD => generalizable solution}. The reviewer now asks whether imposing low rank bias explicitly can also recover generalizable solutions.

To this end, we implemented trace norm regularization on weight matrices. Specifically, we decouple it from the Spectral GD step, and use SVD to get an accurate gradient wrt the normalization term (in larger scale settings this would be intractable, but one could use e.g. Newton-Schulz).

The results are in the updated version (in Q&A and Figure 21), and show that while low rank regularization somewhat helps, the mechanism used is different. Namely, Spectral GD + low rank regularization leads to many singular values rising in the beginning to reduce the loss first, and then falling back to being 0.

We hope this addresses your concerns and the question, and we’re happy to discuss anything further. If all your concerns are addressed, we’d be grateful if you would revise your score accordingly.


[1] ‘Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Less’, 2026, authors: Yuxing Liu, Jianyu Wang, Tong Zhan

I thank the authors for their detailed response. I appreciate the additional low rank ablation and Adam clarification. The clarification I believe is reasonable. Overall, I agree also with the other reviewers that the conceptual contribution is strong, I do believe that the theory is relatively simple and the experimental scale is also quite limited, therefore I revise my score to 3.

Reviewer_3PnCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that Muon's optimization speed comes from removing the saddle-to-saddle simplicity bias of gradient descent (GD), which can sometimes hurt generalization. It analyzes the training dynamics of the spectral GD and GD in theory on two-layer linear networks and multi-head linear attention. The theoretical results are supported by the small-scale experiments designed. It provides a new perspective when comparing optimizers in practice.

Critical strengths
  1. The trade-off between optimization and generalization studied in the paper is interesting and meaningful. It provides a new perspective on comparing different optimizers and on designing and applying new ones.

  2. The paper aligns theory with experiments. It designs several experiments, with analyses of training dynamics, to illustrate the insights. They can be used for further analysis in later works.

Critical weaknesses
  1. Although the insights seem straightforward and reasonable, it seems that Muon still generalizes well in most cases when training current LLMs in practice. I didn't find a clear connection between this paper's results and the many empirical results. For instance, can the analysis in this paper explain why the generalization issue with Muon doesn't happen in those cases, and explain why it indeed happens? This limits the significance of this work. It would be good if the analysis yielded insights that could be directly reflected in practice.

  2. The depth and comprehensiveness of the work are not very clear to me. The theoretical parts in Section 2 are based on prior work, and many assumptions are made when proving the results in Section 5. On the empirical side, all experiments are limited in scale, and it's unclear whether all hyperparameters are well-tuned, which seems important for the training dynamics, as shown in Appendix E.3. Also, I think in practice, people mostly compare Adam and Muon, where Muon is found to be faster than Adam.

Questions for authors
  1. Could you elaborate on the hyperparameters chosen in the experiments and verify the solidity of the results?

  2. Could you connect your results to the LLM training and analyze observations there?

2 comments

We thank the reviewer, and are happy they appreciate how our theory aligns with and motivates our experiments! We’re glad they found our representation learning viewpoint of optimizers novel, interesting and meaningful!

Could you elaborate on the hyperparameters chosen in the experiments and verify the solidity of the results?

  • For the digit and pixel feature learning example with a CNN, the details of the learning rate sweep are in Appendix D.2

In the routing task and cycle task with transformers, the experiments highlight that even though the training looks successful and the loss reduces, it can happen that the representations learned are not generalizing. Nonetheless, we agree with the reviewer that providing multiple hyperparameter values solidifies the results further. In the updated PDF submission we added:

  • Learning rate sweep for routing task in Appendix D.3
  • Learning rate sweep for cycle task in Appendix D.4
  • Different number of heads are already present in Figure 12 (used to be 10) We see that Muon struggles to learn generalizable structures while SGD succeeds across settings.

Could you connect your results to the LLM training and analyze observations there?

Our main takeaway is that representations learned with Muon may encourage memorization over generalization, but in LLMs it is harder to extract and analyze representations. LLM setting doesn't lend itself to a clean controlled experiment. For example, there is no obvious knob in the training data that controls the strength of spurious features, and thus their effect can't be separated from confounding factors.

For preliminary evidence of Muon encouraging memorization, we refer the reviewer to concurrent works [1] and [2]. Both contain examples where Muon pre-trained model finetuned on math performs worse than AdamW counterpart, contrary to popular belief that Muon is better. We agree that understanding this further is an important next step, and hope that our paper provides a useful point of view towards answering it.

As detailed in our Discussion section, our paper hints at one-by-one feature learning being beneficial when the features are coupled, i.e. uncovering what the second feature is can happen only after the first feature is fully learned. E.g, in the cycle task, it was important for generalization to first learn to encode the tokens in cyclical representations, and only then start learning what to do for each specific skip. Muon’s uniform learning does not do that, and learns all skips separately, at the same time, failing to recognize the common underlying cyclical structure that greatly simplifies the task. In these LLM experiments, we expect coupled features are also present in math, but not in memorization tasks, hence Muon performs worse in [1], but better in [3].

Also, I think in practice, people mostly compare Adam and Muon

Yes, we agree. This is the case because training at a larger scale with SGD is infeasible due to instabilities. We comment on Adam in Appendix E, and show empirically that it also loses simplicity bias (or perhaps one can say it’s in the middle between SGD and Muon, as shown in [1] Figure 7). We now added this clarification in the conclusion.

Because Adam is not a spectral optimizer, its theoretical analysis draws on a different set of tools and results (e.g. [4]). Fitting Muon, Adam, and GD into a single framework in which they are directly comparable would therefore call for a separate work, and we see this as a valuable future direction.

We're happy to further explain any of the points above, please let us know if there are any concerns left. If not, we'd be grateful if you would reconsider your score.


[1] ‘Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Less’, 2026, authors: Yuxing Liu, Jianyu Wang, Tong Zhan

[2] ‘Pretraining Data Scale Reverses the Muon-AdamW Fine-Tuning Gap’, 2026, authors: Catherine Li, Jingchu Gai, Andrej Risteski, Jacob Mitchell Springer, Aditi Raghunathan

[3] ‘Muon Outperforms Adam in Tail-End Associative Memory Learning’, 2025, authors: Shuche Wang, Fengzhuo Zhang, Jiaxiang Li, Cunxiao Du, Chao Du, Tianyu Pang, Zhuoran Yang, Mingyi Hong, Vincent Y. F. Tan

[4] ‘Adam Exploits l∞l_\infty-geometry of Loss Landscape via Coordinate-wise Adaptivity’, 2025, authors: Shuo Xie, Mohamad Amin Mohamadi, Zhiyuan Li

Thanks for the response. The additional learning-rate sweeps address my concern about the robustness of the results, and the discussion of coupled features gives a plausible account of when the loss of simplicity bias may matter in practice. Personally, I would still like to see some evidence from more realistic settings, even if a rigorous study there is harder to conduct. Nevertheless, I believe the current findings are useful to the community, and I have raised my score to 3.

Reviewer_FbmiOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

While gradient descent learns features at a rate inversely proportional to their eigenvalues, the orthogonalization in Muon causes all features to be learned at the same rate. In gradient descent, this leads to a simplicity bias where the problem is learned with a small number of features. Any practical amount of training is essentially "early stopping" that acts as implicit regularization, which they frame as an "implicit curriculum".

In contrast, the orthogonalization step in Muon makes the optimizer take the "same size step" in all directions, removing the simplicity bias. The paper argues that this simplicity bias is important for generalization and that the theory suggests it could cause generalization failures in Muon.

They show theoretically that gradient flow in deep linear networks illustrate this idea. They use an MNIST setup with a spurious pixel feature added, and show that the strength of the informative / spurious features learned depends on the optimizer. They use a routing task that maps numbers to vectors and show that Muon fails to learn a simple generalizable representation while SGD does.

For the multi-head attention setting, they show that this simplicity bias makes gradient descent learn attention heads one at a time, whereas (idealized) Muon learns all the heads simultaneously. They argue that, for generalization, is it better for a single concept to be learned by a single head and closely related concepts to be learned by the same head for generalization.

Critical strengths
  1. This paper was early to the area of understanding differences between Muon and other optimizers not just measured in terms of final pretraining losses but in terms of the representations learned and that Muon has memorization and generalization disadvantages. The core idea is very nice: it connects the spectral bias perspective (eigenmodes are learned sequentially), the optimizer (Muon vs SGD) differences, and the effects on generalization.
  2. The theory and experiments are well-designed to support the core idea of the paper.
  3. The paper presentation is clear and well-organized. Which is refreshing in the current era of AI-written papers.
Critical weaknesses

The experiments are relatively small-scale, but I don't think it's necessary for every paper to have large-scale experiments. This paper is a nice example of how a clear idea can be investigated in well-designed small experiments and add understanding to the literature.

Questions for authors

I don't have questions for the authors at this time.

2 comments

We thank the reviewer for the positive feedback! We’re glad they liked our investigation of Muon optimizer beyond just training loss and speed. We’re happy to hear they found the exposition clear, the theory and experiments well-designed, and that overall, that the paper brings a deeper understanding in the field.

Thanks to the authors for the rebuttal clarifications. I read through the discussions with the other reviewers and I'm happy their concerns were addressed. I will keep my original positive score and hope to see this work accepted.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.