- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 3(+1)
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work investigates the simplicity bias of the Muon optimizer. The theory is concerned with the spectral flow proxy for Muon and it shows that singular directions are learned together instead of in order which contrasts with gradient flow. The consequences are demonstrated in controlled experiments. It is shown that Muon picks up spurious features easier than SGD. Moreover, in a cyclic task spectral GD is not able to find the generalizable solution and instead memorizes in contrast to GD.
Critical strengths
The work makes an important point regarding optimizer choice affecting the evolution of the singular values and therefore the solution reached. This is well substantiated by a theoretical toy setting and clear controlled experimental design. It furthermore highlights that optimization speed should not be the sole criterion for using an optimization procedure.
Critical weaknesses
Empirical evaluation and guidance is limited. As pointed out in the limitations the work is focused more on tractable settings, however, while the theory describes the corresponding settings well it provides limited predictive power for large scale settings or the considered control experiments for the cyclic task. Concretely, the theory is concerned with singular value growth which eventually would reach a similar solution but for example Section 4 relies on the final solution reached which is of higher rank in case of spectral GD.
Moreover, the work highlights other empirical works to substantiate the relevance to more practical settings. These settings compare mainly with Adam introducing an important confounder as Adam may also not follow a saddle-to-saddle dynamics.
Questions for authors
Can the authors provide an ablation adding explicit regularization to promote low rank (not weight decay) in the task considered in Section 4, to isolate the incremental learning from the rank bias?
2 comments
I thank the authors for their detailed response. I appreciate the additional low rank ablation and Adam clarification. The clarification I believe is reasonable. Overall, I agree also with the other reviewers that the conceptual contribution is strong, I do believe that the theory is relatively simple and the experimental scale is also quite limited, therefore I revise my score to 3.
We thank the reviewer for their feedback, and are glad they recognize that optimization speed alone doesn’t tell us the full story, as shown in our paper! We’re happy to hear they found the paper well substantiated by theory and clear controlled experiments!
There might have been a misunderstanding in what piece of theory we use for our experiments:
If this makes it clearer, we will make sure to add it to the main text.
The reviewer’s concern is justified, and we indeed show (empirically) in Appendix E that Adam and AdamW also lose the simplicity bias. However, the work we referenced [1] does contain a memorization study including SGD in Figure 7. They empirically show that SGD memorizes less, Muon more, and AdamW is somewhere in the middle. We now included this additional explanation in the conclusion.
This is an interesting question! We showed that {the simplicity bias of SGD => low rank bias} and {simplicity bias of SGD => generalizable solution}. The reviewer now asks whether imposing low rank bias explicitly can also recover generalizable solutions.
To this end, we implemented trace norm regularization on weight matrices. Specifically, we decouple it from the Spectral GD step, and use SVD to get an accurate gradient wrt the normalization term (in larger scale settings this would be intractable, but one could use e.g. Newton-Schulz).
The results are in the updated version (in Q&A and Figure 21), and show that while low rank regularization somewhat helps, the mechanism used is different. Namely, Spectral GD + low rank regularization leads to many singular values rising in the beginning to reduce the loss first, and then falling back to being 0.
We hope this addresses your concerns and the question, and we’re happy to discuss anything further. If all your concerns are addressed, we’d be grateful if you would revise your score accordingly.
[1] ‘Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Less’, 2026, authors: Yuxing Liu, Jianyu Wang, Tong Zhan