- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how momentum affects the convergence of the Muon optimizer. The authors prove convergence guarantees for sufficiently large momentum and construct a strongly convex quadratic problem on which Muon diverges for small momentum. They consider two sampling strategies and show that both behaviors can occur on the same problem. The paper also includes experiments on quadratic problems, MNIST, and CIFAR-10.
Critical strengths
The main strength of the paper is the divergence counterexample. The authors show that Muon can diverge even on a smooth, strongly convex quadratic problem where all component functions share the same minimizer. They also establish a stationarity guarantee for sufficiently large momentum on the same problem.
Critical weaknesses
The theoretical results cover only narrow ranges of momentum. For the quadratic problem with divergence is guaranteed below and stationarity above but the experiments show a transition around Thus, the theory does not explain the observed behavior. The convergence guarantees are for the best iterate and require additional assumptions for vanishing stationarity bounds. This makes the practical relevance of the theoretical results unclear.
The divergence example uses a specific quadratic problem where Muon behaves like SignSGD. It is unclear whether the same behavior occurs in real neural-network training. The MNIST and CIFAR-10 experiments mainly show that higher momentum reduces training loss, but do not directly support the theoretical results. The models are relatively small, and the CIFAR-10 experiments use approximate orthogonalization instead of the exact from the theory.
Questions for authors
I do not have specific questions for the authors. My main concerns are about how useful the theoretical results are in practice and whether they apply to other problems.