- Rating
- 4
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes MuonM for language model pretraining. It combines fast and slow momentum along estimated flat directions with sphere constraints, and tests the method on dense and mixture-of-experts models. The experiments compare several optimizers and individual components across model sizes and training lengths, and report measured step time.
Critical strengths
- Gains across model sizes and training lengths support value beyond one small setting.
- The sphere-only baseline helps separate the effects of sphere constraints and the added momentum design.
- Measured step time shows about 0.9% overhead for the 5.1B mixture-of-experts model, supporting practical use.
Critical weaknesses
- Appendix B.2 supports the direction proxy with a small example. The author can check curvature directly on a trained model and strengthen the evidence for the mechanism.
- Appendix B.3 reports step time, but total time to reach a target loss would make the speed gains easier to judge in practice.
- The paper notes that the best learning-rate schedule under sphere constraints remains an open question. The authors should provide brief practical guidance based on the tested schedules.
Questions for authors
Please respond to the concerns raised in Critical Weaknesses.
Response to Reviewer JwCq
We appreciate the reviewer for the valuable comments and recognition to our work. Our response are as follows. Should there be any additional questions, we are more than willing to provide further clarification.
W1. Appendix B.2 supports the direction proxy with a small example. The author can check curvature directly on a trained model and strengthen the evidence for the mechanism.
Re1: Thanks for the constructive comment. We evaluate how well our sharp-direction projection estimate covers the top subspace of the Hessian (i.e., the sharp directions) in the 0.12B dense model (dmodel=768) trained with MuonM in our experiments. Equivalently, this assesses the extent to which the orthogonal complement (i.e., the flat-direction projection produced by MuonM) lies within the Hessian flat directions. Details are provided in Appendix B.2, and the main results are summarized here for brevity.
For $W \in \{ W_{\text{up}}, W_{\text{gate}}, W_{\text{down}}, W_{\text{q}}, W_{\text{k}}, W_{\text{v}}, W_{\text{o}}\} $ in the second transformer layer, using a batch size of 32 and a sequence length of 1024, we estimate the top subspace of the Hessian using restarted Lanczos methods, an efficient randomized linear algebra technique that does not require computing the full Hessian matrix. Figure 11 (a) in Appendix B.2 (Pages 16-17) shows that throughout training, for each transformer block, the top-2000 subspace (sharp directions) of the Hessian exhibits rapid eigenvalue decay, concentrating sharp curvature into a low-dimensional subspace. Figure 11 (b) demonstrates that the top-2000 sharp subspace is well covered by our sharp-direction estimation (i.e., the top 50-200 left or right eigenspaces of the gradient), with most blocks exhibiting a coverage score of at least 0.8 on at least one side. Note that our sharp-direction subspace is formed by the product projection of both the left and right top eigenspaces. Thus, substantial coverage by the top eigenspace on either side alone is sufficient to demonstrate the reliability of our sharp-direction projection.
W2: Appendix B.3 reports step time, but total time to reach a target loss would make the speed gains easier to judge in practice.
Re2: Thanks for the suggestion. We have added the loss-time plots in Figure 12 (Appendix B.3, page 18). Given the nearly negligible additional time overhead of MuonM, its loss advantage is also maintained with respect to training time.
W3: The paper notes that the best learning-rate schedule under sphere constraints remains an open question. The authors should provide brief practical guidance based on the tested schedules.
Re3: Thanks for the comment. In this study, we primarily focus on cosine decay for spherical-constraint methods and find that it performs better than the WSD schedule in our experimental settings (e.g., the 0.6B MoE experiments in the left panel of Fig. 7 and in Fig. 14), which is also consistent with [1]. Therefore, cosine decay or a similar-shaped decay is an appropriate choice for spherical-constraint methods.
References:
[1] Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors, arXiv preprint arXiv:2606.25971, 2026.