- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors point out that the Frobenius normalization used in existing Muon implementations does not improve the condition number and can make singular values unnecessarily small. To address this issue, they propose Turbo-Muon, which incorporates Almost Orthogonal Layer (AOL) preconditioning into Newton-Schulz iterations with negligible additional computational overhead. Through comparisons with Muon and Muon+ (which uses Triton kernels and adaptive polynomial coefficients), the authors demonstrate that Turbo-Muon achieves faster training while maintaining comparable performance on CIFAR-10 speed-run, NanoGPT speed-run, and ImageNet benchmarks.
Critical strengths
The motivation behind Turbo-Muon is compelling. I agree that the existing Frobenius normalization, which ensures that all singular values are at most one, does not improve the condition number and can make the singular values unnecessarily small. Therefore, adapting the Almost Orthogonal Layer (AOL) from prior work to Newton-Schulz iterations with negligible additional computational overhead is a meaningful contribution.
Critical weaknesses
- The overall gains achieved by Turbo-Muon are marginal. As far as I understand, the main contribution of this paper is to use preconditioning to obtain a better initialization of the momentum matrix for NS iterations. However, as shown in Table 1, a larger portion of the runtime improvement comes from the additional Triton kernel rather than removing one NS iteration. Furthermore, the end-to-end improvements reported in Table 2 are marginal, with only a 0.011s reduction in CIFAR-10 and a 7.8s reduction in the NanoGPT speed-run task. Although there is a more noticeable improvement on ImageNet, I suspect that a substantial portion of this gain may also come from the third Triton kernel rather than the reduced number of NS iterations. Overall, I think much of the practical improvement of Turbo-Muon comes from engineering optimizations rather than AOL preconditioning itself.
- I have concerns about whether reducing the polar error is sufficient to improve Muon's training performance, particularly regarding the role of small singular values. In Figure 1b and Appendix B, Turbo-Muon achieves a lower polar error than the five-step Muon baseline even with fewer than four NS iterations in some settings. However, Table 2a shows that reducing the number of NS iterations below four leads to worse training performance compared to the five-step Muon baseline. This discrepancy suggests that either the actual gradient (momentum) matrices encountered during training are more ill-conditioned than those used in the synthetic experiments, or that reducing the polar error alone does not necessarily capture the properties important for training performance. In fact, [1] shows that replacing the singular values with random positive values can still achieve performance comparable to Muon, suggesting that making all singular values close to one may not be necessary. On the other hand, [2] emphasizes the importance of amplifying small singular values as a key motivation for Muon. Therefore, I think it is important to investigate how AOL preconditioning affects small singular values, rather than focusing only on the polar error.
References
[1] Shumaylov et al., Muon is Not That Special: Random or Inverted Spectra Work Just as Well. https://arxiv.org/abs/2605.11181
[2] Keller Jordan, Muon: An optimizer for hidden layers in neural networks. https://kellerjordan.github.io/posts/muon/
Questions for authors
- Could the authors provide an end-to-end training ablation study that separates the contribution of AOL preconditioning and the reduced number of NS iterations from that of the additional Triton kernel? While Table 1 provides a runtime breakdown for the NS routine, it would be helpful to understand how much each component contributes to the overall training speedup reported in Table 2.
- Could the authors evaluate the convergence of NS iterations using actual gradient(momentum) matrices collected during training, rather than only synthetic matrices sampled from Gaussian or Levy distributions? This would help clarify whether the discrepancy between polar error and training performance is related to differences in the underlying matrix distributions.
- Could the authors provide an analysis of how singular values evolve across NS iterations for Muon+ and Turbo-Muon, particularly focusing on the smallest singular values? It would be interesting to understand whether AOL preconditioning specifically improves the amplification of small singular values or primarily reduces the average polar error.
Minor Requests
- Please format the abstract as a single paragraph.
- When citing references at the end of sentences, please use
\citep{}instead of\citet{}for better readability. - Matrix notation is inconsistent throughout the paper. For example, matrices such as and are written in boldface in Section 3, but boldface notation is not consistently used in later sections including in . Please ensure consistent matrix notation throughout the paper.
- In Eq. (3), please replace the period (
.) with a multiplication dot (\cdot).
Answer to reviewer VfCL
We thank the reviewer for their detailed review and their suggestion to study the singular values in more detail than the polar error metric.
Q1. End to end training time
These results are the same as in the common response for readiness.
Using Muon with the triton kernel optimizations for lines 1, 2 and 3 (which we denote as Muon++) yields NanoGPT end-to-end training speeds of:
Using a machine with 4xH100, note that using AOL preconditioning with one less Newton-Schulz iteration gives a same order of magnitude speedup as our NS line 3 implementation.
Q2. Regarding the gradient matrices’ distributions
As we wanted a general measure of orthogonalization quality, we prioritized synthetic heavy tailed samples to avoid focusing on a specific training modality which could bias results (as argued in our answer to reviewer cfup).
We evaluate the methods on checkpoints of pre-NS updates on a GPT-2 training run on Fineweb, we compare the polar error of the orthogonalized update using either Muon (Frobenius), Turbo-Muon or MuonEq initializations (in mode R as recommended for transformer training). We report the best mean polar error across 48 parameter update matrices taken at step 1000, 2000 and 3000 using PolarExpress coefficients with hyperparameter ℓ=10−3 computed for 5 NS iterations. We get the following results (these results are the same as in the common response for readiness):
Turbo-Muon significantly outperforms other methods on real GPT model gradients.
Q2 / Q3. Measuring singular value spread on real gradients.
We agree that while the polar error measure is adapted to mean polar factor approximation quality estimation, it could be valuable to characterize the effect of AOL preconditioning on smaller singular values. Therefore, on CIFAR-10, we save weight updates at training steps 1, 40, 80, 120 and 160, then compute the quantiles of singular values of the 2D reshaped convolutional weight update (the quantiles are denoted as QXX) when running either standard Muon NS or AOL preconditioned NS (Turbo-Muon). These 2D matrices of the neural network updates are of shape (64x216, 64x576, 256x576, 256x2304). The results are as follow:
This shows the effect of AOL preconditioning on the smallest singular values. AOL allows for lower quantile spread around 1. In this experiment, AOL leaves the smallest singular value 3.06× and Q1 2.96× larger than Frobenius normalization on average (paired ratios), indicating better small singular value orthogonalization.