acceptodds
Under review as a conference paper at ICLR 2027

Turbo-Muon: Almost-Orthogonal Pre-Conditioning for Fast Muon Updates

Abstract

Orthogonality-based optimizers, such as Muon, have recently shown strong performance across large-scale training and community-driven efficiency challenges. However, these methods rely on a costly gradient orthogonalization step. Even efficient iterative approximations such as Newton-Schulz remain expensive, typically requiring dozens of matrix multiplications to converge. We introduce a pre-conditioning procedure that improves the initialization of the Newton–Schulz iterations while incurring negligible overhead. Furthermore, our pre-conditioning reduces the initial polar error and enables the removal of one Newton-Schulz iteration (out of the five iterations usually used in practice). The resulting implementation significantly reduces Muon’s overhead. At the end-to-end training level, we observe consistent runtime improvements across speed-run and standard benchmarks, including  3% reductions in training time on multiple fast training benchmarks, while matching reference performance on both language and vision tasks. Crucially, these improvements require no hyperparameter tuning and can be adopted as a simple drop-in replacement. Beyond empirical gains, we provide theoretical insight into the geometry of the update and its potential robustness against feature collapse. Our code is publicly available at https://anonymous.4open.science/r/flash-newton-schulz-D67B/anonymous-github.

Primary area:
optimization
Keywords:
Muon, optimization, AOL, pre-conditioning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VfCLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors point out that the Frobenius normalization used in existing Muon implementations does not improve the condition number and can make singular values unnecessarily small. To address this issue, they propose Turbo-Muon, which incorporates Almost Orthogonal Layer (AOL) preconditioning into Newton-Schulz iterations with negligible additional computational overhead. Through comparisons with Muon and Muon+ (which uses Triton kernels and adaptive polynomial coefficients), the authors demonstrate that Turbo-Muon achieves faster training while maintaining comparable performance on CIFAR-10 speed-run, NanoGPT speed-run, and ImageNet benchmarks.

Critical strengths

The motivation behind Turbo-Muon is compelling. I agree that the existing Frobenius normalization, which ensures that all singular values are at most one, does not improve the condition number and can make the singular values unnecessarily small. Therefore, adapting the Almost Orthogonal Layer (AOL) from prior work to Newton-Schulz iterations with negligible additional computational overhead is a meaningful contribution.

Critical weaknesses
  • The overall gains achieved by Turbo-Muon are marginal. As far as I understand, the main contribution of this paper is to use preconditioning to obtain a better initialization of the momentum matrix for NS iterations. However, as shown in Table 1, a larger portion of the runtime improvement comes from the additional Triton kernel rather than removing one NS iteration. Furthermore, the end-to-end improvements reported in Table 2 are marginal, with only a 0.011s reduction in CIFAR-10 and a 7.8s reduction in the NanoGPT speed-run task. Although there is a more noticeable improvement on ImageNet, I suspect that a substantial portion of this gain may also come from the third Triton kernel rather than the reduced number of NS iterations. Overall, I think much of the practical improvement of Turbo-Muon comes from engineering optimizations rather than AOL preconditioning itself.
  • I have concerns about whether reducing the polar error is sufficient to improve Muon's training performance, particularly regarding the role of small singular values. In Figure 1b and Appendix B, Turbo-Muon achieves a lower polar error than the five-step Muon baseline even with fewer than four NS iterations in some settings. However, Table 2a shows that reducing the number of NS iterations below four leads to worse training performance compared to the five-step Muon baseline. This discrepancy suggests that either the actual gradient (momentum) matrices encountered during training are more ill-conditioned than those used in the synthetic experiments, or that reducing the polar error alone does not necessarily capture the properties important for training performance. In fact, [1] shows that replacing the singular values with random positive values can still achieve performance comparable to Muon, suggesting that making all singular values close to one may not be necessary. On the other hand, [2] emphasizes the importance of amplifying small singular values as a key motivation for Muon. Therefore, I think it is important to investigate how AOL preconditioning affects small singular values, rather than focusing only on the polar error.

References

[1] Shumaylov et al., Muon is Not That Special: Random or Inverted Spectra Work Just as Well. https://arxiv.org/abs/2605.11181

[2] Keller Jordan, Muon: An optimizer for hidden layers in neural networks. https://kellerjordan.github.io/posts/muon/

Questions for authors
  1. Could the authors provide an end-to-end training ablation study that separates the contribution of AOL preconditioning and the reduced number of NS iterations from that of the additional Triton kernel? While Table 1 provides a runtime breakdown for the NS routine, it would be helpful to understand how much each component contributes to the overall training speedup reported in Table 2.
  2. Could the authors evaluate the convergence of NS iterations using actual gradient(momentum) matrices collected during training, rather than only synthetic matrices sampled from Gaussian or Levy distributions? This would help clarify whether the discrepancy between polar error and training performance is related to differences in the underlying matrix distributions.
  3. Could the authors provide an analysis of how singular values evolve across NS iterations for Muon+ and Turbo-Muon, particularly focusing on the smallest singular values? It would be interesting to understand whether AOL preconditioning specifically improves the amplification of small singular values or primarily reduces the average polar error.

Minor Requests

  • Please format the abstract as a single paragraph.
  • When citing references at the end of sentences, please use \citep{} instead of \citet{} for better readability.
  • Matrix notation is inconsistent throughout the paper. For example, matrices such as XX and QQ are written in boldface in Section 3, but boldface notation is not consistently used in later sections including ZZ in ϵpolar(Z,X)\epsilon_{\text{polar}}(Z, \bf{X}). Please ensure consistent matrix notation throughout the paper.
  • In Eq. (3), please replace the period (.) with a multiplication dot (\cdot).
1 comment
AuthorsOpenReview ↗

Answer to reviewer VfCL

We thank the reviewer for their detailed review and their suggestion to study the singular values in more detail than the polar error metric.

Q1. End to end training time

These results are the same as in the common response for readiness.

Using Muon with the triton kernel optimizations for lines 1, 2 and 3 (which we denote as Muon++) yields NanoGPT end-to-end training speeds of:

MethodTime Difference from Switching Version
Muon --> Muon+- 0.60s
Muon+ --> Muon++- 0.41s
Muon++ --> Turbo-Muon-0.42s

Using a machine with 4xH100, note that using AOL preconditioning with one less Newton-Schulz iteration gives a same order of magnitude speedup as our NS line 3 implementation.

Q2. Regarding the gradient matrices’ distributions

As we wanted a general measure of orthogonalization quality, we prioritized synthetic heavy tailed samples to avoid focusing on a specific training modality which could bias results (as argued in our answer to reviewer cfup).

We evaluate the methods on checkpoints of pre-NS updates on a GPT-2 training run on Fineweb, we compare the polar error of the orthogonalized update using either Muon (Frobenius), Turbo-Muon or MuonEq initializations (in mode R as recommended for transformer training). We report the best mean polar error across 48 parameter update matrices taken at step 1000, 2000 and 3000 using PolarExpress coefficients with hyperparameter ℓ=10−3\ell = 10^{-3} computed for 5 NS iterations. We get the following results (these results are the same as in the common response for readiness):

MethodMean Polar Error
Muon0.583744
Turbo-Muon0.567255
Front-NorMuon0.588192
MuonEq-R0.635988

Turbo-Muon significantly outperforms other methods on real GPT model gradients.

Q2 / Q3. Measuring singular value spread on real gradients.

We agree that while the polar error measure is adapted to mean polar factor approximation quality estimation, it could be valuable to characterize the effect of AOL preconditioning on smaller singular values. Therefore, on CIFAR-10, we save weight updates at training steps 1, 40, 80, 120 and 160, then compute the quantiles of singular values of the 2D reshaped convolutional weight update (the quantiles are denoted as QXX) when running either standard Muon NS or AOL preconditioned NS (Turbo-Muon). These 2D matrices of the neural network updates are of shape (64x216, 64x576, 256x576, 256x2304). The results are as follow:

MethodNS IterationQ01Q25Q50Q99
Turbo-Muon10.3080850.5469530.7959941.904775
Turbo-Muon20.3112270.8739991.3012931.711664
Turbo-Muon30.7206420.8022330.9945981.278196
Turbo-Muon40.9841230.9887210.9977821.015556
Muon10.1178700.2039310.3179931.627282
Muon20.3259660.6500530.9267111.705509
Muon30.6655250.8642971.0662961.279003
Muon40.9383910.9881840.9987051.016522

This shows the effect of AOL preconditioning on the smallest singular values. AOL allows for lower quantile spread around 1. In this experiment, AOL leaves the smallest singular value 3.06× and Q1 2.96× larger than Frobenius normalization on average (paired ratios), indicating better small singular value orthogonalization.

Reviewer_m8vHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Turbo-Muon, which replaces Frobenius normalization with Almost Orthogonal (AOL) pre-conditioning before Newton–Schulz orthogonalization. The proposed method provides a better-conditioned initialization, allowing the number of Newton–Schulz iterations to be reduced while maintaining comparable optimization performance with little additional computational overhead. Experiments on language and vision tasks demonstrate modest end-to-end training speedups with largely preserved model performance.

Critical strengths

• Clear and well-motivated design: The paper targets the initialization of Newton–Schulz iterations and reuses the Gram matrix already computed in the procedure, resulting in a simple modification with little additional overhead. • Practically meaningful efficiency gains: Although the end-to-end speedup is modest, achieving several percent improvement on top of an already highly optimized training pipeline is practically valuable. • Useful and reasonably thorough ablations: The paper includes comparisons under the same number of NS iterations, component-wise ablations, and comparisons with alternative normalization schemes and polynomial coefficients, which provide clear evidence for the contribution.

Critical weaknesses

• Limited algorithmic novelty: The technical contribution is relatively incremental, primarily consisting of incorporating AOL pre-conditioning into Muon’s Newton–Schulz initialization and leveraging the improved initialization to remove one iteration. While the design is simple and effective, the resulting gains in practical training settings remain modest. • Modest end-to-end improvement: The optimizer-level speedup is considerably larger than the final end-to-end training gain. In full training pipelines, the reported improvement is only a few percent, making the practical benefit sensitive to how much of the overall runtime is actually dominated by Muon updates. • Limited large-scale validation: Although the paper motivates Turbo-Muon partly by the computational cost of Muon in large-scale training, the end-to-end experiments remain relatively limited in scale. Validation on billion-scale language-model pretraining would provide stronger evidence for the claimed scalability and practical impact.

Questions for authors

• Performance on non-square matrices: Apart from the end-to-end training experiments, the numerical evaluations are primarily conducted on synthetic square matrices. How does Turbo-Muon perform on non-square matrices, especially those with large aspect ratios that are common in practical neural network layers? It would be helpful to evaluate whether the improvement in Newton–Schulz convergence and approximation quality remains consistent in these settings. • Lack of training curves: The end-to-end experiments report only the final performance in tabular form, without showing the optimization trajectory. Since Turbo-Muon directly modifies the optimizer update, could the authors provide training and validation curves versus both training steps and wall-clock time to demonstrate that the method preserves convergence behavior and stability throughout training?

1 comment
AuthorsOpenReview ↗

Answer to reviewer m8vH

We thank the reviewer for their insightful review.

Q1. Experiments on square matrices:

Regarding performance on non-square matrices, sampling 25 matrices of shape (256, 512) of Levy distributions with parameters alpha = 1.0 and beta = 0.0, we obtain the following polar error results:

ShapeMethodPolar ErrorRuntime (microseconds)
256x512Muon (5 iterations)0.19525 ± 0.1292659.82
256x512Muon+ (5 iterations)0.19498 ± 0.1295552.03
256x512Turbo-Muon (5 iterations)0.08388 ± 0.0067351.81
256x512Turbo-Muon (4 iterations)0.08450 ± 0.0066942.34

Computed on NVIDIA RTX 4090 using the average polar error across 25 random samplings. All methods use PolarExpress for Newton-Schulz coefficients and report the best value obtained by sweeping the ℓ\ell hyperparameter of the method among {0.1,0.05,0.01,0.005,0.001}\{0.1, 0.05, 0.01, 0.005, 0.001\} values. Additionally, we refer the reviewer to our response to Reviewer VfCL where we study singular value quantiles throughout Newton-Schulz iterations on a CIFAR-10 model (with mostly rectangular update matrices).

Training curves.

We will report the training curves of the nano-gpt experiment in the revised version (figure 10).

Large scale validation

While we acknowledge the value of billion-scale pretraining experiments, the ICLR 2027 Reviewer Guidelines explicitly state that additional experiments "should be limited in scope." Furthermore, we believe that the improvements demonstrated on NanoGPT (a “modest” improvement in a worldwide project aiming to find the most highly optimized baseline), together with the speedups reported in Table 4 for training steps of a 1.3B-parameter model, provide sufficient evidence of the effectiveness of our approach without requiring additional end-to-end billion-scale pretraining experiments.

Limited algorithmic novelty

Regarding limited algorithmic novelty, the paper argues that allowing for faster Newton-Schulz convergence even by introducing some bias in the estimator is beneficial. To our knowledge we are the first method introducing this bias-approximation dynamic for efficient Newton-Schulz computations, making our method truly novel and opening new opportunities for the design of biased yet highly efficient preconditioning algorithms.

Reviewer_cfupOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

Muon orthogonalizes each momentum matrix GG with five Newton-Schulz (NS) iterations, after dividing GG by its Frobenius norm so that ∥X0∥2≤1\|X_0\|_2 \le 1. The paper replaces this scalar division by the column rescaling of the Almost Orthogonal Layer (AOL). Since the Gram matrix G⊤GG^\top G is the first product of the NS iteration anyway, the rescaling costs no extra matrix multiplication and only O(min⁡(m,n))O(\min(m,n)) extra memory. The authors show on random matrices that the first NS iterate is closer to the polar factor and that one of the five iterations can then be dropped. With a third Triton kernel for the last line of the iteration, the NS routine runs up to 2.8×2.8\times faster than the public Muon implementation at size 40964096. Used as a drop-in replacement in end-to-end training, Turbo-Muon saves about 2.8 % of wall-clock time in the modded-nanoGPT speed-run with the same loss to two decimal places, 0.8 % in the CIFAR-10 speed-run and 3.6 % for ViT-Tiny on ImageNet-1K, where top-1 is 0.2 percentage points lower.

Critical strengths

∙\bullet The idea is simple and really is free in matrix multiplications and memory. I checked the released code, and it reuses the Gram matrix exactly as described.

∙\bullet The polar error in Figures 1a, 3a and Table 1 is measured against the polar factor of the unconditioned matrix, so it includes the bias the method introduces.

Critical weaknesses

W1. The comparison with other methods is incomplete

In the main text, the paper compares only against the standard Newton-Schulz iteration with the original Frobenius normalization of the matrix. The main comparison is run on Gaussian matrices. For such matrices the diagonal matrix SS is close to the identity times a constant cc from the start. This follows from the distribution of the Gram matrix H=G⊤GH = G^\top G of a Gaussian matrix GG. The diagonal entries Hjj=∑igij2H_{jj} = \sum_i g_{ij}^2 are χn2\chi^2_n distributed with mean nn. Every off-diagonal entry Hjk=∑igijgikH_{jk} = \sum_i g_{ij} g_{ik} is a sum of nn independent products of two standard normals, so it has mean 00 and variance nn and is approximately N(0,n)\mathcal{N}(0,n). Its absolute value has mean close to 2n/π\sqrt{2n/\pi}, so the row sums are Λj≈n+(n−1)2n/π\Lambda_j \approx n + (n-1)\sqrt{2n/\pi} for every jj. As a result, on average the normalization does not change the condition number. It divides by a smaller coefficient of order n3/4n^{3/4}, whereas the usual Frobenius normalization divides by a coefficient of order nn. On such an example εbias\varepsilon_{\text{bias}} is close to zero, and the gain of the proposed method may be explained by this difference in the scaling coefficient. In addition, the Appendix does not show explicitly at which iteration εbias\varepsilon_{\text{bias}} becomes comparable to the approximation error. Finally, there is no comparison with methods that normalize the matrix before orthogonalization, such as the Front variant of NorMuon [1] and MuonEq [2], although they are very close in methodology.

W2. There is no end-to-end decomposition showing what produces the speedup

In the full training run of Table 3, with nearly identical losses, the training time differs by about 2.6 %. From the description of the protocol I cannot tell whether this speedup comes only from the proposed initialization or also from the Triton kernels and the other changes described in the paper. For a valid comparison I would like to see:

(1) runs that are identical except for the initialization method,

(2) runs that are identical except for the Triton kernel.

W3. It is unclear for me what the theoretical contribution is

Lemma 1 is a direct consequence of the definitions: for X=GSX = GS one has ⟨G,PolarFactor(X)⟩=tr(S−1(X⊤X)1/2)\langle G, \mathrm{PolarFactor}(X) \rangle = \mathrm{tr}\big(S^{-1}(X^\top X)^{1/2}\big), which is positive because S−1S^{-1} is positive definite and (X⊤X)1/2(X^\top X)^{1/2} is positive semidefinite and nonzero.

Proposition 1 does not explain the method that is used in practice. Its solution is −η PolarFactor(GS) S-\eta\,\mathrm{PolarFactor}(GS)\,S (equation 14), while Turbo-Muon applies PolarFactor(GS)\mathrm{PolarFactor}(GS) without the factor SS. The paper itself notes in the Discussion of Appendix A.1 that applying SS after the polar factorization "would yield an optimizer with dynamics different from Muon's". The relation between the theory and the practical update is not analyzed beyond this remark.

[1] Z. Li et al. NorMuon: Making Muon more efficient and scalable. arXiv:2510.05491, 2025.

[2] D. Chang et al. MuonEq: Balancing Before Orthogonalization with Lightweight Equilibration. arXiv:2603.28254, 2026.

Questions for authors

Q1 Could the authors provide the comparison with the NorMuon (Front) and MuonEq initializations?

Q2 Could the authors report the per-step time split (optimizer step versus forward and backward) for Muon+ and Turbo-Muon in the speed-run, and the time to reach 3.283.28 rather than the time for a fixed number of steps?

If the weaknesses above are addressed and these questions answered, I will be glad to raise my scores.

3 comments
AuthorsOpenReview ↗

Answer to reviewer cfup (pt 1/2)

We thank the reviewer for their careful review. We will answer their main concerns below and hope it might influence their appreciation of the paper:

W1. A. Incomplete comparisons (Gaussian matrices).

Regarding the reviewer’s comment on Gaussian distributed matrices for evaluating our method. Experiments across 100 independently sampled 64×6464\times64 Gaussian matrices, show an improved mean condition-number ratio κAOL/κoriginal=0.969\kappa_{\mathrm{AOL}}/\kappa_{\mathrm{original}}=0.969 (lower is better) with improved conditioning in all 100 samples. While, the reviewer is correct that their asymptotic analysis explains AOL's more favorable scalar normalization, the conclusion that “on average the normalization does not change the condition number” is incorrect as demonstrated experimentally. Theoretically, while the diagonal scaling coefficients concentrate around a common value as the reviewer demonstrated, the condition number is nonlinear, and the scaling matrix SS depends directly on XX. Thus E[S]≈cI\mathbb{E}[S]\approx cI does not imply E[κ(XS)]≈E[κ(X)]\mathbb{E}[\kappa(XS)]\approx\mathbb{E}[\kappa(X)].

Furthermore, as is the case in the paper’s Appendices (Figure 4; 5; 8; 9), we use heavy tailed distributions, as they depict the distribution of deep neural network gradients with fidelity according to Simsekli et al. (2019). Their results also show Turbo-Muon performs orthogonalization more efficiently. This is an additional argument in favor of a general speedup across tasks, especially as the gradients of different training modalities (diffusion models, RL gradients, LLM pretraining, vision models) can exhibit very different properties (different tail index) as shown in the GPT-2 experiments below.

W1. B. Incomplete comparisons (+ Q1 question: NorMuon / MuonEq comparisons).

The purpose of our method is to improve the approximate orthogonalization step of the Muon optimizer. Therefore, we only compare our performance to Muon as our objective is to accelerate Muon without sacrificing the quality of its update. NorMuon has a different and orthogonal vocation to our method, as it uses running statistics of the orthogonalized matrix to improve training performance, and does not aim to improve the approximation of the polar factor (except maybe for the Front variant noted by the reviewer which we will benchmark).

However, we thank the reviewer for their remark regarding the comparison to the recently published MuonEq method, which aims to improve the approximation of the polar factor by also introducing bias. We evaluate the methods on checkpoints of pre-NS updates on a GPT-2 training run on Fineweb, we compare the polar error of the orthogonalized update using either Muon (Frobenius), Turbo-Muon or MuonEq initializations (in mode R as recommended for transformer training). We report the best mean polar error across 48 parameter update matrices taken at step 1000, 2000 and 3000 using PolarExpress coefficients with hyperparameter ℓ=10−3\ell = 10^{-3} computed for 5 NS iterations. We get the following results (these results are the same as in the common response for readiness):

MethodMean Polar Error
Muon0.583744
Turbo-Muon0.567255
Front-NorMuon0.588192
MuonEq-R0.635988

Turbo-Muon significantly outperforms other methods on real GPT model gradients.

W.1. C. Incomplete comparisons (budget where εapprox≤εbias\varepsilon_\mathrm{approx} \leq \varepsilon_\mathrm{bias})

Regarding the iteration budget where the convergence error of standard Muon becomes comparable to Turbo-Muon’s error bias, we refer the reviewer to the experiments of Figure 6, which shows an inversion of performance from t≥7t\geq7 onwards, which is a superior budget to what most methods use in practice. Also, we additionally sanity check the influence of bias in Table 6, where we show that it has no visible effect on CIFAR-10 training performance in very high Newton-Schulz iteration settings.

W2. End to End Speedup Comparison (+ Q2 question)

These results are the same as in the common response for readiness.

Using Muon with the triton kernel optimizations for lines 1, 2 and 3 (which we denote as Muon++) yields NanoGPT end-to-end training speeds of:

MethodTime Difference from Switching Version
Muon --> Muon+- 0.60s
Muon+ --> Muon++- 0.41s
Muon++ --> Turbo-Muon-0.42s

Using a machine with 4xH100, note that using AOL preconditioning with one less Newton-Schulz iteration gives a same order of magnitude speedup as our NS line 3 implementation.

AuthorsOpenReview ↗

Answer to reviewer cfup (pt 2/2)

Q2. Relative FLOP Saving wrt Forward and Backward Passes.

In order to best quantify the effect of Turbo-Muon on training efficiency, we provide a breakdown of the FLOP demanding operations of the nanogpt training experiment shown in Table 4. Each component represents approximately: Forward pass: 83Tflops Backward pass: 166Tflops Muon NS (5 iterations): 62Tflops (25% of total training flops) Turbo-Muon NS (4 iterations): 34.7Tflops (13.9% of total training flops) These values are measured for a heavily optimized pipeline that minimizes the overall cost of Muon.

W3. Unclear theoretical contribution

The purpose of Appendix A.1 is to provide the following insight on AOL bias: AOL biased orthogonalized updates still correspond to following a valid descent direction, since ⟨PolarFactor⁡(AOL⁡(G)),G⟩>0\langle \operatorname{PolarFactor}(\operatorname{AOL}(G)), G \rangle > 0 for all non-zero gradients GG.

This insight is mostly curated to high NS iteration regimes, which are not prevalent in practice, where we show that AOL bias does not pose any convergence issues (as tested in Table 6). While the reviewer notes that “Proposition 1 does not explain the method that is used in practice“: nearly all Muon-like norm constrained inspired methods do not account for approximation error of the practical LMO update. In an empirical sense, Proposition 1 does not need to explain the method used in practice, as we show that Turbo-Muon approximates the Muon ideal update (the polar factor) better than what Muon does in practice.

I thank the authors for their response. However, I feel that most of my concerns have not been addressed. In particular, the time to reach 3.28 is still not reported, the per-step split is given in FLOPs rather than in wall-clock time, and the point about the missing factor SS in Proposition 1 remains open. I believe the paper needs substantial further work, so I keep my original score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

First, of all, we would like to thank all reviewers for their time. Finding our method a “meaningful contribution” (VfCL) with a “clear and well-motivated design” (m8vH). However, we think that some misconceptions prevent the correct appreciation of our method. Here we provide some shared answers related to some common themes among reviewer questions and concerns.

Reviewer cfup and VfCL (Evaluation on real-gradients): Some reviewers have questions regarding our use of synthetic matrices (both Gaussian and heavy-tailed random matrices) to benchmark methods. Given that the principal domain that could benefit from our method’s efficiency is LLM pretraining, we save update matrices for a GPT-2 style model pretrained on Fineweb, then we use the update matrices at steps 1000, 2000 and 3000 for the 44 parameter matrices and report the mean polar error. We benchmark Muon and Turbo-Muon, along with MuonEq and NorMuon (front variant), as requested by Reviewer cfup. On these matrices, the results are as follows for 5 Newton-Schulz iterations with PolarExpress coefficients (with hyperparameter ℓ=10−3\ell=10^{-3}):

MethodMean Polar Error
Muon0.583744
Turbo-Muon0.567255
Front-NorMuon0.588192
MuonEq-R0.635988

On these heavy-tailed matrices, Turbo-Muon significantly outperforms concurrent methods by benefiting from better approximation speed despite its inherent bias. We offer the following insights in the individual reviewer responses: AOL on Gaussian distributed matrices improves the condition number (contrary to the remark of Reviewer cfup), AOL reduces singular value spread across iterations on CIFAR-10 gradients (c.f. Reviewer VfCL) which are mostly rectangular matrices (c.f. Reviewer m8vH).

Reviewer cfup and VfCL (End-to-End Training Comparison): Using Muon with the triton kernel optimizations for lines 1, 2 and 3 (which we denote as Muon++) yields NanoGPT end-to-end training speeds of:

MethodTime Difference from Switching Version
Muon --> Muon+- 0.60s
Muon+ --> Muon++- 0.41s
Muon++ --> Turbo-Muon-0.42s

Using a machine with 4xH100, note that using AOL preconditioning with one less Newton-Schulz iteration gives a same order of magnitude speedup as our NS line 3 implementation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.