acceptodds
Under review as a conference paper at ICLR 2027

Scaling Muon for Diffusion Transformers

Abstract

The matrix-aware optimizer Muon improves large-model training by balancing updates across singular directions, yet its scaling behavior and end-to-end efficiency on large Diffusion Transformers (DiTs) remain unclear. We first establish Muon's scaling behavior on DiTs from 1.3B to 15B parameters, showing that its optimization and generative quality advantages over AdamW persist across model scales. However, at scale, the 5-step Newton–Schulz iteration (NS5) performed at every optimization step, together with full-momentum materialization, introduces substantial computation and communication overhead that can offset Muon's step-efficiency advantage. We introduce Periodic Row-wise Muon, which performs a full NS5 spectral update once every \(K\) steps and applies a low compute and communication cost row-wise constrained update based on the current momentum at the remaining steps. We further co-design a distributed implementation that operates directly on sharded momentum during non-refresh steps and accelerates spectral refreshes through bucketed all-gather and communication–computation overlap. Across all scales, Muon improves the best observed generative quality over AdamW by 12.9–19.1%. Compared with vanilla Muon, Periodic Row-wise Muon remains within 0.5% in best generative quality on the 1.3B–4B models and improves it by 4.5% at 9B. It reduces optimizer time by 46.9–54.3%, end-to-end step time by 15.7–24.3%, and logical communication volume by 66.7%, while reaching its respective best generative quality with 33.7–64.8% less active training time. These results show that Periodic Row-wise Muon preserves Muon's generative quality advantage while translating it into end-to-end training efficiency for large DiTs.

Primary area:
optimization
Keywords:
Diffusion Transformers, Scalability, Muon, Optimizer

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XuGhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies Muon optimization for text-to-image Diffusion Transformers ranging from 1.3B to 15B parameters. It finds that Muon improves validation loss and generation quality over AdamW, but incurs substantial overhead. Authors proposed Periodic Row-wise Muon performs five Newton–Schulz iterations every third step and uses scaled row-normalized momentum updates otherwise. Its distributed implementation computes these inexpensive updates on momentum shards and overlaps communication with computation during spectral refreshes. Experiments train four model sizes on GPIC for 60,000 steps. Compared with vanilla Muon, the method reduces optimizer time by 46.9–54.3% and total step time by 15.7–24.3%. These results show that Periodic Row-wise Muon preserves Muon’s gener- ative quality advantage while translating it into end-to-end training efficiency for large DiTs

Critical strengths

Substantial experiments at relevant scale. Evaluating four models up to 15B parameters provides meaningful evidence beyond small-model optimizer benchmarks. Matching training examples and reporting multiple generation metrics strengthen the study.

Effective algorithm and systems integration. RowNorm’s with optimizations reduces optimizer time by 46.9–54.3%

Critical weaknesses

The comparisons do not establish the benefit over closely related efficient Muon approaches. The experiments compare primarily against AdamW and an implementation that gathers full momentum and replicates NS5 computation on every rank. Yet periodic spectral updates combined with cheap intermediate updates already appear in LionMuon https://arxiv.org/pdf/2605.19811, and DMuon https://arxiv.org/pdf/2606.27153 specifically addresses distributed Muon overhead. Both are acknowledged in the paper but lack experimental comparisons. Consequently, the results establish an improvement over the selected vanilla implementation, while leaving the incremental benefit of RowNorm and its execution strategy unresolved.

Quality preservation and the largest time savings lack sufficient robustness evidence. No repeated training runs or uncertainty estimates are reported, despite substantial checkpoint variation. At 9B, the method’s best FD-DINO improves from 33.97 to 32.44, but its final FD-DINO worsens from 33.97 to 36.70 and FID from 5.61 to 6.13. At 15B, final GenEval2 AM falls from 57.93 to 52.26, and Table 10 shows a late FD-DINO spike to 52.25.

The large-scale study and systems improvements are valuable. However, the current evidence does not sufficiently establish the proposed method’s advantage over relevant efficient Muon alternatives or the reliability of its quality-preservation claims. Focused additional comparisons and robustness evidence could change my recommendation.

Questions for authors

On Figure 1 (c) Validation loss versus wall-clock time. Validation loss for Muon looks worse than AdamW with longer training, e.g 15B at step 200.0K, please explain.

What explains the late quality fluctuations, particularly the 15B FD-DINO increase to 52.25 at 50k steps (shown in Table 10: All checkpoint evaluation metrics for the 15B model.)? Do the selected best-FD-DINO checkpoints also preserve text alignment and compositionality?

3 comments
AuthorsOpenReview ↗

Response to Reviewer XuGh

We apologize for the late response, as the additional large scale experiments took considerable time to complete. Thank you in advance for considering the new results and sharing your feedback. We will incorporate the relevant results and clarifications into the camera-ready version.

Thank you for asking how much of the improvement can be attributed to the choice of intermediate update and how reliably the method preserves generation quality. We compared Periodic Row-wise Muon (PRowMuon) with LionMuon at the same refresh frequency and with DMuon, which assigns matrices to individual workers. We also examined independent training runs, traced the 15B anomaly to the affected training interval, and measured fidelity and prompt adherence on the same checkpoints.

1. PRowMuon and LionMuon at the same spectral refresh frequency

We trained both methods for 60k updates at 1.3B and 4B, with a spectral refresh every third update. We evaluated both methods every 5k updates, matching the phase of their refresh cycles at each checkpoint. On the two intervening steps, PRowMuon normalizes each row of the current momentum and scales the update by γ = 0.15. LionMuon uses an elementwise sign update. For LionMuon, the spectral branch used peak learning rates of 0.005 at 1.3B and 0.003 at 4B. The sign branch used 2 × 10⁻⁵ at both sizes. The branches shared a momentum state, with coefficients of 0.9 for the update direction and 0.99 for updating the state. We used a 2k step warmup, cosine decay, and decoupled weight decay of 0.01. LionMuon updated the two-dimensional weights within each block. Other parameters used AdamW with a learning rate of 10⁻⁴ and β = (0.9, 0.999).

Scale / methodBest stepBest FD-DINO ↓FID at best FD-DINO ↓60k FD-DINO ↓60k FID ↓
1.3B PRowMuon35k41.915.5345.656.287
1.3B LionMuon25k44.825.57850.886.753
4B PRowMuon20k39.425.6142.256.350
4B LionMuon55k41.986.79048.267.509

At 1.3B and 4B, PRowMuon’s best FD-DINO was lower than LionMuon’s by 2.91 and 2.56 points, respectively. At the common 60k endpoint, its FD-DINO was lower by 5.23 and 6.01 points, respectively.

2. DMuon step time and checkpoint quality

We benchmarked owner-based DMuon, which computes each matrix’s full spectral update once and overlaps communication with training. At 1.3B, the observed step time ratios relative to PRowMuon (1.00×) were 0.80× for DMuon and 1.01× for NorMuon.

The table compares image fidelity after 30k updates for all four methods.

1.3B DiT at 30kFID ↓FD-DINO ↓
PRowMuon6.11444.77
DMuon6.2045.79
NorMuon6.36646.76
Vanilla Muon6.83652.86

At 30k updates, PRowMuon achieved the lowest FID (6.114) and FD-DINO (44.77) of the four methods. DMuon reached 6.20 and 45.79, respectively, and NorMuon reached 6.366 and 46.76.

3. How much do the results vary across training seeds?

We added two independent seed pairs to the original 1.3B runs. Within each pair, Muon and PRowMuon used the same training seed. PRowMuon had a lower FD-DINO at 60k for all three seeds. The mean paired difference (PRowMuon minus Muon) was −0.88, with a sample standard deviation of 0.52. The direction of the endpoint difference was thus consistent across the three pairs tested at this scale.

Training seedBest FD-DINO MuonBest FD-DINO PRowMuon60k FD-DINO Muon / PRowMuon
4241.7341.9146.08 / 45.65
2026110743.8242.5945.76 / 44.31
2026110844.5143.7644.51 / 43.76
1.3B at 60kFID ↓FD-DINO ↓GenEval2 AM ↑
Muon6.350 ± 0.19445.45 ± 0.8347.24 ± 2.57
PRowMuon6.325 ± 0.05844.57 ± 0.9748.19 ± 2.02
Paired difference (PRowMuon − Muon)−0.025 ± 0.247−0.88 ± 0.52+0.95 ± 1.73

Values are means ± sample standard deviations across three training seeds. For the paired differences, we first subtracted the two observations within each seed.

4. Does the 15B spike at 50k recur?

Thank you for drawing attention to the 50k checkpoint. The training logs record job preemptions at 47k and 48k, in the interval where the training loss fluctuates in Figure 6. The original resume procedure restarted the data stream from the beginning of the epoch and reset the random seed. The following 50k checkpoint had an FD-DINO of 52.25. We apologize for overlooking these events in the submission.

We repeated this segment from the 45k checkpoint preceding the preemptions, using the original learning rate and training recipe. At 50k, the new PRowMuon checkpoint had an FD-DINO of 34.28, close to Muon’s 34.03. The spike did not recur at 55k or 60k either.

Update after resumingPRowMuon FID / FD-DINO ↓Muon FID / FD-DINO ↓
50k5.565 / 34.285.571 / 34.03
55k5.825 / 35.365.508 / 33.80
60k5.758 / 35.095.575 / 33.52

5. Do the best-FD-DINO checkpoints also preserve text alignment and compositionality?

For each optimizer, we selected the checkpoint with the lowest measured FD-DINO under the paper’s selection rule, then evaluated HPSv2 and GenEval2 on that same checkpoint. The table gives the results from the original 15B runs and includes matched comparisons at 25k and 60k.

15B checkpointFD-DINO ↓HPSv2 ↑GenEval2 AM ↑GenEval2 GM ↑
PRowMuon 25k (best)32.3522.9049.6811.74
Muon 25k33.4822.9254.3014.21
Muon 45k (best)33.2522.5357.5013.76
PRowMuon 60k33.9921.3552.2611.97
Muon 60k33.5122.3757.9314.55

At 25k, PRowMuon had better FD-DINO (32.35 versus 33.48) and nearly identical HPSv2 (22.90 versus 22.92). Its GenEval2 arithmetic and geometric means were 4.62 and 2.47 points lower. Comparing each method’s best FD-DINO checkpoint, PRowMuon scored 0.37 points higher on HPSv2, while Muon scored higher on both GenEval2 aggregates. Thus, the 15B checkpoint that gave PRowMuon its best fidelity retained a comparable preference score, with a measurable gap in compositional accuracy.

FD-DINO measures the overall distribution of generated images in feature space. An image can have realistic features while depicting the wrong number of objects or placing them in the wrong positions. GenEval2 tests these prompt specific requirements, and HPSv2 estimates human preference. Denoising pretraining minimizes prediction error conditioned on text and noise level, so an improvement in distributional fidelity need not improve every prompt level score. Section 5.2 of GenEval also reports that further pretraining does not consistently improve compositional performance. Methods such as Diffusion-DPO introduce a separate post-training objective for preference alignment. Here we evaluate the original pretrained checkpoints.

6. Why does the 9B model finish with a higher FD-DINO than Muon?

Thank you for identifying the 9B endpoint difference in Figure 2. At 60k, FD-DINO was 36.70 for PRowMuon and 33.97 for Muon. Decomposing this 2.73-point gap shows that PRowMuon more closely matched the real images’ mean DINO features (a 0.29-point lower mean term). Its covariance term was 3.01 points higher, which accounts for Muon’s better endpoint score.

9B checkpointDenoising val. loss ↓Mean term ↓Covariance term ↓FD-DINO ↓
PRowMuon 25k0.3238352.71529.72332.44
Muon 25k0.3213854.67734.29238.97
PRowMuon 60k0.3171995.05631.64036.70
Muon 60k0.3139505.34228.62633.97

The ordering changed late in training. At 25k, PRowMuon achieved its best FD-DINO of 32.44, compared with 38.97 for Muon. From 25k to 60k, PRowMuon’s mean and covariance terms rose by 2.34 and 1.92, respectively, accounting for the 4.26-point rise in its FD-DINO. Over the same interval, Muon’s covariance term fell from 34.29 to 28.63. PRowMuon’s validation loss also fell, from 0.323835 to 0.317199. Denoising loss measures velocity prediction error at a sampled noise level, whereas FD-DINO measures the feature distribution after multistep sampling. The 9B checkpoints show how denoising can continue to improve after the best measured image fidelity has been reached.

Over the same 25k–60k interval, PRowMuon’s GenEval2 AM improved from 51.25 to 57.33, and its GM improved from 11.75 to 15.97. These checkpoints therefore improved on the compositional evaluation despite the decline in distributional fidelity. Vanilla Muon also showed a late decline in fidelity at 1.3B, where its FD-DINO rose from 41.73 at 40k to 46.08 at 60k. A related late decline in fidelity is reported in EDM2, Appendix A.2, Figure 12.

7. Reading the 15B curves near 200k in Figure 1(c)

We appreciate your careful reading of this figure. The 200k tick marks elapsed training time in units of the average 1.3B AdamW step. Both 15B runs ended at 60k updates. The table gives the validation records immediately before and after the 200k time tick for each method.

15B methodBefore 200k (step / time / val. loss)After 200k (step / time / val. loss)
AdamW52k / 197,501 / 0.31784453k / 201,299 / 0.317779
Muon30k / 195,821 / 0.31771331k / 202,348 / 0.317344

Near that tick, both recorded Muon losses were lower than either neighboring AdamW loss. AdamW did lead earlier on the time axis, as you observed. For example, it reached a validation loss of 0.340 in 30,385 time units, compared with 39,164 for Muon. At the common endpoint of 60k updates, validation losses were 0.31205 for Muon and 0.31759 for AdamW.

Reviewer_PFq7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores the feasibility of using Muon to train Diffusion Transformer (DiT) models, including 1.3B, 4B, 9B, and 15B. To solve the system scaling issue caused by Muon's computation and communication operations, the authors propose Periodic Row-wise Muon (PRowMuon), which periodically replaces the expensive full all-gather on momentum shards and Newton-Schulz iterations (NS5) with cheap row norm and all-reduce.

Critical strengths
  1. The empirical evidence showing Muon's higher convergence rate over AdamW in training DiTs is very useful.

  2. The proposed algorithm is well motivated and has been shown to be effective with empirical evidence on relative speedup.

Critical weaknesses
  1. The technical solution mismatches the target problem. The proposed scaling method does not leverage any unique feature of the diffusion transformer. The scaling technique can be generally applied to LLM training, too. Please justify the choice of the scope of this paper.

  2. The most important issue with this manuscript is that it lacks comparison to the closely related methods, including MuonBP [MuonBP], CacheMuon [cachemuon], and the concurrent Dion3 [Dion3], although the first two are cited in the related work. There is also no detailed profiling of vanilla Muon's cost: Table 2 reports only total optimizer time and logical communication volume, with no breakdown into NS5 compute, exposed all-gather communication, and full-matrix materialization, so the reader cannot tell which component PRowMuon's savings come from. Appendix B.1 shows that the standard NS5 recurrence is used; if NS5 computation is the main bottleneck, the authors may want to include a comparison to Gram-NS [Gram-NS] as well.

  3. Figure 2 shows performance degradation on the final trained model using the same train iterations as vanilla Muon. However, its discussion is missing.

  4. The intra- and inter-node network configuration is unclear.

  5. Using bucketed all-gather to achieve computation-communication overlapping is a reasonable design. However, the underlying execution path (Section 2.3) all-gathers the full momentum to every rank and runs NS5 on every matrix on every rank, keeping only the local slice. Considering that the model uses FSDP2 and 256 GPUs, each layer's NS result is computed 256 times per step on every rank. We would expect a comparison between PRowMuon and an owner-based Muon design (e.g., NorMuon [NorMuon] and DMuon [DMuon]), in which each worker performs NS5 on a subset of layers, and the workload should be balanced to avoid slowdown from one or two ranks' NS5.

  6. This paper misses the time-to-same-quality comparison between PRowMuon and vanilla Muon.

[MuonBP] Ahmed Khaled, Kaan Ozkara, Tao Yu, Mingyi Hong, and Youngsuk Park. MuonBP: Faster Muon via block-periodic orthogonalization. arXiv preprint arXiv:2510.16981, 2025.

[cachemuon] Bishnu Dev, Sushil Bohara, Martin Takáč, and Samuel Horváth. CacheMuon: Using temporal preconditioning to approximate polar factor. arXiv preprint arXiv:2606.16371, 2026.

[Dion3] Noah Amsel, Jack Zhang, Kwangjun Ahn, Ali Naeimi, Austin Feng, Berlin Chen, Tri Dao, and John Langford. Dion3: Full-stack orthogonal updates. arXiv preprint arXiv:2608.11612, 2026.

[Gram-NS] Jack Zhang, Noah Amsel, Berlin Chen, and Tri Dao. Gram Newton-Schulz. DAO AI Lab Blog, 2026. URL \url{https://dao-ailab.github.io/blog/2026/gram-newton-schulz/}.

[DMuon] Vincent Chen, Starrick Liu, Regis Cheng, Dance Yang, Shalfun Li, Ryan Yu, Lucy Liang, Hang Su, Roy Gan, Hao Wang, and Qian Wang. DMuon: Efficient distributed Muon training with near-Adam overhead. arXiv preprint arXiv:2606.27153, 2026.

[NorMuon] Zichong Li, Liming Liu, Chen Liang, Weizhu Chen, and Tuo Zhao. NorMuon: Making Muon more efficient and scalable. arXiv preprint arXiv:2510.05491, 2025.

Questions for authors

Please address my concerns in Critical Weakness.

5 comments
AuthorsOpenReview ↗

Response to Reviewer PFq7

We apologize for the late response, as the additional large scale experiments took considerable time to complete. Thank you in advance for considering the new results and sharing your feedback. We will incorporate the relevant results and clarifications into the camera-ready version.

Thank you for asking us to compare Periodic Row-wise Muon (PRowMuon) with efficient Muon implementations, identify where it saves time within a training step, and measure how quickly it reaches the same image fidelity. We address each question below.

W1. Why study diffusion transformers?

For text-to-image diffusion transformers (DiTs), the practical question is what image fidelity a model can achieve within a given training time budget. The model learns to denoise at multiple noise levels, and evaluating image fidelity requires multistep sampling. Validation loss alone therefore cannot answer this question. The workload also affects how much a spectral update costs relative to the rest of training. Within the same DiT, a time conditioning projection processes one position per image, while an image projection processes many patches. Weight matrices of identical shape have comparable costs for five Newton–Schulz iterations (NS5), even as forward and backward costs vary with the number of positions. We measure both training cost and image fidelity across four DiT sizes.

To test whether the update transfers to language modeling, we also trained a 618M-parameter language model for 30k updates at approximately 1M or 2M tokens per step. In each comparison, Muon and PRowMuon used the same data, training seed (42), and learning rate schedule. PRowMuon retained the DiT settings of K = 3 and γ = 0.15. The table gives final per-token validation cross entropy (CE), perplexity, and the reductions in optimizer time and full step time. At both token budgets, PRowMuon reduced optimizer and end-to-end step time relative to Muon, and the two methods achieved almost the same final validation CE.

618M model, 30k updatesValidation CE ↓ Muon / PRowMuonPerplexity ↓ Muon / PRowMuonTime saved by PRowMuon Optimizer / full step
1M tokens/step2.7665 / 2.784115.90 / 16.1933.6% / 5.7%
2M tokens/step2.8264 / 2.850016.88 / 17.2925.6% / 6.9%

W2. Direct comparisons with MuonBP and Dion3

All four methods used the same 1.3B text-to-image DiT, training data, seed 42, and 60k step budget. We evaluated them on the same 50k prompts. PRowMuon used the paper’s settings, K = 3 and γ = 0.15.

For MuonBP, we retained the published block orthogonalization and periodic full-matrix refresh configuration. The base peak learning rate for matrix parameters was 0.005. At the same budget of five iterations, we compared the published coefficients (2, −1.5, 0.5) with the standard NS5 coefficients (3.4445, −4.7750, 2.0315). The validation losses averaged over 4k and 6k steps were 0.35135 and 0.35320, respectively. We used the published coefficients for the full 60k steps.

For Dion3, we ran the authors’ implementation for 60k steps. At each step, it ranks momentum rows by their L1 norm within each shard, targets a selected fraction f = 0.25, and applies Gram-NS to the selected rows. The coefficient μ = 0.9 controls the residual momentum retained on selected rows after an update, and β₂ = 0.95 smooths the row-wise second moments. The full matrix reference learning rate was η₁ = 0.005, giving a base peak learning rate of η₁/√f = 0.01 after row selection.

1.3B methodStep at best FD-DINOBest FD-DINO ↓FID at that step ↓60k FD-DINO ↓60k FID ↓
Muon40k41.736.01246.086.545
PRowMuon35k41.915.5345.656.287
MuonBP50k48.496.44665.508.353
Dion360k41.825.92841.825.928

PRowMuon reached FD-DINO 41.91 at 35k steps, within 0.09 of Dion3’s best value of 41.82 at 60k. Dion3 had the lowest FID and FD-DINO at the fixed 60k endpoint. Compared with the tested MuonBP configuration, PRowMuon’s final FID was lower by 2.066 and its FD-DINO was lower by 19.85.

Figure 9 (Appendix F of the revised manuscript, p. 30) shows how the comparison with Dion3 changes with the training budget. In the left panel, PRowMuon reaches FD-DINO 44.77 by 30k updates and 41.91 by 35k. Dion3 reaches 50.70 by 30k and continues to improve, attaining 41.82 at 60k. The right panel shows the best checkpoint available against estimated training time. As elsewhere in the paper, one time unit is the mean duration of one 1.3B AdamW training step.

UpdatesVanilla MuonPRowMuonMuonBPDion3
5k152.03172.46——
10k73.3582.29131.1972.65
15k51.5658.96——
20k48.0667.2265.0052.24
25k46.1247.37——
30k52.8644.7754.6250.70
35k46.6541.91——
40k41.7347.2949.6146.62
45k48.2649.77——
50k50.4145.0148.4944.12
55k47.9245.75——
60k46.0845.6565.5041.82

The table gives FD-DINO at each evaluated checkpoint. The curves in Figure 9 give the lowest value observed up to each budget.

How quickly do PRowMuon and Dion3 reach the same fidelity?

We also compared cumulative active training time directly in the paired run logs. On the shared evaluation grid, PRowMuon first reached FD-DINO ≤ 50 at 30k updates and Dion3 at 40k, a 13.0% time saving. At the stricter target of 46.08 (vanilla Muon’s 60k score), PRowMuon reached the target at 30k and Dion3 at 50k, saving 30.8%. Among all available checkpoints for each method, the first scores below 42 were 41.91 at 35k for PRowMuon and 41.82 at 60k for Dion3. PRowMuon used 33.0% less active training time to reach those checkpoints.

NS5 computation, exposed communication, and matrix materialization

We identified the following events in the original traces underlying Figure 4 and Figure 8. Each cell reports Muon / PRowMuon in the paper’s time units, where one unit is the mean step time of 1.3B AdamW. For PRowMuon, each per-step figure averages one refresh step and two intermediate steps.

SizeNS5 computeFull-momentum all-gatherExposed all-gatherExplicit packing / reordering
1.3B0.1530 / 0.04520.5693 / 0.10410.4299 / 0.07780.0058 / 0.0019
4B0.5654 / 0.18330.8786 / 0.28620.3022 / 0.10490.0124 / 0.0041
9B1.3138 / 0.43660.4179 / 0.13770.0192 / 0.01390.0276 / 0.0093
15B2.4001 / 0.79620.6200 / 0.26340.0390 / 0.02040.0462 / 0.0150

NS5 compute measures GPU time spent on orthogonalization kernels, counting overlapping intervals once. For communication, we report total optimizer all-gather time for the full momentum matrix and the portion exposed after overlap with computation. The final column covers explicit concatenation, conversion to contiguous storage, and cloning to materialize matrices. Some intervals overlap, so summing the columns would double count time.

At 1.3B, exposed communication fell by 0.3521 normalized units, from 0.4299 to 0.0778. At 15B, NS5 computation fell by 1.6039 units, from 2.4001 to 0.7962. The time spent explicitly materializing matrices was small at all four scales. Table 2 of the submission reports corresponding reductions in end-to-end step time of 24.3%, 20.2%, 15.7%, and 23.4% at 1.3B, 4B, 9B, and 15B.

W2. Gram-NS for the periodic refresh

We replaced the NS5 solver on full refresh steps, retaining sharded RowNorm on the two intervening steps. Let Gⱼ = XⱼXⱼᵀ and Hⱼ = aⱼI + bⱼGⱼ + cⱼGⱼ². The standard iteration gives Xⱼ₊₁ = HⱼXⱼ. The Gram formulation starts with Q₀ = I, updates Qⱼ₊₁ = HⱼQⱼ and Gⱼ₊₁ = HⱼGⱼHⱼᵀ, and returns Q₅X₀. With the same coefficients, the two formulations therefore implement the same five transformations in exact arithmetic.

1.3B refresh solverMean full step time ratioMean refresh step time ratio
Standard NS51.0001.000
Gram-NS0.9590.998

Both columns are relative to PRowMuon with standard NS5. Over the 300 step timing window, PRowMuon with Gram-NS used 4.1% less time per step on average. Refresh steps themselves were only 0.2% faster. Most of the measured difference came from intermediate steps, which used the same RowNorm implementation in both runs. This experiment establishes that Gram-NS can implement the periodic refresh. The refresh step timings are the relevant comparison for the effect of changing the solver.

W2. CacheMuon hit rate and wall-clock cost

We compared CacheMuon with vanilla Muon over the first 6,000 updates of 1.3B DiT training. At 6,000 updates, the two optimizers achieved almost the same validation loss. To compare wall-clock time, we timed both methods over the 4,000 updates following their shared 2,000 update learning rate warmup. With the same validation and checkpointing schedule, CacheMuon took 24.4% longer over this interval.

To understand why caching did not speed up training, we measured how often it avoided a fresh solve. At each update, CacheMuon tested a cached transformation for each of 468 momentum matrices. With the residual threshold set to τ = 5, only 7.74% of checks passed between resuming from the 1k checkpoint and reaching 6k updates. Every failed check still required five Gram-NS iterations. In a separate three step trace after resumption, 85 of 1,404 checks passed. Forming and testing the candidates together cost GPU time equivalent to 2.59% of full step time. Using the mean cost of a fresh solve in that trace, we estimate that the 85 cache hits saved 1.12% of full step time. Thus, in the profiled steps, testing for reuse cost more GPU time than reuse saved. Section 7 of the CacheMuon paper also discusses the synchronization cost of checking for reuse at every step.

Reviewer_JGriOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a new optimiser, called Periodic Row-wise Muon, framed as an efficiency correction to Muon. Instead of incurring the computational penalty of Newton Schultz every step of optimisation, it instead breaks the this operation into two period. For a certain number of K steps, the full-spectral update is approximated by a cheaper row-wis contained update, after which point the momentum calculation is refreshed appropriately by the original Newton Schultz operation. This periodic property allows the optimisation algorithm to be co-designed with a distributed training implementation that reduces the amount of communication between parallel workers. As such, Periodic Row-wise Muon retrains the optimisation benefits over Muon, whilst decreasing the total wall-clock time of training relative to Muon.

Critical strengths

S1. Good Empirical Validation of The Method. The paper has put considerable effort in producing results from 1.5-15B model scales. The fact that the results generalise across these sizes are encouraging.

S2. Clear, Effective and Seemingly Novel Contribution. The controbtioon of exploiting efficiency in Muon on the temporal direction is novel. It is well explained and justified by the choice of expreriments.

Critical weaknesses
  1. Overly Specific Framing and Inadequate Comparison to Other Efficient Implementations

    1. The first weakness in that the paper's framing is overly specific. That is, this paper examines the applicability for Muon in Diffusion Transformers, but it is not clear why the scaling issues are uniquely a property of this domain.
    2. Building on this, the paper fails to consider appropriate efficient Muon baselines (such as Dion(1\2\3)) which have been developed in the LLM community but address very similar problems (albeit in an orthogonal way).
    3. The reason this is a major weakness is that it feels that the solution is somewhat arbitrarily presented, where any effective solutions already exist in the literature, and it is difficult to extract a usable insight when one would want to compare these methods.
  2. Insufficient Empirical Justification of Theoretical Choices

    1. A second weakness that detracts from the quality of this paper is not validate its theoretical arguments in Section 3. Namely, the authors define a bound in equation 6 for periodic spectral correction, but they never show that this happens in the models they are experimenting on (nor on toy models). As such, this detracts from the quality of the argument being made in the paper.
Questions for authors
  1. Why do you expect that the benefit of scaling with Muon to be different. in Diffusion Transformers when compared to LLM experiments which had popularised the method?
    1. Why is the diminishing efficiency advantage a property of DiTs, and not of the Transformer architecture generally?
  2. Can you explain why this periodic row-wise Muon was selected in favour of other possible alternatives? That is, why does this framing from a temporal perspective solve the issue that other efficient Muon implementations, that approach this from a spatial perspective, do not?
  3. Building on the above, please could you compare your method with Dion (any of the variants, ideally the most recent version) and with MuonBP? Of course, running the full scale of results may be computationally challenging during the rebuttal window, but a comparison at the smallest scale of your experiments would be sufficient.
  4. Can you show that the benefit of Muon decreases with scale as predicted? I am aware of Figure 1c that you have used, but it would perhaps be easier to present this as a difference in wall-clock time as a function of the scale to improve readability.
  5. Can you please provide an experimental result that shows that Equation 6 materialised in practice, and how this corroborates with the selection of K?
  6. How were the hyperparameters found in Table 5? Was this based on related work? Additionally, you made a choice to fix the K and gamma factors across model scales; what reason is there to believe that these are quantities that remain fixed across model size?
  7. The experimental results would also benefit from experiments being run across additional seeds for statistical significance. Of course, doing this in the short rebuttal window is not feasible for the full suite. However this would be improve the paper's quality significantly.
  8. Observing the results, especially at the 15B scale, it seems that Period Row-Wise Muon introduced some training instabilities as observed in the FD-DINO metric. Could the authors please clarify why this is the case and whether their optimiser design directly attributes this? This would also be clarified by verifying these results at different seeds.
4 comments
AuthorsOpenReview ↗

Response to Reviewer JGri

We apologize for the late response, as the additional large scale experiments took considerable time to complete. Thank you in advance for considering the new results and sharing your feedback. We will incorporate the relevant results and clarifications into the camera-ready version.

Thank you for recognizing the value of evaluating four model sizes and for asking us to compare Periodic Row-wise Muon (PRowMuon) with efficient Muon variants and examine its theoretical motivation. We completed 60k step comparisons with MuonBP and Dion3, measured how the polar factor of the momentum changes during training, and used the original learning curves to calculate the time each method takes to reach a common validation loss or image fidelity target.

Q1. Why study text-to-image diffusion transformers?

Our question is whether Muon can bring a larger text-to-image diffusion transformer (DiT) to a given level of image fidelity sooner under a fixed training time budget. A dual-stream DiT processes image latents and text conditioning at varying noise levels. Generating an image requires multiple network evaluations. Token prediction results in a language model therefore leave the image fidelity question open. We measure it directly at four model sizes from 1.3B to 15B using the same image resolution and sampling protocol.

Q1, continued. Which aspects of the scaling behavior are specific to DiTs?

Other transformers can incur the same spectral update overhead. Vanilla Muon applies five Newton–Schulz iterations (NS5) to each matrix in its parameter group at every step. Their cost depends on matrix shape, whereas the cost of the forward and backward passes also depends on how many tokens use that matrix. In our DiT, the diffusion timestep modulation is computed once per image, while an image projection is applied at every patch. In the 1.3B trace, projections of the same shape (1,216 × 1,216) process 1, 154, or 1,024 positions per example. The diffusion conditioning path thus provides a concrete case in which the same spectral operation accounts for very different fractions of a layer’s computation.

At fixed resolution and batch size, increasing model width can raise that fraction. For square matrices, doubling the width increases the arithmetic in a dense matrix product approximately eightfold and that in a per-token linear projection approximately fourfold. Table 2 measures the resulting step costs on our DiTs. Our response to Q4 below gives the corresponding time to a common validation loss.

We also tested whether the update rule transfers to a 618M-parameter language model. We trained for 30k updates at approximately 1M or 2M tokens per step. Within each comparison, the methods shared the data, training seed (42), and learning rate schedule. PRowMuon used the DiT settings K = 3 and γ = 0.15. We report final validation cross entropy (CE) per token, perplexity, and the reductions in optimizer time and full-step time relative to Muon.

618M model, 30k updatesValidation CE ↓ Muon / PRowMuonPerplexity ↓ Muon / PRowMuonTime saved by PRowMuon Optimizer / full step
1M tokens/step2.7665 / 2.784115.90 / 16.1933.6% / 5.7%
2M tokens/step2.8264 / 2.850016.88 / 17.2925.6% / 6.9%

PRowMuon reduced both optimizer time and full step time at both token budgets. At 30k updates, its validation CE was higher by 0.0176 and 0.0236, respectively. This experiment establishes that the update can be used in a language model and quantifies its trade-off there.

Q2. Why reduce the frequency of spectral updates?

Thank you for asking what a temporal update saves. As a brief reminder, PRowMuon uses a full matrix NS5 update every third step. On the two intervening steps, it updates every row using sharded row normalization, avoiding both the full momentum all-gather and the spectral solve. Dion3 applies a spectral update to selected rows at each step, while MuonBP orthogonalizes blocks between full refreshes.

The controlled comparison in Table 3(a) explains why we retained periodic full refreshes. At 1.3B, row normalization alone yielded a best FD-DINO of 51.322. Adding a full refresh every third step brought it to 41.913, close to 41.733 with NS5 at every step. Executing the intermediate updates on shards also reduced logical optimizer communication volume by 66.7% (Table 3(b)).

Q3. How do MuonBP and Dion3 compare under the same training budget?

All four methods used the same 1.3B text-to-image DiT, training data, seed 42, and 60k step budget. We evaluated them on the same 50k prompts. PRowMuon used the paper’s settings, K = 3 and γ = 0.15.

For MuonBP, we retained the published block orthogonalization and periodic full matrix refresh configuration. The base peak learning rate for matrix parameters was 0.005. At the same budget of five iterations, we compared the published coefficients (2, −1.5, 0.5) with the standard NS5 coefficients (3.4445, −4.7750, 2.0315). The validation losses averaged over 4k and 6k steps were 0.35135 and 0.35320, respectively. We used the published coefficients for the full 60k steps.

For Dion3, we ran the authors’ implementation for 60k steps. At each step, it ranks momentum rows by their L1 norm within each shard, targets a selected fraction f = 0.25, and applies Gram-NS to the selected rows. The coefficient μ = 0.9 controls the residual momentum retained on selected rows after an update, and β₂ = 0.95 smooths the row-wise second moments. The full matrix reference learning rate was η₁ = 0.005, giving a base peak learning rate of η₁/√f = 0.01 after row selection.

1.3B methodStep at best FD-DINOBest FD-DINO ↓FID at that step ↓60k FD-DINO ↓60k FID ↓
Muon40k41.736.01246.086.545
PRowMuon35k41.915.5345.656.287
MuonBP50k48.496.44665.508.353
Dion360k41.825.92841.825.928

PRowMuon reached FD-DINO 41.91 at 35k steps, within 0.09 of Dion3’s best value of 41.82 at 60k. Dion3 had the lowest FID and FD-DINO at the fixed 60k endpoint. Compared with the tested MuonBP configuration, PRowMuon’s final FID was lower by 2.066 and its FD-DINO was lower by 19.85.

Image fidelity across training budgets

Figure 9 (Appendix F of the revised manuscript, p. 30) shows how the comparison with Dion3 changes with the training budget. In the left panel, PRowMuon reaches FD-DINO 44.77 by 30k updates and 41.91 by 35k. Dion3 reaches 50.70 by 30k and continues to improve, attaining 41.82 at 60k. The right panel shows the best checkpoint available against estimated training time. As elsewhere in the paper, one time unit is the mean duration of one 1.3B AdamW training step.

UpdatesVanilla MuonPRowMuonMuonBPDion3
5k152.03172.46——
10k73.3582.29131.1972.65
15k51.5658.96——
20k48.0667.2265.0052.24
25k46.1247.37——
30k52.8644.7754.6250.70
35k46.6541.91——
40k41.7347.2949.6146.62
45k48.2649.77——
50k50.4145.0148.4944.12
55k47.9245.75——
60k46.0845.6565.5041.82

The table gives FD-DINO at each evaluated checkpoint. The curves in Figure 9 give the lowest value observed up to each budget.

How quickly do PRowMuon and Dion3 reach the same fidelity?

We also compared cumulative active training time directly in the paired run logs. On the shared evaluation grid, PRowMuon first reached FD-DINO ≤ 50 at 30k updates and Dion3 at 40k, a 13.0% time saving. At the stricter target of 46.08 (vanilla Muon’s 60k score), PRowMuon reached the target at 30k and Dion3 at 50k, saving 30.8%. Among all available checkpoints for each method, the first scores below 42 were 41.91 at 35k for PRowMuon and 41.82 at 60k for Dion3. PRowMuon used 33.0% less active training time to reach those checkpoints.

Q4. How does the wall-clock difference in Figure 1(c) vary with scale?

We appreciate the suggestion to express the result as a time difference. At each model size, a horizontal line at a fixed validation loss intersects the two curves in Figure 1(c). The difference between the times at which they first reach that line measures how much sooner one method reaches the target. The right panel of Figure 1(b) reports generation quality.

Figure 10 (Appendix F of the revised manuscript, p. 30) plots Muon’s time to a given validation loss minus AdamW’s time, across the four model sizes. We used the original validation records at 1k update intervals and two intermediate losses reached at every size (0.340 and 0.335). For each method, we multiplied the first recorded update count at or below the target by the per-step time used in Figure 1(c). One time unit is the mean duration of one 1.3B AdamW training step. A positive difference means that Muon took longer to reach the target despite requiring fewer updates.

SizeSteps to loss ≤ 0.340 Muon / AdamWTime difference at 0.340Steps to loss ≤ 0.335 Muon / AdamWTime difference at 0.335
1.3B10k / 16k+2,73117k / 28k+3,842
4B7k / 10k+2,08410k / 15k+1,950
9B6k / 8k+4,3198k / 11k+4,901
15B6k / 8k+8,7798k / 11k+10,440

For example, at 15B, Muon reached loss ≤ 0.340 in 2,000 fewer updates but took 8,779 additional time units. At both targets, the time penalty is larger at 15B than at 1.3B, while the 4B penalty is smaller than the 1.3B penalty. Thus per-step overhead can offset faster convergence in updates at large scale. The measured effect varies nonmonotonically across the four sizes.

We also used the validation loss attained by AdamW at 20k steps as a separate target for each size. On the Figure 1(c) time scale, Muon/AdamW time-to-target ratios were 1.218, 1.145, 1.129, and 1.117 at 1.3B, 4B, 9B, and 15B. For example, the 4B target was 0.331475. Muon reached it at 14k steps and AdamW at 20k, corresponding to 32,924 and 28,756 time units. AdamW reached this early target sooner at every size.

Q5. How tight is Equation (6) on training momentum matrices?

We restored the complete 1.3B PRowMuon training state at 30k steps and recorded updates 30,001–30,012 under the original training recipe. We selected blocks 0, 9, and 18 in advance and collected two full matrices per block, an attention query projection and an MLP projection. We computed 72 double precision thin SVDs of the momentum defined in the paper. For each pair of updates, we compared the measured Frobenius distance between the exact polar factors (L) with the right-hand side of Eq. (6) (B).

All 264 pairs satisfied L ≤ B. An additional 264 comparisons using the tensors actually passed to NS5 also satisfied the inequality. The smallest normalized bound B/√r was 6.4. For polar factors with orthonormal rows, geometry alone gives L/√r ≤ 2, and the largest observed L/√r was 0.98. The six matrices had condition numbers ranging from approximately 150 to 6.7 × 10⁷. Equation (6) thus held in this window, but small singular values made the bound too loose to select a numerical refresh interval.

Lag h (updates)12346
Exact polar drift, L/√r0.520.620.730.790.91
L/√r starting at a refresh0.490.620.730.790.91
Exact polar cosine similarity0.8650.8060.7310.6850.586
NS5 output cosine similarity0.640.660.570.570.50

Entries are medians over the six matrices. Here r is the smaller matrix dimension. Refreshes occurred at updates 30,001, 30,004, 30,007, and 30,010.

With K = 3, the two row-normalized updates after a refresh correspond to h = 1 and 2. Their measured normalized polar drifts were 0.49 and 0.62, compared with 0.91 six updates after a refresh. This growth provides empirical motivation for a short interval. The final row separately reports similarity for the directions produced by NS5. We used downstream image fidelity to choose K.

Q6. How did we select K and γ at 1.3B?

Figure 5 compares 12 configurations, with K ∈ {2, 3, 4} and γ ∈ {0.10, 0.15, 0.25, 0.35}. We trained each for 30k steps and averaged FD-DINO over the 20k, 25k, and 30k checkpoints. Increasing K from 2 to 3 removed one-third of the full refreshes and increased the best mean FD-DINO for that interval by 2.8%. Increasing K from 3 to 4 removed another quarter of the refreshes and increased that metric by 8.2%. We used K = 3 and γ = 0.15 in the main experiments. In the separate ablation in Table 3(a), holding K = 3 and changing γ from 1 to 0.15 improved best FD-DINO from 44.294 to 41.913.

Why use the same values at larger scales?

K is measured in optimizer updates. Expanding the momentum recurrence over h updates gives the previous momentum a coefficient of βʰ. Holding β = 0.95 fixes that decay factor across widths and provides a reason to start from the same interval. The actual change in direction also depends on the incoming gradients and the singular spectrum of the momentum.

The multiplier γ sets the size of an intermediate update relative to the model’s own update scale. Orient a full-row-rank matrix as r × c. If every row norm exceeds the numerical cutoff, RowNorm has r unit-norm rows, while the exact polar factor has r unit singular values. Both therefore have Frobenius norm √r. Scaling the row-normalized direction by γ gives an exact norm ratio of γ relative to the polar direction. Both branches share the scale specific base learning rate and shape scaling. NS5 is a finite approximation, but this calculation explains why the same γ is a natural starting point at different widths.

Each size may admit better values, and a full search at each size would require multiple large model training runs. With the same settings, the best FD-DINO at 15B was 32.35 for PRowMuon and 33.25 for vanilla Muon, while PRowMuon reduced full-step time by 23.4% in the paper’s timing window.

How were the base configurations in Table 5 selected?

We tuned AdamW and vanilla Muon independently. For AdamW we jointly varied peak learning rate and weight decay. For vanilla Muon we jointly varied the peak learning rate for matrix parameters and the momentum coefficient. In the initial search, we trained each configuration for 10k steps, evaluated on held out data every 1k steps, and ranked configurations by their lowest validation loss. PRowMuon inherited vanilla Muon’s base configuration at each model size.

Q7. How much variation is there across independent training runs?

We added two paired 1.3B runs per method. Within each pair, Muon and PRowMuon used the same seed and completed 60k steps. Across the three training seeds, PRowMuon’s final FD-DINO was lower by 0.43, 1.45, and 0.75. For each pair, we selected the best checkpoint of each method among the steps at which both methods were evaluated.

Training seedBest FD-DINO MuonBest FD-DINO PRowMuon60k FD-DINO Muon / PRowMuon
4241.7341.9146.08 / 45.65
2026110743.8242.5945.76 / 44.31
2026110844.5143.7644.51 / 43.76

Mean and variability at 60k steps

1.3B at 60k stepsFID ↓FD-DINO ↓
Muon6.350 ± 0.19445.45 ± 0.83
PRowMuon6.325 ± 0.05844.57 ± 0.97
Paired difference (PRowMuon − Muon)−0.025 ± 0.247−0.88 ± 0.52

Entries are means ± sample standard deviations across the three independent training seeds. We computed each paired difference within a seed before aggregation.

Q8. Does the 15B spike at 50k steps recur?

Thank you for drawing attention to this checkpoint. The training logs show job preemptions at 47k and 48k steps, in the interval where the training loss briefly fluctuates in Figure 6. The original recovery procedure replayed data from the beginning of the epoch and reset the random seed. The subsequent 50k checkpoint had FD-DINO 52.25. We regret that the submitted paper did not identify or explain this restart.

We retrained this segment from the 45k checkpoints preceding preemption, using the original learning rate and training recipe. At 50k, PRowMuon obtained FD-DINO 34.28, close to Muon’s 34.03 at the same step. The next two evaluations also remained near this range.

Update after resumingPRowMuon FID / FD-DINO ↓Muon FID / FD-DINO ↓
50k5.565 / 34.285.571 / 34.03
55k5.825 / 35.365.508 / 33.80
60k5.758 / 35.095.575 / 33.52

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

Thank you for your thoughtful reviews. Your questions helped us clarify when Periodic Row-wise Muon (PRowMuon) saves training time and how that translates into image quality.

  1. Summary of Recognized Strengths
  • Large scale evaluation. JGri and XuGh highlighted the evaluation across four DiT sizes up to 15B parameters. PFq7 emphasized the evidence for Muon’s convergence advantage over AdamW.

  • Periodic spectral updates. JGri highlighted the novelty of reducing the refresh frequency. PFq7 and XuGh recognized the efficiency gains from combining periodic updates with sharded row normalization.

  1. Main Concerns and Our Responses

Efficient Muon baselines (JGri, PFq7, XuGh)

  • We evaluated MuonBP and Dion3 at 1.3B and LionMuon at 1.3B and 4B, each over 60k updates. PRowMuon achieved lower best checkpoint and 60k FD-DINO than MuonBP at 1.3B and LionMuon at both sizes. On the shared evaluation grid, PRowMuon reached vanilla Muon’s final FD-DINO of 46.08 in 30.8% less active training time than Dion3.

System costs and time to quality (JGri, PFq7, XuGh)

  • We profiled training steps across four DiT sizes and measured time to a common fidelity target from Figure 3. PRowMuon reduced exposed communication at 1.3B and spectral computation at 15B. At 4B, it reached FD-DINO ≤ 40 in 36.1% less time than vanilla Muon. At 1.3B and 30k updates, PRowMuon achieved lower FID and FD-DINO than both DMuon and NorMuon.

Robustness and checkpoint quality (JGri, PFq7, XuGh)

  • We trained both optimizers on two additional 1.3B seeds and repeated the interrupted 15B segment at the original learning rate. Against vanilla Muon, PRowMuon had lower best FD-DINO in two of three seed pairs and lower 60k FD-DINO in all three. At 15B, the rerun yielded a 50k FD-DINO of 34.28, compared with 52.25 originally. We also examined the 9B endpoint gap and compositionality at the best-fidelity checkpoints.

Scope and refresh interval (JGri, PFq7)

  • We tested PRowMuon on a 618M language model at two token budgets and measured spectral drift in six DiT momentum matrices. At both token budgets, PRowMuon reduced step time and achieved almost the same final validation cross entropy as Muon. All 264 measured DiT momentum pairs satisfied Eq. (6). The bound was loose, but the observed increase in drift with the interval supports frequent spectral refreshes.

We apologize for the delay while we completed the additional large scale training runs and checkpoint evaluations. Thank you for your patience and understanding. Our individual replies give the supporting data for each question.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.