- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies Muon optimization for text-to-image Diffusion Transformers ranging from 1.3B to 15B parameters. It finds that Muon improves validation loss and generation quality over AdamW, but incurs substantial overhead. Authors proposed Periodic Row-wise Muon performs five Newton–Schulz iterations every third step and uses scaled row-normalized momentum updates otherwise. Its distributed implementation computes these inexpensive updates on momentum shards and overlaps communication with computation during spectral refreshes. Experiments train four model sizes on GPIC for 60,000 steps. Compared with vanilla Muon, the method reduces optimizer time by 46.9–54.3% and total step time by 15.7–24.3%. These results show that Periodic Row-wise Muon preserves Muon’s gener- ative quality advantage while translating it into end-to-end training efficiency for large DiTs
Critical strengths
Substantial experiments at relevant scale. Evaluating four models up to 15B parameters provides meaningful evidence beyond small-model optimizer benchmarks. Matching training examples and reporting multiple generation metrics strengthen the study.
Effective algorithm and systems integration. RowNorm’s with optimizations reduces optimizer time by 46.9–54.3%
Critical weaknesses
The comparisons do not establish the benefit over closely related efficient Muon approaches. The experiments compare primarily against AdamW and an implementation that gathers full momentum and replicates NS5 computation on every rank. Yet periodic spectral updates combined with cheap intermediate updates already appear in LionMuon https://arxiv.org/pdf/2605.19811, and DMuon https://arxiv.org/pdf/2606.27153 specifically addresses distributed Muon overhead. Both are acknowledged in the paper but lack experimental comparisons. Consequently, the results establish an improvement over the selected vanilla implementation, while leaving the incremental benefit of RowNorm and its execution strategy unresolved.
Quality preservation and the largest time savings lack sufficient robustness evidence. No repeated training runs or uncertainty estimates are reported, despite substantial checkpoint variation. At 9B, the method’s best FD-DINO improves from 33.97 to 32.44, but its final FD-DINO worsens from 33.97 to 36.70 and FID from 5.61 to 6.13. At 15B, final GenEval2 AM falls from 57.93 to 52.26, and Table 10 shows a late FD-DINO spike to 52.25.
The large-scale study and systems improvements are valuable. However, the current evidence does not sufficiently establish the proposed method’s advantage over relevant efficient Muon alternatives or the reliability of its quality-preservation claims. Focused additional comparisons and robustness evidence could change my recommendation.
Questions for authors
On Figure 1 (c) Validation loss versus wall-clock time. Validation loss for Muon looks worse than AdamW with longer training, e.g 15B at step 200.0K, please explain.
What explains the late quality fluctuations, particularly the 15B FD-DINO increase to 52.25 at 50k steps (shown in Table 10: All checkpoint evaluation metrics for the 15B model.)? Do the selected best-FD-DINO checkpoints also preserve text alignment and compositionality?
3 comments
4. Does the 15B spike at 50k recur?
Thank you for drawing attention to the 50k checkpoint. The training logs record job preemptions at 47k and 48k, in the interval where the training loss fluctuates in Figure 6. The original resume procedure restarted the data stream from the beginning of the epoch and reset the random seed. The following 50k checkpoint had an FD-DINO of 52.25. We apologize for overlooking these events in the submission.
We repeated this segment from the 45k checkpoint preceding the preemptions, using the original learning rate and training recipe. At 50k, the new PRowMuon checkpoint had an FD-DINO of 34.28, close to Muon’s 34.03. The spike did not recur at 55k or 60k either.
| Update after resuming | PRowMuon FID / FD-DINO ↓ | Muon FID / FD-DINO ↓ |
|---|---|---|
| 50k | 5.565 / 34.28 | 5.571 / 34.03 |
| 55k | 5.825 / 35.36 | 5.508 / 33.80 |
| 60k | 5.758 / 35.09 | 5.575 / 33.52 |
5. Do the best-FD-DINO checkpoints also preserve text alignment and compositionality?
For each optimizer, we selected the checkpoint with the lowest measured FD-DINO under the paper’s selection rule, then evaluated HPSv2 and GenEval2 on that same checkpoint. The table gives the results from the original 15B runs and includes matched comparisons at 25k and 60k.
| 15B checkpoint | FD-DINO ↓ | HPSv2 ↑ | GenEval2 AM ↑ | GenEval2 GM ↑ |
|---|---|---|---|---|
| PRowMuon 25k (best) | 32.35 | 22.90 | 49.68 | 11.74 |
| Muon 25k | 33.48 | 22.92 | 54.30 | 14.21 |
| Muon 45k (best) | 33.25 | 22.53 | 57.50 | 13.76 |
| PRowMuon 60k | 33.99 | 21.35 | 52.26 | 11.97 |
| Muon 60k | 33.51 | 22.37 | 57.93 | 14.55 |
At 25k, PRowMuon had better FD-DINO (32.35 versus 33.48) and nearly identical HPSv2 (22.90 versus 22.92). Its GenEval2 arithmetic and geometric means were 4.62 and 2.47 points lower. Comparing each method’s best FD-DINO checkpoint, PRowMuon scored 0.37 points higher on HPSv2, while Muon scored higher on both GenEval2 aggregates. Thus, the 15B checkpoint that gave PRowMuon its best fidelity retained a comparable preference score, with a measurable gap in compositional accuracy.
FD-DINO measures the overall distribution of generated images in feature space. An image can have realistic features while depicting the wrong number of objects or placing them in the wrong positions. GenEval2 tests these prompt specific requirements, and HPSv2 estimates human preference. Denoising pretraining minimizes prediction error conditioned on text and noise level, so an improvement in distributional fidelity need not improve every prompt level score. Section 5.2 of GenEval also reports that further pretraining does not consistently improve compositional performance. Methods such as Diffusion-DPO introduce a separate post-training objective for preference alignment. Here we evaluate the original pretrained checkpoints.
6. Why does the 9B model finish with a higher FD-DINO than Muon?
Thank you for identifying the 9B endpoint difference in Figure 2. At 60k, FD-DINO was 36.70 for PRowMuon and 33.97 for Muon. Decomposing this 2.73-point gap shows that PRowMuon more closely matched the real images’ mean DINO features (a 0.29-point lower mean term). Its covariance term was 3.01 points higher, which accounts for Muon’s better endpoint score.
| 9B checkpoint | Denoising val. loss ↓ | Mean term ↓ | Covariance term ↓ | FD-DINO ↓ |
|---|---|---|---|---|
| PRowMuon 25k | 0.323835 | 2.715 | 29.723 | 32.44 |
| Muon 25k | 0.321385 | 4.677 | 34.292 | 38.97 |
| PRowMuon 60k | 0.317199 | 5.056 | 31.640 | 36.70 |
| Muon 60k | 0.313950 | 5.342 | 28.626 | 33.97 |
The ordering changed late in training. At 25k, PRowMuon achieved its best FD-DINO of 32.44, compared with 38.97 for Muon. From 25k to 60k, PRowMuon’s mean and covariance terms rose by 2.34 and 1.92, respectively, accounting for the 4.26-point rise in its FD-DINO. Over the same interval, Muon’s covariance term fell from 34.29 to 28.63. PRowMuon’s validation loss also fell, from 0.323835 to 0.317199. Denoising loss measures velocity prediction error at a sampled noise level, whereas FD-DINO measures the feature distribution after multistep sampling. The 9B checkpoints show how denoising can continue to improve after the best measured image fidelity has been reached.
Over the same 25k–60k interval, PRowMuon’s GenEval2 AM improved from 51.25 to 57.33, and its GM improved from 11.75 to 15.97. These checkpoints therefore improved on the compositional evaluation despite the decline in distributional fidelity. Vanilla Muon also showed a late decline in fidelity at 1.3B, where its FD-DINO rose from 41.73 at 40k to 46.08 at 60k. A related late decline in fidelity is reported in EDM2, Appendix A.2, Figure 12.
7. Reading the 15B curves near 200k in Figure 1(c)
We appreciate your careful reading of this figure. The 200k tick marks elapsed training time in units of the average 1.3B AdamW step. Both 15B runs ended at 60k updates. The table gives the validation records immediately before and after the 200k time tick for each method.
| 15B method | Before 200k (step / time / val. loss) | After 200k (step / time / val. loss) |
|---|---|---|
| AdamW | 52k / 197,501 / 0.317844 | 53k / 201,299 / 0.317779 |
| Muon | 30k / 195,821 / 0.317713 | 31k / 202,348 / 0.317344 |
Near that tick, both recorded Muon losses were lower than either neighboring AdamW loss. AdamW did lead earlier on the time axis, as you observed. For example, it reached a validation loss of 0.340 in 30,385 time units, compared with 39,164 for Muon. At the common endpoint of 60k updates, validation losses were 0.31205 for Muon and 0.31759 for AdamW.
Response to Reviewer XuGh
We apologize for the late response, as the additional large scale experiments took considerable time to complete. Thank you in advance for considering the new results and sharing your feedback. We will incorporate the relevant results and clarifications into the camera-ready version.
Thank you for asking how much of the improvement can be attributed to the choice of intermediate update and how reliably the method preserves generation quality. We compared Periodic Row-wise Muon (PRowMuon) with LionMuon at the same refresh frequency and with DMuon, which assigns matrices to individual workers. We also examined independent training runs, traced the 15B anomaly to the affected training interval, and measured fidelity and prompt adherence on the same checkpoints.
1. PRowMuon and LionMuon at the same spectral refresh frequency
We trained both methods for 60k updates at 1.3B and 4B, with a spectral refresh every third update. We evaluated both methods every 5k updates, matching the phase of their refresh cycles at each checkpoint. On the two intervening steps, PRowMuon normalizes each row of the current momentum and scales the update by γ = 0.15. LionMuon uses an elementwise sign update. For LionMuon, the spectral branch used peak learning rates of 0.005 at 1.3B and 0.003 at 4B. The sign branch used 2 × 10⁻⁵ at both sizes. The branches shared a momentum state, with coefficients of 0.9 for the update direction and 0.99 for updating the state. We used a 2k step warmup, cosine decay, and decoupled weight decay of 0.01. LionMuon updated the two-dimensional weights within each block. Other parameters used AdamW with a learning rate of 10⁻⁴ and β = (0.9, 0.999).
At 1.3B and 4B, PRowMuon’s best FD-DINO was lower than LionMuon’s by 2.91 and 2.56 points, respectively. At the common 60k endpoint, its FD-DINO was lower by 5.23 and 6.01 points, respectively.
2. DMuon step time and checkpoint quality
We benchmarked owner-based DMuon, which computes each matrix’s full spectral update once and overlaps communication with training. At 1.3B, the observed step time ratios relative to PRowMuon (1.00×) were 0.80× for DMuon and 1.01× for NorMuon.
The table compares image fidelity after 30k updates for all four methods.
At 30k updates, PRowMuon achieved the lowest FID (6.114) and FD-DINO (44.77) of the four methods. DMuon reached 6.20 and 45.79, respectively, and NorMuon reached 6.366 and 46.76.
3. How much do the results vary across training seeds?
We added two independent seed pairs to the original 1.3B runs. Within each pair, Muon and PRowMuon used the same training seed. PRowMuon had a lower FD-DINO at 60k for all three seeds. The mean paired difference (PRowMuon minus Muon) was −0.88, with a sample standard deviation of 0.52. The direction of the endpoint difference was thus consistent across the three pairs tested at this scale.
Values are means ± sample standard deviations across three training seeds. For the paired differences, we first subtracted the two observations within each seed.