- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The authors present a method to remove optimizer-specific communication over tensor-parallel ranks during training with matrix-aware optimization. They do this by projecting the momentum into a block-diagonal basis, built from the input-activation second moment. Each block lies within a single tensor-parallel shard, so the update can be computed locally. They motivate the selection of the basis and derive the model updates. Finally they validate their approach on GPT-2 against AdamW, Muon and Dion, and show the speedup relative to Dion on Qwen3 models.
Critical strengths
The authors present a simple approach to an important distributed training bottleneck - the additional communication required by matrix-aware optimizers under tensor parallelism. The paper is well written, and the motivation and implementation of the design are clear. The authors also provide a thorough ablation study in the appendix.
Critical weaknesses
My main concern is mostly limited evaluations and comparisons:
- The speedup of the method is overstated in the abstract - practitioners mostly care about total iteration time.
- While Dobb consistently beats adamw, the margin is very small (For example, for B=16). Can the authors rerun training with additional seeds, to add error approximations and show that the difference is significant? Furthermore, testing a different model architecture can further strengthen the accuracy results.
- Can the authors run evaluations on downstream tasks, and not only validation loss? I understand that due to limited computational resources the models that are trained are small and will not be performant on complex tasks, but perhaps perplexity on wikitext or accuracy on easy tasks like hellaswag can be a valid test at this model size?
Questions for authors
- Can the authors explain why normalizing all weight updates to have the same RMS constitutes a fair comparison? Can they supply evidence for this claim?
3 comments
Thank you for your response to my concerns. I increased my soundness score and maintained my overall score. I think this is a solid paper and recommend its acceptance.
Dear Reviewer nYDZ,
Thank you for reviewing our response and for raising your soundness score! We appreciate your recommendation for acceptance and your helpful comments throughout the review process.
Best regards, The Authors of Submission 49151
Response to Reviewer nYDZ
We thank the reviewer for the careful reading and the concrete suggestions.
Speedup and total iteration time. We agree that the abstract should report the effect on total iteration time. We removed the 7.8× optimizer-call claim. The revised abstract reports that, on GPT-2 XL with 32 H100 GPUs, Dobb reduces pooled iteration time from 1184.0 ms for Dion to 1069.1 ms, a reduction of 114.9 ms (9.7%). The corresponding optimizer-call times are 124.1 ms and 9.2 ms. Pooled iteration time is defined in Appendix G.1. Table 3 reports pooled iteration times across the main timing configurations; Dobb's is comparable with AdamW's and below Muon's and Dion's in every GPT-2 configuration (General Response, item 3 and continued).
Margin over AdamW: seeds and architecture. On GPT-2 Small the margin holds across seeds. At the reviewer's example of B=16, Dobb reaches 3.028 ± 0.001 against 3.045 ± 0.002 for AdamW, a statistically significant difference (paired t-test over the three seeds, p=0.001), and Dobb is below AdamW at every block count in every seed. We reran all eleven GPT-2 Small configurations with two additional seeds; Table 2 reports the means and standard deviations, and Table 10 lists each seed (General Response, item 1).
We also added GPT-2 XL (1.56B parameters), where Dobb at B=16 reaches 2.476 against 2.497 for AdamW, a reduction of 0.021. Additional seeds at Large and XL and training on a different architecture were beyond our compute budget for the discussion period. Our quality evaluation therefore covers the GPT-2 family at four sizes, from 124M to 1.56B parameters.
Downstream tasks. WikiText perplexity agrees with the validation-loss comparison: Dobb's is lower than AdamW's at every block count. HellaSwag points the same way, but by only 0.3 to 0.5 points. We also evaluated ARC-Easy and PIQA. Appendix F.6 and Table 19 report zero-shot evaluation of the final GPT-2 Small checkpoints for all eleven configurations. Accuracies are in percent.
Dobb's WikiText perplexity rises from 40.2 to 41.5 as B increases from 1 to 64, while HellaSwag accuracy remains between 29.5% and 29.7%. For Dion, perplexity rises from 39.4 to 56.2 and HellaSwag accuracy falls from 30.2% to 27.6% as the rank fraction decreases from 1 to 1/64. ARC-Easy and PIQA show no consistent ordering among the optimizers.
Fairness of the shared update scale. The optimizers share one scale formula, which gives low-rank Dion a smaller update RMS than the others. Neither raising its update RMS to 0.2 nor tuning its learning rate closes the gap: Dion at ρ=1/16 and 1/64 stays above every Dobb entry on GPT-2 Small. Our submission described Dion's scale incorrectly as the rank-dependent τmn/r; Section 6 and Appendix E.1 now state the shared scale that the runs used (General Response, item 2).
The scale formula is 0.2max(m,n). Muon, RMNP, Dobb, and full-rank Dion have update RMS values in the range 0.2 to 0.4, whereas low-rank Dion has update RMS 0.2ρ (Table 11).
The comparison evaluates the methods under a common learning rate and weight decay transferred from AdamW, without optimizer-specific retuning. Muon achieves lower validation loss than AdamW at all four model sizes under this scheme. AdamW's selected rate is also close to its optimum: in our AdamW sweeps, a rate off by a factor of 2 raises AdamW's final loss by about 0.002 (Appendix E.2), below every margin of Dobb over AdamW in Table 2 (0.006 or more).
We separately tested whether low-rank Dion's smaller update RMS explains its loss increase. Table 12 raises its update RMS to 0.2 using a rank-corrected scale while keeping the learning rate and weight decay unchanged. On GPT-2 Small at ρ=1/16, this improves validation loss from 3.108 to 3.089, recovering 0.019 of the 0.106 gap to full-rank Dion (3.002).
We also tuned Dion's learning rate directly at ρ=1/16 and 1/64 (Table 13). The best rates lower its loss from 3.108 to 3.071 and from 3.265 to 3.227; both remain above every Dobb entry on GPT-2 Small (at most 3.036), with Dobb not retuned.
Taken together, the margin over AdamW holds across seeds, WikiText perplexity agrees with validation loss, and Dion's higher loss persists when its update RMS is raised to 0.2 or its learning rate is tuned.