- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper targets the communication and orthogonalization costs of the Muon optimizer in synchronous distributed training. It builds on EF21-Muon, whose RankK compressor transmits new singular factors every iteration. The proposed method keeps shared left and right rank-r bases from a global momentum approximation. Workers project momentum onto these bases and quantize differences with error feedback. Every T iterations compressed full-momentum differences refresh the bases; a diagonal option transmits r entries. Experiments pretrain NanoGPT-124M on FineWeb10B (four B200 GPUs) against EF21-Muon with RankK 15% plus Natural compression under fixed tokens, communication, time, and target loss, reporting up to 47.8% lower worker-to-server communication and up to 53.8% less wall-clock time. Code is released.
Critical strengths
S1. Well-written paper.
S2. Strong theoretical results.
S3. Good experimental evaluation on modern hardware. Shows substantial gains, especially in terms of wall-clock execution time.
Critical weaknesses
W1. The gains are unattributed: all runs use one four-GPU B200 node where the counted traffic (about 15 MiB/s; Table 1, Appendix D.2) is far below intra-node bandwidth, so communication is not shown to be a bottleneck.
Questions for authors
Q1. Explain W1.
Dear Reviewer bzET,
We are writing to follow up on the rebuttal that we posted in response to your review. We have also uploaded a revised version of the paper with all changes highlighted in blue. We welcome any further discussion and would value your feedback on our rebuttal and the revised paper.
Many thanks for your time and your helpful comments.
Sincerely,
The Authors of Submission 44076