- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposed StructMuon, which reuses Muon's internal states as communication bases. Every worker can rebuild left and right bases locally, so no SVD or basis exchange is needed. On the theory side, the paper proves that communication error contracts whenever the residual compressor is contractive. It also gives a nonconvex stationarity bound whose error term shrinks as the subspaces capture more of the gradient. Experiments on GPT-2 Large and NanoGPT-124M demonstrated that at matched validation perplexity, StructMuon needs 92 - 97.5% less communication than Full Muon.
Critical strengths
- The idea is well-motivated: The bases come from optimizer states every worker already shares, so only r*r values are sent and the basis-maintenance cost of PowerSGD is avoided.
- Careful and honest evaluation: The paper uses five seeds, and reports wall-clock and bidirectional traffic. This comprehensive result makes the claims concrete.
Critical weaknesses
- The practical gain over baseline is small: To reach the same targets, StructMuon is only 2–18% faster than Full Muon (see Table 12). Counting the dense broadcast, total traffic savings fall to about 47–49% (Table 13).
- Key baselines are missing: Dion and PowerSGD+, which were designed for distributed Muon, are cited but not compared, making the experimental results less convincing.
Questions for authors
- How does StructMuon compare against EF-PowerMuon with the same Top-k residual budget, and against Dion? These results would show whether the history bases themselves are responsible for the gains.
- The bases require every worker to compute bitwise-identical momentum and Newton–Schulz outputs. How do you guarantee this under nondeterministic GPU kernels or mixed precision, and what happens to the training performances if bases vary across workers?
3 comments
Dear Reviewer,
We have added the requested baseline comparisons and basis-consistency checks.
We would be grateful if you could review our response and share your feedback. Thank you for your time!
Dear Reviewer xZQ4,
As the discussion period is nearing its end, we would be grateful if you could revisit your assessment in light of our responses and the added experiments. Please let us know if any concerns remain; we would be happy to clarify.
Thank you again for your time and consideration.
Best regards, The Authors
Thank you for your careful reading. Time constraints limited the original experiments; we have now expanded the evaluation to address your concerns.
1. Practical runtime benefits.
The runtime benefits of compression depend on the setting. Agarwal et al. [1] found speedups in only 6 of over 200 configurations in typical data centers. The Dion paper also reports 14.5% longer training time than Muon in its 350M NanoGPT speedrun experiment [2].
Appendix Table 12 reports NanoGPT-124M times calculated from recorded local costs plus upload/download times. Targets use five-seed mean validation-loss curves. Both methods accumulate gradients before one upload per update, with the same effective batch size and one return per update.
With one upload, speedup at PPL ≤ 130 rises from 2.34× to 4.01× as bandwidth decreases. At reference bandwidth, the stricter PPL ≤ 85 target reduces the gain to 1.85×.
2. Communication savings including downlink traffic.
The original manuscript counted full-model broadcasts after each update. The revised protocol returns combined compressed messages once per update; workers reconstruct the aggregated update locally. Appendix Table 11 reports communication over 25K updates:
Total communication falls by approximately 98.9% with four uploads and 99.1% with one upload. The runtime results above use this revised protocol.
3. Additional baseline comparisons.
We added EF-PowerMuon(Top-k) and Dion. EF-PowerMuon(Top-k) uses the same residual budget as StructMuon: k=150 on GPT-2 Large and k=100 on NanoGPT-124M. Table 1 reports the final PPL at 25K updates:
StructMuon outperforms both compressed baselines, including EF-PowerMuon with the same Top-k budget.
PowerSGD+ [3] uses periodic SVD-based subspace refreshes. In our tested implementation, these caused out-of-memory (OOM) errors during large-model training, preventing completion.
4. Basis consistency across workers.
One worker can construct the bases and broadcast them to all other workers at each refresh, ensuring a shared coordinate system. Related work such as TSR-Adam [4] also uses explicit basis communication.
StructMuon constructs bases locally from shared optimizer states with consistent row selections, QR sign conventions, and refresh schedules. We measure cross-worker numerical differences and their effect on training. Appendix A.8, Figure 14(d) reports the deviation across the tested workers and matrix instances, with (r,k)=(123,100).
On NanoGPT-124M/OpenWebText at (r,k)=(94,100), we compare local construction on each worker against construction on one worker followed by broadcast to the others. Local results at 10K and 15K correspond to Table 14:
In practice, one can choose between local basis construction and construction on one worker followed by broadcasting, depending on the system setup.
References
[1] Agarwal et al. On the Utility of Gradient Compression in Distributed Training Systems. MLSys 2022.
[2] Ahn et al. Dion: Distributed Orthonormalized Updates. 2025.
[3] Xie, Chen, and Yuan. From PowerSGD to PowerSGD+: Low-Rank Gradient Compression for Distributed Optimization with Convergence Guarantees. 2025.
[4] Dang et al. From O(mn) to O(r²): Two-Sided Low-Rank Communication for Adam in Distributed Training with Memory Efficiency. 2026.