- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper aims to reduce the computation and communication costs of the Muon optimizer. Its contributions can be summarized as follows:
- Gram Newton-Schulz that decouples non-symmetric ratio and the number of Newton-Schulz iterations , reducing the computation cost.
- CuteDSL symmetric GEMM kernels on Hopper and Blackwell, for faster Gram Newton-Schulz.
- subsampling rows or columns of the momentum matrix for orthogonalization to reduce the computation and communication cost.
- megabatched communication to reduce the communication cost.
Experiments suggest that these techniques bring the optimizer step time close to that of AdamW while matching or exceeding the performance of standard Muon.
Critical strengths
- suggested techniques reduce the computation and communication cost of Muon to that of AdamW, regarded as tolerable in general.
- it provides pip-installable packages dion and gram-newton-schulz, that are expected to have a strong impact on the community.
Critical weaknesses
None
Questions for authors
- convergence behavior of EF: theoretically, error feedback aims to preserve the convergence guarantee for biased compression methods. Is there any theoretical reason why the method in the paper is named EF? In other words, does it guarantees the same convergence behavior?
Thank you for the kind feedback!
As for your question, we don't have a theoretical analysis, but the name "error feedback" (which we adopt from the Dion paper of Ahn et al.) is indeed a reference to a technique used in federated learning to combat biased compression. We are doing the same thing here, if you think of our "compression" scheme as selecting some rows and ignoring the rest. Error feedback helps ensure that the other rows get their due. (And it does help; see Figure 25.)