- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a mixed-optimizer strategy, assigning hidden weight matrices to either MuonBP's spectral-norm linear minimization oracle (LMO) or an element-wise max-norm LMO. The former retains periodic full-matrix orthogonalization, while the latter avoids Newton-Schulz computation and optimizer-specific gather/scatter operations. A heuristic determines each matrix's optimizer using estimated descent surrogates. Evaluations include a 160M-model trained on eight B200 GPUs, and virtual-sharding of 4GPUs to simulate 128 and 256 GPU parallelism for a 960M and 1.2B model respectively. The 160M experiments show a throughput increase of 3.0% above MuonBP. The simulated experiments show a throughput increase of 4% for 128GPUs, and 6.5% for 256 GPUs.
Critical strengths
Assigning certain matrices to an element-wise optimizer removes orthogonalization computation and optimizer-specific communication, which is seen in the evaluation to lead to increased throughput.
Critical weaknesses
- In Appendix A.1, it is mentioned that training comparisons do not establish runtime savings if profiling is included, but it is not discussed how well the selected layer assignments transfer across different training runs. It would be helpful to see individual training runs and what the variance in both throughput and loss is across runs.
- Results are insufficient to confidently establish a general selection principle, since all model sizes are tested using one Modded-NanoGPT architecture.
Questions for authors
- How transferable are selected layer assignments to training runs with different hyperparameters? Can you report the variability across individual runs and how they compare against a standalone AdamW/Signum implementations?
- How transferable are selected layer assignments to different architectures? How does this static assignment compare to profiling the new architecture directly, and can you draw a conclusion relating to when it is feasible to keep the same assignment and when re-profiling is necessary?
Dear Reviewer 21PU,
We are writing to follow up on the rebuttal that we posted in response to your review. A revised version of the paper is now also available, with all changes highlighted in blue. We would value your feedback on our rebuttal and the revised paper at your convenience. We are happy to discuss any further questions.
Thank you again for your time and consideration.
Best regards,
The Authors of Submission 36827