- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper establishes theoretical guarantees regarding training loss for the Muon algorithm, which employs a momentum mechanism and a five-step Newton–Schulz orthogonalization method. It demonstrates that this finite-step mapping maintains alignment with the momentum buffer and bounds the spectral norm of the update; leveraging gradient stability near initialization, this ensures descent along the current gradient direction. For sufficiently wide two-layer ReLU networks with fixed output weights trained via full-batch gradient descent, the Muon algorithm—using a target-dependent learning rate—attains any target loss within steps with high probability. Results from synthetic experiments further confirm that these mechanisms of alignment and kernel stability hold true even when the network width is small.
Critical strengths
- This paper examines the significant gap between exact orthogonalization analysis and the five-step update method employed by the Muon algorithm. It establishes the sufficient properties required for the tuned map, without necessitating that the map strictly approximate the exact polar factor.
- The proof follows a clear logical progression: it first establishes trajectory containment via update bounds, then utilizes kernel stability to control momentum lag and derive loss descent, thereby avoiding circular reasoning.
Critical weaknesses
- There remains a significant gap between this theory and practical training protocols. The theory relies on assumptions such as two-layer networks, fixed random output weights, full-batch updates, and exact arithmetic operations. However, such theoretical analysis is difficult to scale to truly large-scale models.
- This framework is equally applicable to the normalized momentum method without orthogonalization (i.e., ); thus, the theorem does not establish an advantage unique to the Newton–Schulz update. An end-to-end comparison—tracking both loss values and runtime—among the baseline, tuned NS, classic NS, and the exact polar factor would help clarify the specific contribution of orthogonalization.
- I am personally unclear on the actual value provided by this type of theoretical analysis in the papers related to Muon. Furthermore, current observations do not offer sufficient insight; I would find empirical results from pre-trained models in the range of hundreds of millions to billions of parameters more informative.
Questions for authors
No