- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This work introduces GMBI, a polynomial state-space parameterization for approximating the orthogonal factor in matrix polar decomposition for the Muon optimizer. Unlike traditional Gram iteration, GMBI lets you learn the transition rule for the auxiliary state and the accumulation rule for the output separately. It also uses mid-way restarts and staged fitting, and calibrates using the first-half output from actual FP16 computation
Critical strengths
The problem addressed is practical and relevant to real-world model training.
Critical weaknesses
-
The end-to-end gain they show is pretty small, and it's honestly unclear if it matters in practice. In Table 4, the four-layer restarted variant only brings mean validation loss down from 3.280700 to 3.279417 compared to Direct PE — that's about 0.128% lower perplexity. Its mean step-time reduction is only around 0.15%, and the paired timing interval even includes zero.
-
The way it's written makes the contribution and how it actually works way harder to judge than it needs to be.
Questions for authors
-
How was the four-layer restarted Variant selected for the downstream conclusion?
-
Can the existing results more clearly separate the effect of decoupling from the effect of the phasewise fitting recipe?