- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies how the coefficient schedule of the quintic Newton–Schulz iteration in Muon behaves in bf16. It uses an idealized scalar model, with singular values spread uniformly over [1/κ, 1] and bf16 rounding applied to every scalar operation. On this model, it compares the fixed Muon coefficients, a per-step minimax schedule (P1, referred to as Polar Express), a variant with a boundedness constraint (P2), and a hybrid of the two (P3). At κ = 100, P1 and P2 diverge in bf16 while P3 remains finite. Two controls then show that P1 also diverges in fp32 on real gradient matrices, and that the deployed Muon reaches similar loss whether the iteration runs in bf16 or fp32. The authors therefore present the work as a characterization of an idealized design space rather than a fix for Muon.
Critical strengths
The paper clearly states the scope of its conclusions. It also tests whether the idealized results carry over, using a control on real gradient matrices (E4) and an end-to-end training run (E5). The coordinate reduction and the step-by-step error curves are clearly presented.
Critical weaknesses
The paper's description of Polar Express does not match the original work. The abstract states that no prior analysis characterizes these iterations under low precision, and the main text describes Polar Express as considering only exact (or fp32) arithmetic. However, since its first arXiv version, Polar Express has included a section on blowups in bf16. It handles them with a 1.01 safety factor, cushioning of the early polynomials, and ℓ = 10⁻³, and its experiments are run in bf16. The P1 schedule in this paper does not include these measures, so the divergence of P1 in bf16 does not represent the behavior of the published Polar Express. A substantial part of the paper's stated contribution has already been addressed in Polar Express.
The experimental results provide limited support for the paper's contribution, and the results of the bounded variant are inconsistent. Based on E4 and E5, the authors conclude that the bf16-specific instability does not occur on real gradients and that the deployed Muon is safe in bf16. The proposed P3 is evaluated only in the idealized simulation. In addition, the identity map (1, 0, 0) satisfies the constraint in Eq. (5), so with an exact solution a bounded step cannot increase the error. In Table 3, however, the fp32 error of P2 at κ = 10 increases from 3.14×10⁻³ at T = 3 to 1.61×10⁻¹ at T = 4. At κ = 100, the error of P3 increases from 8.66×10⁻² after the free steps to 2.35×10⁻¹ at T = 8. This indicates that the numerical solver does not reach the optimum of Eq. (5), so the P2 and P3 results do not reflect the behavior of the bounded design itself.
Questions for authors
- The "Finite precision considerations" section of Polar Express discusses blowups in bf16 and uses a 1.01 safety factor, cushioning of the early polynomials, ℓ = 10⁻³, and Frobenius normalization. Does P1 include these measures? If not, please explain how P1 differs from the published Polar Express and what the description of Polar Express in the main text is based on.
- In Table 3, the fp32 error of P2 increases between consecutive steps, and the error of P3 after its bounded steps is higher than after its free steps, although (1, 0, 0) is a feasible point of Eq. (5). Does this mean that the Nelder–Mead method with a penalty term does not find the optimum of Eq. (5)?
- Could you rerun E2 and E4 with the published Polar Express coefficients, including the measures above.
- Could you re-solve P2 and P3 with a reliable solver (e.g., Remez), and evaluate P3 on the real gradient matrices of E4.