acceptodds
Withdrawn from ICLR 2027

Precision-Aware Optimal Newton–Schulz Iterations for the Matrix Sign

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The Muon optimizer orthogonalizes each gradient matrix by running a short quintic Newton–Schulz (NS) iteration that drives its singular values to one—an approximate matrix sign function, msign(M) = UV^T. Recent work (Polar Express) shows that the per-step coefficients of this iteration should not be hand-tuned but solved for: a per-step minimax (Remez) fit yields provably fastest convergence in exact arithmetic. But Muon actually runs the iteration in bfloat16, and no prior analysis characterizes these learned iterations under low precision. We give a precision-aware characterization of the coefficient design space, deliberately scoped to the idealized scalar reduction of the iteration—the singular-value band [1/κ, 1] with per-scalar rounding—and bounded honestly against real gradients and real training. We study the sign iteration as an optimal–structured–hybrid family—P1, a free per-step minimax fit (Polar Express); P2, the same minimax under a boundedness constraint; and P3, a hybrid that runs free steps early and bounded steps late—and measure error-versus-step curves in fp32 and bf16 across κ ∈ 10, 30, 100. Three findings. (i) On the idealized band, the fp32 per-step-optimal P1 crushes fixed Muon at a matched budget (1e-8 vs. 0.318 at κ = 10; 7.8e-6 vs. 0.318 at κ = 100). (ii) On that same idealized band, bf16 is fragile at high conditioning: at κ = 100 the aggressive P1 coefficients make the bf16 scalar iteration diverge (terminal 4×10^18; P2 overflows to +∞), while the hybrid P3 stays finite (bf16 floor 0.141, beating fixed’s 0.328); we give a first-order model of this floor. (iii) This bf16-specificity is a feature of the idealized band, not of real gradients. Running the true matrix iteration on real 124M gradient matrices (fp32-accumulate tensor cores), the optimal P1 coefficients are fragile in both precisions—blowing up on 8/8 matrices in bf16 and 7/8 in fp32, so the clean bf16-only divergence appears for just 1/8—while fixed Muon stays stable (residual ≈ 0.32). And the deployed iteration is bf16-safe end-to-end: a nanoGPT-124M premise test trains equally well in bf16 and fp32 (a null, −0.038 nats). Our contribution is thus a precision-aware map of the coefficient design space in the idealized setting, honestly bounded: the optimal-coefficient instability does not clearly transfer to real gradients bf16-specifically, and today’s Muon is already bf16-safe—not a fix for a real deployment problem. Extending precision-aware optimal iterations to general spectral functions is left to future work.

Primary area:
optimization
Keywords:
Muon, Newton-Schulz iteration, matrix sign, polar decomposition, low precision, bfloat16, minimax approximation, optimization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_vsi9OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how the coefficient schedule of the quintic Newton–Schulz iteration in Muon behaves in bf16. It uses an idealized scalar model, with singular values spread uniformly over [1/κ, 1] and bf16 rounding applied to every scalar operation. On this model, it compares the fixed Muon coefficients, a per-step minimax schedule (P1, referred to as Polar Express), a variant with a boundedness constraint (P2), and a hybrid of the two (P3). At κ = 100, P1 and P2 diverge in bf16 while P3 remains finite. Two controls then show that P1 also diverges in fp32 on real gradient matrices, and that the deployed Muon reaches similar loss whether the iteration runs in bf16 or fp32. The authors therefore present the work as a characterization of an idealized design space rather than a fix for Muon.

Critical strengths

The paper clearly states the scope of its conclusions. It also tests whether the idealized results carry over, using a control on real gradient matrices (E4) and an end-to-end training run (E5). The coordinate reduction and the step-by-step error curves are clearly presented.

Critical weaknesses

The paper's description of Polar Express does not match the original work. The abstract states that no prior analysis characterizes these iterations under low precision, and the main text describes Polar Express as considering only exact (or fp32) arithmetic. However, since its first arXiv version, Polar Express has included a section on blowups in bf16. It handles them with a 1.01 safety factor, cushioning of the early polynomials, and ℓ = 10⁻³, and its experiments are run in bf16. The P1 schedule in this paper does not include these measures, so the divergence of P1 in bf16 does not represent the behavior of the published Polar Express. A substantial part of the paper's stated contribution has already been addressed in Polar Express.

The experimental results provide limited support for the paper's contribution, and the results of the bounded variant are inconsistent. Based on E4 and E5, the authors conclude that the bf16-specific instability does not occur on real gradients and that the deployed Muon is safe in bf16. The proposed P3 is evaluated only in the idealized simulation. In addition, the identity map (1, 0, 0) satisfies the constraint in Eq. (5), so with an exact solution a bounded step cannot increase the error. In Table 3, however, the fp32 error of P2 at κ = 10 increases from 3.14×10⁻³ at T = 3 to 1.61×10⁻¹ at T = 4. At κ = 100, the error of P3 increases from 8.66×10⁻² after the free steps to 2.35×10⁻¹ at T = 8. This indicates that the numerical solver does not reach the optimum of Eq. (5), so the P2 and P3 results do not reflect the behavior of the bounded design itself.

Questions for authors
  1. The "Finite precision considerations" section of Polar Express discusses blowups in bf16 and uses a 1.01 safety factor, cushioning of the early polynomials, ℓ = 10⁻³, and Frobenius normalization. Does P1 include these measures? If not, please explain how P1 differs from the published Polar Express and what the description of Polar Express in the main text is based on.
  2. In Table 3, the fp32 error of P2 increases between consecutive steps, and the error of P3 after its bounded steps is higher than after its free steps, although (1, 0, 0) is a feasible point of Eq. (5). Does this mean that the Nelder–Mead method with a penalty term does not find the optimum of Eq. (5)?
  3. Could you rerun E2 and E4 with the published Polar Express coefficients, including the measures above.
  4. Could you re-solve P2 and P3 with a reliable solver (e.g., Remez), and evaluate P3 on the real gradient matrices of E4.
Reviewer_DCEQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes changes to the Polar Express method for approximating the polar factor that are more numerically stable in bfloat16. They introduce a heuristic, and validate it only on 1-d polynomials, with no benchmarks against other schemes to safe guard against precision issues.

Critical strengths

The objective of understanding the best polar approximation under a fixed precision (e.g bfloat16) makes total sense. The polar express in bfloat16 does run into numerical rounding issues, which I would like to see resolved.

Critical weaknesses

The paper has a few serious issues. 

1) No consideration of prior work on numerical stability. There are established notions and techniques for improving numerical stability. Consider for instance proving backwards stability. Furthermore the paper that proposes Polar Express also proposes a cushioning technique to improve numerical stability (and an additional safety factor). You would have to compare your work against other such techniques for improving numerical stability. 

  1. Informal treatment: Reading the paper I was lead to believe there would be a formal treatment of numerical stability, perhaps drawing clear links between bfloat16 and how to compute the polynomials. This was not the case. The only proposition 1, is not a proposition, and more of an informal statement. Furthermore, the premise that only the singular values will change is wrong in finite precision. In bfloat16 polynomials can also change the singular vectors.

  2. Solvability and no Global solutions: First, on line 251 you write "[Polar Express] choses each step's quintic greedily". Note the main contributions of this polar express is to show that this greedy selection leads to the globally optimal composition of polynomials. This global optimality is lost when you introduce constraints in (4), or at least you make no comments. Furthermore, this greedy problem in (5) is now a non-convex problem for which you have no guarantees that it can be solved. I read your section B about this about using Nelder-Mead, but still, this is a heuristic.

  3. Experiment. The experiments only test just the isolated polynomials on an interval. This is not enough, because finite precision also changes the singular vector spaces. To really validate your approach you would need many more experiments, comparing on real gradient matrices from deep learning, and validate the method within a pre-training run iwht Muon.

  4. Legibility. It's hard to read this paper. The sentences are very long and contain too many interjecting parenthetical statements. It's also full of imprecise statements that could be removed entirely. Finally, the original content of the paper only starts  halfway down page 5. For a 9 page paper, you need to better use the space and get to the point more quickly. If you are using an LLM to help in the writing, I recommend asking in the prompt that it use a  "The Tongue and Quill" style, which is a style of  advocated by the Air Force which uses clear, concise, specific writing, plain language, and active voice.

Questions for authors

I have a few questions and suggestions:

First the abstract: This abstract is almost illegible. You are trying to convey your main experimental results in the abstract. But the reader does not have enough context to know what you are talking about. You should use the abstract just to highlight the main high level ideas and concepts.

  1. 052 " The same orthogonalization primitive underlies preconditioners such as Shampoo" this is not explicitly true. I know there is some confusion conflating Shampoo and Muon, but nowhere does Shampoo need to (semi)-orthogonalize a matrix.

  2. Line 183, for the definition of polar factor you need the reduced SVD which only uses strictly positive singular values σi>0.\sigma_i >0.

  3. Line 198 "The reasons such itterations are tractable ..." This sentence makes no sense. This is not the reason the iterations are tractable, instead you explaining that the polynomial only acts on the singular values (in infinite precision)

  4. Line 202 and 210. "Each coordinated updated independelty"... "is coordinate-wise".. This reference to coordinate-wise updates is not clear, and also not the correct term. What you mean is that these are singular-value matrix functions, also known as generalzied matrix functions.

  5. Line 225-233: This paragraph contains a lot of things that have, at this point, already been repeated several times.

  6. Line 227: You haven't defined ht(x)h_t(x)

  7. Line 247: "We show its real error-vs-steo curve alongside every learned schedule"... where do you show this?

  8. Line 252: You have to define properly what is ϕt\phi_t.

  9. Line 264: Finally your new contribution is described, but how is B related to bfloat16? How do you choose B or relate it to the mantissa, dynamic range, and all of these things you mentioned before? 

  10. Proposition 1: This is not a proposition, it is informal and almost illegible. I checked appendix D, there is not clear proof or statement of this. In fact, you don't even define ete_t here.

  11. Table 3. This has to be a plot, it is ilegible as a massive table of numbers.

Reviewer_YRqPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes the Muon optimizer’s Newton-Schulz iteration, specifically its polynomial coefficients, from the perspective of low precision settings such as bf16. While Polar Express improves the convergence speed, the authors focus on the numerical instability with bf16 precision. To examine the idea, they experiment with per-step minimax optimization (P1), its bounded version (P2), and a schedule of P1 and P2 (P3). They show that coefficient schedules achieving high accuracy in fp32 can become numerically unstable under simulated bf16 arithmetic in an idealized scalar setting. In conclusion, they investigate the precision dependent behavior of the Newton-Schulz coefficients.

Critical strengths

Strengths 1: This research focuses on the Newton-Schulz coefficients under low precision, from the perspective that optimality under exact arithmetic does not imply optimality under finite precision. The research idea is close to real world application under bf16 which most large language models use.

Strengths 2: Since the authors fixed all conditions other than the coefficient choice, the differences among the three methods can be analyzed clearly. Also the experiment on the iteration budget, conditioning, precision provides detailed explanation about trade-off between three controlled methods.

Critical weaknesses

Weaknesses 1: Although the paper main contribution is characterization under idealization, the practical significance of this characterization remains unclear. In particular, the bf16 specific instability observed in the idealized setting does not transfer to real gradient, while the deployed Muon iteration already remains stable in bf16. Furthermore, the proposed P3 schedule is not evaluated on real gradient or in end-to-end training. The only end-to-end experiment compares fp32 and bf16 precision using the existing fixed Muon iteration on nanoGPT, and does not establish any practical benefit of the proposed schedules. Thus, the paper provides insufficient evidence that the proposed framework addresses a meaningful practical limitation or that reducing bf16 numerical error leads to improved training performance.

Weaknesses 2: The per-step minimax formulation largely builds on prior work, while the proposed hybrid schedule relies on a heuristic halfway switching rule without sufficient theoretical justification. And the relationship between greedy per-step optimality and terminal approximation error remains unclear for P3. Furthermore, the first-order stability analysis relies on several assumptions, which makes applicability of proposed schedules not sufficiently justified, limiting the strength of its theoretical conclusions.

Questions for authors

Questions 1: What evidence supports the practical importance of reducing worst-case orthogonalization error, particularly when the existing Muon iteration already operates stably in bf16?

Questions 2: P1 is unstable in both precisions on real gradients while the existing Muon iteration remains stable. What effect do you expect P3 to have on real gradient and in end-to-end LLM training, and what supports this expectation in detail?

Questions 3: How is the greedy per-step minimax objective related to terminal optimality, and what justifies switching from P1 to P2 specifically halfway through the iteration?

Question 4: How can P2 exhibit numerical overflow despite its boundedness constraint, and are the assumptions underlying the first-order stability model satisfied by the proposed coefficient schedules?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.