acceptodds
Under review as a conference paper at ICLR 2027

Decoupled Polynomial State-Space Iterations for Matrix Polar Approximation, with an Application to Muon

Abstract

We introduce Gram-matrix-based iterations (GMBI), a decoupled polynomial state-space parameterization that separately controls propagation of a square auxiliary state and accumulation of the rectangular output under a restricted matrix-computation budget. Building on the finite-depth polynomial framework of Polar Express, GMBI supports both compact short graphs and robust full-restart graphs. For restart graphs, we freeze a guarded two-layer prefix, materialize its outputs from FP16 calibration runs to rebuild true Gram states, and optimize the suffix by scalar rollouts from the realized restart spectra, adapting to prefix-induced distribution shifts and finite-precision effects. In controlled FP16 experiments, the decoupled GMBI parameterization outperforms a learned tied-polynomial control under both robust-only and Muon-aware fitting protocols, while the Muon-aware protocol provides an additional improvement for the decoupled variant. Tests on unseen spectra, matrix shapes, and held-out GPT-2 Medium updates show that these gains are not confined to the calibration set, while restart controls separately identify the benefit of finite-precision Gram-state repair. Kernel benchmarks show that compact GMBI graphs can reduce polar-call latency on large rectangular matrices under both compiled and specialized backends. Multi-seed GPT-2 Small training with Muon matches Direct PE in final validation loss while achieving modest reductions in mean step time, showing that the matrix-level gains can survive end-to-end training.

Primary area:
optimization
Keywords:
matrix polar approximation, Muon optimizer, polynomial iterations, Gram matrices, low-precision numerical linear algebra, GPU acceleration, matrix orthogonalization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nW7MOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This work introduces GMBI, a polynomial state-space parameterization for approximating the orthogonal factor in matrix polar decomposition for the Muon optimizer. Unlike traditional Gram iteration, GMBI lets you learn the transition rule for the auxiliary state and the accumulation rule for the output separately. It also uses mid-way restarts and staged fitting, and calibrates using the first-half output from actual FP16 computation

Critical strengths

The problem addressed is practical and relevant to real-world model training.

Critical weaknesses
  1. The end-to-end gain they show is pretty small, and it's honestly unclear if it matters in practice. In Table 4, the four-layer restarted variant only brings mean validation loss down from 3.280700 to 3.279417 compared to Direct PE — that's about 0.128% lower perplexity. Its mean step-time reduction is only around 0.15%, and the paired timing interval even includes zero.

  2. The way it's written makes the contribution and how it actually works way harder to judge than it needs to be.

Questions for authors
  1. How was the four-layer restarted Variant selected for the downstream conclusion?

  2. Can the existing results more clearly separate the effect of decoupling from the effect of the phasewise fitting recipe?

Reviewer_o8ELOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GMBI removes the constraint g_i(s) = s f_i(s)² that ties state propagation to output accumulation in Gram-form polynomial iterations for the matrix polar factor, so the two maps can be learned separately under a shared budget of matrix intermediates. It adds an FP16 full restart and fits coefficients phasewise with robust and Muon-aware calibration. Five-layer configurations improve approximation accuracy, a four-layer restart configuration attains the lowest mean GPT-2 Small validation loss among nine methods over six matched seeds, and a three-layer configuration has the lowest operator latency.

Critical strengths

S1. The decoupling is a clear contribution with a targeted control. Gram consistency forces g(s) = s f(s)², and GMBI removes that tie while sharing S, S² and H. The comparison against a refitted tied baseline at the same depth, restart position and multiplication budget is the right control for the parameterization (§3.1, Table 1), and in exact arithmetic the gain is real: Variant T5 R3 dominates five-layer Polar Express on small-singular-value gain.

S2. The statistics are handled carefully and the reporting is candid. The training analysis uses six matched seeds with identical step-1 losses, so the design is genuinely paired, and the 108 shipped logs reproduce the reported tables and paired p-values exactly. The paper discloses its own BF16 failures and timing intervals that span zero.

Critical weaknesses

W1. The Muon training evidence runs against the approximation narrative, and the practical benefit is narrow. The best configuration differs for accuracy, latency and training. The training winner, Variant T4 R3 (3.279417), approximates the polar factor worse than Direct PE on 4 of 5 Table 2 groups (Small 0.1965 vs. 0.1652), while the paper's headline approximation recipe, Basic T5 R3, trains worse than Direct PE (3.282583 vs. 3.280700, worse in all six seeds) — an effect visible only in Table 4 and never discussed; across these methods, approximation error and final loss are anti-ordered. The reported improvement is about 0.128% in perplexity, chosen post hoc among five candidates with one-sided p = 0.0484 / 0.0552, and under two-sided Holm correction the most significant effects are harms. All nine methods also share a single Muon learning rate that the paper never states.

W2. The efficiency claim is not a Pareto claim, and the approximation gain is protocol-dependent. App. B compares the four-layer Basic graph (2ρ + 13) against Direct PE (10ρ + 5), the weakest realization, whereas the relevant prior art is Gram PE with restart, against which the restarted GMBI graphs save at most two square GEMMs. Measured against the faster PE realization per shape, T4 R3 is faster on only 1 of 6 shapes and the accurate five-layer graphs are 12–35% slower than Gram PE. On attribution, Basic beats the tied control by 15.6% / 16.4% only under the Muon-aware recipe, which leaves that control worse than its own robust fit; under matched robust-only fitting the reductions fall to 1.0–1.6%, with no fit-to-fit variance reported.

Questions for authors
  1. Which coefficient set does Basic T5 in Table 4 use, and how does it correspond to the final configurations in Tables 1–3? Please report two-sided paired tests for all five candidates, and explain why benchmark.py uses coefficients differing from the shipped final set and why the Small training entry point supports only the two learned-tied methods.
  2. Based on the present results, which single configuration do the authors recommend, what verified advantage does it have at matched compute budget, and how should the reversed ordering between matrix accuracy and downstream training be understood?
  3. With a learning-rate sweep (or update-RMS normalization) and seeds that also vary data order, does the T4 R3 gain survive? Where does each graph lie on a per-shape accuracy–latency front against PE and Gram PE at T = 3, 4, 6?
  4. Under matched robust-only fitting, with a parameter-matched tied control and fit-to-fit variance, how large is the decoupling gain? How do you reconcile Table 2 with the App. E FP16 screen, where Variant T5 R3 (0.073344) and T4 R3 (0.075019) are worse than Direct PE (0.072358)?
Reviewer_e9hROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies matrix polar approximation — the orthogonalization primitive Muon applies to momentum matrices — under a restricted matrix-computation budget and FP16 arithmetic. It introduces GMBI (Gram-matrix-based iterations), a "decoupled" polynomial state-space parameterization: each layer is written as a state transition S_i = g_i(S_{i-1}) and an output update Q_i = f_i(S_{i-1})Q_{i-1}, with g and f learned separately rather than tied by the Gram-consistency relation g_i(s)=s·f_i(s)² that a Gram Newton–Schulz / Polar Express realization imposes, while sharing the square-state intermediates S, S² and H (§1.3 p.2; §2 p.3; §3.1 p.4). Two recurrences are defined — Basic and a Variant with extra scalar reuse paths d_i, e_i (§3, p.4) — together with a full restart before layer 3 that rebuilds a true Gram state from the materialized prefix output (§3.3, p.4). Coefficients are fit phasewise (freeze layers 1–2, then fit 3–5) under two protocols, robust-only and Muon-aware (§3.4, p.5). Validation covers FP16 singular-value-MSE comparisons against a refitted tied control and nine frozen methods (Tables 1–2, p.6–7), FP64 polar-factor audits (§J.4, p.20–21), H100 isolated-call latency across six shapes (Table 3, p.8), and a six-seed GPT-2 Small Muon-training comparison with a paired statistical analysis (Table 4 §4.4 p.8–9; §K p.21–22).

Critical strengths

The experimental design is clear, the statistical treatment is reliable, and the conclusions are honest. This is consequential because it lets the reader take the paper's modest result at face value rather than discount it for overclaiming. The training comparison uses six fixed, published seeds and reports mean±sample s.d. (§4.4, Table 4, p.8–9); the downstream analysis uses paired t-tests with Holm adjustment and two-sided 95% CIs, and is explicitly flagged as "post hoc and provides supportive rather than confirmatory evidence" (§K, p.22). The paper itself discloses that "the nominal paired timing intervals span zero for both compiled-PE comparisons" and ties the small gaps to limited Amdahl headroom (§4.4, p.9); that the worst polar-factor error is still ≈0.9030 on a near-low-rank input (§4.2 p.8; §J.4 p.21); that "the broad unseen-spectrum ranking depends on the chosen error metric" (§4.1 p.7; §5 p.9); that the 92-case aggregate is "descriptive, not a preregistered primary endpoint" (§J.3, p.20); and that stability is empirical and "not a proof for arbitrary inputs" (§D, p.13). This candor makes the evidence trustworthy.

Critical weaknesses

W1 — Improvements over the strongest baseline are statistically insignificant, negligible in magnitude, and limited in scale. In Table 4 (GPT-2 Small, six seeds; §4.4, p.9) the best method, GMBI Variant (4, full R3), reaches validation loss 3.279417±0.000527. The strongest baseline is Gram PE (5, full R3) at 3.280550±0.000894 — the lowest-loss non-GMBI row. Against it the Variant wins only five of six seeds (§K, Table 16, p.21) and the paper's own Holm-adjusted paired-t gives p = 0.0552, above 0.05; only the Direct PE comparison is significant (p = 0.0484, §K, p.22). The effect is tiny: the nine-method loss spread is ≈3.2794–3.2826 and step times 550.000–551.627 ms, the paired timing intervals "span zero for both compiled-PE comparisons" (§4.4, p.9), and the headline downstream gain is ≈0.128% lower perplexity relative to Direct PE only (§K, p.22). All training evidence is one model — GPT-2 Small, 12 layers, width 768, 5100 steps (§4.4, p.8). GMBI's reason to exist is a better polar map inside Muon, yet against the closest existing Gram-state baseline that benefit is not statistically distinguishable. These are limitations the paper itself reports (the p-values, the timing intervals, and the single-model scope are all stated in §4.4 and §K); the concern is the magnitude and scale of the demonstrated benefit, not concealment or a methodological error.

Questions for authors
  1. The paper states no numbered theorem, proposition, or lemma; §D says stability is empirical and "not a proof for arbitrary inputs" (p.13); and §1.3 notes the final map is still an odd polynomial X↦q(XXᵀ)X, so the function class is not enlarged (p.2). What, if anything, can be guaranteed about convergence or stability of the decoupled iteration on spectra outside those used for fitting?
  2. On unseen spectra the FP64 polar-factor metric does not favour the final recipe: Basic robust joint attains the lowest mean (0.1998) versus 0.2136 for the final Basic aware phasewise recipe (Table 14, §J.4, p.20), and you state the unseen ranking is metric-dependent (§4.1, p.7). Which error metric should govern the deployment choice, and is the final recipe the right default given this?
  3. The downstream training evidence is a single small model (GPT-2 Small), with a ≈0.128% perplexity endpoint gain relative to Direct PE (§K, p.22) and paired timing intervals that span zero (§4.4, p.9). What in the present evidence supports reading this as a downstream benefit of the method at the scales where Muon is actually used, rather than an effect within run-to-run noise at one model size?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.