- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper computes , , for Shampoo- and Muon-type optimizers with a rational approximant with positive shifts and weights. Coefficients come from Gauss-Jacobi quadrature of a real-axis Stieltjes integral plus a log-error fit, or from Zolotarev's construction for . The main contribution is a GPU solver for the shifted SPD systems: blocked Cholesky with fp32 diagonal blocks and compensated fp16 GEMMs, then recursive triangular inversion, batched across poles. Three propositions bound approximation, solve, and drift errors. On an RTX 5090 the solver is 3.7-4.1x faster than cuSOLVER at higher error, and 4-pole inverse fourth roots are 2.6-8.1x faster than 10-iteration coupled Newton. In 1B-token GPT-2 runs, Shampoo step time drops by about 24% relative to CN at similar loss, and Muon and DynMuon reach similar or slightly lower loss.
Critical strengths
-
The positive-shift construction is what makes a fast solver possible. Every pole is an SPD system, so one batched Cholesky path serves all poles and most of the flops become fp16 GEMMs. The matrix evidence for this is strong within its setting. Table 7 shows 4-pole ZPoles ahead of 10-iteration CN and DASH's NDB in both runtime and error at all six sizes when all methods use fp16 operands, and Table 9 reproduces the ranking on an RTX 4090.
-
The precision ablations make the error budget traceable. Table 4 shows that CMul matches the error of fp32 GEMMs at 1.8-3.3x their speed, Table 6 shows that the custom solver adds little error beyond the 4-pole approximation itself, and Appendix C.4 isolates the effect of dropping factor residuals. A reader can see which design choice buys the speed and what it costs in accuracy, and the claims in the text are worded to match these measurements.
Critical weaknesses
-
The matrix benchmarks do not match the training regime, and CN is run with a scaling that, by the paper's account, inflates its error. Figure 3 and Table 7 use n = 1024-12288 and one Gaussian matrix per size, processed one per call. The Shampoo runs use blocks of at most 768 (Small) or 512 (Medium, Appendix D), and the only timing I found there is n = 768 in Appendix C.4 (0.226 vs 0.466 ms). CN uses Frobenius scaling with 10 fixed iterations, which p. 17 names as one cause of its error growth. Accuracy-matched timing exists for one n = 2048 matrix (C.2). Runtime-versus-error curves at block sizes 256-768, batched over a real layer, on saved training statistics, with CN given spectral scaling and warm starts, would settle this.
-
The training results do not yet show a reliable end-to-end gain. Appendix D states one seed per configuration, and in Table 2 the Shampoo-vs-CN and Muon-vs-NS loss gaps are 0.0003-0.0021, which I would expect to fall within seed variation at this scale. At a learning rate of 0.001, CN beats ZPoles on both models (Table 14). The 24% Shampoo step-time saving is measured with a root refresh at every update, in separately timed and non-isolated runs (p. 24). Practical Shampoo configurations usually refresh roots every tens to hundreds of steps, where the root share of step time, and the saving, would be smaller. Three seeds, a refresh-interval sweep with a per-step time breakdown, and an eigendecomposition baseline would show whether the solver changes training cost in practice.
Questions for authors
-
In the Shampoo runs, are the roots of all blocks in a layer computed in one batched call or one block at a time, and is CN batched the same way? Section 3.3 says concurrent calls are untested, so I cannot tell whether the 24% step-time gap reflects the solver or the call pattern.
-
What is the 4-pole approximation error on Shampoo statistics saved from your runs, measured against an fp64 eigendecomposition, and which does the bank select for them? With a 0.01 relative ridge and low effective rank, can be much larger than in the Gaussian test, and Proposition 1 gives no guarantee at 4 poles.
-
The cuSOLVER baseline in Table 1 is a full-accuracy fp32 path, while your solver has 57-99x higher error. How does the solver compare with a mixed-precision GPU Cholesky, such as the FP16-enabled SPD solvers of Abdelfattah et al. (2020) that the paper cites?
-
For Muon, which damping and spectral interval are used when ZPoles forms , and what is the polar-factor error on real momentum matrices compared with NS? Section 3.3 says the dynamic momentum experiment picks its interval heuristically, outside the covering guarantee, and I could not find the corresponding choice for the Muon runs.
Thank you to the reviewers and Area Chair for the careful assessment. The reviewers found the solver contribution promising and identified two central questions: whether the numerical results extend to matrices encountered during training, and whether the Shampoo benefit remains meaningful under more controlled comparisons. We performed additional experiments addressing both.
1. Real training covariances and accuracy-matched CN. We benchmarked 48 saved GPT-2-Small Shampoo covariance groups from four training stages, covering attention and MLP parameters in early, middle, and late layers. Each group contains the 3 or 4 blocks of dimension 768 processed together during training. Against FP64 eigendecomposition, ZPoles with 4 poles has a relative inverse-root error of 0.073–0.547%, with median 0.205%. The measured damped condition numbers range from approximately 155 to 75,460; all selected approximation intervals cover the measured spectra.
We scanned coupled Newton (CN) across 2–24 iterations, FP16/FP32 operands, and Frobenius/power-iteration scaling. For each group, we retrospectively selected the fastest measured CN configuration whose error was no greater than ZPoles on the same input. ZPoles was 9.9–20.1× faster, median 13.8×, on an RTX 4090, including normalization and scaling. This is a cold-start, per-input comparison against the scanned configurations; validated warm starts and Medium's 512-dimensional blocks remain outside this experiment.
2. Training quality and refresh frequency. Three paired RTX 4090 Shampoo runs, each trained for approximately 1B tokens, give CN/ZPoles validation losses of 3.711085/3.710105, 3.715699/3.711700, and 3.719212/3.718644. The mean paired ZPoles-minus-CN difference is −0.001849 (sample standard deviation 0.001874). An additional RTX 5090 pair gives 3.713990/3.714418, slightly favoring CN. We therefore emphasize comparable validation loss, rather than a statistically established loss advantage.
We also completed approximately 1B-token runs at matched refresh periods, using seed 42:
The period-1 eigendecomposition run was not performed. In this recipe, less frequent refresh gives higher final loss. Separately, two controlled 100-update timing measurements per configuration, replayed in forward and reverse order from a common checkpoint on an idle RTX 4090, give:
The ZPoles saving versus CN decreases from 24.0% at period 1 to 3.4% at period 10 and 0.35% at period 100. These are CUDA-event replay measurements at one checkpoint, not full-run wall-clock times. Both methods use the same outer batching and make 96 root calls per update at period 1.
3. Prior work and scope. Thank you for identifying PRISM. We have added the citation and evaluated the released routine on the saved covariances. A two-stage FP32 PRISM composition achieves 0.00121% median inverse-fourth-root error at 10 iterations per stage, compared with 0.205% for ZPoles-4; its released calls were evaluated eagerly and sequentially across blocks. The results therefore show a meaningful accuracy difference, while its timing should not be treated as the performance of a separately optimized batched implementation.
We also clarify the limits of the theory: the approximation proposition is an existence result, and the residual bound is conditional; neither certifies the fitted coefficient bank or every mixed-precision operation. We have not completed a validated comparison with the mixed-precision MAGMA solver, a real-Muon momentum polar-error comparison, or a valid warm-start CN comparison. The momentum interval selector is heuristic, and the Shampoo coverage results do not certify it.
Taken together, the new evidence supports ZPoles as a practical solver for the tested Shampoo workload, with comparable training loss and a refresh-frequency-dependent cost advantage. It does not establish universal numerical stability or a general training-speed advantage across optimizers and workloads. We thank the reviewers for helping us clarify both the contribution and its limits.