acceptodds
Under review as a conference paper at ICLR 2027

ZPoles: Fast Fractional Matrix Powers for Optimization via Rational Approximation

Abstract

Fractional matrix powers underpin gradient preconditioning in Shampoo and momentum transforms in Muon, but their repeated computation can be expensive. Rational approximations use a small number of shifted matrix inverses, whose cost has limited their appeal for GPU training. We propose ZPoles, a method for computing fractional powers with a new mixed-precision linear solver. The solver keeps diagonal operations in higher precision and batches half-precision matrix multiplications across the shifted linear systems. On an RTX 5090 GPU, the solver is 3.7–4.1× faster than cuSOLVER, trading numerical accuracy for runtime. For matrices of size 1024×1024 to 12288×12288, ZPoles computes inverse fourth roots 2.6–8.1× faster than coupled Newton iteration at a fixed 10-iteration budget, with lower error when both use half-precision matrix-multiplication operands. In GPT-2 pretraining, ZPoles reduces observed Shampoo step time relative to coupled Newton, with lower or similar validation loss. Muon and fractional-power Muon provide further evidence of applicability to momentum transforms. These results support rational approximation as a practical computational primitive for GPU-based optimization.

Primary area:
optimization
Keywords:
fractional matrix power, orthogonalization, Muon, Newton-Schulz, rational approximation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8SvBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper computes AqA^q, −1<q<0-1<q<0, for Shampoo- and Muon-type optimizers with a rational approximant αI+∑iωi(A+τiI)−1\alpha I+\sum_i \omega_i (A+\tau_i I)^{-1} with positive shifts and weights. Coefficients come from Gauss-Jacobi quadrature of a real-axis Stieltjes integral plus a log-error fit, or from Zolotarev's construction for q=−1/2q=-1/2. The main contribution is a GPU solver for the shifted SPD systems: blocked Cholesky with fp32 diagonal blocks and compensated fp16 GEMMs, then recursive triangular inversion, batched across poles. Three propositions bound approximation, solve, and drift errors. On an RTX 5090 the solver is 3.7-4.1x faster than cuSOLVER at higher error, and 4-pole inverse fourth roots are 2.6-8.1x faster than 10-iteration coupled Newton. In 1B-token GPT-2 runs, Shampoo step time drops by about 24% relative to CN at similar loss, and Muon and DynMuon reach similar or slightly lower loss.

Critical strengths
  1. The positive-shift construction is what makes a fast solver possible. Every pole is an SPD system, so one batched Cholesky path serves all poles and most of the flops become fp16 GEMMs. The matrix evidence for this is strong within its setting. Table 7 shows 4-pole ZPoles ahead of 10-iteration CN and DASH's NDB in both runtime and error at all six sizes when all methods use fp16 operands, and Table 9 reproduces the ranking on an RTX 4090.

  2. The precision ablations make the error budget traceable. Table 4 shows that CMul matches the error of fp32 GEMMs at 1.8-3.3x their speed, Table 6 shows that the custom solver adds little error beyond the 4-pole approximation itself, and Appendix C.4 isolates the effect of dropping factor residuals. A reader can see which design choice buys the speed and what it costs in accuracy, and the claims in the text are worded to match these measurements.

Critical weaknesses
  1. The matrix benchmarks do not match the training regime, and CN is run with a scaling that, by the paper's account, inflates its error. Figure 3 and Table 7 use n = 1024-12288 and one Gaussian matrix per size, processed one per call. The Shampoo runs use blocks of at most 768 (Small) or 512 (Medium, Appendix D), and the only timing I found there is n = 768 in Appendix C.4 (0.226 vs 0.466 ms). CN uses Frobenius scaling with 10 fixed iterations, which p. 17 names as one cause of its error growth. Accuracy-matched timing exists for one n = 2048 matrix (C.2). Runtime-versus-error curves at block sizes 256-768, batched over a real layer, on saved training statistics, with CN given spectral scaling and warm starts, would settle this.

  2. The training results do not yet show a reliable end-to-end gain. Appendix D states one seed per configuration, and in Table 2 the Shampoo-vs-CN and Muon-vs-NS loss gaps are 0.0003-0.0021, which I would expect to fall within seed variation at this scale. At a learning rate of 0.001, CN beats ZPoles on both models (Table 14). The 24% Shampoo step-time saving is measured with a root refresh at every update, in separately timed and non-isolated runs (p. 24). Practical Shampoo configurations usually refresh roots every tens to hundreds of steps, where the root share of step time, and the saving, would be smaller. Three seeds, a refresh-interval sweep with a per-step time breakdown, and an eigendecomposition baseline would show whether the solver changes training cost in practice.

Questions for authors
  1. In the Shampoo runs, are the roots of all blocks in a layer computed in one batched call or one block at a time, and is CN batched the same way? Section 3.3 says concurrent calls are untested, so I cannot tell whether the 24% step-time gap reflects the solver or the call pattern.

  2. What is the 4-pole approximation error on Shampoo statistics saved from your runs, measured against an fp64 eigendecomposition, and which κˉ\bar\kappa does the bank select for them? With a 0.01 relative ridge and low effective rank, u/ϵu/\epsilon can be much larger than in the Gaussian test, and Proposition 1 gives no guarantee at 4 poles.

  3. The cuSOLVER baseline in Table 1 is a full-accuracy fp32 path, while your solver has 57-99x higher error. How does the solver compare with a mixed-precision GPU Cholesky, such as the FP16-enabled SPD solvers of Abdelfattah et al. (2020) that the paper cites?

  4. For Muon, which damping and spectral interval are used when ZPoles forms MM⊤MM^\top, and what is the polar-factor error on real momentum matrices compared with NS? Section 3.3 says the dynamic momentum experiment picks its interval heuristically, outside the covering guarantee, and I could not find the corresponding choice for the Muon runs.

Reviewer_jcEqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ZPoles, a GPU backend for approximating negative fractional matrix powers using rational approximations with positive shifts and weights. The method combines coefficient construction with batched, mixed-precision Cholesky-based inversion, and applies the resulting operators to Shampoo, Muon, and DynMuon. The paper provides approximation and perturbation results, numerical benchmarks evaluating runtime and accuracy, and GPT-2 training experiments.

Critical strengths
  1. The positive-shift rational formulation provides a natural connection between fractional matrix powers and batched shifted inversions. The paper develops this connection into a concrete GPU implementation, with attention to precision, factorization, and accumulation. Evaluating both runtime and approximation error makes the computational trade-offs reasonably clear.

  2. Table 6 compares complete ZPoles computations while keeping the rational coefficients and input processing fixed and changing the inversion backend. The reported approximately 4.9–10.2× speedups over the cuSOLVER-based implementation, with relatively small increases in final matrix-function error, provide convincing evidence that the specialized solver contributes materially to the overall result.

  3. It also shows meaningful end-to-end savings in the Shampoo experiments.

Critical weaknesses

W1. The scope of the computational comparisons remains limited. Most matrix-function benchmarks compare fixed approximation budgets on synthetic Gaussian matrices. The accuracy-matched comparison in Appendix C.2 is helpful, but covers one matrix size and input. The relative performance of CN can depend substantially on iteration count, spectral scaling, and opportunities for warm starts, so the main benchmarks do not fully characterize the accuracy–runtime trade-off against comparably tuned alternatives on matrices encountered during training.

W2. The theoretical guarantees only partially cover the implemented method. The approximation existence result does not directly certify the fitted or interpolated coefficient bank used in practice. Likewise, the residual-based error decomposition is informative, but does not by itself bound the errors produced by the complete mixed-precision implementation. The heuristic spectral interval used for the negative-power DynMuon branch introduces an additional gap between the approximation domain and the matrices encountered during training.

I view the theory as useful supporting analysis. The strongest evidence for the implemented method remains empirical, and additional diagnostics on actual training matrices would make its numerical reliability easier to assess

Questions for authors

1. Does the accuracy-runtime advantage persist on representative matrices encountered during training when compared against an appropriately tuned baselines? Appendix C.2 already provides a useful accuracy-matched comparison. A compact extension to matrices collected during training, reflecting the actual block sizes and using comparable error targets, would help establish the practical scope of the advantage. Relevant baseline choices, including iteration count, precision, scaling, and valid warm starts where applicable, should be accounted for.

2. How robust is the reported end-to-end Shampoo speedup while preserving training quality? For one representative configuration, controlled repeated timing and limited additional paired-seed evidence would help establish the reliability of the observed benefit. It would also be useful to clarify how the practical gain depends on root-refresh frequency; a focused cost breakdown or a small representative comparision would be informative. I would prioritize this validation of the substantial Shampoo improvement.

I find the computational contribution valuable, and the two questions above capture the main uncertainties limiting a stronger recommendation. If the authors convincingly address these points, I would be willing to raise my recommendation.

Reviewer_3UCkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose a new numerical method, ZPoles, for computing fractional matrix powers efficiently on GPUs, motivated by matrix function computations in optimizers like Shampoo and Muon. The main part of this method is a mixed-precision computation of the shifted inverses, based on batched blocked Cholesky, half-precision matrix multiplication, and recursive triangular inverse. The authors also show the theoretical analysis of the error bounds. The paper also empirically compares with coupled Newton (CN), Newton–Denman–Beavers (NDB), Newton–Schulz(NS), Gram-form Newton–Schulz (Gram-NS). For application, this paper also consider training experiments on GPT-2-Small and Medium, approximately one billion tokens per run to compare ZPoles, Shampoo, Muon and DynMuon.

Critical strengths

The motivation of this paper is strong and it has great applications in machine learning. One important thing I learned from the paper is that each shift should require its own factorization.

Critical weaknesses

One problem the paper has is that the simulations only use some random matrices to runtime and errors. There is no accuracy and time comparisons on actual training covariances, momentum matrices or more practical matrices. And this is no universal stability theorem for this proposed method.

Questions for authors
  1. You missed a very related work by Yang et al. 2026. This work also studied matrix function computation using an adaptive method and apply the method for Muon and Shampoo to train neural networks.

Yang et al. 2026. PRISM: Distribution-free Adaptive Computation of Matrix Functions for Accelerating Neural Network Training.

  1. What are the ωi\omega_i and τi\tau_i in (5)?

  2. Wha the specific setting of random matrices in Table 1?

  3. Do you replace independent training seeds for the same training experiment to check the uncertainty?

Comments on the paper

1 comment

Thank you to the reviewers and Area Chair for the careful assessment. The reviewers found the solver contribution promising and identified two central questions: whether the numerical results extend to matrices encountered during training, and whether the Shampoo benefit remains meaningful under more controlled comparisons. We performed additional experiments addressing both.

1. Real training covariances and accuracy-matched CN. We benchmarked 48 saved GPT-2-Small Shampoo covariance groups from four training stages, covering attention and MLP parameters in early, middle, and late layers. Each group contains the 3 or 4 blocks of dimension 768 processed together during training. Against FP64 eigendecomposition, ZPoles with 4 poles has a relative inverse-root error of 0.073–0.547%, with median 0.205%. The measured damped condition numbers range from approximately 155 to 75,460; all selected approximation intervals cover the measured spectra.

We scanned coupled Newton (CN) across 2–24 iterations, FP16/FP32 operands, and Frobenius/power-iteration scaling. For each group, we retrospectively selected the fastest measured CN configuration whose error was no greater than ZPoles on the same input. ZPoles was 9.9–20.1× faster, median 13.8×, on an RTX 4090, including normalization and scaling. This is a cold-start, per-input comparison against the scanned configurations; validated warm starts and Medium's 512-dimensional blocks remain outside this experiment.

2. Training quality and refresh frequency. Three paired RTX 4090 Shampoo runs, each trained for approximately 1B tokens, give CN/ZPoles validation losses of 3.711085/3.710105, 3.715699/3.711700, and 3.719212/3.718644. The mean paired ZPoles-minus-CN difference is −0.001849 (sample standard deviation 0.001874). An additional RTX 5090 pair gives 3.713990/3.714418, slightly favoring CN. We therefore emphasize comparable validation loss, rather than a statistically established loss advantage.

We also completed approximately 1B-token runs at matched refresh periods, using seed 42:

Root refresh periodCN lossZPoles lossEigendecomposition loss
13.711093.71011—
103.739563.732383.73209
1003.790263.789663.79100

The period-1 eigendecomposition run was not performed. In this recipe, less frequent refresh gives higher final loss. Separately, two controlled 100-update timing measurements per configuration, replayed in forward and reverse order from a common checkpoint on an idle RTX 4090, give:

Root refresh periodCN ms/updateZPoles ms/updateEigendecomposition ms/update
1394.34299.891870.82
10276.73267.36423.11
100265.44264.52280.10

The ZPoles saving versus CN decreases from 24.0% at period 1 to 3.4% at period 10 and 0.35% at period 100. These are CUDA-event replay measurements at one checkpoint, not full-run wall-clock times. Both methods use the same outer batching and make 96 root calls per update at period 1.

3. Prior work and scope. Thank you for identifying PRISM. We have added the citation and evaluated the released routine on the saved covariances. A two-stage FP32 PRISM composition achieves 0.00121% median inverse-fourth-root error at 10 iterations per stage, compared with 0.205% for ZPoles-4; its released calls were evaluated eagerly and sequentially across blocks. The results therefore show a meaningful accuracy difference, while its timing should not be treated as the performance of a separately optimized batched implementation.

We also clarify the limits of the theory: the approximation proposition is an existence result, and the residual bound is conditional; neither certifies the fitted coefficient bank or every mixed-precision operation. We have not completed a validated comparison with the mixed-precision MAGMA solver, a real-Muon momentum polar-error comparison, or a valid warm-start CN comparison. The momentum interval selector is heuristic, and the Shampoo coverage results do not certify it.

Taken together, the new evidence supports ZPoles as a practical solver for the tested Shampoo workload, with comparable training loss and a refresh-frequency-dependent cost advantage. It does not establish universal numerical stability or a general training-speed advantage across optimizers and workloads. We thank the reviewers for helping us clarify both the contribution and its limits.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.