acceptodds
Under review as a conference paper at ICLR 2027

Fast Differentiable SVD on GPU via Polar Decomposition

Abstract

We present a fully GPU-oriented SVD pipeline based on polar decomposition, motivated by iterative methods that rely solely on matrix multiplications, such as the Newton-Schulz iteration. We show that this approach enables up to a speedup compared to standard implementations. Furthermore, we derive a numerically stable backward pass for the polar decomposition and leverage it to obtain a fully differentiable SVD. Our methods are released as open-source implementations in both PyTorch and JAX.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Numerical linear algebra, polar decomposition, SVD, singular value decomposition, differentiation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_kaXMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper computes the SVD through the polar decomposition. The polar factor is found with the CANS iteration, which uses only matrix multiplications and can run in tf32/bf16. A symmetric eigendecomposition of the remaining factor then gives the SVD. For the backward pass, the paper derives a gradient formula for the polar decomposition. It reduces to a Lyapunov equation with terms 1/(σᵢ+σⱼ), so it does not need spectral gaps. Tikhonov regularization handles near rank-deficiency, and an “EVD pseudo-gradient” handles repeated eigenvalues. The method is tested on random matrices and on LLaMA-2-7B compression, with up to 2× speedup and better perplexity at 40% compression.

Critical strengths
  1. The problem is real: gradients of SVD are unstable when singular values are close or the matrix is near rank-deficient.
  2. The backward formula for the polar decomposition is clean and has full proofs. The regularized version has a convergence result (Prop. 5.2).
  3. The forward design is sensible, and Appendix B.1 shows matrix multiplication scales better than QR on new GPUs.
  4. The experiments are fairly honest: they report where cuSOLVER POLAR fails, that tf32 is much less accurate, and that CANS fp32 is slower on low-rank matrices.
Critical weaknesses
  1. The 2× speedup is conditional. It comes mainly from tf32, which gives error around 1e-3, about three orders of magnitude worse than fp32 or QR-based methods. In fp32, the gain is small and only for medium and large matrices. At n = 2¹⁴, QDWH and cuSOLVER POLAR catch up, and for small matrices CUDA Jacobi is faster.
  2. The comparison is not fully fair. The method is in JAX and the baselines are cuSOLVER. The Jacobi tolerance was set to match the accuracy of CANS tf32, which is a choice that favors the proposed method.
  3. The “no spectral gap” claim is overstated. It holds only for the polar part. The full SVD backward still goes through the eigendecomposition of H, which has 1/(σᵢ−σⱼ) terms and needs a separate regularization. The abstract and introduction can mislead readers.
  4. Theory and practice do not match well. Prop. 5.2 gives only an ε^{1/4} rate and assumes full rank. Algorithm 4 uses ε^{1/2} and ε^{1/4}, and the link to "ε = 1e-8" (Table 3) and ε = 0.1, 1, 10 (Table 4) is not explained. With ε = 1 or 10, the gradient is clearly biased, not close to the exact one.
  5. The application is narrow. There is one model (LLaMA-2-7B), one dataset (WikiText-2), and no variance over runs. Gains are small at 80% and 60% and large only at 40%, and results change a lot with ε. The source of the 1.35× speedup (forward, backward, or implementation) is not explained.

Finally, the claim about Zhang & Hu (2024) needs care. Appendix C says their formula is wrong. This should be checked carefully, since it is a strong statement.

Questions for authors
  1. What is the speedup and accuracy at fp32 against cuSOLVER, with matched implementation language?
  2. Please compare gradient error against a high-precision reference, not only the norm.
  3. How should ε be chosen, and how does it relate to the theory?
  4. Can you test more models and datasets, with repeated runs?
  5. Where does the 1.35× speedup in the LLM experiment come from?
3 comments

Thank you for taking the time to review our paper and for your positive assessment of our work. We appreciate your thoughtful comments and suggestions.

W1 / Q1 - fp32 speedup and matched implementation language

Thank you for raising these points. We revised the text to clarify which matrix sizes and precision settings yield a speedup over the baselines. The speedup reaches up to 2×2\times for medium-sized matrices (n=1024n=1024 and 20482048).

To address the concern about implementation language, we also implemented our method in CUDA and benchmarked it against the cuSOLVER baselines. The CUDA implementation reproduces the accuracy of our original implementation and achieves similar runtimes on medium and large matrices. On small matrices, it now matches or outperforms the baselines (Figure 2 in the revised version).

W2 - Jacobi tolerance

We note that we do not set Jacobi tolerance by ourselves, we use default Jacobi tolerance provided by cuSOLVER library. Moreover, CUDA Jacobi is even slower than CANS SVD (fp32) for medium and large matrices, despite achieving substantially lower accuracy. Thus, our speedup cannot be attributed to a relaxed accuracy requirement. Moreover, increasing the accuracy of Jacobi would only increase its runtime, as demonstrated in Figure 8. We have added the corresponding comments in work (green text in Appendix B.8).

W3 - Spectral gap dependence of the full SVD backward pass

We would like to clarify that our no-spectral-gap claim refers only to the polar backward pass. The EVD backward requires separate regularization, as already stated in the introduction and Section 5.2. We clarified the wording to avoid confusion in the revised manuscript.

Q2 - Gradient error against a high-precision reference

Thank you for this question. We added a direct evaluation of gradient accuracy in Table 4 (p. 9), reporting the relative Frobenius error against the analytical gradient computed in fp64. For spectral gaps from 10−210^{-2} to 10−410^{-4}, our backward pass with ε=10−8\varepsilon = 10^{-8} achieves near machine precision accuracy in fp32, with relative errors roughly two orders of magnitude smaller than those of the analytical formula evaluated in fp32.

W4 / Q3 - Connection between the theoretical regularization result and practical choice of ε\varepsilon

Proposition 5.2 states that if M~\widetilde M approximates MM with error ε\varepsilon, then our regularized backward formula approximates the exact gradient at MM with error O(ε1/4)O(\varepsilon^{1/4}). Note that the same regularization terms, ε1/2\varepsilon^{1/2} and ε1/4\varepsilon^{1/4}, appear in both the proposition and Algorithm 4.

In practice, ε\varepsilon should reflect the numerical errors arising from finite-precision arithmetic. However, we also explore larger values as a regularization hyperparameter. The improved results with ε=1\varepsilon=1 in Table 4 may be explained by accumulated numerical errors from reduced-precision computations, as well as better control of gradient norms, which can stabilize training in a complex nonconvex optimization problem. We agree that this introduces bias, but a more accurate gradient does not necessarily lead to better optimization outcomes.

W5 / Q4 / Q5 - Limited application experiments, run-to-run variance, and source of the 1.35×1.35\times speedup

Thank you for these comments. We extended our experiments to Qwen3 and report perplexity on WikiText-2, PTB, and C4 (see revised Table 5). Given the limited rebuttal timeline, we will report variance across runs in the camera-ready version.

The overall 1.35×1.35\times speedup comes from faster SVD computation in the forward pass, the main bottleneck in the compression pipeline. Our method uses CANS SVD for this step, and Dobi-SVD uses PyTorch’s default SVD routine.

W6 - Claim concerning the formula of Zhang & Hu (2024)

Thank you for raising this point. We checked Eq. (25) of Zhang & Hu (2024) before submission and revisited it in light of your comment. The matrix in Eq. (25) is symmetric, so its Moore–Penrose pseudoinverse must also be symmetric. The expression given in Eq. (25) is nonsymmetric. Appendix C discusses this discrepancy and gives an example of inconsistent gradients from the resulting backward formula (Example C.1).

Dear Reviewer, we have posted our rebuttal and would be happy to address any further questions. Could you please let us know whether our responses resolve your concerns?

Thanks for the reply from the author. I will keep my score as positive.

Reviewer_8tgKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper computes the SVD through the polar decomposition: CANS iterations produce the polar factor using only matrix multiplications (so tf32 and bf16 are usable), then a symmetric EVD of H = WT M gives the singular values and right vectors, with U = W V. The authors benchmark this on a B200 across sizes, condition numbers, ranks and rectangular shapes, derive a backward pass for the polar decomposition with Tikhonov regularization, add a pseudo-gradient for the EVD at degenerate spectra, and assemble a fully differentiable SVD that they test on a LLaMA-2-7B compression pipeline. Code is released for PyTorch and JAX.

Critical strengths

1The engineering is solid. Four matrix families are benchmarked, and Figures 3 and 4 make the hardware argument empirically - QR stops scaling after A100 while tf32 GEMM keeps improving across B200, H200, H100 and A100. Many "our method is faster" papers never establish why the comparison should look the way it does; this one does. 2Accuracy is not hidden. Tables 1 and 2 separate the ~1e-6 and ~1e-3 regimes and place the tf32 variant honestly, and the failure boundaries are reported too: CUDA POLAR breaks above κ = 1e3, CUDA JACOBI wins on small matrices. Showing the unfavourable results is to the authors' credit. 3The VJP formula in Theorem 5.1 is clean and useful: Mbar = Wbar H-1 + 2 M X with Tij = 1/(σi + σj), degenerate only when σi + σj = 0. That avoids the 1/(σi2 - σj2) pathology of the standard SVD backward, and it is directly relevant to the Muon and orthogonalization community. 4The pseudo-gradient is defined explicitly (minimum Frobenius norm subject to the non-degenerate part of the variational constraint) and the paper states plainly that rotations inside repeated-eigenvalue eigenspaces are disregarded by definition. That is more honest than quietly returning a number.

Critical weaknesses

1.The "up to 2x" in the abstract has no accuracy qualifier. The genuine 2x is at fp32 and medium n, against QDWH and CUDA POLAR; by n = 214 QDWH has caught up, and on small matrices CUDA is faster. The 10x figure belongs to the tf32 variant, whose reconstruction error is 1e-3. A reader of the abstract will come away with "2x at equal accuracy, everywhere", and neither half of that is quite true. 2.The methodological novelty is limited. SVD via the polar decomposition is the old Higham / Nakatsukasa route; what changes is that QDWH is replaced by CANS and lower precision is allowed. That is competent systems work, but a thin methodological contribution. More to the point, Figure 1b shows the bottleneck has already moved to EIGH (35-40% of runtime), so the "fully GPU-oriented pipeline" claim covers only the first half of the computation. 3.The forward and backward passes are inconsistent. The forward pass runs a finite number of CANS iterations with tolerance 1e-5 or 1e-3; the backward pass uses the analytic formula for the exact polar decomposition. The gradient is therefore not the derivative of the computation actually performed, but of an idealized map. At tf32 tolerance this could be a sizeable discrepancy, yet it is neither discussed nor measured. An unrolled-versus-analytic comparison would settle it. 4.The pseudo-gradient has no theoretical standing. Definition 5.3 picks, at a degenerate point, the smallest-Frobenius symmetric matrix satisfying a first-order constraint. It is not the derivative of any function, and it is not shown to be a Clarke subgradient or a limit of nearby gradients. If contributions from rotations within repeated eigenspaces are dropped "by definition", what is training actually optimizing, and can this yield a wrong descent direction? At a minimum, show that when the loss depends on (U, Σ, V) only through M, the pseudo-gradient equals the true gradient. 5.Table 3 works against the paper's own narrative. With τ-regularization the gradient norm matches Townsend exactly for gaps of 1e-4 and above (66, 67, 100, 690, 5500), including the already-explosive 5500; only at gaps of 1e-5 and below does it fall back to 66. So the ε = 1e-8 regularizer does nothing at 1e-4, where the backward pass is already unstable. "Controlled gradients" holds only in a very small window, and that boundary should be stated.

Questions for authors

1.Can you provide a wall-clock comparison at matched accuracy, for example reconstruction error ≤ 1e-6? Which configuration and which n does the 2x in the abstract refer to? 2.How large is the gradient difference between unrolled and analytic backward passes, and does training still converge with tf32 forward? 3.When the loss depends on (U, Σ, V) only through M, does the pseudo-gradient equal the true gradient? Proof or counterexample? 4.How does the backward pass compare numerically with torch.linalg.svd on the same matrices? 5.At gap 1e-4 the gradient norm is 5500 in Table 3. Does that mean ε = 1e-8 is too small? What is the rule for choosing ε, and why is ε = 1.0 best in the compression experiment?

3 comments

We sincerely thank you for your time and effort. We appreciate your valuable comments and feedback.

W1 / Q1 - Scope and reconstruction accuracy of the 2×2\times speedup

Thank you for raising this point. We have clarified the benchmark conditions in Section 4.1. The discussion following Figure 2 compares wall-clock times within groups of methods with comparable reconstruction accuracy. In the higher-accuracy group, fp32 CANS SVD achieves up to a 2×2\times speedup over CUDA POLAR and QDWH SVD on square matrices of size n∈1024,2048n\in{1024,2048} on an NVIDIA B200 (Figure 2a). At n=16384n=16384, QDWH SVD reaches comparable runtime.

We also implemented custom CUDA kernels that reduce the runtime of CANS SVD for small matrices (n≤64n\leq 64), as shown in the updated Figure 2.

W2 - Methodological novelty and the EIGH bottleneck

We respectfully disagree with the assessment of our method’s novelty. Adapting Higham’s approach to better suit GPUs is an important, timely, and nontrivial contribution. To the best of our knowledge, this is the first work to demonstrate competitive end-to-end SVD performance on GPUs using a matrix multiplication–based polar iteration. Our algorithm achieves state-of-the-art performance on modern GPUs. These results highlight the practical value of polar decomposition for GPU-based numerical linear algebra.

As an additional important contribution of our paper, we highlight that we derive a regularized backward pass for the polar decomposition and the SVD and prove a stability bound under perturbations for the polar backward pass. Together, these forward and backward methods yield a practical, differentiable SVD for modern GPUs. As the reviewer notes, replacing QDWH with CANS shifts the main bottleneck to the symmetric eigendecomposition (Figure 1b), and we see accelerating this step as a more challenging problem for future work.

W3 / Q2 - Consistency of the approximate forward pass and analytic backward pass. Unrolled versus analytic gradients and convergence with tf32 forward

Thank you for raising this question. With our chosen stopping tolerances, the CANS polar factors agree with the fp64 reference to within the working precision of each run (fp32 or tf32). To check how the forward approximation affects the backward pass, we repeated the gradient evaluation using the high-precision polar factor in the same analytical backward formula. The resulting gradient errors agreed up to rounding. Table 4 (see the revised version) reports the fp32 gradient errors against the analytical reference computed in fp64.

We also evaluated the configuration in which both the forward and backward passes use tf32 (Appendix B.9, Tables 13 and 14, see revised manuscript). The gradients remained bounded across the tested spectral gaps, with substantially larger errors than in fp32.

W4 / Q3 - Theoretical interpretation of the pseudo-gradient and its descent direction. Pseudo-gradient for losses depending on the SVD only through the reconstructed matrix

Thank you for the comment. When eigenvalues are repeated, the EVD is no longer unique, so the mapping we aim to differentiate is not uniquely defined, and the standard analytical backward formula becomes undefined. Moreover, standard notions of generalized gradients, such as Clarke subgradients, are not directly applicable without a well-defined, locally Lipschitz mapping.

We therefore consider only perturbations of the EVD factors that change the original matrix, excluding rotations within repeated-eigenvalue eigenspaces. Among all symmetric matrices satisfying the chain rule for these perturbations, we select the one with minimum Frobenius norm to ensure uniqueness.

Importantly, when all eigenvalues are distinct, our pseudo-gradient coincides exactly with the classical analytical gradient. This follows directly from Proposition F.1 and Theorem 5.4 in Appendix F. Thus, our definition agrees with standard differentiation in the nondegenerate case while providing a well-defined backward rule at degenerate points.

Q4 - Numerical comparison with torch.linalg.svd backward

Thank you for this question. The backward pass of torch.linalg.svd implements the analytical formulas described by Townsend [1] (see PyTorch source code). We also verified that replacing our implementation of the analytical formula with the backward pass of torch.linalg.svd gave the same results.

Table 3 compares the gradient norms from our backward pass with those obtained using Townsend formulas. Table 4 reports relative gradient errors using the same formulas evaluated in fp64 as the reference. We have clarified the connection between this analytical baseline and PyTorch’s backward pass in the revised text.

W5/Q5 - Gradient stability range and the effect of ε\varepsilon in Table 3. Rule for choosing ε\varepsilon.

Thank you for raising this point. We added a direct comparison with the analytical gradient computed in fp64 (Table 4 in the revised version) and evaluated several values of ε\varepsilon in Tables 3 and 4. At a spectral gap of 10−410^{-4}, our original choice of ε=10−8\varepsilon=10^{-8} gives an accurate gradient whose mean norm is still about 5,500. The large norm is consistent with the accuracy of this approximation because the reference gradient itself has a large norm. Increasing ε\varepsilon to 10−410^{-4} reduces the mean norm to 3,286.43{,}286.4 and raises the mean relative error to 0.400.40. This trade-off is unavoidable because a gradient with a much smaller norm must differ substantially from the reference gradient. While preparing this evaluation, we also derived an equivalent more numerically stable expression for the polar backward pass. In exact arithmetic, the new expression gives the same result as the original formula for every ε≥0\varepsilon \geq 0 (Appendix [J]).

In practice, ε\varepsilon should reflect numerical errors arising from finite-precision arithmetic. However, larger values can also be used to better control gradient norms at the cost of introducing bias. We therefore recommend tuning ε\varepsilon depending on the task. In our LLM compression experiments, ε=1\varepsilon=1 gave the best results, possibly due to accumulated errors from reduced-precision computations and improved gradient stability in a complex nonconvex optimization problem. A more accurate gradient does not necessarily lead to better optimization outcomes.

[1] Townsend J. Differentiating the singular value decomposition. – Technical Report 2016, https://j-towns.github.io/papers/svd-derivative.

Dear Reviewer, we have posted our rebuttal and would be happy to address any further questions. Could you please let us know whether our responses resolve your concerns?

Reviewer_ta73OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper utilizes recent advances in polar decomposition to speed up SVD on GPUs. The traditional method uses QR factorizations which are not well suited to the parallel environment of GPUs. Authors report up to 2x speed up which is very impressive.

Additionally a new stable backpropagation method is proposed for SVD. The merits of this differentiation scheme show themselves when singular values are equal or nearly equal.

Overall the paper makes contributions to a foundational problem for ML. The SVD is utilized heavily, so GPU computation improvement can have a big impact.

Definition 5.3 was hard to understand. Perhaps more explanation about what the optimization objective and constraints mean would help.

Critical strengths

Authors report substantial computation speed up for SVD on GPUs over SOTA. The LLM compression case study also demonstrated the benefits of stable SVD gradients as well as the computation improvement.

Critical weaknesses

The novelty is somewhat limited as CANS and the polar decomposition method of computing SVD already exist.

Table 3 is only a measurement of the magnitude of the gradient, I was not able to find a verification that the direction also matched. I was also not able to find an accompanying accuracy table for rectangular matrices Figure 2 (b) like there was for Figure 2 (a).

Questions for authors

Is the bias from the SVD gradient regularization constant ϵ\epsilon significant? Expansion on Table 3 could help answer this.

2 comments

Thank you for taking the time to review our paper and for your positive assessment of our work. We appreciate your thoughtful comments and suggestions.

W1 - Limited novelty

To the best of our knowledge, this is the first work to demonstrate competitive end-to-end SVD performance on GPUs using a matrix multiplication based polar iteration. Our algorithm builds on several classical components and achieves state-of-the-art performance on modern GPUs. These results highlight the practical value of polar decomposition for GPU based numerical linear algebra.

As an additional important contribution of our paper, we highlight that we derive a regularized backward pass for the polar decomposition and the SVD and prove a stability bound under perturbations for the polar backward pass. We show that combining these forward and backward methods yields a practical, end-to-end differentiable SVD with meaningful performance gains on modern hardware.

W2 - Accuracy of the gradient and accuracy results for rectangular matrices

Thanks for highliting this point. We added Table 4 (p. 9) in revised manuscript, which reports the relative Frobenius error against the analytical gradient computed in fp64. This metric captures differences in both direction and magnitude. Our algorithm with ε=10−8\varepsilon=10^{-8} is accurate up to a machine precision, and even more accurate than analytical formulas in fp32. While preparing this evaluation, we also derived an equivalent more numerically stable expression for the polar backward pass. In exact arithmetic, the new expression gives the same result as the original formula for every ε≥0\varepsilon \geq 0 (Appendix [J]).

To complement Figure 2(b), we added reconstruction and orthogonality errors for 65,536×4,09665{,}536 \times 4{,}096 matrices in Tables 6 and 7 (Appendix B). The reconstruction errors are of the same order as in the square experiments across the tested condition numbers, approximately 10−610^{-6} for fp32 CANS SVD and 10−310^{-3} for its tf32 variant.

Q1 - Bias introduced by the SVD gradient regularization constant ε\varepsilon

Tables 3 and 4 of the revised manuscript quantify how ε\varepsilon affects the Frobenius norm of the gradient and its relative error against the fp64 analytical reference. At a spectral gap of 10−410^{-4}, increasing ε\varepsilon from 10−810^{-8} to 10−410^{-4} reduces the mean gradient norm from 5500.05500.0 to 3286.43286.4, while the mean relative error rises from 4.1×10−64.1\times10^{-6} to 0.400.40. At a larger gap of 10−210^{-2}, both settings give a mean gradient norm of 100.0100.0, and the mean relative errors are 2.7×10−62.7\times10^{-6} and 5.5×10−55.5\times10^{-5}, respectively.

C1 - Clarification of Definition 5.3 and its optimization objective and constraints

When eigenvalues are repeated, the EVD is no longer unique, so the mapping we aim to differentiate is not even uniquely defined. Moreover, the standard analytical backward formula is also undefined in this case.

We therefore propose considering only those perturbations of the EVD factors that actually change the original matrix. Indeed, perturbations that leave the matrix unchanged merely reflect the ambiguity in the choice of eigenvectors, which we cannot control. Finally, among all matrices GG satisfying the chain rule for these perturbations, we select the symmetric one with minimum Frobenius norm to ensure uniqueness.

We added this explanation to clarify the motivation behind Definition 5.3 (see the green text below Definition 5.3).

Thank you for your response. You addressed my concerns. I therefore increased my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.