- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper computes the SVD through the polar decomposition. The polar factor is found with the CANS iteration, which uses only matrix multiplications and can run in tf32/bf16. A symmetric eigendecomposition of the remaining factor then gives the SVD. For the backward pass, the paper derives a gradient formula for the polar decomposition. It reduces to a Lyapunov equation with terms 1/(σᵢ+σⱼ), so it does not need spectral gaps. Tikhonov regularization handles near rank-deficiency, and an “EVD pseudo-gradient” handles repeated eigenvalues. The method is tested on random matrices and on LLaMA-2-7B compression, with up to 2× speedup and better perplexity at 40% compression.
Critical strengths
- The problem is real: gradients of SVD are unstable when singular values are close or the matrix is near rank-deficient.
- The backward formula for the polar decomposition is clean and has full proofs. The regularized version has a convergence result (Prop. 5.2).
- The forward design is sensible, and Appendix B.1 shows matrix multiplication scales better than QR on new GPUs.
- The experiments are fairly honest: they report where cuSOLVER POLAR fails, that tf32 is much less accurate, and that CANS fp32 is slower on low-rank matrices.
Critical weaknesses
- The 2× speedup is conditional. It comes mainly from tf32, which gives error around 1e-3, about three orders of magnitude worse than fp32 or QR-based methods. In fp32, the gain is small and only for medium and large matrices. At n = 2¹⁴, QDWH and cuSOLVER POLAR catch up, and for small matrices CUDA Jacobi is faster.
- The comparison is not fully fair. The method is in JAX and the baselines are cuSOLVER. The Jacobi tolerance was set to match the accuracy of CANS tf32, which is a choice that favors the proposed method.
- The “no spectral gap” claim is overstated. It holds only for the polar part. The full SVD backward still goes through the eigendecomposition of H, which has 1/(σᵢ−σⱼ) terms and needs a separate regularization. The abstract and introduction can mislead readers.
- Theory and practice do not match well. Prop. 5.2 gives only an ε^{1/4} rate and assumes full rank. Algorithm 4 uses ε^{1/2} and ε^{1/4}, and the link to "ε = 1e-8" (Table 3) and ε = 0.1, 1, 10 (Table 4) is not explained. With ε = 1 or 10, the gradient is clearly biased, not close to the exact one.
- The application is narrow. There is one model (LLaMA-2-7B), one dataset (WikiText-2), and no variance over runs. Gains are small at 80% and 60% and large only at 40%, and results change a lot with ε. The source of the 1.35× speedup (forward, backward, or implementation) is not explained.
Finally, the claim about Zhang & Hu (2024) needs care. Appendix C says their formula is wrong. This should be checked carefully, since it is a strong statement.
Questions for authors
- What is the speedup and accuracy at fp32 against cuSOLVER, with matched implementation language?
- Please compare gradient error against a high-precision reference, not only the norm.
- How should ε be chosen, and how does it relate to the theory?
- Can you test more models and datasets, with repeated runs?
- Where does the 1.35× speedup in the LLM experiment come from?
3 comments
Dear Reviewer, we have posted our rebuttal and would be happy to address any further questions. Could you please let us know whether our responses resolve your concerns?
Thanks for the reply from the author. I will keep my score as positive.
Thank you for taking the time to review our paper and for your positive assessment of our work. We appreciate your thoughtful comments and suggestions.
Thank you for raising these points. We revised the text to clarify which matrix sizes and precision settings yield a speedup over the baselines. The speedup reaches up to 2× for medium-sized matrices (n=1024 and 2048).
To address the concern about implementation language, we also implemented our method in CUDA and benchmarked it against the cuSOLVER baselines. The CUDA implementation reproduces the accuracy of our original implementation and achieves similar runtimes on medium and large matrices. On small matrices, it now matches or outperforms the baselines (Figure 2 in the revised version).
We note that we do not set Jacobi tolerance by ourselves, we use default Jacobi tolerance provided by cuSOLVER library. Moreover, CUDA Jacobi is even slower than CANS SVD (fp32) for medium and large matrices, despite achieving substantially lower accuracy. Thus, our speedup cannot be attributed to a relaxed accuracy requirement. Moreover, increasing the accuracy of Jacobi would only increase its runtime, as demonstrated in Figure 8. We have added the corresponding comments in work (green text in Appendix B.8).
We would like to clarify that our no-spectral-gap claim refers only to the polar backward pass. The EVD backward requires separate regularization, as already stated in the introduction and Section 5.2. We clarified the wording to avoid confusion in the revised manuscript.
Thank you for this question. We added a direct evaluation of gradient accuracy in Table 4 (p. 9), reporting the relative Frobenius error against the analytical gradient computed in fp64. For spectral gaps from 10−2 to 10−4, our backward pass with ε=10−8 achieves near machine precision accuracy in fp32, with relative errors roughly two orders of magnitude smaller than those of the analytical formula evaluated in fp32.
Proposition 5.2 states that if M approximates M with error ε, then our regularized backward formula approximates the exact gradient at M with error O(ε1/4). Note that the same regularization terms, ε1/2 and ε1/4, appear in both the proposition and Algorithm 4.
In practice, ε should reflect the numerical errors arising from finite-precision arithmetic. However, we also explore larger values as a regularization hyperparameter. The improved results with ε=1 in Table 4 may be explained by accumulated numerical errors from reduced-precision computations, as well as better control of gradient norms, which can stabilize training in a complex nonconvex optimization problem. We agree that this introduces bias, but a more accurate gradient does not necessarily lead to better optimization outcomes.
Thank you for these comments. We extended our experiments to Qwen3 and report perplexity on WikiText-2, PTB, and C4 (see revised Table 5). Given the limited rebuttal timeline, we will report variance across runs in the camera-ready version.
The overall 1.35× speedup comes from faster SVD computation in the forward pass, the main bottleneck in the compression pipeline. Our method uses CANS SVD for this step, and Dobi-SVD uses PyTorch’s default SVD routine.
Thank you for raising this point. We checked Eq. (25) of Zhang & Hu (2024) before submission and revisited it in light of your comment. The matrix in Eq. (25) is symmetric, so its Moore–Penrose pseudoinverse must also be symmetric. The expression given in Eq. (25) is nonsymmetric. Appendix C discusses this discrepancy and gives an example of inconsistent gradients from the resulting backward formula (Example C.1).