acceptodds
Under review as a conference paper at ICLR 2027

SoftServe: A Scalable Quasi-Newton Method for Deep Learning

Abstract

Quasi-Newton (QN) methods have long been among the most effective methods for large-scale unconstrained convex optimization. Two obstacles have limited their use in deep learning: non-convexity and enormous parameter sizes. We introduce SoftServe, a family of QN methods designed to overcome these obstacles without line searches or ad hoc curvature corrections. SoftServe derives positive-definite curvature estimates from the variational objective of Berglund et al. (2025), even in the presence of negative curvature. We develop diagonal and Kronecker-factored variants that preserve positive definiteness by construction and scale to massive neural networks. Finally, SoftServe relies on the stable coupled Newton-Schulz iteration for the required matrix operations, replacing costly matrix decompositions with GPU-friendly matrix multiplications. SoftServe excels on problems that are severely ill-conditioned, including tasks such as recurrent networks, deep autoencoders, physics-informed neural networks, and a 136M-parameter physics-informed diffusion model, often achieving lower losses than established baselines including Adam, Muon, and SOAP.

Primary area:
optimization
Keywords:
quasi-Newton methods, second-order optimization, nonconvex optimization, stochastic optimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_KnftOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SoftServe, a scalable quasi-Newton optimizer for deep learning. Based on the soft secant formulation of Berglund et al. (2025), the authors develop Kronecker-factored positive-definite preconditioners for scalable training in deep learning. Specifically, it solves equations alternatively and uses GPU-friendly Newton-Schulz iterations. The practical algorithm further incorporates momentum, normalized updates, and periodic curvature estimation. Experiments demonstrate promising performance on several ill-conditioned tasks.

Critical strengths
  1. The motivation of this paper is promising. Extending the soft quasi-Newton method to diagonal or Kronecker-structured inverse-Hessian approximations saves memory and is friendly for practical implementation. This makes quasi-Newton methods possible for large model training.
  2. The combination of Kronecker factorization, alternating matrix updates and Newton–Schulz iterations is a GPU-friendly design.
  3. Experiments show that SoftServe-Kron demonstrates good performance on several ill-conditioned tasks, including recurrent networks, physics-informed neural networks, and diffusion models.
Critical weaknesses
  1. SoftServe is closely related to existing PSGD methods (Li, 2018; Li, 2019). In particular, expanding SoftServe's soft secant penalty (Eq. 5) yields the same fundamental curvature-fitting terms used in PSGD, namely s⊤H−1s+y⊤Hys^\top H^{-1}s + y^\top Hy, with SoftServe additionally introducing a log-determinant proximity regularizer. Moreover, prior PSGD work has already explored positive-definite Kronecker-factored preconditioners for stochastic non-convex optimization (Li, 2019). While SoftServe uses a different update rule, the paper does not sufficiently clarify the benefits of this formulation over existing PSGD approaches. Adding a detailed discussion would be helpful. More importantly, PSGD-Kron is missing from the experimental comparisons.
  2. Beyond the proposed curvature estimation in Section 3, SoftServe incorporates several practical tricks, particularly normalized preconditioned updates and Kronecker-factor rebalancing. The motivation and effectiveness of these are not sufficiently discussed. In particular, baselines (L-BFGS, K-BFGS, K-FAC) can use normalized updates as well, or SoftServe can use an unnormalized version. Overall, it is unclear whether the improvement of SoftServe is from the proposed curvature estimation or these practical tricks.
  3. The paper claims to extend quasi-Newton methods to large-scale deep learning, but the strongest results focus on specialized ill-conditioned tasks. The only language-modeling experiment in Section 6.5 uses a small 16M model, and SoftServe underperforms Muon and SOAP. Evaluations on more representative codebases would be helpful to see the effectiveness of SoftServe, such as cifar10-airbench [1] and modded-nanoGPT [2].

[1] https://github.com/KellerJordan/cifar10-airbench

[2] https://github.com/kellerjordan/modded-nanogpt

Questions for authors
  1. Could the authors clarify the advantages of SoftServe's log-determinant regularization and alternating matrix-equation updates compared with PSGD/PSGD-Kron? A detailed discussion and a direct experimental comparison with PSGD-Kron would be helpful.
  2. Could the authors provide ablations to isolate the effects of update normalization and Kronecker-factor rebalancing? In particular, how does SoftServe perform without normalization, and would applying similar normalization to K-FAC or K-BFGS change the conclusions?
  3. Could the authors evaluate SoftServe on more representative deep learning codebases?
6 comments

Thank you for your helpful feedback; our responses below follow the W (Weaknesses) and Q (Questions) numbering in your review.

W1 / Q1: Relation to PSGD and direct comparisons

We agree here with the reviewer. The connection to PSGD is interesting, and we will include the following detailed discussion in our revision.

Our secant penalty is equivalent to the curvature fitting term (Criteria 3 in Li, 2018) of PSGD. In particular, expanding our secant penalty we have

∥s−Hy∥H−12=s⊤H−1s+y⊤Hy+constants\| s - Hy\|_{H^{-1}}^2 = s^{\top}H^{-1}s + y^{\top}Hy + \text{constants} The right hand side is now criteria 3 in Li, 2018. The difference is in how this penalty is used in PSGD and SoftServe.

PSGD updates the preconditioner by introducing a Cholesky factorization, and applying a step of multiplicative gradient descent to the factors. Thus their method relies on linearizing (in multiplicative coordinates) this penalty. In this sense, they apply a type of first order method for updating the preconditioner.

In contrast, SoftServe avoids first-order approximations by directly minimizing the penalty together with a log-determinant regularizer. This makes SoftServe closely related to proximal point methods, which uses the full objective rather than a local approximation. When an efficient closed-form update is available, proximal point methods are generally faster and easier to tune as compared to first-order alternatives [Parikh & Boyd, 2014].

We also do agree with the reviewer that in both (Li, 2018) and (Li, 2019) they consider Kronecker factorizations, and as such, arrive at methods that are comparable in cost to SoftServe-Kron.

As such, we have now added direct comparisons using the official Kronecker-factored PSGD implementation with momentum whitening (https://github.com/lixilinx/psgd_torch). We swept LR separately for each method and normalization setting, testing six rates for each PSGD setting. PSGD otherwise uses its official defaults.

The table also addresses your normalization question. Norm denotes the update normalization used in SoftServe.

The numbers in the table are the training losses, averaged over the last five logged values, at equal gradient budgets (including SoftServe's secant replays).

TaskSoftServe-Kron (default; with Norm)PSGD (with Norm)SoftServe-Kron (without Norm)PSGD (default; without Norm)
MNIST autoencoder52.334All six LRs failed52.97053.006
RNN Adding3.55×10⁻⁶3.95×10⁻³1.10×10⁻⁴1.28×10⁻³
Stochastic Wave1.34×10⁻⁴1.21×10⁻²1.16×10⁻²1.29×10⁻²
Stochastic Convection2.26×10⁻⁷4.37×10⁻⁵4.57×10⁻⁵1.42×10⁻⁴
Stochastic Reaction7.53×10⁻⁸2.44×10⁻⁷7.30×10⁻⁷1.22×10⁻⁷

Comparing SoftServe's standard normalized update with PSGD's original update, SoftServe has lower training loss on all five tasks. Without normalization on either method, MNIST is nearly tied, and PSGD achieves a lower loss on Reaction.

We have also added convergence curves comparing SoftServe-Kron and PSGD with their standard updates in Appendix F.4 (Figure F.5), with secant replays included in the gradient counts. The figure can also be viewed directly at anonymous repository.

[Parikh&Boyd2014] Neal Parikh and Stephen Boyd (2014). Proximal Algorithms. Foundations and Trends in Optimization, 1(3), 127–239.

W2 / Q2: Normalization and factor rebalancing

We agree with the reviewer, and now have added more experiments testing the effect of normalization. To test the effect of normalization, we switch normalization on and off on SoftServe, K-FAC and K-BFGS(L).

For these new experiments, we swept LR separately for each method with normalization on and off. As in W1, we report MNIST training losses averaged over the last five logged values at 6,000 gradient evaluations.

MethodNormalization OFFNormalization ON
SoftServe-Kron52.97052.334
K-FAC53.65652.980
K-BFGS(L)55.72553.625

Normalization improves the tuned result for all three methods, and SoftServe has the lowest loss in both columns.

Rebalancing has a different role. Replacing (L,R)(L,R) by (L/c,cR)(L/c,cR), with c>0c>0, leaves their Kronecker product unchanged in exact arithmetic; it only balances the sizes of the two factors. In the separate rebalancing ablation, turning it off changes MNIST loss and RNN training MSE only very slightly.

The paper also includes SoftServe-I, which fixes H=IH=I while retaining update normalization. Its update is therefore a normalized momentum direction without learned curvature. Comparing it with SoftServe-Kron tests whether learning the metric provides benefits beyond normalization alone.

W3 / Q3: Broader deep-learning evaluation

We have now run both suggested codebases cifar10-airbench and modded-nanoGPT, comparing SoftServe-Kron with Adam(W) and Muon. Airbench uses a 1.97M-parameter CNN trained for eight epochs on CIFAR-10. The modded-nanoGPT experiment uses a 162M-parameter model trained for 3,250 updates, or 1.70B tokens. We tune LR separately for each method.

The table summarizes the best completed full-length runs selected by validation loss.

BenchmarkAdam(W)MuonSoftServe-Kron
CIFAR-10 Airbench: test accuracy (%) ↑88.5892.8790.11
162M modded-nanoGPT: validation NLL ↓3.42843.27823.4042

For Airbench, we reserve 5,000 training examples for validation and use FP32 training. Test accuracy is measured without test-time augmentation for all methods. GPT uses a single RTX PRO 6000 with gradient accumulation. Within each task, the methods share the architecture, data, parameter-update budget, and fallback optimizer.

SoftServe improves on Adam(W) in these runs, but Muon remains better on both tasks. Our main paper also reports that SoftServe trails Muon and SOAP on language modeling. Its strengths are clearer on physics-informed problems, where curvature-aware optimization is particularly relevant. We observe these benefits from small PINNs to a 136M-parameter physics-informed diffusion model, making such applications a natural use case for SoftServe.

Thanks for the detailed response. I have two remaining questions.

  1. The new experiments use momentum-whitening PSGD-Kron rather than Hessian-fitting PSGD-Kron (KronNewton in the PSGD repo), which is more directly related to SoftServe's curvature-fitting objective. Could the authors directly compare with Hessian-fitting PSGD-Kron?

  2. The PSGD comparisons report only training losses in the response to W1 and W2. Could the authors also provide held-out performance to verify whether the improvements translate into better task performance?

Reviewer_FjEcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The presents SoftServe which builds on the soft-secant quasi-Newton formulation of Berglund et al. (2025), and it replaces the dense inverse-Hessian estimate with diagonal or Kronecker-factored structure. The diagonal rule is closed-form; Kronecker factors are updated through alternating quadratic matrix equations, approximately solved by GPU-friendly Newton–Schulz iterations. The experiments range from RNNs and autoencoders to physics- informed learning, a 136M-parameter diffusion model, and a small language model.

Critical strengths

The method has a clear mathematical reason for remaining well behaved when a secant pair has negative curvature: positive definiteness is built into the exact variational update, not enforced by rejecting problematic pairs. The structured derivation is easy to follow, and the implementationis sufficiently detailed to be useful. The evaluation is also broader than a single favorable task: the authors include strong optimizer baselines, count extra gradient evaluations, report three-seed results, and test an identity-preconditioner control. I especially appreciate the candid language-model result, where SoftServe improves on AdamW but does not beat Muon or SOAP. That helps put the strong physics-informed results in context.

Critical weaknesses

The biggest missing piece for me is the distinction between the exact positive-definite updates and their numerical realization. SoftServe-Kron actually uses a finite number of FP32 Newton–Schulz iterations and a subtractive Gram formula. The paper monitors solver residuals, but I could not find a guarantee or an explicit fallback ensuring that the computed factors remain positive definite in difficult cases. This matters because Appendix F reports reduced solver accuracy under severe conditioning and sensitivity to large secant penalties.

I would also be cautious with claims of general scalability. The refresh cost is still cubic in the individual matrix dimensions (Table D.1), and the large-model experiments use a mixture of Kronecker and simpler parameter updates. Finally, the paper does not establish convergence for the actual normalized stochastic optimizer, so the mathematical properties of the curvature update and the end-to-end training guarantees should stay separate.

Questions for authors
  1. Were the minimum eigenvalues of the computed Kronecker factors tracked during training? What happens if roundoff causes a factor to lose positive definiteness?

  2. Could you provide peak optimizer memory for the 136M run and a controlled comparison against a similarly normalized structured/PSGD-style alternative?

3 comments

Thank you for your helpful feedback; our responses below follow the W (Weaknesses) and Q (Questions) numbering in your review.

W1 & Q1: Eigenvalue monitoring and numerical fallback

This is a good question. We did not however monitor minimum eigenvalues at every refresh in the original experiments, since this is computationally expensive, not to mention estimating smallest eigenvalues is numerically unstable.

In an audit of the original implementation on RNN Adding, we checked both proposed Kronecker factors by FP64 Cholesky at every refresh. At λ=999\lambda=999, the setting used in our reported RNN experiment, all 7,272 factor pairs passed. At the larger value λ=9999\lambda=9999, some pairs failed the check, and training terminated after 7,328 gradient evaluations.

To address the finite-iteration issue, we developed an inverse-root Gram formulation that preserves positive definiteness for any finite number of NS iterations in exact arithmetic. It uses the same two NS routines, with no additional square-root solve. Our common response, “New more numerically stable factor estimation for SoftServe-Kron,” provides the derivation and separate synthetic and RNN comparisons with the original solver, including the remaining effects of floating-point rounding.

W2: Scalability

The reviewer is correct here, and we will adjust our claims about general scalability. To be more precise, SoftServe-Kron is as scalable as other kronecker based methods such a Shampoo and KFAC. But all of these methods do incur a higher O(m2+n2)O(m^2+n^2) storage and O(m2n+mn2)O(m^2n+mn^2) step cost as compared to the O(mn)O(mn) storage and O(mn)O(mn) step cost of Adam.

Though one innovation we introduce for such Kronecker based methods, is the use of Newton–Schulz type iterations, which are much more GPU-friendly (since they need only matrix products). Where-as the current code for Distributed Shampoo, KLShampoo and KFAC mostly rely on direct solvers (such as SVD decompositions) which do not scale well on GPU.

W3: Scope of Theory

We agree that establishing convergence guarantees for the full normalized stochastic optimizer is an important direction for future work. Our theoretical results concern the curvature updates, namely their positive definiteness and the decrease of the soft quasi-Newton objective under alternating minimization. We explicitly distinguish these from end-to-end convergence guarantees in Section 7, and identity that this lack of a convergence theory is an open question. We will clarify this distinction further in the revision.

Q2: Memory, time, and PSGD comparison

We have added the official Kronecker-factored PSGD implementation (https://github.com/lixilinx/psgd_torch) as a baseline.

For this comparison, we used the 136M physics-informed diffusion model(PIDM) already evaluated in the paper, with the same mechanics dataset of 30,000 optimized structures.

We ran Adam, SOAP, SoftServe-Kron, and PSGD for 220 gradient evaluations each, using batch size 16 on RTX PRO 6000 Blackwell GPU. PSGD uses the official momentum-whitening implementation. The budget includes SoftServe's additional secant gradients. These short runs measure peak GPU memory and training time; we did not repeat the full training experiment.

MethodPeak GPU memory (GiB)Training time (s)
Adam22.0592.1
SOAP24.00339.6
SoftServe-Kron24.48143.3
PSGD22.04547.3

The peak includes the model, activations, optimizer, and evaluation; it is not an optimizer-only peak. Kron's stored optimizer tensors occupy 2.43 GiB, excluding replay buffers. The timings include setup and training but leave out evaluation.

SoftServe-Kron's total peak memory is close to SOAP's in this check. The larger timing difference with PSGD needs some context: PSGD keeps its official defaults, including BF16 factors and a refresh at every step, while Kron refreshes every 10 steps. Both use the same matrix blocks and Adam fallback. These numbers therefore describe the cost of these implementations, not how quickly they reach the same loss. No extra normalization was added to PSGD for this memory/time check.

We also compared SoftServe-Kron with the official PSGD implementation on our smaller tasks. Each method has its own LR sweep, with six rates tested for PSGD on each task. We use SoftServe's standard update and reuse its completed tuning results. PSGD keeps its original update and default settings apart from LR.

We compare the selected settings at the same gradient budget for each task, counting secant replays. The table reports training losses, averaged over the last five logged values.

TaskSoftServe-KronPSGD
MNIST autoencoder52.33453.006
RNN Adding3.55×10⁻⁶1.28×10⁻³
Stochastic Wave1.34×10⁻⁴1.29×10⁻²
Stochastic Convection2.26×10⁻⁷1.42×10⁻⁴
Stochastic Reaction7.53×10⁻⁸1.22×10⁻⁷

SoftServe-Kron reaches lower training loss on all five tasks in this comparison. The last-five averages use each run's logging interval. We will include the complete LR sweep of PSGD in the final paper as well.

We have also added convergence curves comparing SoftServe-Kron and PSGD with their standard updates in Appendix F.4 (Figure F.5), with secant replays included in the gradient counts. The figure can also be viewed directly at anonymous repository.

Thank you for the response which answers my questions and concerns. After reading the paper again, I think my rating of the paper is fair.

Reviewer_M2QUOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The primary contribution of this work is a precise diagonal update, as well as alternating Kronecker-factor updates that have a single positive-definite solution under these conditions. These types of matrix operations are made feasible through a Gram formulation and GPU-friendly Newton-Schulz iterations.

In addition to these estimates, the training algorithm uses normalized momentum updates and periodically updates the curvature using the difference in gradients calculated with the same stochastic input.

Critical strengths
  1. The paper evaluates Newton-Schulz accuracy for various matrix condition numbers and numbers of iterations, as well as how solver accuracy influences training. In practice, the algorithm employs approximate matrix operations. These experiments determine whether the implementation accurately represents the derived updates and identify numerical limitations.
  2. SoftServe-Kron achieves the lowest final training loss in five of six small PINN settings, substantially improves RNN Adding's accuracy, and scales to a 136M-parameter diffusion model. Comparisons count additional curvature-gradient evaluations, use three fresh evaluation seeds, and include an identity-preconditioner ablation. These controls strengthen the evidence that learned curvature provides practical benefits, making the empirical results a consequential reason to support acceptance within the demonstrated task scope.
Critical weaknesses
  1. Robustness studies are concentrated on small benchmarks. The detailed ablation of penalty, solver accuracy, and refresh frequency was done on both RNN Adding and MNIST. These studies found sensitivity to the choice of λ\lambda; for instance, λ=9999\lambda=9999 resulted in all three RNN evaluation seeds failing. If there is evidence that appropriate choices can be transferred to larger models, this will provide additional support for the use of such methods in practice.

  2. The central positive-definiteness guarantee is not established for the implemented approximation. The proofs cover exact matrix solutions, while Algorithm C.2 uses finitely many Newton–Schulz iterations and returns a matrix of the form V−4EE⊤V-4EE^\top. Symmetrizing the matrix will not guarantee positive definiteness. And there is no error bound provided in the paper to show that the approximate update will maintain positive definiteness.

  3. Appendix E.7, on the 16M-parameter language model, the validation NLL is 4.661, worse than Muon’s 4.440 and SOAP’s 4.416, although better than AdamW’s 4.786.

Questions for authors

Refer to the weaknesses:

  1. How should practitioners select λ\lambda, the Newton–Schulz iteration counts, and the refresh interval for a new task? Can the authors provide evidence that suitable settings transfer to larger models, or characterize the tuning required?

  2. Can the authors explain the second point of weakness?

  3. Can the authors explain the results on the 16M-parameter language model, validation NLL is 4.661, worse than Muon’s 4.440 and SOAP’s 4.416, although better than AdamW’s 4.786?

2 comments

Thank you for your helpful feedback; our responses below follow the W (Weaknesses) and Q (Questions) numbering in your review.

W1 / Q1: Hyperparameter selection and transfer

For a new task, we suggest starting with K=10, λ=9, and 18 root/10 inverse Newton–Schulz iterations, then tuning the learning rate.

  • Refresh interval KK. Smaller KK gives fresher curvature estimates at greater cost, analogous to the preconditioner refresh trade-off in SOAP. In stochastic training, replay adds approximately 1/K1/K gradient evaluations per parameter update. K=10K=10 therefore adds about 10%. For small networks where more frequent adaptation is affordable, K=5K=5 is worth testing; it gave the lowest mean RNN validation MSE in our refresh ablation.

  • Penalty λ\lambda. We suggest the rule of thumb λ=9\lambda=9 for practitioners. With regard to the sensitivity to λ\lambda, though for very large values such as λ=9999\lambda=9999 we found some runs may fail on the RNN adding problem, we found no such issues when using λ=9\lambda=9. As for evidence of how this rule of thumb transfers to larger models, we used λ=9\lambda=9 on our larger 136M PIDM models, and found SoftServe performed well. For the final paper, we will commit to further experiments on the sensitivity of λ\lambda if the reviewer deems necessary after the discussion.

  • Solver iterations. Appendix F.2 supports 18/10 Newton-Schulz steps as a practical default. Indeed, in Figure F.2 we tested a grid of different numbers of Newton-Schulz iterations for solving the quadratic matrix equation that defines the factors, and found that using 18/10 NewtonSchulz steps was enough to reach a relatively precise solution, across different data. We also found that increasing beyond 18/10 steps did not consistently improve the training or validation error on the RNN adding task.

The same K=10K=10 and 18/10 solver settings are used in the 136M experiment without retuning.

W2 / Q2: Exact versus computed positive definiteness

This is a good question. In the original Gram implementation, finite NS iterations and floating-point rounding can both cause a loss of positive definiteness.

To see if this can happen, we have run two new experiments, and developed a more numerically stable way to compute the factors (using the same two NS routines, with no additional square-root solve).

We tested the original Gram formulation on 96 synthetic QME problems in FP32. With 18 root and 10 inverse NS iterations, one output failed the FP64 Cholesky check.

We also audited the original implementation during RNN Adding training, checking both proposed Kronecker factors by FP64 Cholesky at every refresh. At λ=999\lambda=999, the setting used in our reported RNN experiment, all 7,272 factor pairs passed. At the larger value λ=9999\lambda=9999, some pairs failed the check, and training terminated after 7,328 gradient evaluations. These results concern the original implementation; our common response presents a separate comparison with the inverse-root formulation.

We therefore introduce the inverse-root Gram formulation, which preserves positive definiteness for any finite number of NS iterations in exact arithmetic. It uses the same two NS routines, with no additional square-root solve. Our common response, “New, more numerically stable factor estimation for SoftServe-Kron,” gives the derivation and numerical comparisons.

W3 / Q3: Language-model results

Our explanation here for why SoftServe does not outperform Muon or SOAP on the language model, is that the loss is not ill conditioned. SoftServe, and other quasi-Newton methods, are competitive on problems which are ill conditioned. We choose to show this example for training an LLM just to highlight that SoftServe is not always the most competitive method.

Thanks for the author's detailed responses to all reviews. My questions have been answered, and I will increase my score to 4.

Comments on the paper

1 comment
AuthorsOpenReview ↗

New more numerically stable factor estimation for SoftServe-Kron

New, more numerically stable factor estimation for SoftServe-Kron

Given Reviewer M2QU and FjEc's questions, we have now revisited how to compute the factors, and can now compute the factors in a more numerically stable way that preserves positive definiteness, even with a finite number of Newton–Schulz (NS) iterations.

Define C=F⊤VF+14IC = F^\top V F + \frac14 I and use the coupled NS iteration to compute P≈C−1/2P \approx C^{-1/2} and C1/2C^{1/2}. Instead of directly approximating (12I+C1/2)−1(\frac12 I + C^{1/2})^{-1}, we use the identity (12I+C1/2)−1=C−1/2(I+12C−1/2)−1.(\tfrac12 I + C^{1/2})^{-1} = C^{-1/2} (I+\tfrac12 C^{-1/2})^{-1}. We then compute

W &= P\\,\mathrm{NSInverse}(I+\tfrac{1}{2}P), \\\\ Q &= V-(VFW)(VFW)^\top. \end{aligned}$$ The update can be implemented as follows (see also Algorithm C.3 of the revised manuscript) $$\begin{aligned} C &\leftarrow F^\top V F+\tfrac{1}{4}I,\\\\ (\text{unused},P) &\leftarrow \operatorname{NSRootPair}(C),\\\\ W &\leftarrow P\\,\operatorname{NSInverse}(I+\tfrac{1}{2}P),\\\\ E &\leftarrow VFW,\\\\ \text{return}\\;&\operatorname{sym}(V-EE^\top). \end{aligned}$$ **Why is this numerically stable?** With the NS initialization specified in Appendix C of the updated manuscript, the iterations underestimate the inverse square root and inverse: $$0 \prec P \preceq C^{-1/2}, \qquad 0 \prec \mathrm{NSInverse}(I+\tfrac12 P) \preceq (I+\tfrac12 P)^{-1}.$$ Appendix C proves these inequalities for the stated NS initialization in exact arithmetic. They imply $0\prec W\preceq W_* := (\frac12 I+C^{1/2})^{-1}$. Since these matrices commute, $W^2\preceq W_*^2$, and hence $$Q = V - VFW^2F^\top V \succeq V - VFW_*^2F^\top V \succ 0.$$ Thus, unlike our previous approximation, this formulation preserves positive definiteness for any finite number of NS iterations (in exact arithmetic). We refer to this new method for computing the factors as the "Inverse-root" method. We compared both this new inverse-root and our previous method on 96 synthetic problems, varying the condition number of $V$ and the scale of $F$. The table counts the number of non-SPD outputs. FP32 and FP64 refer to the solver precision. We checked each output matrix by Cholesky factorization in FP64; the table reports the number of failed checks. | Root/inverse iterations | Original FP32 | Inverse-root FP32 | Original FP64 | Inverse-root FP64 | |---|---:|---:|---:|---:| | 6/6 | 57/96 | 0/96 | 57/96 | 0/96 | | 12/10 | 30/96 | 0/96 | 30/96 | 0/96 | | 18/10 (default) | 1/96 | 6/96 | 0/96 | 0/96 | | 24/16 | 6/96 | 6/96 | 0/96 | 0/96 | The inverse-root formulation avoids the low-iteration failures in this test. FP32 rounding can still produce non-SPD outputs in the extreme cases in FP32. Where-as in FP64 our new inverse-root method always produced PSD matrices (as expected by our theory). In RNN Adding, the inverse-root formulation completed all tested runs at $\lambda=999$ and $\lambda=9999$ in FP64 without any loss of positive definiteness. In FP32, rounding errors still produced non-SPD factors at $\lambda=9999$. However, at $\lambda=999$, the value selected for our reported RNN experiment, all 7,272 factor pairs in each run remained positive definite in both precisions (FP32 and FP64). Thus, these numerical issues were not observed at the settings of our reported RNN results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.