- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SoftServe, a scalable quasi-Newton optimizer for deep learning. Based on the soft secant formulation of Berglund et al. (2025), the authors develop Kronecker-factored positive-definite preconditioners for scalable training in deep learning. Specifically, it solves equations alternatively and uses GPU-friendly Newton-Schulz iterations. The practical algorithm further incorporates momentum, normalized updates, and periodic curvature estimation. Experiments demonstrate promising performance on several ill-conditioned tasks.
Critical strengths
- The motivation of this paper is promising. Extending the soft quasi-Newton method to diagonal or Kronecker-structured inverse-Hessian approximations saves memory and is friendly for practical implementation. This makes quasi-Newton methods possible for large model training.
- The combination of Kronecker factorization, alternating matrix updates and Newton–Schulz iterations is a GPU-friendly design.
- Experiments show that SoftServe-Kron demonstrates good performance on several ill-conditioned tasks, including recurrent networks, physics-informed neural networks, and diffusion models.
Critical weaknesses
- SoftServe is closely related to existing PSGD methods (Li, 2018; Li, 2019). In particular, expanding SoftServe's soft secant penalty (Eq. 5) yields the same fundamental curvature-fitting terms used in PSGD, namely , with SoftServe additionally introducing a log-determinant proximity regularizer. Moreover, prior PSGD work has already explored positive-definite Kronecker-factored preconditioners for stochastic non-convex optimization (Li, 2019). While SoftServe uses a different update rule, the paper does not sufficiently clarify the benefits of this formulation over existing PSGD approaches. Adding a detailed discussion would be helpful. More importantly, PSGD-Kron is missing from the experimental comparisons.
- Beyond the proposed curvature estimation in Section 3, SoftServe incorporates several practical tricks, particularly normalized preconditioned updates and Kronecker-factor rebalancing. The motivation and effectiveness of these are not sufficiently discussed. In particular, baselines (L-BFGS, K-BFGS, K-FAC) can use normalized updates as well, or SoftServe can use an unnormalized version. Overall, it is unclear whether the improvement of SoftServe is from the proposed curvature estimation or these practical tricks.
- The paper claims to extend quasi-Newton methods to large-scale deep learning, but the strongest results focus on specialized ill-conditioned tasks. The only language-modeling experiment in Section 6.5 uses a small 16M model, and SoftServe underperforms Muon and SOAP. Evaluations on more representative codebases would be helpful to see the effectiveness of SoftServe, such as cifar10-airbench [1] and modded-nanoGPT [2].
Questions for authors
- Could the authors clarify the advantages of SoftServe's log-determinant regularization and alternating matrix-equation updates compared with PSGD/PSGD-Kron? A detailed discussion and a direct experimental comparison with PSGD-Kron would be helpful.
- Could the authors provide ablations to isolate the effects of update normalization and Kronecker-factor rebalancing? In particular, how does SoftServe perform without normalization, and would applying similar normalization to K-FAC or K-BFGS change the conclusions?
- Could the authors evaluate SoftServe on more representative deep learning codebases?
6 comments
W3 / Q3: Broader deep-learning evaluation
We have now run both suggested codebases cifar10-airbench and modded-nanoGPT, comparing SoftServe-Kron with Adam(W) and Muon. Airbench uses a 1.97M-parameter CNN trained for eight epochs on CIFAR-10. The modded-nanoGPT experiment uses a 162M-parameter model trained for 3,250 updates, or 1.70B tokens. We tune LR separately for each method.
The table summarizes the best completed full-length runs selected by validation loss.
| Benchmark | Adam(W) | Muon | SoftServe-Kron |
|---|---|---|---|
| CIFAR-10 Airbench: test accuracy (%) ↑ | 88.58 | 92.87 | 90.11 |
| 162M modded-nanoGPT: validation NLL ↓ | 3.4284 | 3.2782 | 3.4042 |
For Airbench, we reserve 5,000 training examples for validation and use FP32 training. Test accuracy is measured without test-time augmentation for all methods. GPT uses a single RTX PRO 6000 with gradient accumulation. Within each task, the methods share the architecture, data, parameter-update budget, and fallback optimizer.
SoftServe improves on Adam(W) in these runs, but Muon remains better on both tasks. Our main paper also reports that SoftServe trails Muon and SOAP on language modeling. Its strengths are clearer on physics-informed problems, where curvature-aware optimization is particularly relevant. We observe these benefits from small PINNs to a 136M-parameter physics-informed diffusion model, making such applications a natural use case for SoftServe.
Thanks for the detailed response. I have two remaining questions.
-
The new experiments use momentum-whitening PSGD-Kron rather than Hessian-fitting PSGD-Kron (KronNewton in the PSGD repo), which is more directly related to SoftServe's curvature-fitting objective. Could the authors directly compare with Hessian-fitting PSGD-Kron?
-
The PSGD comparisons report only training losses in the response to W1 and W2. Could the authors also provide held-out performance to verify whether the improvements translate into better task performance?
Thank you for your helpful feedback; our responses below follow the W (Weaknesses) and Q (Questions) numbering in your review.
We agree here with the reviewer. The connection to PSGD is interesting, and we will include the following detailed discussion in our revision.
Our secant penalty is equivalent to the curvature fitting term (Criteria 3 in Li, 2018) of PSGD. In particular, expanding our secant penalty we have
∥s−Hy∥H−12=s⊤H−1s+y⊤Hy+constants The right hand side is now criteria 3 in Li, 2018. The difference is in how this penalty is used in PSGD and SoftServe.
PSGD updates the preconditioner by introducing a Cholesky factorization, and applying a step of multiplicative gradient descent to the factors. Thus their method relies on linearizing (in multiplicative coordinates) this penalty. In this sense, they apply a type of first order method for updating the preconditioner.
In contrast, SoftServe avoids first-order approximations by directly minimizing the penalty together with a log-determinant regularizer. This makes SoftServe closely related to proximal point methods, which uses the full objective rather than a local approximation. When an efficient closed-form update is available, proximal point methods are generally faster and easier to tune as compared to first-order alternatives [Parikh & Boyd, 2014].
We also do agree with the reviewer that in both (Li, 2018) and (Li, 2019) they consider Kronecker factorizations, and as such, arrive at methods that are comparable in cost to SoftServe-Kron.
As such, we have now added direct comparisons using the official Kronecker-factored PSGD implementation with momentum whitening (https://github.com/lixilinx/psgd_torch). We swept LR separately for each method and normalization setting, testing six rates for each PSGD setting. PSGD otherwise uses its official defaults.
The table also addresses your normalization question. Norm denotes the update normalization used in SoftServe.
The numbers in the table are the training losses, averaged over the last five logged values, at equal gradient budgets (including SoftServe's secant replays).
Comparing SoftServe's standard normalized update with PSGD's original update, SoftServe has lower training loss on all five tasks. Without normalization on either method, MNIST is nearly tied, and PSGD achieves a lower loss on Reaction.
We have also added convergence curves comparing SoftServe-Kron and PSGD with their standard updates in Appendix F.4 (Figure F.5), with secant replays included in the gradient counts. The figure can also be viewed directly at anonymous repository.
[Parikh&Boyd2014] Neal Parikh and Stephen Boyd (2014). Proximal Algorithms. Foundations and Trends in Optimization, 1(3), 127–239.
We agree with the reviewer, and now have added more experiments testing the effect of normalization. To test the effect of normalization, we switch normalization on and off on SoftServe, K-FAC and K-BFGS(L).
For these new experiments, we swept LR separately for each method with normalization on and off. As in W1, we report MNIST training losses averaged over the last five logged values at 6,000 gradient evaluations.
Normalization improves the tuned result for all three methods, and SoftServe has the lowest loss in both columns.
Rebalancing has a different role. Replacing (L,R) by (L/c,cR), with c>0, leaves their Kronecker product unchanged in exact arithmetic; it only balances the sizes of the two factors. In the separate rebalancing ablation, turning it off changes MNIST loss and RNN training MSE only very slightly.
The paper also includes SoftServe-I, which fixes H=I while retaining update normalization. Its update is therefore a normalized momentum direction without learned curvature. Comparing it with SoftServe-Kron tests whether learning the metric provides benefits beyond normalization alone.