- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces a new regularized version of the saddle-free Newton method for non-convex optimization. Their new method regularizes the Hessian instead of for typical saddle free Newton. However, a naive implementation of this algorithm is not scale-free. To address this issue, the authors propose to set based on the scale of the Hessian as determined by the operator norm, i.e., where and is some constant. The inverse of the regularized matrix can be computed using Chebyshev approximation, and the authors also provide a stability bound for the preconditioner.
Critical strengths
The paper introduces a scale-free regularized version of the saddle-free Newton method and provides experimental evaluation of its performance and stability on some non-convex problems. It is a well-motivated algorithm.
Critical weaknesses
The novelty of this method seems to be rather limited. Adding to the Hessian is already used to stabilize Newton’s method and obtain regularized Newton. This paper does the same for saddle-free Newton. Moreover, this type of algorithm seems to have already been proposed in the literature [1, 2]. Thus, the main novelty lies in the way that this paper selects .
The empirical validation is rather limited. Only two problems are considered, and it seems like Adam and L-BFGS outperform the saddle-free Newton methods by a large margin. Furthermore, there is no baseline comparing how well a fixed-tuned works against the method proposed in the paper.
There are no convergence theorems, which would show the convergence rate of the method.
The clarity and presentation of the paper require major revisions. It is difficult to understand the paper with the language and terminology that it uses.
[1] Simpson, “Regularized Saddle-Free Newton: Saddle Avoidance and Efficient Implementation”, 2022.
[2] Simpson and Becker, “Regularized Saddle-Free Newton: A Globally Convergent and Efficient Non-Convex Newton Method”, 2023.
Questions for authors
Can the authors add a baseline comparing against a fixed tau? Are there experimental settings where using this saddle-free Newton method works better than Adam/L-BFGS?
Is it possible to obtain either a local or global convergence theorem for this method?
3 comments
Response
Thank you for the reply, and I appreciate the author’s engagement. I have read the updated manuscript and the author’s rebuttal.
However, my concerns have not been addressed. Mainly, the limited novelty and limited theoretical and empirical scope.
The main theorem seems to be an application of standard Chebyshev matrix approximation results. The even-degree Chebyshev expansion can be viewed as an ordinary Chebyshev approximation of the inverse square root of the positive-definite matrix . Similar strategies seem to have been applied in Frommer et al.
There has been closely related work, for example by Simpson & Becker (2023), which chooses . There is no empirical comparison between these two related methods. Furthermore, they also prove scale invariance and provide optimizer convergence theory. Their choice of also does not depend on the operator norm of the Hessian.
Furthermore, the experiments do not seem to convince me that there are any significant practical benefits of choosing this adaptive over fixed or other strategies. This method might help improve scale invariance, but it is not clear whether this has any non-trivial effect on optimization.
Thank you for the follow-up and for highlighting these connections to prior work.
We have further revised Sections 2.2–2.3 to acknowledge that our even-degree Chebyshev approximation can be expressed as a polynomial approximation of the inverse square root of , and to discuss its connection to Frommer et al. (2024). We agree that the underlying matrix-function approximation theory is established. Our focus is its finite-budget realization under relative resolution, including approximation accuracy, HVP cost, and empirical-scale effects.
We also clarify that scale consistency is not unique to Hessian-relative smoothing and acknowledge the regularization and convergence results of Simpson & Becker (2023). Our analysis concerns finite-degree spectral approximation rather than optimizer convergence.
To address the empirical concern, we have added a tuned fixed- Smooth-SFN baseline. On the PINN, fixed- achieves lower endpoint relative errors than fixed- and Adam across all three evaluated seeds, while L-BFGS achieves a substantially lower error on one seed. We do not claim an optimization advantage for Hessian-relative smoothing.
A direct objective-rescaling experiment and a comparison with the complete Simpson & Becker optimizer remain outside the current evaluation. We appreciate the reviewer's comments, which have helped us clarify the contribution and its limitations.
Thank you for the detailed review. We have substantially revised the manuscript to clarify its mathematical contribution, distinguish approximation guarantees from optimization performance, and improve the presentation.
Spectral operator versus approximation analysis. The smooth spectral operator belongs to an existing family of regularized SFN constructions, including those studied by Simpson (2022) and Simpson and Becker (2023). We do not claim a new regularizer.
The revised paper focuses instead on the finite-budget approximation of this prescribed operator. Section 4 establishes a sufficient relationship between relative resolution, Chebyshev approximation accuracy, and polynomial degree. Section 5 distinguishes this theoretical guarantee from its realization using an empirical spectral scale. The contribution is therefore an operator-approximation analysis, not a new optimizer convergence theory.
Mechanism validation and end-to-end evaluation. The experiments have two distinct roles. Section 6.1 tests approximation behavior under controlled normalization and examines scale-induced changes to the exact target. Section 6.2 evaluates realized spectral actions on CURVES and a reaction–diffusion PINN under common computational accounting.
We acknowledge that the benchmarks do not establish superiority over Adam or L-BFGS. The controlled Series-HVP comparison is not a comparison against the full published optimizer, and the unfavorable PINN results remain explicitly reported.
Fixed-width smoothing and optimizer performance. Sections 3.4 and 4.3 establish the scale-consistency of Hessian-relative smoothing and explain how a fixed absolute width changes the effective relative resolution under objective rescaling. Appendix A additionally visualizes the exact spectral-action fields of both smoothing choices on Rosenbrock.
These analyses concern spectral-operator behavior rather than tuned optimizer performance. A direct fixed-(\tau) optimization baseline is not included, and the reported experiments do not establish an advantage over Adam or L-BFGS.
Operator guarantees versus optimizer convergence. Section 4 establishes approximation and sufficient matrix-positivity conditions under admissible normalization. These results do not guarantee descent with a fixed step size or convergence of the empirical-scale optimizer. Such a theorem would require additional control of step sizes, scale estimation, and approximation errors along the optimization trajectory.
We have made this theoretical boundary explicit rather than extending the approximation guarantee beyond its assumptions.
Manuscript organization. The revised Sections 3–5 separate spectral parameterization, finite-degree approximation, and empirical-scale realization. Section 6 distinguishes controlled mechanism tests from downstream optimization behavior. We have also clarified mathematical notation, assumptions, and the roles of supplementary results.