acceptodds
Under review as a conference paper at ICLR 2027

Smooth Spectral Operators for Indefinite Hessians: Relative Resolution, Stability, and Matrix-Free Approximation

Abstract

Curvature-aware optimization in high-dimensional non-convex problems relies on matrix-free spectral transformations of indefinite Hessians. A fundamental difficulty in designing these operators is that a regularization parameter measured in absolute curvature units changes its operational meaning when the overall objective scale changes. Under homogeneous curvature rescaling, a fixed absolute parameter alters the relative transition geometry and arbitrarily changes the matrix-free computational difficulty. To resolve this ambiguity, we introduce relative resolution, parameterizing the spectral transition scale relative to an admissible reference spectral scale. This dimensionless formulation yields a normalized spectral geometry whose polynomial approximation complexity is independent of the absolute curvature scale under admissible normalization. Our analysis connects relative resolution to fixed- operator-perturbation stability, finite polynomial-approximation budgets, and the operational scale mismatch. Experiments support the predicted approximation-complexity scaling, quantify sensitivity to operational scale mismatch, and characterize the distinct realized-action regimes observed across controlled non-convex and stiff physics-informed optimization settings.

Primary area:
optimization
Keywords:
non-convex optimization, saddle-free Newton, Hessian-vector products, matrix-free optimization, spectral matrix functions, Chebyshev approximation, second-order optimization, relative resolution

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_WBVwOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces a new regularized version of the saddle-free Newton method for non-convex optimization. Their new method regularizes the Hessian (H2+τI)−½(H^2 + \tau I)^{-½} instead of (H2)−½(H^2)^{-½} for typical saddle free Newton. However, a naive implementation of this algorithm is not scale-free. To address this issue, the authors propose to set τ\tau based on the scale of the Hessian as determined by the ℓ2\ell_2 operator norm, i.e., τ=αL\tau = \alpha L where L>∣∣H∣∣2L > ||H||_2 and α>0\alpha > 0 is some constant. The inverse of the regularized matrix can be computed using Chebyshev approximation, and the authors also provide a stability bound for the preconditioner.

Critical strengths

The paper introduces a scale-free regularized version of the saddle-free Newton method and provides experimental evaluation of its performance and stability on some non-convex problems. It is a well-motivated algorithm.

Critical weaknesses

The novelty of this method seems to be rather limited. Adding τI\tau I to the Hessian is already used to stabilize Newton’s method and obtain regularized Newton. This paper does the same for saddle-free Newton. Moreover, this type of algorithm seems to have already been proposed in the literature [1, 2]. Thus, the main novelty lies in the way that this paper selects τ\tau.

The empirical validation is rather limited. Only two problems are considered, and it seems like Adam and L-BFGS outperform the saddle-free Newton methods by a large margin. Furthermore, there is no baseline comparing how well a fixed-tuned τ\tau works against the method proposed in the paper.

There are no convergence theorems, which would show the convergence rate of the method.

The clarity and presentation of the paper require major revisions. It is difficult to understand the paper with the language and terminology that it uses.

[1] Simpson, “Regularized Saddle-Free Newton: Saddle Avoidance and Efficient Implementation”, 2022.

[2] Simpson and Becker, “Regularized Saddle-Free Newton: A Globally Convergent and Efficient Non-Convex Newton Method”, 2023.

Questions for authors

Can the authors add a baseline comparing against a fixed tau? Are there experimental settings where using this saddle-free Newton method works better than Adam/L-BFGS?

Is it possible to obtain either a local or global convergence theorem for this method?

3 comments

Thank you for the detailed review. We have substantially revised the manuscript to clarify its mathematical contribution, distinguish approximation guarantees from optimization performance, and improve the presentation.

The novelty of this method seems to be rather limited.

Spectral operator versus approximation analysis. The smooth spectral operator belongs to an existing family of regularized SFN constructions, including those studied by Simpson (2022) and Simpson and Becker (2023). We do not claim a new regularizer.

The revised paper focuses instead on the finite-budget approximation of this prescribed operator. Section 4 establishes a sufficient relationship between relative resolution, Chebyshev approximation accuracy, and polynomial degree. Section 5 distinguishes this theoretical guarantee from its realization using an empirical spectral scale. The contribution is therefore an operator-approximation analysis, not a new optimizer convergence theory.

The empirical validation is rather limited.

Mechanism validation and end-to-end evaluation. The experiments have two distinct roles. Section 6.1 tests approximation behavior under controlled normalization and examines scale-induced changes to the exact target. Section 6.2 evaluates realized spectral actions on CURVES and a reaction–diffusion PINN under common computational accounting.

We acknowledge that the benchmarks do not establish superiority over Adam or L-BFGS. The controlled Series-HVP comparison is not a comparison against the full published optimizer, and the unfavorable PINN results remain explicitly reported.

Can the authors add a baseline comparing against a fixed tau? Are there experimental settings where using this saddle-free Newton method works better than Adam/L-BFGS?

Fixed-width smoothing and optimizer performance. Sections 3.4 and 4.3 establish the scale-consistency of Hessian-relative smoothing and explain how a fixed absolute width changes the effective relative resolution under objective rescaling. Appendix A additionally visualizes the exact spectral-action fields of both smoothing choices on Rosenbrock.

These analyses concern spectral-operator behavior rather than tuned optimizer performance. A direct fixed-(\tau) optimization baseline is not included, and the reported experiments do not establish an advantage over Adam or L-BFGS.

Is it possible to obtain either a local or global convergence theorem for this method?

Operator guarantees versus optimizer convergence. Section 4 establishes approximation and sufficient matrix-positivity conditions under admissible normalization. These results do not guarantee descent with a fixed step size or convergence of the empirical-scale optimizer. Such a theorem would require additional control of step sizes, scale estimation, and approximation errors along the optimization trajectory.

We have made this theoretical boundary explicit rather than extending the approximation guarantee beyond its assumptions.

The clarity and presentation of the paper require major revisions.

Manuscript organization. The revised Sections 3–5 separate spectral parameterization, finite-degree approximation, and empirical-scale realization. Section 6 distinguishes controlled mechanism tests from downstream optimization behavior. We have also clarified mathematical notation, assumptions, and the roles of supplementary results.

Reviewer_WBVwOpenReview ↗

Response

Thank you for the reply, and I appreciate the author’s engagement. I have read the updated manuscript and the author’s rebuttal.

However, my concerns have not been addressed. Mainly, the limited novelty and limited theoretical and empirical scope.

The main theorem seems to be an application of standard Chebyshev matrix approximation results. The even-degree Chebyshev expansion can be viewed as an ordinary Chebyshev approximation of the inverse square root of the positive-definite matrix A=X2+α2IA=X^2+\alpha^2I. Similar strategies seem to have been applied in Frommer et al.

There has been closely related work, for example by Simpson & Becker (2023), which chooses τ=M∣∣gt∣∣\tau = \sqrt{M||g_t||}. There is no empirical comparison between these two related methods. Furthermore, they also prove scale invariance and provide optimizer convergence theory. Their choice of τ\tau also does not depend on the operator norm of the Hessian.

Furthermore, the experiments do not seem to convince me that there are any significant practical benefits of choosing this adaptive τ\tau over fixed tautau or other strategies. This method might help improve scale invariance, but it is not clear whether this has any non-trivial effect on optimization.

Thank you for the follow-up and for highlighting these connections to prior work.

We have further revised Sections 2.2–2.3 to acknowledge that our even-degree Chebyshev approximation can be expressed as a polynomial approximation of the inverse square root of X2+α2IX^2+\alpha^2I, and to discuss its connection to Frommer et al. (2024). We agree that the underlying matrix-function approximation theory is established. Our focus is its finite-budget realization under relative resolution, including approximation accuracy, HVP cost, and empirical-scale effects.

We also clarify that scale consistency is not unique to Hessian-relative smoothing and acknowledge the regularization and convergence results of Simpson & Becker (2023). Our analysis concerns finite-degree spectral approximation rather than optimizer convergence.

To address the empirical concern, we have added a tuned fixed-τ\tau Smooth-SFN baseline. On the PINN, fixed-τ\tau achieves lower endpoint relative L2L^2 errors than fixed-α\alpha and Adam across all three evaluated seeds, while L-BFGS achieves a substantially lower error on one seed. We do not claim an optimization advantage for Hessian-relative smoothing.

A direct objective-rescaling experiment and a comparison with the complete Simpson & Becker optimizer remain outside the current evaluation. We appreciate the reviewer's comments, which have helped us clarify the contribution and its limitations.

Reviewer_KiqWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to efficiently approximate regularised saddle-free Newton updates using Hessian-vector products. Its main proposal is to express the regularisation parameter relative to the Hessian's spectral scale, making the target update and its approximation complexity invariant to changes in the objective's scale. The authors derive stability and polynomial approximation bounds and evaluate the approach on synthetic matrices, neural-network Hessians, an autoencoder, and a physics-informed neural network (PINN).

Critical strengths
  1. The analysis establishes how polynomial approximation accuracy depends on relative regularisation and approximation degree. The mathematical arguments largely follow established results and appear correct.
  2. The paper recognises that estimating the Hessian's spectral scale introduces errors and can invalidate the approximation guarantees.
  3. The controlled experiments confirm the predicted relationship between approximation degree, relative regularisation, and approximation error. The authors also acknowledge limitations in the optimisation results.
Critical weaknesses
  1. If I understand correctly, the main contribution appears to be largely a reformulation using relative regularisation, followed by standard matrix approximation results. Scale-aware regularisation and gradient-dependent regularisation are not entirely new concepts (Simpson 2022, Martens 2010). It would be beneficial to correctly position this paper wrt other relevant works,
  2. The method is competitive with a simplified Series-HVP baseline on CURVES, but Adam seems to perform better. On the PINN, both Adam and L-BFGS achieve better objectives, while Smooth-SFN produces poor solution accuracy across all three seeds. Therefore, it is unclear what the practical benefits of this method are.
  3. I couldn't find a direct comparison of relative versus absolute regularisation under objective rescaling.
Questions for authors

see abve

2 comments

Thank you for the careful review and constructive feedback. We address the concerns below and have revised the manuscript to clarify the contribution, experimental evidence, and scope of the analysis.

The main contribution appears to be largely a reformulation using relative regularisation, followed by standard matrix approximation results.

Finite-budget spectral approximation. Smooth-SFN uses a regularized spectral operator closely related to existing SFN constructions. Our contribution is not the regularizer itself, but the analysis of its finite-degree matrix-free realization. Specifically, relative resolution connects the Chebyshev approximation error, sufficient polynomial degree, and HVP budget under admissible normalization. The analysis also identifies why these guarantees cannot be transferred unconditionally to an empirically estimated spectral scale. Sections 2–5 now make this contribution and its relationship to prior work more explicit.

It is unclear what the practical benefits of this method are.

Controlled approximation and realized-action behavior. Section 6.1 evaluates the predicted finite-degree approximation behavior and separately measures changes to the exact spectral target under scale mismatch. Section 6.2 examines the resulting actions under a common computational budget. On CURVES, Smooth-SFN reduces the training objective faster than the controlled Series-HVP realization within the evaluated budget; this does not establish an advantage over Adam or L-BFGS. On the PINN, Adam and L-BFGS perform better, and the physical-space solution errors remain high.

The controlled tests support the approximation analysis, while the end-to-end experiments characterize realized-action behavior and its limitations. Neither establishes general optimizer superiority.

I couldn't find a direct comparison of relative versus absolute regularisation under objective rescaling.

Relative versus fixed-width smoothing. Sections 3.4 and 4.3 establish the scale-consistency of Hessian-relative smoothing and explain how a fixed absolute width changes the effective relative resolution under objective rescaling. Appendix A complements this analysis by visualizing exact spectral-action fields on Rosenbrock under fixed-width and Hessian-relative smoothing.

These results examine spectral-operator behavior rather than tuned optimizer performance. The scale-mismatch test in Sec. 6.1 holds HH, gg, and α\alpha fixed while varying the normalization scale. Neither this test nor the Appendix A visualization is a direct objective-rescaling experiment comparing fixed-width and Hessian-relative smoothing, which remains absent from the current evaluation.

The revised manuscript separates these results and their evidential scope more clearly.

Thank you again for your constructive feedback. Following our previous response, we have made further revisions to address the concerns about novelty and empirical evidence.

Positioning relative to prior work. Sections 2.2–2.3 now more explicitly acknowledge the connections to Simpson & Becker (2023) and established inverse-square-root matrix-function approximation theory, including Frommer et al. (2024). We clarify that scale consistency is not unique to Hessian-relative smoothing and that our contribution concerns finite-budget approximation analysis rather than a new regularizer or convergence theory.

Additional empirical comparison. Section 6.2 now includes a tuned fixed-τ\tau Smooth-SFN baseline under common computational accounting. On the reaction–diffusion PINN, fixed-τ\tau achieves lower endpoint relative L2L^2 errors than fixed-α\alpha and Adam across all three evaluated seeds, although L-BFGS achieves a substantially lower error on one seed. These results do not establish an optimization advantage for Hessian-relative smoothing.

We acknowledge that the new comparison does not constitute a direct objective-rescaling experiment, which remains outside the current evaluation.

We appreciate your feedback, which has helped us clarify the contribution and the scope of the evidence.

Comments on the paper

1 comment

We thank the reviewers for their detailed feedback. The revisions clarify the paper's contribution, its relationship to prior work, and the scope of the empirical evidence.

We have revised the title, abstract, introduction, related work, and conclusion to position the contribution as a finite-budget approximation analysis of an existing smooth spectral operator, rather than a new regularized SFN method or a superior optimizer. The revised Sections 2–5 clarify the connections to established matrix-function approximation and regularized SFN research, the guarantees under admissible normalization, and the effects of empirical spectral-scale estimation.

Section 6 distinguishes controlled approximation tests from downstream optimization behavior and includes a tuned fixed-width smoothing baseline under common computational accounting.

These revisions clarify the theoretical contribution and provide additional empirical evidence while keeping the conclusions within the scope of the evaluated methods.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.