acceptodds
Under review as a conference paper at ICLR 2027

Gauss–Newton is Blind to Embedded Saddles: Value-Trained Surrogates as a Derivative Oracle for Amortized Inversion

Abstract

Amortized inversion—one nonlinear least-squares solve per target, across many targets sharing one forward model—commonly initialises the full model from the fit of a reduced one. By the embedding principle of nested models, such a fit, when interior and stationary, is an exact critical point of the richer model's landscape; we characterise it in closed form as a strict saddle whenever a sign test on the residual holds, and for coalesced Gaussian components generically. The saddle is invisible to the Gauss–Newton matrix : it carries exactly zero curvature in the escape subspace, so Gauss–Newton, Levenberg–Marquardt and secant-corrected methods report convergence at the non-minimiser. The escape lives in the residual-weighted term that Gauss–Newton discards; taking it from the true model costs evaluations per finite-difference probe or an interpolation model, restarts multiply the cost, and under evaluation noise a single-sample stencil faces a bias–variance trade-off. We show the term can be learned: a neural surrogate of the single-component shape, trained on function values alone with input dimension independent of , supplies the Jacobian and the curvature at zero true-evaluation cost, and the fidelity that matters is low leakage into the well-determined subspace, not value accuracy. On Voigt-profile families with –, Surrogate-Guided LM matches the strongest classical escapes to within a few points (twelve behind at transmission ) at a cost flat in , – below them at ; an ablation attributes the cost to the free Jacobian and the first-attempt escape to the curvature (a single split perturbation reaches –, the curvature step from it –). With evaluation noise it meets the accuracy criterion on – of targets against at most for single-sample alternatives. On an external stellar-synthesis code at  s per call, initialised at the single-star fit of a synthetic binary, it meets the success criterion on all non-trivial targets at – fewer syntheses than the tested classical escapes—a saving the free Jacobian supplies, since a perturbed split without the curvature step does as well—the surrogate's cost repaid after about seven solves.

Primary area:
optimization
Keywords:
neural surrogate models, nonlinear least squares, inverse problems, second-order optimization, saddle points, Gauss–Newton, surrogate-based optimization, Fisher information

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_yshxOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies nonlinear least-squares fits of two-component models f=a1φ(θ1)+a2φ(θ2)f=a_1\varphi(\theta_1)+a_2\varphi(\theta_2) initialised by splitting a converged one-component fit into two coincident halves. It shows that this point is critical, that it is a strict saddle iff γ=ϕ(θ∗)⋅r<0\gamma=\phi(\theta^*)\cdot r<0 (for Gaussian peaks, whenever Mμσ≠0M_{\mu\sigma}\neq0), that the Gauss–Newton matrix has zero curvature along the escape direction, and it bounds finite-difference curvature estimates under evaluation noise (Prop. 1, Cor. 1, Lemma 1, Prop. 2). It then proposes SG-LM: Levenberg–Marquardt with the Jacobian and the residual curvature C=∑iri∇2fiC=\sum_i r_i\nabla^2 f_i taken from a neural surrogate of the single-component shape, trained on function values, followed by a negative-curvature line search on the true model. Experiments cover biexponential, Gaussian and K-Voigt families (n=4n=4–3232), a transmission variant, grown tanh networks, and a synthetic binary-star problem with the Korg code (n=10n=10). Costs are counted in true evaluations, with surrogate derivatives and surrogate training excluded.

Critical strengths
  • The computations of Prop. 1 and Remark 3 are explicit and checked numerically. The identity ∂μ2φ∈span{φ,∂σφ}\partial_\mu^2\varphi\in\mathrm{span}\{\varphi,\partial_\sigma\varphi\} behind Cor. 1 is a clean observation.
  • Table 1 separates the contribution of the surrogate Jacobian from that of the surrogate curvature, and the limitations section reports the wall-clock break-even. Code and result logs are provided.
Critical weaknesses
  • The theory is elementary and largely known. The criticality of the embedded point and its classification by the sign of the residual-weighted matrix MM are Fukumizu & Amari (2000, Thm. 3) and the splitting matrix of Liu et al. (2019), as the paper acknowledges (l. 80–84). The "blindness" claim reduces to J⊤J⪰0J^\top J\succeq0 and J⊤Ju=0J^\top Ju=0 for u∈ker⁡Ju\in\ker J, which holds by definition. Prop. 1 is a 4×44\times4 Hessian computation, Lemma 1 expands a Rayleigh quotient in an eigenbasis, and Prop. 2 is the standard bias–variance trade-off of a central difference.
  • The phenomenon is not specific to Gauss–Newton. At a critical point J⊤r=0J^\top r=0, so the LM step −(J⊤J+μI)−1J⊤r-(J^\top J+\mu I)^{-1}J^\top r is zero whatever the curvature model. The same holds for GD, Adam, natural gradient and K-FAC (Table 5). Footnote 1 concedes this. The obstruction comes from initialising exactly at a stationary point (median ∥J⊤r∥∞∼10−8\|J^\top r\|_\infty\sim10^{-8}, l. 320); from a loosely converged fit, the escape is "accidental" (l. 357–358, Table 18).
  • Obvious baselines are missing. These include: a non-coincident split θ1,2=θ∗±δ\theta_{1,2}=\theta^*\pm\delta with plain LM/TRF; zero-amplitude insertion, which by Remark 2 has no such saddle; greedy insertion (sliding Frank–Wolfe); exact Hessian–vector products with a negative-curvature step on the true model, where autodiff is available; and Korg's native autodiff (l. 461).
  • Scope and presentation. The theory covers two components of a single shape. Experiments are synthetic, with n≤32n\le32. The abstract and contributions consist largely of numerical ranges with parenthetical qualifications (l. 29–38, 98–117), which makes the main claim hard to extract.
Questions for authors
  • Which statement of Prop. 1 about Gauss–Newton is not equally true of any method whose step vanishes with the gradient at a critical point?
Reviewer_YkfeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The work claims to identify a structural problem in Gauss-Newton (GN) method that GN stalls on saddle points because GN sees zero curvature along a downhill direction for a stationary-reduced model fit embedded into a richer model in amortized solving of optimization problems. To overcome the problem in GN, the authors proposed Surrogate-Guided modification to Levenberg-Marquardt (SG-LM) method by addition of a surrogate-based approximation to the true objective function’s Hessian using Jacobians and second derivatives of the surrogate to provide the direction to escape the saddle point in the loss landscape.

Critical strengths

The authors propose a well-motivated solution: the surrogate-guided Levenberg-Marquardt (SG-LM) solver to the existing problem with existing second order solvers: the GN or LM solvers often stall at the saddle points. The exposition for the problem setup appears correct and clear. The empirical evidence supporting the proposed SG-LM method seems ample.

Critical weaknesses

-- The theoretical contribution is overstated or poorly distinguished from existing works. A good portion of the abstract and motivation is devoted to the idea that saddle points in nonlinear least-sqares problems can stall solvers. But is this not already ingrained into the way people think about these problems, at least at a high level?

For example, Mizutani & Dreyfus (2010) explicitly establishes the nonlinear least-squares framework in Equation 3, partitioning the full Hessian matrix into the standard Gauss–Newton component and its residual-weighted second-order adjustment in Equation 4. The prior work investigates the negative curvature directions ignored by the Gauss–Newton formulation (page 2), details stationary saddle points created by overlapping parameters in Equation 13 and Theorem 4, and presents the reduced-to-richer exponential setup in Example 1. This overemphasis weakens my confidence in the contributions described in the work.

-- The experiments seem ample but do not sufficiently establish that SG-LM can improve under an accuracy-cost tradeoff. The main method SG-LM is a second order algorithm, therefore the cost is an important factor in determining the significance of the proposed method. On line 1636, the authors score a per-target solution of the forward model as successful if the error is within 55\\% of the pooled reference error. It provides useful information but does not directly answer the following questions. (a). Given the same computational budget, which method produces a better solution by what percentage? (b) How much more computation does each method need to achieve the same accuracy? Without the full information, a solver can have lower cost because failed runs terminate early. I appreciate the inclusion in the mean failure/success numbers for transparency but it does not help in a fair evaluation for same evaluation counts. For example, Table 1 compares 63% success at 64 calls against 86% at 97 calls. The comparison does not show whether the no-curvature solver, using additional perturbations or restarts at a comparable total evaluation cost, could match the higher success rate. The work in line 417-422 concedes the point.

--The work needs comparison against more relevant baselines. [2] directly targets derivative-free NLS formulation by building a quadratic residual model that includes an approximation to the residual-weighted curvature discarded by GN. [3] is cited but not benchmarked. [3] finds negative curvature information from function values without reconstructing a full Hessian. In addition to the external reference I found, in F.3, line 1455: “even a matrix-free Lanczos pass on FD Hessian–vector products costs O(nk)\mathcal{O}(nk) true evaluations” — they concede the O~(n)\tilde{\mathcal{O}}(n) route and never run it, while the headline compares against 2n2+2n2n^{2}+2n.

References:

Mizutani, E., & Dreyfus, S. (2010). An analysis on negative curvature induced by singularity in multi-layer neural-network learning. In Advances in Neural Information Processing Systems, 23.

Wild, S. M. (2017). POUNDERS in TAO: Solving derivative-free nonlinear least-squares problems with POUNDERS. In T. Terlaky, M. F.

Anjos, & S. Ahmed (Eds.), Advances and Trends in Optimization with Engineering Applications (pp. 529–539). SIAM.

Zhang, H., Xiong, H., & Gu, B. (2022). Zeroth-order negative curvature finding: Escaping saddle points without gradients. Advances in Neural Information Processing Systems, 35.

Questions for authors

No questions at this time.

Reviewer_xUX5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper considers nonlinear least-squares problems in a nested-model setting, where a solution of a reduced model is used to initialize a more complex model by splitting one component into two identical components. Such an initialization can produce a strict saddle point whose negative curvature is not detected by Gauss–Newton. The authors propose a surrogate-based Levenberg–Marquardt method (SG-LM), that uses a neural surrogate trained on function values to approximate the derivatives needed to escape such saddle points, while evaluating the objective using the original forward model. Numerical experiments on synthetic profile-fitting problems, noisy forward models, neural-network growth and a stellar-spectrum simulator compare SG-LM with classical optimization methods.

Critical strengths
  • It gives an explicit characterization of embedded saddle points whose negative curvature is invisible to the Gauss–Newton approximation, and proposes a surrogate-based approach to recover the missing curvature information.
  • The empirical evaluation is quite extensive, ablation studies, experiments under evaluation noise, ncluding comparisons with classical optimization methods and an external stellar spectrum simulator.
Critical weaknesses
  • I find the presentation unnecessarily verbose and using too much technical jargon. Often the main ideas are obscured by unnecessary technical details. A more concise and precise presentation would significantly improve readability.
  • The theoretical novelty appears limited. The existence and characterization of embedded saddle points are already known (see Fukumizu and Amari 2000, Zhang et al 2021), while the Gauss–Newton „blindness“ follows directly from the structure of its Hessian approximation. I evaluate the main contribution as algorithmic rather than theoretical. Moreover, the practical necessity of the explicit curvature correction is not fully convincing, as simpler perturbation-based approaches perform comparably in several experiments.
Questions for authors
  • In Prop. 1, for the balanced case a1=a2=aa_1=a_2=a it is stated that aγa\gamma is the smallest Hessian eigenvalue. However, when γ>0\gamma>0, the Hessian also has a zero eigenvalue. Should this claim be restricted to the strict-saddle case γ<0\gamma<0
  • The main theoretical result is derived for two components initialized at exactly the same location. Which parts of the result extend to models with more components, vector valued shape parameters or neural networks? In particular, does the same IFF characterization still hold?
  • For Gaussian components, Cor 1 states that the point where the two components coincide is generically a strict saddle. Could you clarify the assumptions under which this claim holds, particularly regarding the sampling grid and the residual?
  • Prop 2 assumes that the residual is known exactly, although in practice it is also affected by noise. How would this additional noise affect the results?
  • Table 1 suggests that the surrogate Jacobian combined with a small split perturbation or restarts already achieves much of the practical improvement, although the explicit curvature correction further improves reliability. Could the authors clarify in which settings this additional curvature step is essential, rather than just beneficial?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.