acceptodds
Withdrawn from ICLR 2027

MINN: A Math-Informed Neural Optimizer with Global First-Order Convergence

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We present MINN (Math-Informed Neural Network optimizer), a hybrid learned–classical optimizer for unconstrained nonlinear optimization. Purely learned optimizers can provide fast amortized progress, but often stall or fail under distribution shift; classical methods are more reliable, but may be slower and less adaptive. MINN bridges this gap through a principled two-stage architecture that separates neural proposal from classical certification. At each outer iteration, a coordinate-wise LSTM direction is combined with a steepest-descent direction through a bounded scalar trust mechanism. A descent-preserving projection safeguard and Armijo backtracking line search then produce a safeguarded Stage-1 step with sufficient decrease. The safeguarded point is further refined by an inner loop of learnable L-BFGS-style steps, where curvature-pair weights and step sizes are trainable. An accept/reject monotonicity filter accepts this refinement only if it decreases the objective relative to the safeguarded point, allowing the full method to inherit monotone decrease and global first-order convergence under standard smoothness assumptions. All parameters, including the LSTM weights, trust-head parameters, L-BFGS memory weights, and step-size parameters, are trained end-to-end using an unsupervised amortized loss with an increasing-horizon curriculum. Experiments on least-squares, neural network training, and nonconvex Rastrigin benchmarks show that MINN improves the speed–accuracy trade-off, generalizes across dimensions and distributions, and avoids the stalling and out-of-distribution failures of purely learned optimizers.

Primary area:
optimization
Keywords:
Neural solver, nonlinear optimization, convergence, first-order method

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Aqu7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work presents MINN, a hybrid learned‑classical optimizer for unconstrained nonlinear optimization. In each outer iteration, a coordinate‑wise LSTM outputs a trial trajectory. Its terminal direction is normalized and fused with the negative gradient via bounded learned weights. Sufficient‑descent projection and Armijo backtracking yield a certified Stage1 point. A learnable L‑BFGS‑like refinement, initialized with curvature pairs from the trial trajectory, is then performed in Stage2; its output is adopted only if it maintains Stage1 objective reduction. Learned modules are jointly trained with an amortized trajectory loss.

Under given assumptions, MINN converges to first‑order stationarity (not global minima). Evaluations include random least‑squares problems, MNIST MLP training and generalized Rastrigin minimization, covering dimension generalization and distribution shifts. Experiments demonstrate better speed‑accuracy trade‑offs against classical and fully learned baselines.

Critical strengths

1. A coherent interface between learned proposals and certified progress.

The strongest aspect is the separation between proposing an update and certifying its acceptance. In particular, the Stage2 endpoint is compared against the already-certified point x^t\hat{x}_t, not merely against xtx_t. Consequently,

\leq f(\hat{x}_t) \leq f(x_t)-c_1\kappa\eta_t\|\nabla f(x_t)\|^2.$$ As formalized in Section 3.5 and Appendix G, this preserves the sufficient decrease established by Stage1 regardless of whether the refinement is beneficial. Together with the gradient-proportional direction bound and the lower bound on Armijo step lengths, this yields a coherent first-order convergence argument under the stated assumptions. This is consequential because it allows learned components to be incorporated without requiring them to produce reliable updates on every instance. **2. A plausible mechanism for transferring information between the neural and classical components.** The refinement is initialized using displacement–gradient-change pairs collected along the LSTM trial trajectory (Section 3.1, Equation (3)). These are measured curvature quantities rather than predicted curvature, and constructing the pairs reuses gradients already evaluated during the trial run. This gives the two stages a substantive connection beyond simply placing a neural optimizer before a classical solver. The reported least-squares results provide encouraging evidence: at $n=1000$, Table 7 reports runtimes of $1.58$ seconds for MINN and $4.39$ seconds for L-BFGS, with both methods reaching mean best gradient norms near $10^{-2}$. If confirmed under matched evaluation, this would be a meaningful practical benefit.
Critical weaknesses

1. The Rastrigin results appear statistically inconsistent with the stated objective and sampling distribution.

Section 5(iii) defines the Rastrigin objective using independent standard-normal cosine coefficients and a scaling factor of 0.5. A direct sanity check of this definition gives an instance-wise lower bound: half the dimension minus half the sum of the absolute cosine coefficients. This follows from the nonnegative squared-residual term and the fact that each cosine is bounded in magnitude by one. The bound applies to every iterate, including the best objective attained along an optimization trajectory.

Under the stated sampling distribution, my calculation gives an expected instance-wise lower bound of approximately 101.06 at dimension 1,000. However, Table 6 reports a mean best objective of 46.40 for MINN, with similar values for several classical baselines. Assuming independent sampling of the stated 100 test instances, the standard deviation of the sample-average lower bound is approximately 0.95. Although the expected bound is not a deterministic lower bound for a realized test set, the discrepancy is far too large to be plausibly explained by ordinary sampling variability.

This suggests a mismatch between the objective or sampling distribution described in Section 5(iii) and the implementation or metric underlying Table 6. An unreported shift or normalization could explain the issue, but the manuscript does not establish such an explanation.

2. The experiments do not adequately isolate the advantage of the proposed architecture.

Appendix H evaluates a different deployment from Algorithm 4: it runs up to three learned outer iterations with 25 refinement steps per iteration, then switches to standard L-BFGS with retained curvature memory if the stopping criterion has not been reached. Algorithm 4 instead repeatedly applies the safeguarded two-stage iteration and discards refinement memory after each call; Table 3 specifies 20 refinement steps. Moreover, the "w/o LSTM" variant replaces the entire certified Stage-1 procedure in Eqs. (6)-(10) with the identity, rather than removing only the LSTM proposal in Eq. (2), and changes the memory policy. These comparisons therefore do not isolate the LSTM's contribution within the submitted algorithm.

There is also a concrete inconsistency in the fixed-L-BFGS ablation. Appendix H claims that fixing the parameters makes Eqs. (11)-(12) reduce to standard L-BFGS with unit steps. However, Eq. (11) applies a softmax: equal logits produce uniform weights that sum to one, rather than a unit weight for each stored pair. The resulting update coefficients are therefore scaled by the number of stored pairs and retain the denominator safeguard; they do not recover the classical coefficients. Appendix D.1 explicitly acknowledges that this parameterization does not contain classical L-BFGS for every memory size. Thus, either the ablation replaces the operator in Eq. (11) beyond the stated parameter freezing, or its description is incorrect. This directly affects the interpretation of the claimed benefit of learning the refinement.

Questions for authors

1. Clarification of the Rastrigin benchmark.

What exact objective and coefficient distribution were used to generate Table 6? Are the reported values raw objectives, shifted objectives, or normalized quantities? Please provide the empirical mean of ∥cq∥1\|c_q\|_1 and verify the instance-wise inequality

fq(xq)≥12(n−∥cq∥1).f_q(x_q)\geq \frac12(n-\|c_q\|_1).

2. Clarification of the evaluated algorithms and ablation operators.

Which deployment procedure generated each main result: Algorithm 4 or the learned-warm-start-plus-L-BFGS procedure in Appendix H? For the fixed-L-BFGS ablation, does the implementation bypass the softmax and substitute classical coefficients, or does it simply freeze the logits? What evidence establishes the component contributions when the safeguard and memory policy are held fixed?

3. Evidence needed to assess the efficiency claim.

What are the exact baseline hyperparameters, stopping rules, and wall-clock measurement procedures? Using common objective and gradient targets, what speed advantage remains, and are all trial-trajectory evaluations, line-search evaluations, and rejected refinements included?

Reviewer_W93iOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MINN, a two-stage learned optimizer for smooth unconstrained optimization. In Stage 1, a coordinate-wise LSTM generates a trial direction and curvature information. The learned direction is normalized, combined with the negative gradient through bounded trust coefficients, projected onto a sufficient-descent half-space, and followed by Armijo backtracking. In Stage 2, the certified point is refined using a weighted L-BFGS procedure with learned memory-slot weights and step sizes. The refinement endpoint is accepted only if it does not increase the objective relative to the certified Stage-1 point. The paper establishes first-order stationarity under lower-boundedness, globally Lipschitz gradients, and bounded trust coefficients. Experiments on least squares, MNIST MLP training, and nonconvex Rastrigin problems evaluate generalization across dimensions and problem distributions.

Critical strengths

S1. The LSTM and learned L-BFGS components propose potentially efficient updates, while projection, Armijo backtracking, and endpoint rejection prevent unreliable learned outputs from destroying the descent property. The resulting algorithm is conceptually clear and relatively easy to understand.

S2. The projection guarantees sufficient descent, the direction norm is controlled by the gradient norm, and Armijo backtracking admits a uniform positive lower bound on the accepted step size. Together with lower-boundedness, these properties imply that the gradient norms vanish. The appendix proves convergence of the entire gradient-norm sequence to zero, which is stronger than the subsequential stationarity statement given in the main propositions.

S3. The projection guarantees sufficient descent, the direction norm is controlled by the gradient norm, and Armijo backtracking admits a uniform positive lower bound on the accepted step size. Together with lower-boundedness, these properties imply that the gradient norms vanish. The appendix proves convergence of the entire gradient-norm sequence to zero, which is stronger than the subsequential stationarity statement given in the main propositions.

Critical weaknesses

W1. Descent-half-space projection, Armijo search, positive-curvature filtering, L-BFGS, and accept/reject fallback all have established foundations. More importantly, the Stage-2 candidate can be arbitrary without changing Proposition 2. The theorem therefore establishes the safety of the outer wrapper rather than convergence or acceleration specifically attributable to the learned L-BFGS refinement.

W2. The convergence guarantee applies to an idealized smooth, infinite-horizon algorithm and does not directly characterize the finite-tolerance implementation or the nonsmooth ReLU experiment. Moreover, the main propositions state only subsequential stationarity even though the appendix proves convergence of the full gradient-norm sequence. The relationship between these theoretical and practical settings should therefore be clarified.

W3. The classical baselines use PyTorch default settings, whereas MINN is trained and configured for the relevant task families. The reported 17–67× Rastrigin speedup compares termination times under method-specific stopping behavior; the paper acknowledges that L-BFGS reaches a similar objective much earlier, making the advantage at common accuracy potentially much smaller. The Table 4 ablation also uses a deployment and memory-handoff protocol different from Algorithm 4. Finally, the stated fixed-L-BFGS setting does not appear to recover the classical L-BFGS coefficient after the softmax operation.

Questions for authors
  1. Proposition 2 allows the Stage-2 candidate to be arbitrary, with convergence depending only on the certified Stage-1 decrease and the final accept/reject safeguard. Could the authors provide stronger evidence that the learned L-BFGS refinement itself is responsible for the observed gains? In particular, it would be helpful to report the Stage-2 acceptance rate, its contribution to objective decrease, and comparisons against non-learned or randomly parameterized Stage-2 proposals under the same gradient-evaluation budget.

  2. The classical baselines appear to use default PyTorch settings, while MINN is trained and configured for the evaluated task families. Could the authors provide separately tuned classical baselines and report time to a common objective value or gradient-norm threshold? This seems particularly important for Rastrigin, where the paper notes that L-BFGS reaches a comparable objective much earlier than its reported termination time.

  3. The fixed-L-BFGS ablation states that setting vi=1v_i=1 recovers the classical coefficient ρi=1/(si⊤yi)\rho_i=1/(s_i^\top y_i), but Equation (11) applies a softmax to vv, which would instead introduce a normalization factor. Does this ablation bypass the softmax? More generally, could the authors provide an ablation following the actual Algorithm 4 deployment protocol, with identical outer iterations and curvature-memory handling across variants?

Reviewer_QWPkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MINN, a hybrid learning-to-optimize method that combines an LSTM-based optimizer with L-BFGS. It uses safeguard methods like projection to ensure non-increasing update and provide first-order convergence guarantees.

Critical strengths
  1. Hybrid solvers that combine learned optimizers with classical optimization methods are important for learning-to-optimize.
  2. The experiments show promising performance compared with both classical and learned optimizers.
Critical weaknesses
  1. The safeguard and certification methods look more like engineering tricks rather than new optimization techniques. Projection, line search, and accept/reject mechanisms are already well studied in classical optimization, so the novelty of combining them is not very clear.

    For non-convex objectives, such as neural network training, MINN only guarantees convergence to stationary points rather than optimal solutions. I wonder what types of objective functions MINN is most suitable for, and whether there are conditions under which it can provide stronger guarantees.

  2. In the experiments, MINN outperforms other methods, but the reason why it converges faster and achieves better results is not well explained or theoretically justified. The paper also lacks discussion about when and why MINN should be better than classical solvers or other learned optimizers.

  3. MINN requires multiple LSTM trial steps and L-BFGS refinement steps in each outer iteration, which introduces extra computational cost. It would be helpful to discuss whether the performance improvement comes from the learned components or the additional optimization steps, especially under the same gradient evaluation budget.

In sum, the combination is useful, but my main concern is whether the learned parts give a clear gain over a similar safeguarded solver. Clarifying the test protocol and showing the gain under the same compute budget would make me more positive.

Minor: The figures’ font size is too small to see.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.