- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This work presents MINN, a hybrid learned‑classical optimizer for unconstrained nonlinear optimization. In each outer iteration, a coordinate‑wise LSTM outputs a trial trajectory. Its terminal direction is normalized and fused with the negative gradient via bounded learned weights. Sufficient‑descent projection and Armijo backtracking yield a certified Stage1 point. A learnable L‑BFGS‑like refinement, initialized with curvature pairs from the trial trajectory, is then performed in Stage2; its output is adopted only if it maintains Stage1 objective reduction. Learned modules are jointly trained with an amortized trajectory loss.
Under given assumptions, MINN converges to first‑order stationarity (not global minima). Evaluations include random least‑squares problems, MNIST MLP training and generalized Rastrigin minimization, covering dimension generalization and distribution shifts. Experiments demonstrate better speed‑accuracy trade‑offs against classical and fully learned baselines.
Critical strengths
1. A coherent interface between learned proposals and certified progress.
The strongest aspect is the separation between proposing an update and certifying its acceptance. In particular, the Stage2 endpoint is compared against the already-certified point , not merely against . Consequently,
\leq f(\hat{x}_t) \leq f(x_t)-c_1\kappa\eta_t\|\nabla f(x_t)\|^2.$$ As formalized in Section 3.5 and Appendix G, this preserves the sufficient decrease established by Stage1 regardless of whether the refinement is beneficial. Together with the gradient-proportional direction bound and the lower bound on Armijo step lengths, this yields a coherent first-order convergence argument under the stated assumptions. This is consequential because it allows learned components to be incorporated without requiring them to produce reliable updates on every instance. **2. A plausible mechanism for transferring information between the neural and classical components.** The refinement is initialized using displacement–gradient-change pairs collected along the LSTM trial trajectory (Section 3.1, Equation (3)). These are measured curvature quantities rather than predicted curvature, and constructing the pairs reuses gradients already evaluated during the trial run. This gives the two stages a substantive connection beyond simply placing a neural optimizer before a classical solver. The reported least-squares results provide encouraging evidence: at $n=1000$, Table 7 reports runtimes of $1.58$ seconds for MINN and $4.39$ seconds for L-BFGS, with both methods reaching mean best gradient norms near $10^{-2}$. If confirmed under matched evaluation, this would be a meaningful practical benefit.Critical weaknesses
1. The Rastrigin results appear statistically inconsistent with the stated objective and sampling distribution.
Section 5(iii) defines the Rastrigin objective using independent standard-normal cosine coefficients and a scaling factor of 0.5. A direct sanity check of this definition gives an instance-wise lower bound: half the dimension minus half the sum of the absolute cosine coefficients. This follows from the nonnegative squared-residual term and the fact that each cosine is bounded in magnitude by one. The bound applies to every iterate, including the best objective attained along an optimization trajectory.
Under the stated sampling distribution, my calculation gives an expected instance-wise lower bound of approximately 101.06 at dimension 1,000. However, Table 6 reports a mean best objective of 46.40 for MINN, with similar values for several classical baselines. Assuming independent sampling of the stated 100 test instances, the standard deviation of the sample-average lower bound is approximately 0.95. Although the expected bound is not a deterministic lower bound for a realized test set, the discrepancy is far too large to be plausibly explained by ordinary sampling variability.
This suggests a mismatch between the objective or sampling distribution described in Section 5(iii) and the implementation or metric underlying Table 6. An unreported shift or normalization could explain the issue, but the manuscript does not establish such an explanation.
2. The experiments do not adequately isolate the advantage of the proposed architecture.
Appendix H evaluates a different deployment from Algorithm 4: it runs up to three learned outer iterations with 25 refinement steps per iteration, then switches to standard L-BFGS with retained curvature memory if the stopping criterion has not been reached. Algorithm 4 instead repeatedly applies the safeguarded two-stage iteration and discards refinement memory after each call; Table 3 specifies 20 refinement steps. Moreover, the "w/o LSTM" variant replaces the entire certified Stage-1 procedure in Eqs. (6)-(10) with the identity, rather than removing only the LSTM proposal in Eq. (2), and changes the memory policy. These comparisons therefore do not isolate the LSTM's contribution within the submitted algorithm.
There is also a concrete inconsistency in the fixed-L-BFGS ablation. Appendix H claims that fixing the parameters makes Eqs. (11)-(12) reduce to standard L-BFGS with unit steps. However, Eq. (11) applies a softmax: equal logits produce uniform weights that sum to one, rather than a unit weight for each stored pair. The resulting update coefficients are therefore scaled by the number of stored pairs and retain the denominator safeguard; they do not recover the classical coefficients. Appendix D.1 explicitly acknowledges that this parameterization does not contain classical L-BFGS for every memory size. Thus, either the ablation replaces the operator in Eq. (11) beyond the stated parameter freezing, or its description is incorrect. This directly affects the interpretation of the claimed benefit of learning the refinement.
Questions for authors
1. Clarification of the Rastrigin benchmark.
What exact objective and coefficient distribution were used to generate Table 6? Are the reported values raw objectives, shifted objectives, or normalized quantities? Please provide the empirical mean of and verify the instance-wise inequality
2. Clarification of the evaluated algorithms and ablation operators.
Which deployment procedure generated each main result: Algorithm 4 or the learned-warm-start-plus-L-BFGS procedure in Appendix H? For the fixed-L-BFGS ablation, does the implementation bypass the softmax and substitute classical coefficients, or does it simply freeze the logits? What evidence establishes the component contributions when the safeguard and memory policy are held fixed?
3. Evidence needed to assess the efficiency claim.
What are the exact baseline hyperparameters, stopping rules, and wall-clock measurement procedures? Using common objective and gradient targets, what speed advantage remains, and are all trial-trajectory evaluations, line-search evaluations, and rejected refinements included?