acceptodds
Under review as a conference paper at ICLR 2027

Optimizer selection based on function-proxy fidelity

Abstract

Optimizing highly nonconvex loss functions is central to modern machine learning, making optimizer selection both important and difficult. Many training algorithms can be interpreted as minimizing local surrogates of the loss, typically induced by Taylor approximations, but it is often unclear which surrogate is most appropriate for a given problem. At the same time, Hessian drift is poorly understood, and existing second-order methods typically rely on Tikhonov regularization to cope with concavity and departures from local quadratic structure. This regularization introduces additional tuning and can weaken the very second-order information these methods seek to exploit. We present a novel eigendecomposition-centric analysis of Hessian drift and use it to derive a new Hessian regularization method together with a meta-algorithm for optimizer selection. Experiments validate the effectiveness of our algorithms.

Primary area:
optimization
Keywords:
optimization, arc, adaptive regularization with cubics, neural networks, artificial intelligence, training, second-order optimizers, quasi-newton, newton's method, newton-raphson

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DsAvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework for dynamically switching between first- and second-order optimizers based on the local Hessian Lipschitness. The authors first introduce ELMO, a Hessian eigenspace-based optimization method that uses a cubic surrogate model with eigenspace-wise Lipschitz constants. Building on this analysis, they develop AMOS, a meta-optimizer that dynamically assigns different parameter blocks to first- or second-order optimizers according to estimated curvature variation. Experiments on language modeling tasks demonstrate promising improvements over several existing optimizers.

Critical strengths
  1. The paper explores an interesting direction by considering Hessian variation, rather than only curvature magnitude, as a criterion for optimizer selection.

  2. The proposed framework provides a theoretically motivated connection between local quadratic surrogate fidelity and optimizer selection.

  3. The experiments cover several language modeling datasets and compare against a relatively broad collection of optimizers, with promising empirical results.

Critical weaknesses

1. The eigenspace-wise Lipschitz assumption appears overly restrictive.

Assumption 2 imposes a strong structural condition on the variation of Hessian eigenvalues:

∣λi(θ)−λi(ϕ)∣≤Lˉi∣(θ−ϕ)⊤vi(θ)∣.|\lambda_i(\theta)-\lambda_i(\phi)|\leq \bar L_i |(\theta-\phi)^\top v_i(\theta)|.

This assumption seems to be different from standard Hessian Lipschitz continuity and appears substantially stronger than the conventional assumption. I wonder if this influences the reliability of the analysis.

It would be important to clarify which generic classes of functions satisfy this assumption and whether it is reasonable for practical neural network training. Since the theoretical development relies heavily on this assumption, its restrictiveness raises concerns about the general applicability of the results.

2. The additional computational overhead is not sufficiently quantified.

Although the paper reports wall-clock performance and discusses amortizing the cost of Hessian computations, the additional overhead introduced by AMOS (e.g., overhead of maintaining and switching between optimizer states) is not thoroughly evaluated.

3. Missing ablation studies to establish the necessity of Lipschitz-aware optimizer selection.

The experiments show that dynamically combining SGD and Sophia can outperform either optimizer individually. However, it remains unclear whether these improvements arise from the proposed Lipschitz-based selection criterion or simply from assigning a subset of parameters to SGD instead of Sophia. It seems that simply replacing some Sophia updates with SGD may help. I would like to see comparisons against simpler hybrid baselines, such as random or fixed parameter assignments.

Questions for authors

See weakness

Reviewer_gtvNOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies adaptive optimizer selection based on the reliability of local quadratic approximations. It derives an idealized method, ELMO, from an eigenspace-wise cubic model and proposes AMOS, which uses estimated curvature variation to route parameter tensors between SGD and Sophia. Experiments evaluate the approach primarily on small Transformer models under GPU memory constraints.

Critical strengths
  1. Adaptive allocation of first- and second-order updates addresses a practically relevant problem.

  2. Curvature stability provides an intuitive criterion for deciding when second-order information is useful.

  3. The evaluation includes an additional random seed and hyperparameter-sensitivity experiments, and the authors acknowledge limitations beyond the primary experimental setting.

Critical weaknesses
  1. Assumption 2 is stronger than claimed. It requires an eigenvalue to remain unchanged under any displacement orthogonal to its current eigenvector. Standard Hessian Lipschitz continuity does not imply this property, contrary to Remark 1 and Appendix P.2.

  2. ELMO’s preconditioner equivalence fails, and its minmax optimality is not established. When gi=0g_i=0 and λi<0\lambda_i<0, the inverse in Lemma 3.2 is undefined, although ELMO takes a nonzero step. Appendix P.2 also derives an eigenvector-rotation term whose incorporation into the main model is unclear. Finally, minimizing an upper bound does not establish minmax optimality without proving its tightness.

  3. Theorem C.1 is false as stated. The admissible step s=0s=0 gives Mti(0)=0M_t^i(0)=0 and δ=−1\delta=-1, contradicting the claimed bound ∣Mti(s)∣=Θ((1+∣δ∣2)∣Mti(s⋆)∣)|M_t^i(s)|=\Theta((1+|\delta|^2)|M_t^i(s^\star)|) whenever Mti(s⋆)<0M_t^i(s^\star)<0.

  4. The convergence analysis does not cover the evaluated Sophia-based method. Equation (26) omits Sophia’s momentum and update clipping. The main experiments’ tensor-average slope estimates do not control the maximum directional drift appearing in the theory, and Appendix O’s quantile guarantee does not establish Assumption 7’s estimation-error bound.

5.The experiments do not isolate the benefit of curvature-based routing. Controls using matched-proportion random routing, fixed tensor assignments, or alternative routing criteria are missing, leaving effective step-size changes as a plausible explanation for the gains. Baseline search ranges and tuning budgets are also insufficiently specified.

Questions for authors
  1. What function class satisfies Assumption 2, and how should its relationship to standard Hessian Lipschitz continuity be corrected?

  2. How can Lemma 3.2 handle stationary negative-curvature directions, and how should Theorem C.1 be corrected?

  3. Can the authors establish tightness of the proposed worst-case envelope and connect the analysis to the implemented Sophia-based updates?

  4. Can matched-proportion routing controls and a clearly specified baseline-tuning comparison be provided?

Reviewer_jdKFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies when curvature information is worth its computational cost. It derives an idealized cubic-model optimizer, ELMO, and uses its behavior to motivate AMOS, which switches between SGD and Sophia based on estimated curvature drift. The experiments focus on language-model training with limited GPU resources.

Critical strengths
  1. I find the idea of combining SGD with second-order updates interesting. Deciding when the extra curvature information is worth the cost is an important practical problem.
  2. The authors discuss the limitations openly, including the extra memory cost and the less favorable exploratory large-model result.
  3. The authors provide extensive theoretical analysis, with detailed derivations and proofs supporting the motivation behind the proposed method.
Critical weaknesses
  1. I wonder how much the curvature-based routing itself helps. Comparing AMOS with pure SGD and Sophia is useful, but it does not show whether the routing score is better than a simpler way of mixing the two. A fixed or random assignment with the same fraction of Sophia updates and the same optimizer settings would make this clearer.

  2. The main evaluation uses a six-layer Transformer with batch size 4, which is quite small. I wonder how this affects training stability and the comparison between optimizers. Could the authors include an ablation with larger batch sizes? I would also like to see a more controlled comparison on a larger model.

Questions for authors
  1. Appendix L states that the Figure 1 runs use 16,010 iterations. Pure SGD seems to have a similar total runtime to AMOS; can you explain why the extra computations add so little overhead?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.