- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework for dynamically switching between first- and second-order optimizers based on the local Hessian Lipschitness. The authors first introduce ELMO, a Hessian eigenspace-based optimization method that uses a cubic surrogate model with eigenspace-wise Lipschitz constants. Building on this analysis, they develop AMOS, a meta-optimizer that dynamically assigns different parameter blocks to first- or second-order optimizers according to estimated curvature variation. Experiments on language modeling tasks demonstrate promising improvements over several existing optimizers.
Critical strengths
-
The paper explores an interesting direction by considering Hessian variation, rather than only curvature magnitude, as a criterion for optimizer selection.
-
The proposed framework provides a theoretically motivated connection between local quadratic surrogate fidelity and optimizer selection.
-
The experiments cover several language modeling datasets and compare against a relatively broad collection of optimizers, with promising empirical results.
Critical weaknesses
1. The eigenspace-wise Lipschitz assumption appears overly restrictive.
Assumption 2 imposes a strong structural condition on the variation of Hessian eigenvalues:
This assumption seems to be different from standard Hessian Lipschitz continuity and appears substantially stronger than the conventional assumption. I wonder if this influences the reliability of the analysis.
It would be important to clarify which generic classes of functions satisfy this assumption and whether it is reasonable for practical neural network training. Since the theoretical development relies heavily on this assumption, its restrictiveness raises concerns about the general applicability of the results.
2. The additional computational overhead is not sufficiently quantified.
Although the paper reports wall-clock performance and discusses amortizing the cost of Hessian computations, the additional overhead introduced by AMOS (e.g., overhead of maintaining and switching between optimizer states) is not thoroughly evaluated.
3. Missing ablation studies to establish the necessity of Lipschitz-aware optimizer selection.
The experiments show that dynamically combining SGD and Sophia can outperform either optimizer individually. However, it remains unclear whether these improvements arise from the proposed Lipschitz-based selection criterion or simply from assigning a subset of parameters to SGD instead of Sophia. It seems that simply replacing some Sophia updates with SGD may help. I would like to see comparisons against simpler hybrid baselines, such as random or fixed parameter assignments.
Questions for authors
See weakness