Optimizer selection based on function-proxy fidelity

Optimizing highly nonconvex loss functions is central to modern machine learning, making optimizer selection both important and difficult. Many training algorithms can be interpreted as minimizing local surrogates of the loss, typically induced by Taylor approximations, but it is often unclear which surrogate is most appropriate for a given problem. At the same time, Hessian drift is poorly understood, and existing second-order methods typically rely on Tikhonov regularization to cope with concavity and departures from local quadratic structure. This regularization introduces additional tuning and can weaken the very second-order information these methods seek to exploit. We present a novel eigendecomposition-centric analysis of Hessian drift and use it to derive a new Hessian regularization method together with a meta-algorithm for optimizer selection. Experiments validate the effectiveness of our algorithms.