Navigating Potholes with Geometry-Aware Sharpness Minimization

Sharpness-aware minimization (SAM) biases optimization toward flat neighborhoods by evaluating gradients at adversarially perturbed parameters, but standard SAM defines these perturbations in Euclidean geometry. Geometry-aware optimizers instead precondition the descent direction using curvature information. We combine these ideas using LLQR, which learns a structured, slowly varying inverse metric from layerwise network dynamics. LLQR+SAM uses this metric both to shape the SAM perturbation and to precondition the gradient evaluated at the perturbed parameters. On a two-scale quadratic model, we show that the learned geometry enlarges the SAM exploration scale in directions that are flat under the background geometry but locally sharp, providing a mechanism for escaping narrow sharp wells (“potholes”). Empirically, LLQR+SAM consistently improves over SAM and LLQR across CIFAR-10/100, TinyImageNet, ImageNet, and IWSLT14, with analogous gains when combined with recent SAM variants such as F-SAM. These results support the view that slow learned geometry and fast sharpness-aware perturbations can be complementary.