Sharpness Corrections Need Not Act Immediately in Momentum-Based SAM
Abstract
Sharpness-Aware Minimization (SAM) evaluates gradients at perturbed parameters, but under momentum the resulting sharpness correction acts through an immediate zero-lag route and a delayed route carried by the momentum state. The central question is whether a generated correction must act immediately to remain useful. Temporal Feedback SAM (TF-SAM) isolates this choice by removing the zero-lag route while preserving correction generation and its momentum-mediated influence. An exact original-iterate identity shows that this intervention replaces the fresh correction at the target update with its momentum-weighted previous counterpart. A target-matched first-order criterion then shows that the preferred route depends on the corrections' relative alignment with the current descent direction. At regular training states, the normalized-SAM correction has an explicit Jacobian, and the two routes enter different coefficients of the local closed-loop characteristic matrix polynomial. Across the evaluated image and text settings, removing zero-lag feedback does not produce a consistent loss in predictive accuracy or aggregate corruption robustness; target-matched measurements show that the relative first-order value of the routes varies across regimes. Generating sharpness information and applying it immediately are therefore separable design decisions in momentum-based SAM, while holding correction generation fixed.
est. 32% chance this paper gets accepted at ICLR 2027.
What do you think this paper will get?
All positions stay anonymous.