Fine-grained Analysis of Adam's Implicit Bias Near the Manifold

Understanding the implicit biases of different optimizers is one of the key challenges in studying the training dynamics of deep learning. However, the implicit bias of Adam, one of the most widely used adaptive gradient methods (AGMs), is still not fully understood theoretically. li2025adam studied the implicit bias of Adam with a second-moment momentum coefficient $1-{\beta }_2=\Theta ({\eta }^2)$, a setting they termed the 2-scheme. They proved that, under the 2-scheme, Adam evolves near the manifold of training loss minimizers and implicitly reduces a form of sharpness different from that associated with SGD over an $O(1/{\eta }^2)$ timescale. However, in practice, ${\beta }_2$ is much larger than ${\eta }^2$. This leaves an open theoretical question: what is the implicit bias of Adam in the $\alpha$-regime with $\alpha <2$? In this work, we resolve this question. We first extend the convergence results of li2025adam to $1\le \alpha <2$ and provide counterexamples showing that such convergence can fail for $\alpha <1$. We then characterize Adam's implicit bias for $1\le \alpha <2$ by developing a new framework for analyzing the limiting flows of AGMs, including Adam, over long training horizons. This framework generalizes the ideas in katzenberger1991solutions and also provides insight into a broader class of stochastic processes beyond AGMs.