Memory Renormalizes Stabilizer Annealing in Adam
Abstract
Adam's memory parameters and denominator stabilizer both shape its dynamics, raising a natural question: when can a memory change be compensated by retuning the stabilizer, and what can such compensation preserve? For deterministic full-batch Adam on separable linear data, with sufficiently small fixed positive steps and an exponentially annealed outside-root stabilizer with polynomial modulation, the two moment filters renormalize the schedule into an effective rate and shape . These coordinates determine all divergent terms of the parameter trajectory, including iterated-logarithmic corrections on degenerate margin geometries. Matching is necessary and sufficient for the trajectory difference to converge to a finite vector under the prescribed effective clocks, while rate matching alone can leave logarithmic drift. Yet trajectory equivalence need not imply relative-risk equivalence: the residual finite offset can yield a nonunit asymptotic risk ratio. Preserving relative risk at equal effective time additionally requires an amplitude coordinate, whereas equal-update risk is governed by a distinct complete invariant; the two preservation targets can be incompatible. Experiments on actual discrete Adam iterates support these separating predictions across regular, degenerate, and native-softmax geometries. Together, these results give a clock- and observable-dependent classification of equivalence under Adam memory changes.
est. 32% chance this paper gets accepted at ICLR 2027.
What do you think this paper will get?
All positions stay anonymous.