Newer Is Not Always Better: The Joint Role of AdamW Moment Histories

Controlled AdamW moment transplants across model families from 34M to 7B parameters reveal that a newer first moment paired with an older second moment can be far more disruptive than an older, coherent moment pair. We explain this asymmetry through history coverage: when the second moment covers the gradients represented in the first, a weighted Cauchy–Schwarz bound controls the normalized first moment. Equal-radius interventions in a 70M output head show that update magnitude alone cannot explain the damage; extended continuations reveal effects that outlast inherited moment differences. These findings motivate REBUILD_M: retain the second moment and reaccumulate the first from zero. In a common-gradient recovery model, this rule is minimax-optimal for cumulative absolute adaptive-update error relative to full-state recovery on a coverage-defined source class, given the retained second moment. For natural continuation, an exact reset representation links history coverage to a local validation-loss perturbation bound. Under uniform local regularity and a sufficiently small learning-rate scale $\eta$, complete first-moment deletion induces $O(\eta )$ loss perturbations over $O(1/\eta )$ updates, with an $O({\eta }^2)$ remainder in the signed response expansion. In 1B/7B recovery with the original batches, schedules, and data cursors, the policy saves one third of checkpoint payload with validation loss close to full-state recovery, including repeated 1B recovery. Constructing a compatible second moment repairs much of the damage from zeroing it, while retaining its actual history improves recovery further at the same payload.