acceptodds
Under review as a conference paper at ICLR 2027

Newer Is Not Always Better: The Joint Role of AdamW Moment Histories

Abstract

Controlled AdamW moment transplants across model families from 34M to 7B parameters reveal that a newer first moment paired with an older second moment can be far more disruptive than an older, coherent moment pair. We explain this asymmetry through history coverage: when the second moment covers the gradients represented in the first, a weighted Cauchy–Schwarz bound controls the normalized first moment. Equal-radius interventions in a 70M output head show that update magnitude alone cannot explain the damage; extended continuations reveal effects that outlast inherited moment differences. These findings motivate REBUILD_M: retain the second moment and reaccumulate the first from zero. In a common-gradient recovery model, this rule is minimax-optimal for cumulative absolute adaptive-update error relative to full-state recovery on a coverage-defined source class, given the retained second moment. For natural continuation, an exact reset representation links history coverage to a local validation-loss perturbation bound. Under uniform local regularity and a sufficiently small learning-rate scale , complete first-moment deletion induces loss perturbations over updates, with an remainder in the signed response expansion. In 1B/7B recovery with the original batches, schedules, and data cursors, the policy saves one third of checkpoint payload with validation loss close to full-state recovery, including repeated 1B recovery. Constructing a compatible second moment repairs much of the damage from zeroing it, while retaining its actual history improves recovery further at the same payload.

Primary area:
optimization
Keywords:
AdamW, adaptive optimization, optimizer-state compatibility, moment histories, training recovery, selective checkpointing, checkpoint compression

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VwMAOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates reusing old first and second AdamW moments. The paper finds that when the first moment estimate is newer than the second moment, continuing to pretrain using these moments as a starting place can cause substantial forgetting. However, using first moments that are older than the second moment is often not consequential for forgetting. The paper takes advantage of this by proposing a method that retains only the second moment in its checkpoint and re accumulates the first moment, with little-to-no impact on performance.

Critical strengths

The main observation that in order for the first and second moments to be compatible, the first moment must be at least as old as the second moment is interesting.

The experimental methodology seems sound.

Critical weaknesses

Presentation: Writing is poor and would need substantial revision to be acceptable for publication.

  • Beginning in the abstract, the basic terms aren't clear and don't introduce the problem precisely, for example, it's not clear what is meant precisely by "disruptive" or what an "equal-radius intervention" might be (this is two examples, there are multiple).
  • Contributions have a similar issue where it's not clear what they mean
  • Section 2 largely uses notation and terminology that aren't defined, nor is the basic problem introduced

Impact: While the results are interesting, it's not clear exactly what the purpose of this study is.

  • The main benefit is saving only 1/3 of the checkpoint size. The introduction doesn't introduce why this is important, and so it's not clear that this is impactful.
  • If this disk-saving benefit is in fact the purpose of this study, it would be important to compare it against other simple methods for saving disk space, such as quantizing the moments, or via standard compression.

Soundness/completeness:

  • One of the major missing pieces, in my opinion, is to better understand how the learning rate schedule interacts with the mismatch of moments. Since the OLMo and I believe Pythia family of models uses a cosine learning rate, taking local moment histories from multiple steps would select moments computed at different learning rates. The dynamics of model training may differ substantially a different learning rates, especially as the learning rate decays (see, e.g., [1]). I think it would be important to disentangle whether or not the observation about mismatched moments causing more forgetting arises as a result of a factor like decaying learning rate, or even potentially some other hyperparameter.

[1] Cohen et al. Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability. ICML 2022.

Questions for authors

See weaknesses

Reviewer_6e1FOpenReview ↗
Rating
2
Confidence
3(+1)
Informativeness
2
Presentation
1(−2)
Soundness
2
Read review
Paper summary

The paper studies how AdamW's two moments should be paired when training resumes from a checkpoint. The authors run controlled interventions that fix weights, clock, data order, and randomness, and only swap the age of the first and second moments. Across models from 34M to 7B, a new first moment with an old second moment hurts recovery far more than the reverse. They explain the asymmetry with history coverage, saying that when the second moment covers the gradients in the first, a weighted Cauchy-Schwarz bound controls the first. From this they derive REBUILD_M, which keeps the second moment and rebuilds the first from zero. On 1B and 7B runs with the original batches and schedules, it cuts checkpoint size by one third while validation loss stays close to full recovery, and it composes with BF16 and 8 bit quantization.

Critical strengths
  1. The controlled design isolates source pairing from everything else. Weights, clock, and future batches are fixed while only the two moment ages vary, and the equal-radius output-head tests separate direction from magnitude. This makes the result credible rather than a correlation read off training curves.
  2. The coverage explanation gives a testable prediction, and they actually test it. Raising q from 0.95 to 1.05 increases covered-side damage, matching the Cauchy-Schwarz bound. The final rule REBUILD_M is concrete and useful for large-model training, saving one third of the payload with near-FULL quality.
Critical weaknesses
  1. The theory is idealized. The minimax statement assumes scalar Adam, common future gradients, no weight decay, and epsilon equal to zero. The loss bound also needs local smoothness and a small enough learning rate. The gap to real 1B/7B training is large, and the paper does not say whether the result survives when these are relaxed.
  2. The experimental scope is narrow. Recovery is tested over 128 to 512 updates, always on autoregressive language-model pretraining. The 27B experiment covers only FULL versus REBUILD_M at one checkpoint. Vision, fine-tuning, reinforcement learning, or longer recovery windows remain open.
Questions for authors
  1. The theory assumes common future gradients, no weight decay, and epsilon equal to zero. If these are relaxed, could deleting the first moment become costly enough that REBUILD_M is no longer near-lossless?
  2. You report only 128 to 512 updates. If the window extends to thousands of updates so the rebuilt first moment fully catches up, does the gap between REBUILD_M and FULL shrink or grow? Does REBUILD_MV catch up?
  3. In Table 1 REBUILD_M has a negative delta NLL, so it sometimes beats FULL by a small amount. Is that noise, or does zeroing the first moment remove a stale numerator bias? This affects how the method should be read.
Reviewer_YsCzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper explores how AdamW's first-moment (m) and second-moment (v) histories jointly affect training recovery. Based on controlled moment transplants across models from 34M to 7B parameters, the authors find that pairing a newer m with an older v is far more damaging than using an older but source-coherent (m, v) pair. They explain this via history coverage and propose REBUILD_M: retain v, zero m, and reaccumulate. This method saves ~1/3 of checkpoint payload while maintaining near-FULL validation loss.

Critical strengths
  • Moment-source transplants fix weights, clock, scheduler, RNG, and future batches, cleanly isolating the effect of moment pairing.

  • The source-pairing asymmetry is reproduced across Pythia (1B/6.9B), Mistral (7B), and OLMo-2 (1B/7B), strengthening generalizability.

  • The coverage bound, minimax analysis, and loss-perturbation theorem provide a principled rationale for a simple storage-saving policy.

Critical weaknesses
  • REBUILD_M's NLL differences from FULL are ~−0.0003 nats/token (~0.03% perplexity). These are effectively zero, and the policy is slightly better than FULL, i.e., a counterintuitive result likely due to noise, undermining the claim of near-FULL quality.

  • Experiments run only 128–512 updates. For real training runs spanning hundreds of thousands of steps, this offers no evidence that the benefit persists.

  • Proposition 1 relies on several restrictive assumptions, i.e., a scalar Adam setting, zero weight decay, ϵ=0\epsilon = 0, and identical future gradients, none of which hold in practice. Proposition 2 predicts O(η)O(\eta) loss perturbations, yet experiments show NLL spikes from ~6 to ~260 nats/token. The theory does not explain the paper's most striking empirical phenomenon.

  • "Keep v, drop m" is a simple heuristic. History coverage reduces to a direct Cauchy–Schwarz application. The paper wraps a lightweight idea in heavy formalism without commensurate depth of new insight.

Questions for authors
  • In both 1B (−0.000300) and 7B (−0.000369) native recovery, REBUILD_M achieves slightly lower NLL than FULL. Since REBUILD_M has strictly less information, this should not happen systematically. Is this purely noise, or is there a regularization-like effect from zeroing m that the authors can characterize?

  • The experiments use 128–512 update windows. In practice, training may run 10k–100k steps between faults. Could the accumulated parameter displacement from repeated m-resets compound over such horizons, eventually causing divergence or loss spikes that are absent in these short-window tests?

Comments on the paper

1 comment
Area_Chair_UvC6OpenReview ↗

Reminder: halfway through discussion period

This is a friendly reminder to the reviewers that we are halfway through the discussion period. The authors have posted responses to the reviews, so please acknowledge the rebuttal, and discuss their response and remaining questions.

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.