- Rating
- 1(−1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper investigates reusing old first and second AdamW moments. The paper finds that when the first moment estimate is newer than the second moment, continuing to pretrain using these moments as a starting place can cause substantial forgetting. However, using first moments that are older than the second moment is often not consequential for forgetting. The paper takes advantage of this by proposing a method that retains only the second moment in its checkpoint and re accumulates the first moment, with little-to-no impact on performance.
Critical strengths
The main observation that in order for the first and second moments to be compatible, the first moment must be at least as old as the second moment is interesting.
The experimental methodology seems sound.
Critical weaknesses
Presentation: Writing is poor and would need substantial revision to be acceptable for publication.
- Beginning in the abstract, the basic terms aren't clear and don't introduce the problem precisely, for example, it's not clear what is meant precisely by "disruptive" or what an "equal-radius intervention" might be (this is two examples, there are multiple).
- Contributions have a similar issue where it's not clear what they mean
- Section 2 largely uses notation and terminology that aren't defined, nor is the basic problem introduced
Impact: While the results are interesting, it's not clear exactly what the purpose of this study is.
- The main benefit is saving only 1/3 of the checkpoint size. The introduction doesn't introduce why this is important, and so it's not clear that this is impactful.
- If this disk-saving benefit is in fact the purpose of this study, it would be important to compare it against other simple methods for saving disk space, such as quantizing the moments, or via standard compression.
Soundness/completeness:
- One of the major missing pieces, in my opinion, is to better understand how the learning rate schedule interacts with the mismatch of moments. Since the OLMo and I believe Pythia family of models uses a cosine learning rate, taking local moment histories from multiple steps would select moments computed at different learning rates. The dynamics of model training may differ substantially a different learning rates, especially as the learning rate decays (see, e.g., [1]). I think it would be important to disentangle whether or not the observation about mismatched moments causing more forgetting arises as a result of a factor like decaying learning rate, or even potentially some other hyperparameter.
[1] Cohen et al. Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability. ICML 2022.
Questions for authors
See weaknesses
Reminder: halfway through discussion period
This is a friendly reminder to the reviewers that we are halfway through the discussion period. The authors have posted responses to the reviews, so please acknowledge the rebuttal, and discuss their response and remaining questions.
Best, AC