- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The manuscript focuses on the training effect by using the same mini-batch. The authors study the coupling relationship between update, data order. They also provide somehow a little bit "formula-style" theorectical results and corresponding supportive experiments.
Critical strengths
The pipeline design is clean. The authors have acknowledged about the limitations of their work.
Critical weaknesses
My main concern is the theorecitical novelty since it is already known that Adam update dynamics is bounded (Kingma & Ba, 2015, Zou et al., 2019, Defossez et al., 2022). That is because Adam optimizer uses the second-moment of gradient as the denominator. So, if the gradient is large, then the update should be should, but the second-moment of gradients are larger than the numerator making the update magnitude small, that is why these "large updates" are bounded. That is why Adafactor and StableAdamW use additional mechanisms to control this thing. I do not understand what new understanding here and what "different" pratical benefits this manuscript can offer.
Another concern is Theorem 1 only consider one-step first order effect. This means it is unclear why and whether "this work" can explain the long traning runs behavior. Especially, it is unclear whether it work in large models or not.
Questions for authors
Can the authors add experiments that controls for update size? e.g., in ResNet-18 setting? If we do not control this, we can not tell whether the increased order dispersion comes from which part, e.g., covariance term C or large update?
Does the Pythia divergence occur under the gradient clipping and even under realistic \epsilon? The problem is that cipping can rescure the decoupled arm, if so, what these results can tell us beyond previous existing methods?
In Theorem 1, damping requires the formula [Consequently: (i) on page 3] per coordinate. To do an alabtion experiemnt, I think the authors can measure how many coordinates actually satidfy the consition during training?