- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper characterizes failure modes of the quadratic approximation in analyzing neural network training dynamics. Specifically, the authors identify Edge of Convexity (EoC): the dynamics do not diverge even though negative curvature persists. The authors also generalize Edge of Stability (EoS, the dynamics keeping the curvature near the threshold required for the descent lemma) to stochastic training with adaptive optimizers. The generalization defines EoS as the training learning rate being close to the mean-square stability threshold on the local stochastic quadratic.
The authors evaluate EoC and EoS in 3000 checkpoints from more than 100 pretraining runs. Across LLM pretraining, they observe EoC consistently and EoS whenever the learning rate and the batch size are not too small. They also report that the batch size below which EoS disappears correlates positively with the critical batch size.
Critical strengths
The experiments cover a larger scale than prior studies of EoS.
Critical weaknesses
The stability notion does not make sense for a nonconvex objective (soundness 1)
The authors define stability as boundedness of the distance to the stationary point of the local quadratic and argues that negative curvature is problematic because it makes the dynamics unstable in this definition. However, this definition of stability is not a useful notion in nonconvex optimization, and negative curvature does not break the optimization.
This notion of stability is not desired in nonconvex optimization. Stationary points with negative curvature directions are saddles. Near the stationary point that is a saddle, it is desired to move away from the point (be unstable in the defined notion of stability) to decrease the loss. So it is not problematic at all to be “unstable.” Refer to the literature on escaping saddles.
Optimization still works with negative curvature. A nonconvex function can be decreased even if the iterate moves in the opposite direction to (or away from) a stationary or target point. Therefore, when analyzing nonconvex optimization, it is customary to track the objective value rather than the distance to a specific point. Refer to the literature on smooth nonconvex optimization. The descent lemma works (so SGD decreases the objective in expectation) even with negative curvature as long as smoothness (spectral radius of the Hessian) is bounded.
Taken together, the main empirical analysis focuses on the wrong problem: EoC labels expected dynamics in nonconvex optimization as a failure.
The paper re-iterates well-known results (informativeness 1)
All analysis in the paper builds on SGD dynamics for local quadratic approximation of the objective function. The main conclusions are based on the observation that deep learning dynamics deviate from the quadratic optimization dynamics. However, all these analyses are elementary results from classic optimization theory, and the paper offers no new explanation of why the real dynamics deviate from the quadratic optimization dynamics.
Refer to the analysis of GD/SGD dynamics for quadratic functions:
https://www.damtp.cam.ac.uk/user/hf323/M21-II-NA/Lect18.pdf
https://www.cs.toronto.edu/~rgrosse/courses/csc2541_2021/slides/lec07.pdf
https://ernestryu.com/courses/MLT/chapter4.pdf
Zhang, G., et al., "Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model," in Advances in Neural Information Processing Systems, 2019.
Stability of SGD on quadratic approximations has been studied extensively:
L. Wu, C. Ma, W. E, "How SGD Selects the Global Minima in Over-parameterized Learning: A Dynamical Stability Perspective," in Advances in Neural Information Processing Systems, 2018.
L. Wu, W. Su, "The Implicit Regularization of Dynamical Stability in Stochastic Gradient Descent," in Proceedings of the 40th International Conference on Machine Learning, 2023, pp. 37656–37684.
R. Mulayoff, T. Michaeli, "Exact Mean Square Linear Stability Analysis for SGD," in Proceedings of Thirty Seventh Conference on Learning Theory, 2024, pp. 3915–3969.
Questions for authors
Doesn’t quadratic approximation still provide useful quantitative estimates locally, even with negative curvature directions?
What new result or insight does the paper provide beyond the existing stability analyses of SGD on quadratics and the original EoS observation?
We thank the reviewer for a careful report. The revised paper makes these changes. New text is marked in blue.