- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work investigates why momentum helps normalized optimizers in the deterministic regime (evidenced empirically in Figure 1), while existing theory on noise reduction or acceleration on quadratics does not predict improvement in that setting. This work shows that in a two-dimensional valley loss landscape, the use of normalization creates two regimes of training: first a stability window succeeded by oscillations. The use of momentum both delays the latter and levels up the overall learning rate in both regimes, which leads to improvement. Based on this, the authors introduce the switch flow, which transitions between these two regimes using gradient and sharpness estimates in neural network training. They empirically show that this flow indeed transitions between these two regimes with the predicted increase of the effective learning rate when using momentum. Finally, the authors rigorously prove that NormGD-M follows the switch flow on general losses in the oscillating regime and that, on a quadratic loss, NormGD-M involves these two regimes for almost every initialization and that momentum delays the oscillating regime and increases the effective learning rate in that regime.
Critical strengths
A main strength of that work is the clarity and the flow of the paper. The motivation is very well explained in the introduction: momentum benefits are not currently well explained by either deterministic studies or through the lens of acceleration on quadratics when using normalization, which motivates a clear question on understanding 'why does momentum help normalized optimizers in deterministic training '". Additionally, the structure of the paper is very clear, with a heuristic/theoretical derivation of the switch flow in Section 2, an empirical study of the predictions of that framework in practice on deterministic neural network training, and finally theoretical rigorous results on high-dimensional quadratics further supporting the study.
A second main strength is the novelty of the switch flow, which helps explain why momentum helps the normalized optimizer by evidencing two regimes of training whose transition and levels are dictated by the momentum parameter .
Critical weaknesses
A weakness of that work is the limited empirical validation of that framework. The experiments are done on an MLP with only 20k parameters and on synthetic data, which is at odds. It would have been very interesting to push that empirical investigation a bit further to test the limits of that framework, especially in more realistic settings with real data.
Additionally, the novelty of the multiplicative factor on the effective learning rate for momentum at the edge of stability is unclear, as it already appeared in [1] in a similar context.
[1] Cohen, Jeremy M., et al. "Adaptive gradient methods at the edge of stability." arXiv preprint arXiv:2207.14484 (2022).
Questions for authors
-
In Section 2.3, the authors explain that the previous parameter from the loss-valley model in equation 2 can be recovered by the gradient . However, the gradient , when applied to the model in Equation 2, has two components: one from the linear part and one from the quadratic gradient, which is not optimal: . Can the authors explain whether that would introduce a discrepancy between the previous theoretical discussion and the switch flow in Equation 7?
-
A similar factor was evidenced in [1] in the context of Adam at the edge of stability. Could the authors comment on the link between their study and that work and the novelty of this factor regarding momentum methods at the edge of stability?
4 comments
Response to Reviewer bsbK (Part II)
W2 & Q2. The factor .
The factor is known. We agree that the same stability threshold with momentum has already been studied in the literature for different settings. Throughout, we write heavy-ball momentum in the EMA form, , as in our paper. For example:
- Polyak (1964); Goh (2017): on a quadratic, heavy-ball momentum is stable if and only if .
- Cohen et al. (2021): gradient descent with heavy-ball momentum diverges on a quadratic once the sharpness exceeds (their Appendix B), and in neural network training the sharpness rises to this threshold and plateaus there (see Figure 7 and Appendix N of their paper).
- Cohen et al. (2022) [1]: for full-batch Adam, the preconditioned sharpness equilibrates near , derived with the preconditioner held fixed.
- Lu et al. (2026): on river-valley losses, gradient descent with heavy-ball momentum tolerates learning rates up to about , that is, times the threshold of plain gradient descent. At a fixed learning rate, its speed along the valley barely depends on , so the acceleration comes primarily from the larger admissible learning rate.
We remark that the appearance of the same in these stability thresholds over sharpness and learning rates is rooted in the same linearization: about straight motion along the valley floor, the transverse dynamics of NormGD-M follow the heavy-ball recurrence with effective learning rate (Appendix E). And we do not claim novelty over identifying this specific stability threshold.
Acceleration of training progress. Instead, our finding that momentum can accelerate the optimization by a factor of for the same learning rate is novel. Specifically, we show that when both have entered the oscillation phase, NormGD-M moves times faster than NormGD at the same learning rate (Eq. (6); Theorem 4.2; Figure 4). Indeed, this is a joint effect of normalization and momentum. Without normalization, gradient descent with momentum moves per update for every , and momentum only raises the threshold (Lu et al., 2026). To our knowledge, this acceleration at a fixed learning rate has not been characterized before. Moreover, based on this characterization of the oscillation phase, we are able to derive a unified switch-flow framework that describes the whole training dynamics (Eq. (7); Figure 5).
I thank the authors for their detailed answer to my questions.
The additional experiments strengthen the theoretical results in particular using transformers and real data experiments. I appreciate the discussion on where the gradient is taken between the discrete updates and the flow.
I also appreciate the discussion on the novelty of the factor in particular the emphasis on the acceleration of training progress.
I will keep my score and recommend acceptance, as the paper's results are strong and clear. Reasons for not a higher score are the novelty and significance of the results.
Thank you for taking the time to read our response. We greatly appreciate your help to improve the quality of our work, especially on your questions about the the presentation gap and the suggestions to improve experiments.
Response to Reviewer bsbK (Part I)
We thank the reviewer for the constructive suggestions on extending experiments, presentation gaps and highlighting novelty against prior literatures. Below we respond to your questions and weaknesses.
W1. Realistic setting experiments.
We thank the reviewer for this suggestion. We added two settings with real data, a fully connected network on CIFAR-10 and a two-layer Transformer on C4, and tested the predictions of Section 3 on them (Table R1; see Appendices B.7 and B.8 in the revision for details).
Table R1. Predictions of Section 3 on three settings. Each entry holds for every β; the loss error is defined as in Figure 5 (Appendix B.1).
Table R2 reports this error over a range of learning rates: in most cases, the switched flow predicts the loss to within a few percent.
Table R2. Relative error of the loss predicted by the switched flow of Eq. (7), over a range of learning rates η.
We also include the 14.6M-parameter language model of Figure 1 (Appendix B.4) as qualitative evidence at a larger scale.
Q1. Gradient on the switch flow.
We thank the reviewer for pointing out the gap between the discrete iterates and the flow, especially where the gradient is taken. We answer the question in two parts:
Center of the discrete iterations. At an individual iterate at Eq. (2), the gradient is (λuk,b), whose norm exceeds b during the oscillation. On the period-two orbit, the buffer is mk=(±μ,b), where μ is its sharp component: μ flips sign at every update, so it cancels from the two-step average, while ∥mk∥ stays fixed. The center xˉk=(xk+xk+1)/2 therefore moves by
xˉk+1−xˉk=−∥mk∥η(0,b),∥mk∥η=λ2Kβ,
where the second identity is the period-two condition, which holds exactly on the period-two orbit: the transverse multiplier 1−ηλ/(Kβ∥mk∥) must equal −1. The norm λ2uk2+b2 thus enters only the normalizer, and the progress is carried by b alone (see Appendix B.6 in the revision for an illustration).
From center to switch-flow. We introduce the continuous flow in order to capture the movement of the oscillation center in the training process. In our formulation in Eq. (7), the gradient is evaluated on the flow point itself. This is valid for two reasons:
We verify the second point empirically on our teacher–student task: along the gradient-flow path, the cosine between the gradient and the top Hessian eigenvector remains negligible, below 3×10−4 in absolute value (see Appendix B.6 in the revision). Since the ∥∇L∥ in Eq. (7) is evaluated at the center, where it equals b, Eq. (7) gives the progress speed min{η,2Kβb/λ}, which matches the displacement per update in Eq. (6). Hence there is no discrepancy; we now state explicitly in Section 2.3 of the revision that the gradient is evaluated at the center.