- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates deep learning optimization dynamics in terms of the effective learning rate. Specifically, the authors find that different optimizers share similar trends, and the effective learning rate essentially determines the regime. They categorize three regimes: low-LR softening, mid-LR sharpening, and high-LR edge. In the low-LR regime, the loss and the sharpness follow the same traces across optimizers, which indicates that all optimizers follow continuous dynamics. In the high-LR regime, the optimizer determines the sharpness on the edge of stability through the escape gain of its root locus. The mid-LR regime is where progressive sharpening happens, bridging the other two regimes.
The authors evaluate 10 optimizers with varying learning rates across 3 architectures on the CIFAR-10 dataset. They observe that the traces overlap across optimizers below the edge, and that the sharpness on the edge is reciprocal to the effective learning rate and the escape gain.
Critical strengths
The experiments cover a large range of learning rates that reveal the three regimes.
Critical weaknesses
The framework and the experiments are not explained (presentation 1)
The paper builds on the filter view of optimizers from Lee & Lee (2026), but it does not explain this framework fully. As a result, many terms are used before/without definition, including transfer function, dc gain, root locus, unit-circle escape, and escape angle. Without this background, it is difficult to appreciate the presented results fully.
The experimental settings are also not explained properly, so it is difficult to understand what is plotted and how the experiments are conducted. The paper does not state how the sharpness is computed, whether the loss is evaluated on the full training set or on mini-batches, and how random seeds affect the result. It is unclear how the final losses of different optimizers before the edge of stability can match so exactly.
Taken together, the reader cannot understand and verify the main observations from the paper alone.
Most observations are implied by Lee & Lee (2026) (informativeness 1)
Lee & Lee (2026) normalize the dc gain to Q(1)=1 so that the learning rate alone sets the gain. The effective learning rate of this paper essentially incorporates the effect of this normalization. They derive the stability condition based on the first unit-circle crossing of the root locus, covering the same set of optimizers in this paper. The high-LR relationship in this paper is equivalent to this condition.
The overlap below the edge is not in Lee & Lee (2026), but it is the well-known fact that, with small learning rates, gradient-based optimizers all follow the gradient flow with time rescaled:
https://francisbach.com/gradient-flows/
W. Su, S. Boyd, E. J. Candes. "A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights," in Journal of Machine Learning Research, vol. 17, no. 153, pp. 1–43, 2016.
N. B. Kovachki, A. M. Stuart. "Continuous Time Analysis of Momentum Methods," in Journal of Machine Learning Research, vol. 22, no. 17, pp. 1–40, 2021.
Questions for authors
Why do the final losses of different optimizers coincide so closely below the edge? Do the regimes persist when runs are compared at the same flow time, rather than at the same number of iterations?
What new result does the paper provide that is not implied by Lee & Lee (2026) and the effective learning rate of momentum methods?
2 comments
(Continued)
[W1, Q1] Why do the final losses coincide below the edges?
The first answer is that, because we reported the last 100 training losses averaged only in Figures 1 and 2 in order to summarize the entire training run in a single point, the losses are nicely stabilized in a visible trend. You can reproduce the results from our uploaded code. However, this accounts only for the losses after the edge. Before the edge, however, the raw training loss values themselves are actually smooth and coincide regardless of the optimizer or learning rate if carefully aligned in the coordinate (Figures 3d and 3e).
Therefore, by this question, the reviewer has actually pointed out one of our contributions: throughout this extensive optimizer and lr sweep, we found that the learning dynamics are very sharply decomposed into two regimes: the flow regime (previously split into low-LR softening and mid-LR sharpening), where all trajectories converge to the common canonical gradient flow , and the edge regime, where individual optimizers constrain the training dynamics through the edge of stability. Our experiments show that the flow regime covers almost the entire interval of learning rates below the edge of any optimizer (Figures 1, 2, and 4c). This means that the choice of optimizers to operate below the edge of stability has a negligible effect on training dynamics--they all converge to the same gradient flow dynamics.
We additionally provide numerical and theoretical analysis of this coincidence in the flow regime.
- Numerically: (lines 365-368). We can fit 0.9M data points from 146 different runs across 10 different optimizers with effective lr drawn in Figure 3d (training progress vs. loss ) onto a single shallow power law with great accuracy. We have with , a clear coincidence.
- Theoretically: (Theorem 3.1). As the reviewer has pointed out, it is known that previous works (Francis Bach, 2020, Su et al., 2016, Kovachki & Stuart, 2021) identified this small-lr flow limit in specific optimizers: GD, Heavy Ball, Nesterov's momentum. Our Theorem 3.1 extends this to general first-order optimizers with any number of momenta or gradient history buffers, including multi-momentum (AdEMAMix, AggMo, Grokfast) and gradient-augmented (QHM) optimizers, and quantifies the deviation of optimizer-induced dynamics from the gradient flow as . Hence coincidence in low lr.
We hope this answers your question.
[W2,Q2] Overlapping contributions with previous works
We thank the reviewer for the meticulous comment. Our contribution is neither a rediscovery of the gradient flow limit of a specific type of GD nor a new static root locus criterion for stability. Rather, it is the empirical, theoretical, and mechanistic connection between these two established explanations: when different types of optimizers become salient for the learning dynamics, and how these optimizer-dependent trajectories branch from the common flow evolution. We compare our contribution to the previous works as follows:
| Previous work | What was established | What is new in this work |
|---|---|---|
| Cohen et al. (2021) | Discovery of the edge of stability & progressive sharpening, flow-to-edge transition with a fixed optimizer type: GD and GD with momentum. | We study the mechanism of multiple optimizer-dependent trajectories that branch from a single gradient flow trajectory in the common low-lr limit. |
| Lee & Lee (2026) | Derived the root locus stability criterion from static analysis, full-batch training | We extend this approach to dynamic analysis on the root locus in the flow-to-edge transition of dynamics. |
| Su et al. (2016) | Continuous-time second-order ODE limit for NAG | We study general causal gradient filters and relate their small-step dynamics to a common first-order gradient flow. |
| Kovachki & Stuart (2021) | Rescaled gradient flow limit for two types of single-momentum optimizers, HB and NAG. | We extend this continuous-time flow limit for single momentum-based optimizers to general causal linear optimizers that include any parameter update from linear combinations of one or more momenta and historical gradients. |
We hope this clarification helps identify our contributions. We have updated our Sections 1 and 2 to sharpen this claim. Again, we thank the reviewer for helping us make this work clearer.
Overall, the reviewer's comments have been of great help to us in reforming this manuscript to sharpen our contributions and improve our presentation. We would like to ask the reviewer to have a look at the revised manuscript as well as the summary we provided in the general remark above. We are grateful to listen to any additional concern from the reviewer.
We deeply appreciate the reviewer for taking the time to leave thoughtful and constructive feedback. We acknowledge that the previous version of the manuscript has obscured our contribution and the implications of our experimental results, despite the extensiveness of its coverage. Based on the reviewer's feedback, we have strengthened the theoretical and mechanistic explanations in the manuscript, improved the presentation, and further clarified our contribution claims in the uploaded revision. We answer each of the raised concerns below:
[W1] Presentation.
We thank the reviewer for pointing out this issue.
Regarding signal processing jargon: We fully revised our manuscript (especially Section 1) to make sure that no undefined terminology disrupts the reading flow. Moreover, we enhanced our explanation of the optimizer-as-a-filter framework in Section 2, which is relatively uncommon in this field yet plays an important role in our paper. We made sure that the definitions of the technical terms are included. We respectfully note that every concept used in this work is from well-established undergraduate-level signal processing and control systems theory. Please take a look at the changes made in the revised manuscript, and please feel free to raise any additional concern.
Regarding experimental settings: We enhanced the implementation details in Appendices C.1 and C.2 on the network architecture, the training process, the computation of the optimizer-related quantities, and how we computed the sharpness value. Specifically, the losses are computed for a training batch throughout the run, which is rather a standardized practice. We used an averaged value of the loss only for Figures 1 and 2 from the last 100 iterations of the 200k run to better summarize the results and to visualize the trends. We included these details in the affected figures' captions. Other figures, e.g., training traces of Figure 3 and full training traces in Appendix C.7, do not use this aggregation, and show the raw training losses instead. We gently inform the reviewer that the entire reproducible code for this work has been uploaded as supplementary material from the first submission of this work.
We would like to ask the reviewer to take another look at this revised manuscript, which has been greatly reformed in full respect of the reviewer's comment. If you find any incomprehensible or unverifiable point in our work, please let us know at any time. We are grateful to receive any comment.
[Q1] Do the regimes persist when runs are compared at the same flow time?
We deeply thank the reviewer for this concern. After receiving the reviews, we find that merging the three-regime classification of low-LR softening, mid-LR sharpening, and high-LR edge into two regimes of (1) flow (= softening + sharpening, Section 3) and (2) edge (=edge, Section 5) more clearly delivers our core contribution, to empirically and theoretically explain how optimizer-dependence in the latter regime emerges from the trajectory coincidence in the former regime. In the revised version of the manuscript, we provide a dedicated section, Section 4, with a mechanistic explanation of this emergence of optimizer-dependence as the learning rate increases.
And, with this simplified classification, the answer is yes. Please see Figure 4. To answer the reviewer's question further, we conducted a 100M-iteration (×500 longer than the main experiments) run on the small-lr SGD, representing the flow regime, and aligned it with the main sweep results in the flow time τ-coordinate. The first thing we notice is that the edge regime and the flow regime are clearly different classes. With higher learning rates, depending on the type of optimizer, the sharpness curve escapes from the same curve of the flow regime down to the reciprocal slope of the edge of stability. Furthermore, the progressive sharpening effect is a property of the common gradient flow, rather than individual optimizers. The choice of optimizers and higher learning rates only reduce the upper-bounded sharpness curve through their own edges of stability. The sharpness from the flow regime, on the other hand, saturates to the maximal value, once the progressive sharpening takes place. We hope this clarifies the question.
(Continued)