acceptodds
Under review as a conference paper at ICLR 2027

A Journey to the Edge of Stability

Abstract

It has recently been found that deep learning often occurs at the "edge of stability (EoS)," where the maximum Hessian eigenvalue of the model is stabilized at a value reciprocal to the learning rate. However, what happens before we reach that regime? We fix a deep learning problem and vary first order optimization methods with dense learning rate sweeps. We then track the characterizing quantities of a learning trajectory: the loss, the sharpness, and the alignment between consecutive gradients. To our surprise, if we scale the learning rate by the dc gain of the optimizer, these traces from the sweeps from different optimizers almost perfectly overlap across a large range of learning rates. The dc-normalized optimizers have another role that only becomes apparent in high learning rates: they select when the sharpness value detaches from this universal curve and enters the edge of stability. Upon this discovery, we specify three distinct regimes with respect to the dc-adjusted learning rate: the low-LR regime where the trajectory is nearly insensitive to the optimizer, the high-LR regime, where the optimizer governs the sharpness according to the EoS reciprocal rule, and the in-between mid-LR regime where so-called progressive sharpening originates independently of the optimizer. This distinguishes the role of the optimizer, the learning rate, and the model in shaping the learning progress.

Primary area:
learning theory
Keywords:
optimization, edge of stability, root locus, sharpness, Hessian, curvature curriculum, learning rate, learning rate sweep, material property

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bMFiOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates deep learning optimization dynamics in terms of the effective learning rate. Specifically, the authors find that different optimizers share similar trends, and the effective learning rate essentially determines the regime. They categorize three regimes: low-LR softening, mid-LR sharpening, and high-LR edge. In the low-LR regime, the loss and the sharpness follow the same traces across optimizers, which indicates that all optimizers follow continuous dynamics. In the high-LR regime, the optimizer determines the sharpness on the edge of stability through the escape gain of its root locus. The mid-LR regime is where progressive sharpening happens, bridging the other two regimes.

The authors evaluate 10 optimizers with varying learning rates across 3 architectures on the CIFAR-10 dataset. They observe that the traces overlap across optimizers below the edge, and that the sharpness on the edge is reciprocal to the effective learning rate and the escape gain.

Critical strengths

The experiments cover a large range of learning rates that reveal the three regimes.

Critical weaknesses

The framework and the experiments are not explained (presentation 1)

The paper builds on the filter view of optimizers from Lee & Lee (2026), but it does not explain this framework fully. As a result, many terms are used before/without definition, including transfer function, dc gain, root locus, unit-circle escape, and escape angle. Without this background, it is difficult to appreciate the presented results fully.

The experimental settings are also not explained properly, so it is difficult to understand what is plotted and how the experiments are conducted. The paper does not state how the sharpness is computed, whether the loss is evaluated on the full training set or on mini-batches, and how random seeds affect the result. It is unclear how the final losses of different optimizers before the edge of stability can match so exactly.

Taken together, the reader cannot understand and verify the main observations from the paper alone.

Most observations are implied by Lee & Lee (2026) (informativeness 1)

Lee & Lee (2026) normalize the dc gain to Q(1)=1 so that the learning rate alone sets the gain. The effective learning rate of this paper essentially incorporates the effect of this normalization. They derive the stability condition based on the first unit-circle crossing of the root locus, covering the same set of optimizers in this paper. The high-LR relationship in this paper is equivalent to this condition.

The overlap below the edge is not in Lee & Lee (2026), but it is the well-known fact that, with small learning rates, gradient-based optimizers all follow the gradient flow with time rescaled:

https://francisbach.com/gradient-flows/

W. Su, S. Boyd, E. J. Candes. "A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights," in Journal of Machine Learning Research, vol. 17, no. 153, pp. 1–43, 2016.

N. B. Kovachki, A. M. Stuart. "Continuous Time Analysis of Momentum Methods," in Journal of Machine Learning Research, vol. 22, no. 17, pp. 1–40, 2021.

Questions for authors

Why do the final losses of different optimizers coincide so closely below the edge? Do the regimes persist when runs are compared at the same flow time, rather than at the same number of iterations?

What new result does the paper provide that is not implied by Lee & Lee (2026) and the effective learning rate of momentum methods?

2 comments

We deeply appreciate the reviewer for taking the time to leave thoughtful and constructive feedback. We acknowledge that the previous version of the manuscript has obscured our contribution and the implications of our experimental results, despite the extensiveness of its coverage. Based on the reviewer's feedback, we have strengthened the theoretical and mechanistic explanations in the manuscript, improved the presentation, and further clarified our contribution claims in the uploaded revision. We answer each of the raised concerns below:


[W1] Presentation.

We thank the reviewer for pointing out this issue.

  • Regarding signal processing jargon: We fully revised our manuscript (especially Section 1) to make sure that no undefined terminology disrupts the reading flow. Moreover, we enhanced our explanation of the optimizer-as-a-filter framework in Section 2, which is relatively uncommon in this field yet plays an important role in our paper. We made sure that the definitions of the technical terms are included. We respectfully note that every concept used in this work is from well-established undergraduate-level signal processing and control systems theory. Please take a look at the changes made in the revised manuscript, and please feel free to raise any additional concern.

  • Regarding experimental settings: We enhanced the implementation details in Appendices C.1 and C.2 on the network architecture, the training process, the computation of the optimizer-related quantities, and how we computed the sharpness value. Specifically, the losses are computed for a training batch throughout the run, which is rather a standardized practice. We used an averaged value of the loss only for Figures 1 and 2 from the last 100 iterations of the 200k run to better summarize the results and to visualize the trends. We included these details in the affected figures' captions. Other figures, e.g., training traces of Figure 3 and full training traces in Appendix C.7, do not use this aggregation, and show the raw training losses instead. We gently inform the reviewer that the entire reproducible code for this work has been uploaded as supplementary material from the first submission of this work.

We would like to ask the reviewer to take another look at this revised manuscript, which has been greatly reformed in full respect of the reviewer's comment. If you find any incomprehensible or unverifiable point in our work, please let us know at any time. We are grateful to receive any comment.


[Q1] Do the regimes persist when runs are compared at the same flow time?

We deeply thank the reviewer for this concern. After receiving the reviews, we find that merging the three-regime classification of low-LR softening, mid-LR sharpening, and high-LR edge into two regimes of (1) flow (= softening + sharpening, Section 3) and (2) edge (=edge, Section 5) more clearly delivers our core contribution, to empirically and theoretically explain how optimizer-dependence in the latter regime emerges from the trajectory coincidence in the former regime. In the revised version of the manuscript, we provide a dedicated section, Section 4, with a mechanistic explanation of this emergence of optimizer-dependence as the learning rate increases.

And, with this simplified classification, the answer is yes. Please see Figure 4. To answer the reviewer's question further, we conducted a 100M-iteration (×500\times 500 longer than the main experiments) run on the small-lr SGD, representing the flow regime, and aligned it with the main sweep results in the flow time τ\tau-coordinate. The first thing we notice is that the edge regime and the flow regime are clearly different classes. With higher learning rates, depending on the type of optimizer, the sharpness curve escapes from the same curve of the flow regime down to the reciprocal slope of the edge of stability. Furthermore, the progressive sharpening effect is a property of the common gradient flow, rather than individual optimizers. The choice of optimizers and higher learning rates only reduce the upper-bounded sharpness curve through their own edges of stability. The sharpness from the flow regime, on the other hand, saturates to the maximal value, once the progressive sharpening takes place. We hope this clarifies the question.


(Continued)

(Continued)


[W1, Q1] Why do the final losses coincide below the edges?

The first answer is that, because we reported the last 100 training losses averaged only in Figures 1 and 2 in order to summarize the entire training run in a single point, the losses are nicely stabilized in a visible trend. You can reproduce the results from our uploaded code. However, this accounts only for the losses after the edge. Before the edge, however, the raw training loss values themselves are actually smooth and coincide regardless of the optimizer or learning rate if carefully aligned in the τ=ηQ(1)t\tau = \eta Q(1)t coordinate (Figures 3d and 3e).

Therefore, by this question, the reviewer has actually pointed out one of our contributions: throughout this extensive optimizer and lr sweep, we found that the learning dynamics are very sharply decomposed into two regimes: the flow regime (previously split into low-LR softening and mid-LR sharpening), where all trajectories converge to the common canonical gradient flow dθ/dτ=−g\mathrm d\theta / \mathrm d\tau = -g, and the edge regime, where individual optimizers constrain the training dynamics through the edge of stability. Our experiments show that the flow regime covers almost the entire interval of learning rates below the edge of any optimizer (Figures 1, 2, and 4c). This means that the choice of optimizers to operate below the edge of stability has a negligible effect on training dynamics--they all converge to the same gradient flow dynamics.

We additionally provide numerical and theoretical analysis of this coincidence in the flow regime.

  • Numerically: (lines 365-368). We can fit 0.9M data points from 146 different runs across 10 different optimizers with effective lr ηQ(1)≤5×10−5\eta Q(1) \le 5\times 10^{-5} drawn in Figure 3d (training progress τ\tau vs. loss LL) onto a single shallow power law with great accuracy. We have L(τ)∝0.487⋅τ−0.188L(\tau) \propto 0.487 \cdot \tau^{-0.188} with R2=0.999R^2 = 0.999, a clear coincidence.
  • Theoretically: (Theorem 3.1). As the reviewer has pointed out, it is known that previous works (Francis Bach, 2020, Su et al., 2016, Kovachki & Stuart, 2021) identified this small-lr flow limit in specific optimizers: GD, Heavy Ball, Nesterov's momentum. Our Theorem 3.1 extends this to general first-order optimizers with any number of momenta or gradient history buffers, including multi-momentum (AdEMAMix, AggMo, Grokfast) and gradient-augmented (QHM) optimizers, and quantifies the deviation of optimizer-induced dynamics from the gradient flow as O(η)O(\eta). Hence coincidence in low lr.

We hope this answers your question.


[W2,Q2] Overlapping contributions with previous works

We thank the reviewer for the meticulous comment. Our contribution is neither a rediscovery of the gradient flow limit of a specific type of GD nor a new static root locus criterion for stability. Rather, it is the empirical, theoretical, and mechanistic connection between these two established explanations: when different types of optimizers become salient for the learning dynamics, and how these optimizer-dependent trajectories branch from the common flow evolution. We compare our contribution to the previous works as follows:

Previous workWhat was establishedWhat is new in this work
Cohen et al. (2021)Discovery of the edge of stability & progressive sharpening, flow-to-edge transition with a fixed optimizer type: GD and GD with momentum.We study the mechanism of multiple optimizer-dependent trajectories that branch from a single gradient flow trajectory in the common low-lr limit.
Lee & Lee (2026)Derived the root locus stability criterion from static analysis, full-batch trainingWe extend this approach to dynamic analysis on the root locus in the flow-to-edge transition of dynamics.
Su et al. (2016)Continuous-time second-order ODE limit for NAGWe study general causal gradient filters and relate their small-step dynamics to a common first-order gradient flow.
Kovachki & Stuart (2021)Rescaled gradient flow limit for two types of single-momentum optimizers, HB and NAG.We extend this continuous-time flow limit for single momentum-based optimizers to general causal linear optimizers that include any parameter update from linear combinations of one or more momenta and historical gradients.

We hope this clarification helps identify our contributions. We have updated our Sections 1 and 2 to sharpen this claim. Again, we thank the reviewer for helping us make this work clearer.


Overall, the reviewer's comments have been of great help to us in reforming this manuscript to sharpen our contributions and improve our presentation. We would like to ask the reviewer to have a look at the revised manuscript as well as the summary we provided in the general remark above. We are grateful to listen to any additional concern from the reviewer.

Reviewer_pj5SOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how training dynamics evolve across learning rates before reaching the Edge of Stability (EoS). By interpreting optimizers as gradient filters and normalizing their DC gains, the authors identify three empirical regimes: low-learning-rate softening, mid-learning-rate progressive sharpening, and high-learning-rate EoS behavior. The paper presents a systematic experimental study across several optimizer families and architectures. The research question is well motivated and the overall construction provides a coherent perspective on the relationship between optimizer dynamics, learning rate, and loss landscape geometry.

Critical strengths
  1. Comprehensive experiments setup: The work provides experiments across different opimizers (SGD, SGDM, SGDN and so on) and architectures (MLP, CNN and VIT) to demonstrate their findings of optimization behavior (sharpness, loss and gradient alignment) under small, medium and large learning rate regime.

  2. Clear motivation and observation: The work discovers that the EOS phenomenon is not general across learning rates. It happens only at large learning rate regime where the sharpness (normalized) is inversely propotional to the learning rate so is the progressive sharpening phenomenon. Those phenomenons will vanish under small learning rate regime and the distinction between optimizers will also reduced.

Critical weaknesses
  1. Whether the observed behaviors will hold in language tasks or more complicated vision tasks? Despite the extensive studies within the work for CIFAR10 dataset, it is unclear that whether this phenomenon will continue to hold under language tasks that are of interest for most modern application. Additionally, with most focus dedicated to MLP, CNN and simplified VIT, we are also not sure whether this is also true for larger scale vision tasks. Different degree of matchness between complexity of models and datasets can potentially lead to very different conclusion experimentwise.

  2. Regarding the small learning rate regime: In the figure 3 of the work, the x-axis is not aligned between different plots. I find that the small learning rate regime can potentially not trained enough compared to the large learning rate counterpart if we look at the loss and the x-axis scale. There be simple reasoning behind the separation of three different learning rate regime that it might just the not sufficiently trained enough to enter EOS. If we instead plot the sharpness against loss, it might show that the sharpness is more correlated to the loss. This can also explain the reasoning behind the reduced separation of different optimizers as the training only progresses limittedly.

  3. Rational for the observed behavior: Despite the demonstration of these new optimization finding, the reason behind those findings are still missing. I appreciate the effort devoted to the work by authors for those new findings but the reasons explaining the mechanism is still missing. We are still not sure about the core reason for this correlation between the step size and sharpness under different learning rate regimes or how they interact with progressive sharpening. The work will be stronger if authors can provide deeper mechanism understanding of these claims.

Questions for authors
  1. Would the small learning rate trajectories eventually exhibit progressive sharpening if trained to the same effective time as the mid learning rate? How would this affect the proposed regime classification?

  2. Can the authors compare sharpness trajectories with training loss or other measures of optimization progress to distinguish differences in learning rate from differences in the regions of the loss landscape visited?

  3. Beyond the known continuous-time limit, progressive sharpening, and optimizer-dependent EoS stability boundaries, what new mechanism does the three-regime characterization establish?

4 comments

We deeply appreciate the reviewer for considerate reviews, which greatly helped us restructure the manuscript into the current form. We also thank the reviewer for acknowledging the comprehensiveness of the experiments and the clarity of our motivation. Each of the raised concerns is answered below:


[W1] Generalization to language tasks and larger vision models.

We very much agree that studying large-scale architectures and language tasks is valuable. In this work, we focus on the mechanistic explanation of how different optimizer-effects emerge from a common low-lr gradient flow limit. For this mechanistic study, we deliberately chose to fix the task and instead to vary the optimization algorithms and learning rates in an extensive manner. Rather than exploring full-dimensional control for every possible deep learning problem, our controlled setting more precisely delivers the key evidence for our theoretical and mechanistic interpretations (in the updated Sections 3, 4, and 5) on this emergence of optimizer-dependence. For example, we do not need to run thousands of Chinchilla-scale language model training runs to rediscover that designing optimizers for operating in the learning rate below the edge of stability is generally meaningless because it will operate in the common gradient flow limit. As the reviewer suggests, we agree that applying the knowledge we established here to larger practical scenarios is a promising research direction, especially for designing better optimizers.


[W2, Q1] Will the small learning rate trajectories eventually exhibit progressive sharpening if trained longer?

Yes, progressive sharpening happens for smaller-lr trajectories trained long enough, at the exact same training progress τ=ηQ(1)t\tau = \eta Q(1) t predicted by larger-lr runs. We additionally trained a small-lr SGD (lr=5e-5) for 100M iterations (×500\times 500 more iterations than main experiments). Our new Figure 4 provides four additional observations on the progressive sharpening. This motivated us to change our regime classification. After a deep consideration, we find that merging low-LR softening and mid-LR sharpening regimes into the same flow regime and simplifying our regime structure from three to two (i.e., low-LR flow and high-LR edge) better delivers our findings. We restructured our manuscript accordingly. With this change, we could invest more pages on providing better theoretical and mechanistic interpretation, which we address in the next question. We sincerely thank the reviewer for providing this intuition.

In addition to this, we want to share that this elongated run in the common gradient flow regime upper bounds the sharpness S(τ)S(\tau) from optimizer-driven runs (Figure 4c). All our 338 runs across dense learning rate sweeps on 10 different optimizers have not exceeded this bound. This not only indicates that different optimizers only reduce the sharpness through the edge of stability (in other words, progressive sharpening is a property of the common gradient flow, independent of optimizers), but also provides a crossover timing Sflow(τ)=k⋆(Q)/hS_\text{flow}(\tau) = k_\star(Q) / h (line 450) and mechanistic insights.


[W3, Q3] Deeper mechanistic explanations of the observations.

We thank the reviewer for the warm regards and thoughtful recommendation. In our revised manuscript, we greatly strengthened the flow-to-edge crossover mechanisms with added theory in a dedicated Section 4. In this paper, we question how optimizer-dependence takes place from the common gradient flow. We find that the sharpness speed F=dS/dτF = \mathrm dS/\mathrm d\tau from the common gradient flow is transferred into the optimizer-specific root locus traversal speed dz/dτ\mathrm dz/\mathrm d\tau through the optimizer-specific quantity of root sensitivity TQ(z,k)T_Q(z, k). As a consequence, the optimizer's roots (closed-loop poles) parked near their initial points in the low-lr flow regime begin to move along the prescribed trajectory. This happens in a relatively short interval in learning rates. Eventually, the roots are shot outside the unit circle (the discrete-time stability threshold). At the edge, the stabilized sharpness SS is determined by the optimizer and learning rates, rather than drifting along the training progress τ\tau, i.e., F=0F = 0. As the learning rate increases, once the root locus leaves the unit circle, it immediately loses its speed and is pinned down at the escape point of the unit circle and stays for the higher learning rates until training blows up. We included these mechanistic explanations in our revised manuscript on pages 8 and 9. Again, we thank the reviewer for helping us improve this work.


(Continued)

(Continued)


[Q2] Sharpness vs. loss curve for visualizing loss landscape.

We thank the reviewer for this insightful idea. We provided this sharpness vs. loss curve in Figure 11 of the revised manuscript. All families of optimizers start from initial softening and later sharpening in the common gradient flow regime on the left side of the figure. Then, different optimizers with different root loci exit this common trajectory and go through an optimizer-specific edge of stability on the right side of the figure. It is noteworthy that different families or optimizers occupy clearly different regions of the loss landscape, partially drawn by the (L,S)(L, S) curve.


We are eager to resolve any additional concern. As mentioned above, reviewer's comments have been of great influence in this reform of the manuscript. We would like to ask the reviewer to check the revised manuscript as well as our summary in the general remarks above of the updates. Please feel free to share any additional thoughts.

Thank you for substansive update on the manuscript to address the issues. The manuscript has changed a lot after the update including the summary of the regimes to flow based diagnosis and addtional mechanism explanation. Most of my concerns are addressed and I will maintain my score.

We truly appreciate your constructive and encouraging review. We are grateful for the opportunity to receive your feedback and will remain available until the very end of the discussion phase should any additional questions arise.

Reviewer_vah2OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper compares several momentum-based optimizers over a wide range of learning rates. After adjusting the learning rate to account for each optimizer’s overall amplification of gradients, the authors find that loss and sharpness curves are similar before reaching the edge of stability. Near the stability boundary, the curves separate in ways that depend on the optimizer.

Critical strengths

The experiments cover many learning rates and several optimizer configurations. The figures clearly show where their behavior agrees and where it differs. Comparing optimizers after correcting for their overall gradient amplification is also useful.

Critical weaknesses
  1. The theoretical contribution seems limited. The optimizers studied here are expected to approach the same gradient flow at small learning rates, after accounting for their overall gradient amplification. The similarity of their training curves is therefore not particularly surprising. The paper would be stronger if it explained quantitatively how far this agreement extends beyond the small-learning-rate limit and when it breaks down.

  1. The experimental novelty over Cohen et al. is unclear. Their work already showed that gradient descent with different learning rates approximately follows a shared gradient-flow trajectory before reaching the edge of stability. It also studied progressive sharpening and momentum-dependent stability boundaries. This paper extends the comparison to more optimizer configurations, but it is unclear whether this reveals a substantially new phenomenon.
Questions for authors

What is the main finding that goes beyond the expected shared gradient-flow limit?


What is the main experimental observation that was not already established by Cohen et al.?


References

[1] Jeremy M. Cohen, Simran Kaur, Yuanzhi Li, J. Zico Kolter, and Ameet Talwalkar. Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability. ICLR, 2021. https://arxiv.org/abs/2103.00065

2 comments

We deeply thank the reviewer for taking the time to read our work and leaving valuable comments. We are also encouraged by the reviewer's recognition of the comprehensiveness of our experiments and clarity of presentation. We will answer each concern below.


[W1, Q1] The theoretical contribution seems limited.

We acknowledge that our focus on the low-lr coincidence of the graphs obscured other important findings and mechanistic explanations that are more significant than this continuous-time flow limit. We have fully revised our manuscript to strengthen our theoretical and mechanistic explanations as requested.

  • Quantitative explanation of the small-lr agreements: We answer this question in two ways.

    • Theoretically: Please see new Theorem 3.1. As the reviewer correctly noted, previous works (Francis Bach, 2020, Su et al., 2016, Kovachki & Stuart, 2021) identified this small-lr flow limit in specific optimizers: GD, Heavy Ball, Nesterov's momentum. Our Theorem 3.1 extends this to general first-order optimizers with any number of momenta or gradient history buffers, including multi-momentum (AdEMAMix, AggMo, Grokfast) and gradient-augmented (QHM) optimizers. Theorem 3.1 quantifies the deviation of optimizer-induced dynamics from the gradient flow dθ/dτ=−g\mathrm d\theta/\mathrm d\tau = -g as −ηQ(1)(12−Q′(1)Q(1))Hg+O(η2Q(1)2).-\eta Q(1) \left(\frac{1}{2} - \frac{Q'(1)}{Q(1)}\right) H g + O(\eta^2 Q(1)^2).
    • Numerically: Please see lines 365-368. We can fit 0.9M data points from 146 different runs across 10 different optimizers with effective lr ηQ(1)≤5×10−5\eta Q(1) \le 5\times 10^{-5} drawn in Figure 3d (training progress τ\tau vs. loss LL) onto a single shallow power law with great accuracy. We have L(τ)∝0.487⋅τ−0.188L(\tau) \propto 0.487 \cdot \tau^{-0.188} with R2=0.999R^2 = 0.999. This provides clear empirical evidence for Theorem 3.1 of how well this small-lr coincidence agrees across distinct types of optimization algorithms.
  • Main finding beyond expected shared gradient flow limit: Actually, our main contribution is more than a shared gradient flow limit. We utilize dynamic analysis on optimizer root loci to provide a mechanistic explanation of how optimizer-dependent trajectories emerge from the common gradient flow as learning rate increases. To highlight this contribution, we (1) revised the introduction (Section 1), (2) merged two regimes low-LR softening and high-LR sharpening into one flow regime (Section 3), and (3) provided a dedicated section (Section 4) on flow-edge crossover dynamics through additional theoretical and mechanistic explanation (Lemma 4.1, Theorem 4.2, Section 5 "Spectral edge pinning"). To put this in one sentence, we find that a quantity called speed of root locus couples two quantities from the flow and edge regimes, converting the model-driven sharpening speed from the common gradient flow into the optimizer-specific dynamics through the optimizer root sensitivity.

We hope this answer clarifies the issue.


[References]

  • [1] Francis Bach. Effortless optimization through gradient flows. Machine Learning Research Blog, May 2020.
  • [2] W. Su, S. Boyd, E. J. Candes. A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights, in Journal of Machine Learning Research, 2016.
  • [3] N. B. Kovachki, A. M. Stuart. Continuous Time Analysis of Momentum Methods, in Journal of Machine Learning Research, 2021.

(Continued)

(Continued)


[W2, Q2] The experimental novelty over Cohen et al. is unclear.

We do not claim to rediscover the onset of the edge of stability from the small-lr gradient flow limit or the progressive sharpening effect, which are already established by Cohen et al. We revised our manuscript to more clearly deliver our findings through the extensive experiments:

  • Trajectory divergence from the common gradient flow by different optimizers: We empirically observed how different optimizers yield different learning trajectories from their common gradient flow limit in a relatively short interval of effective learning rates specific to the optimizer.

  • Probes for mechanistic explanation of optimizer-dependency emergence: Mainly focused on how optimizer-dependency arises from increasing learning rates, we devised consecutive gradient alignment ctc_t (Figure 1d) and first order alias ρt\rho_t (Figure 1e) to quantify the trajectory deviation from the common gradient flow, with their theoretical accuracy margin (Proposition B.3 and Proposition B.4). We additionally provide a dynamic trajectory of the root loci mapped with respect to the effective lr (Figure 1c); these loci have only been statically analyzed. Our dynamic mapping is central to our mechanistic findings in Section 4 and Section 5.

  • Flow-driven progressive sharpening as an upper bound of the sharpness in optimization-driven trajectories: In new Figure 4, we conducted a 100M-iteration run (×500\times 500 extension) with a small-lr (5e-5) SGD. This reveals four additional facts:

    1. Progressive sharpening eventually happens even with a smaller learning rate if we train this long enough.
    2. Initial softening and progressive sharpening all happen in the common continuous-time gradient flow evolution, and optimizer does not play a role.
    3. Progressive sharpening eventually stops after reaching a certain value. This upper limit is from the gradient flow, and is therefore a model-intrinsic property, rather than the optimization algorithm.
    4. Progressive sharpening gives an upper limit of sharpness of the training progress. All our 338 runs with 10 different optimizers did not sharpen the model more than this flow regime curve. Rather, optimizers only reduce the sharpness through their edges of stability (Figure 4c).

We would like to ask the reviewer to take a look at the revised manuscript as well as the general remarks above summarizing the updates, which were greatly influenced by the reviewer's comment. Please feel free to share any remaining concerns. We would like to resolve them as best as we can.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Reviewers,

We deeply appreciate the reviewers for their time and effort in reading our work and providing valuable comments. Most of all, we thank the reviewers for identifying that our focus on trajectory coincidence and multi-regime presentation obscured our central contribution: to provide a mechanistic explanation of how optimizer-dependence emerges from a common gradient flow as the learning rate increases. Upon careful reconsideration, we updated our manuscript to better deliver what we have found. We outline the changes as follows (All references are to the revised manuscript):

  • Three (Low-LR|Mid-LR|High-LR) regimes are merged into two (Flow|Edge): The previous low-LR softening and mid-LR sharpening regimes are better understood as two consecutive stages of a common gradient flow evolution, rather than separate governing dynamics (new Figure 4). We merged these two into the model-intrinsic flow regime (new Section 3), in contrast to the optimizer-dependent edge regime (new Section 5), and reorganized the manuscript around the flow-to-edge crossover dynamics (new Section 4). We thank Reviewer pj5S for comment [Q1], which helped us make this distinction clearer.

  • Theoretical and mechanistic explanations strengthened: With this regime dichotomy, we devoted more space to providing intuitive mechanistic and theoretical explanations of how the common gradient flow dynamics branch into distinct trajectories specific to individual optimization algorithms (new Sections 3, 4, and 5). These are summarized as follows:

    • Generalized continuous-time gradient flow limit for an arbitrary linear-memory optimizer (new Theorem 3.1, previously B.2): As Reviewer bMFi nicely pointed out [W2], this small-lr approximation has been studied for vanilla GD (Francis Bach's Blog), Nesterov's acceleration (Su et al., 2016), and fixed-momentum methods (Kovachki & Stuart, 2021). We extend this result to general optimizers with any linear memory structure. This includes an arbitrary number of momenta (e.g., AdEMAMix, Grokfast, AggMo) and/or delayed/buffered gradients (e.g., Nesterov, QHM). This theoretically justifies how our extensive experiments converge to the same gradient flow dynamics in a wide range of learning rates.
    • Strengthened mechanistic explanations of the flow-to-edge traversal (new Lemma 4.1, Theorem 4.2, and Section 5: "Spectral edge pinning"): As Reviewers vah2 [W1] and pj5S [W3] requested, we extended our mechanistic explanation of how optimizer-specific trajectories diverge from their common gradient flow limit using root locus traversal dynamics. In summary, we found that a characteristic quantity of the optimizer called the root sensitivity TQ(z,k)T_Q(z,k) translates the optimizer-independent sharpness speed F=dS/dτF = \mathrm dS/\mathrm d\tau of the common gradient flow to the optimizer-specific root locus traversal speed dz/dτ\mathrm dz/\mathrm d\tau, eventually leading to distinct trajectories arriving at optimizer-specific edges of stability. To our knowledge, this is the first use of the dynamic analysis of the root locus of general first-order optimizers in the literature.
  • Contributions sharpened: With this reformed manuscript, we sharpened our contribution claims and clarified what separates our work from previous studies (new Section 1). The distinction is summarized in our answer to Reviewer bMFi [W2, Q2] into a table. In summary, our contribution is not rediscovering the common gradient flow regime and the edge of stability (Cohen et al. 2021) or using static root locus analysis to study the onset of the edge of stability (Lee & Lee, 2026), but is to explain theoretically and mechanistically how high-lr optimizer-dependent trajectories diverge from the common low-lr gradient flow using dynamic analysis on the optimizer root locus.

We thank the reviewers again for their constructive reviews, which greatly helped us improve this manuscript. We are excited to have an opportunity to communicate with the reviewers during this discussion phase. Please feel free to share any additional or unresolved issues. We will remain attentive to your concerns to the very last minute.

We would also be very grateful if you could take another look at the updated manuscript.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.