- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on EoS, and argues that its focus on maximum curvature mode of the loss landscape is limiting. It instead argues for a variant called realized edge of stability that takes into account the effect of time-varying stateful optimizers that actually take the updates.
Critical strengths
The authors perform a nice systemic empirical comparison of different optimizers. That said, it would be nice to better contrast these results with previous empirical results. As noted in the questions below, the detailed emprical studies from the origianl EoS paper were the core contribution, and if this paper is indeed arguing that those EoS experiments were wrong or insufficient, more careful discussion should be given here.
Critical weaknesses
It was not completely clear to me whether the authors have discovered a new phenomenon. Part of my confusion here is that the entire claim of the initial EoS paper is that it better characterized the behavior of full GD, and follow up work aimed to generalize some of these ideas to other optimizers. Please see questions below for more details.
Questions for authors
Is this a new phenomena?
- The plots in the original EoS paper show that EoS nicely tracks optimizer behavior, though notably less so for cross-entropy than MSE. The EoS paper does seem use a different scale for their plots though. How to reconcile the difference here? Are you plotting something different? Were the results wrong in the original EoS paper? Are you using a different setup? Is this just a function of the scale of the plot?
- Have any subsequent papers since the original EoS paper also noticed this behavior?
"The original EoS (Cohen et al., 2021) and adaptive EoS (Cohen et al., 2023) are special cases of this general stability condition.": What does this generality buy us? Doesn't the original eos paper and subsequent follow ups already consider several types of optimizers? if so, why do the authors need further generality to make their claims?
5 comments
(Continued)
Then what is the optimizer-specific discrepancy?
Perhaps one concern the reviewer might raise from our last answer above would be like: "If optimizer-specific factors are identified by Proposition 2.1, what are the remaining sections? Why are there remaining discrepancies reported in Section 3 and so forth?"
Cohen's (2021, 2023) as well as our generalized Proposition 2.1's formulae of the original edge of stability (will be explained in the subsequent answer) give an exact static stability margin for the worst-case direction. These thresholds are mathematically laid lines that a stable gradient-based optimization under a fixed quadratic basin must not trespass. And Cohen's first work (2021) reports that a vanilla GD violates that bound. Cohen's second work (2023) and other subsequent works like Andreyev et al. (2026) then again find analogous lines for other preconditioned or momentum-based optimizers that can be trespassed, yet remained as a faithful boundary for a fixed quadratic case.
This calls to another, dynamic, stability threshold that reflects the actual online trajectory. As the reviewer VuQ7 has mentioned, these lines of work primarily rely on a directional Hessian (Mishkin et al., 2024 and Islamov et al., 2026). Alternatively to Cohen's and our Proposition 2.1's formulae, these works are based on the directional Hessian with respect to the actual parameter progress. The second part (Sections 4, 5, and 6) of the paper defines the learning impedance and the realized edge of stability , in line with these latter works. These two types of formulations differ in their directions on which the stability margin is measured (worst-case available vs. real updates). Most importantly, these two types of edges of stability do not compete with each other. They describe different aspects of the stability of a learning system.
Regarding the plot scale differences between ours and Cohen's
In a recap, there are two types of stability thresholds: The first is the static threshold, whose formulae are proposed by Cohen and subsequent works, yet these thresholds are consistently violated in actual trajectories (hence the name, "edge of stability"). And, these overflowing gaps are known since Cohen's original work (starkly depicted as a series of dots over the line in Figure 1 of Cohen et al., 2021). And, yes, we do not claim to rediscover them. However, these gaps are not measured in an extensive study with respect to optimizers before us.
Our first observation starts from the fact that there is an optimizer-specific quantity hidden in the amount of violation of the static threshold at the edge of stability. To visualize such optimizer-dependency, the following preliminary jobs were required:
-
We needed to extend Cohen's and subsequent works' formulae for the worst-case stability margin to arbitrary optimizers in order to show this optimizer-dependent gap in a wide range of optimizers. Hence Proposition 2.1's generalization.
-
We needed to normalize the formulation of edge of stability to make the thresholds aligned at 1, so that the optimizer-dependent relative gaps are visualized correctly. Hence our formulation of the EoS as . We then drew them in the same y-aligned fashion.
Then, we can start to present our central contribution to bridge this worst-case marginality with a realized dynamic marginality through the subsequent sections (Sections 4, 5, and 6).
So, yes. The reviewer has understood correctly. Ours and Cohen's plots are differently scaled. We choose this particular normalization for visualizing the optimizer-specificity in the edge of stability. Yet, the contents are the same as Cohen's report, only differently scaled.
(Continued)
(Continued)
Then what is our novelty?
So far, we have answered the reviewer's main concerns by highlighting the connection between our work and the prior works. Especially we confirm that ours are not meant to replace Cohen's and others' works, but to build a bridge between two different lines of explanations, i.e., the worst-case marginality (mainly Cohen's) and the realized dynamic marginality (mainly Mishkin's), that describe the same concept of the stability of the gradient-based optimization in a different aspect (direction).
Our contribution centers around the introduced quantity of the realized edge of stability. This is defined as a running median of the online curvature-to-alignment ratio . We call the instantaneous quantity a learning impedance, and the statistical quantity a realized edge of stability. Our theoretical analysis, now strengthened with a new Section 6, says that
- (Section 4) The learning impedance is interpretably decomposable into the original edge of stability (in our normalized form of ) and the spatial and temporal calibration factors.
- (Section 5) The learning impedance and the worst-case marginality can be the same quantity only under specific conditions, and by exploiting this idea, we can make diagnostics on the running learning systems, understanding how much of the potential loss decrease is actually converted into the real loss drop.
- (Section 6) By taking a running median of the learning impedance , we get a robust estimator of the dynamic stability threshold, which we name the realized edge of stability .
We hope our answers appropriately address the reviewer's concerns.
We, again, appreciate the valuable feedback. Please feel free to share any additional concern. We would like to resolve them as best as we can.
[References]
- Cohen, et al. Gradient descent on neural networks typically occurs at the edge of stability, in ICLR, 2021.
- Cohen, et al. Adaptive gradient methods at the edge of stability, in NeurIPS Workshop, 2023.
- Andreyev et al. Momentum further constrains sharpness at the edge of stochastic stability, in ICML, 2026.
- Mishkin et al., Directional smoothness and gradient methods: Convergence and adaptivity, in NeurIPS, 2024.
- Islamov et al., Non-euclidean gradient descent operates at the edge of stability. In ICML, 2026.
We deeply thank the reviewer for taking time and effort reviewing our work. We are pleased to address the raised concerns below:
Is this a new phenomenon?
As the reviewer might have noticed, the original EoS paper of Cohen et al. (2021) has already shown the progressive sharpening (increasing) and hovering of the sharpness value above so-called classically predicted boundary S>2/η. Subsequent works are devoted to extending this finding to other optimizers, including GD, GD with momentum, and some widely-used preconditioned optimizers like Adam (e.g., Cohen et al., 2023). We are fully aware of this as we have summarized these trends in our Appendix A: Related Work and more briefly in Section 1: Introduction since the initial upload of this paper. Therefore, we confirm that we do not claim a rediscovery of the edge of stability phenomenon, nor do we falsify these prior works. Moreover, our realized edge of stability is not superseding the original edge of stability; they are complementary quantities that measure different aspects of the stability boundary. Simply put, the original EoS accounts for the available, worst-case stability margin, concerning the scenarios where the parameters proceed towards the sharpest direction with the maximal curvature (the maximal Hessian eigenmode direction), whereas our realized EoS accounts for the dynamical stability threshold with respect to the direction of actual parameter updates. In order not to confuse readers, we adjusted our wording in the paper, especially the mentioned quote: "The original EoS (Cohen et al., 2021) and adaptive EoS (Cohen et al., 2023) are special cases of this general stability condition." We thank the reviewer for pointing out this issue.
Regarding the "generalization" of Cohen's formula
We would like to discuss more on this "generalization" claim of ours in that mentioned quote. As the reviewer has pointed out, many follow-up works of Cohen's original work of 2021 have aimed for providing extensions to the formula S≈2/η by introducing optimizer-specific factors, e.g., for Heavy Ball or Nesterov's momenta. For example, Cohen et al. (2023) and Andreyev et al. (2026) provide empirically verified and theoretically backed amendments of the formula S≈2/η by incorporating the momentum coefficient β of the optimizer. However, these works focused on specific families of optimizers. Our Proposition 2.1 uses a control theory-inspired approach to derive another version of this extended formula, that applies to a wider family of (optionally preconditioned) first-order optimization algorithms with linear memory structures. Linear memory means that the parameter update is produced by a linear combination of any number of momenta or delayed/buffered gradients. Although the original EoS paper and subsequent follow-ups already consider several types of optimizers, they do not extend to this extent, e.g., QHM or two-pole optimizers (AggMo, AdEMAMix, Grokfast). Thus our formulation generalizes the formulae given from the previous works, e.g., as proved in Corollary 2.2. Our Section 2 generalizes this subset of the previous works in this particular sense. We adjusted our claims in the revised manuscript accordingly.
(Continued)