acceptodds
Under review as a conference paper at ICLR 2027

The Road Taken: The Role of Optimizers at the Edge of Stability

10 views

Abstract

The edge of stability refers to a phenomenon in deep learning with gradient-based optimizers where the Hessian eigenvalues of the loss remain stable above a threshold that the classical descent lemma predicts to be unstable. Previous works formulate the edge of stability with respect to the maximum Hessian eigenvalue and the learning rate. However, we observe that many first-order methods, including gradient descent, significantly violate the stability bound predicted by these theories by a factor as large as . Moreover, this deviation turns out to be systematic and highly dependent on the underlying optimizer, which is not captured by previous formulations. This calls for a new formulation of the stability threshold, which we derive from the directional Hessian and the gradient-alignment score with respect to the actual update taken by the optimizer, rather than the maximum curvature mode. Our new formulation of the realized edge of stability not only removes optimizer-dependent offsets and provides more consistent predictions of the stability threshold, but also introduces new diagnostic tools that reveal the unique role of the optimizer in actively balancing between the temporal and spatial budgets in first-order optimization.

Primary area:
learning theory
Keywords:
optimization, edge of stability, gradient descent, root locus, optimizer filter, alignment, directional Hessian, learning power

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_zieaOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on EoS, and argues that its focus on maximum curvature mode of the loss landscape is limiting. It instead argues for a variant called realized edge of stability that takes into account the effect of time-varying stateful optimizers that actually take the updates.

Critical strengths

The authors perform a nice systemic empirical comparison of different optimizers. That said, it would be nice to better contrast these results with previous empirical results. As noted in the questions below, the detailed emprical studies from the origianl EoS paper were the core contribution, and if this paper is indeed arguing that those EoS experiments were wrong or insufficient, more careful discussion should be given here.

Critical weaknesses

It was not completely clear to me whether the authors have discovered a new phenomenon. Part of my confusion here is that the entire claim of the initial EoS paper is that it better characterized the behavior of full GD, and follow up work aimed to generalize some of these ideas to other optimizers. Please see questions below for more details.

Questions for authors

Is this a new phenomena?

  • The plots in the original EoS paper show that EoS nicely tracks optimizer behavior, though notably less so for cross-entropy than MSE. The EoS paper does seem use a different scale for their plots though. How to reconcile the difference here? Are you plotting something different? Were the results wrong in the original EoS paper? Are you using a different setup? Is this just a function of the scale of the plot?
  • Have any subsequent papers since the original EoS paper also noticed this behavior?

"The original EoS (Cohen et al., 2021) and adaptive EoS (Cohen et al., 2023) are special cases of this general stability condition.": What does this generality buy us? Doesn't the original eos paper and subsequent follow ups already consider several types of optimizers? if so, why do the authors need further generality to make their claims?

5 comments

We deeply thank the reviewer for taking time and effort reviewing our work. We are pleased to address the raised concerns below:


Is this a new phenomenon?

As the reviewer might have noticed, the original EoS paper of Cohen et al. (2021) has already shown the progressive sharpening (increasing) and hovering of the sharpness value above so-called classically predicted boundary S>2/ηS > 2/\eta. Subsequent works are devoted to extending this finding to other optimizers, including GD, GD with momentum, and some widely-used preconditioned optimizers like Adam (e.g., Cohen et al., 2023). We are fully aware of this as we have summarized these trends in our Appendix A: Related Work and more briefly in Section 1: Introduction since the initial upload of this paper. Therefore, we confirm that we do not claim a rediscovery of the edge of stability phenomenon, nor do we falsify these prior works. Moreover, our realized edge of stability is not superseding the original edge of stability; they are complementary quantities that measure different aspects of the stability boundary. Simply put, the original EoS accounts for the available, worst-case stability margin, concerning the scenarios where the parameters proceed towards the sharpest direction with the maximal curvature (the maximal Hessian eigenmode direction), whereas our realized EoS accounts for the dynamical stability threshold with respect to the direction of actual parameter updates. In order not to confuse readers, we adjusted our wording in the paper, especially the mentioned quote: "The original EoS (Cohen et al., 2021) and adaptive EoS (Cohen et al., 2023) are special cases of this general stability condition." We thank the reviewer for pointing out this issue.

Regarding the "generalization" of Cohen's formula

We would like to discuss more on this "generalization" claim of ours in that mentioned quote. As the reviewer has pointed out, many follow-up works of Cohen's original work of 2021 have aimed for providing extensions to the formula S≈2/ηS \approx 2 / \eta by introducing optimizer-specific factors, e.g., for Heavy Ball or Nesterov's momenta. For example, Cohen et al. (2023) and Andreyev et al. (2026) provide empirically verified and theoretically backed amendments of the formula S≈2/ηS \approx 2 / \eta by incorporating the momentum coefficient β\beta of the optimizer. However, these works focused on specific families of optimizers. Our Proposition 2.1 uses a control theory-inspired approach to derive another version of this extended formula, that applies to a wider family of (optionally preconditioned) first-order optimization algorithms with linear memory structures. Linear memory means that the parameter update is produced by a linear combination of any number of momenta or delayed/buffered gradients. Although the original EoS paper and subsequent follow-ups already consider several types of optimizers, they do not extend to this extent, e.g., QHM or two-pole optimizers (AggMo, AdEMAMix, Grokfast). Thus our formulation generalizes the formulae given from the previous works, e.g., as proved in Corollary 2.2. Our Section 2 generalizes this subset of the previous works in this particular sense. We adjusted our claims in the revised manuscript accordingly.


(Continued)

(Continued)


Then what is the optimizer-specific discrepancy?

Perhaps one concern the reviewer might raise from our last answer above would be like: "If optimizer-specific factors are identified by Proposition 2.1, what are the remaining sections? Why are there remaining discrepancies reported in Section 3 and so forth?"

Cohen's (2021, 2023) as well as our generalized Proposition 2.1's formulae of the original edge of stability χ\chi (will be explained in the subsequent answer) give an exact static stability margin for the worst-case direction. These thresholds are mathematically laid lines that a stable gradient-based optimization under a fixed quadratic basin must not trespass. And Cohen's first work (2021) reports that a vanilla GD violates that bound. Cohen's second work (2023) and other subsequent works like Andreyev et al. (2026) then again find analogous lines for other preconditioned or momentum-based optimizers that can be trespassed, yet remained as a faithful boundary for a fixed quadratic case.

This calls to another, dynamic, stability threshold that reflects the actual online trajectory. As the reviewer VuQ7 has mentioned, these lines of work primarily rely on a directional Hessian (Mishkin et al., 2024 and Islamov et al., 2026). Alternatively to Cohen's and our Proposition 2.1's formulae, these works are based on the directional Hessian with respect to the actual parameter progress. The second part (Sections 4, 5, and 6) of the paper defines the learning impedance ζ\zeta and the realized edge of stability ζ^\hat\zeta, in line with these latter works. These two types of formulations differ in their directions on which the stability margin is measured (worst-case available vs. real updates). Most importantly, these two types of edges of stability do not compete with each other. They describe different aspects of the stability of a learning system.

Regarding the plot scale differences between ours and Cohen's

In a recap, there are two types of stability thresholds: The first is the static threshold, whose formulae are proposed by Cohen and subsequent works, yet these thresholds are consistently violated in actual trajectories (hence the name, "edge of stability"). And, these overflowing gaps are known since Cohen's original work (starkly depicted as a series of dots over the line in Figure 1 of Cohen et al., 2021). And, yes, we do not claim to rediscover them. However, these gaps are not measured in an extensive study with respect to optimizers before us.

Our first observation starts from the fact that there is an optimizer-specific quantity hidden in the amount of violation of the static threshold at the edge of stability. To visualize such optimizer-dependency, the following preliminary jobs were required:

  1. We needed to extend Cohen's and subsequent works' formulae for the worst-case stability margin to arbitrary optimizers in order to show this optimizer-dependent gap in a wide range of optimizers. Hence Proposition 2.1's generalization.

  2. We needed to normalize the formulation of edge of stability to make the thresholds aligned at 1, so that the optimizer-dependent relative gaps are visualized correctly. Hence our formulation of the EoS as χ=ηSΓ/2\chi = \eta S \Gamma / 2. We then drew them in the same y-aligned fashion.

Then, we can start to present our central contribution to bridge this worst-case marginality with a realized dynamic marginality through the subsequent sections (Sections 4, 5, and 6).

So, yes. The reviewer has understood correctly. Ours and Cohen's plots are differently scaled. We choose this particular normalization for visualizing the optimizer-specificity in the edge of stability. Yet, the contents are the same as Cohen's report, only differently scaled.


(Continued)

(Continued)


Then what is our novelty?

So far, we have answered the reviewer's main concerns by highlighting the connection between our work and the prior works. Especially we confirm that ours are not meant to replace Cohen's and others' works, but to build a bridge between two different lines of explanations, i.e., the worst-case marginality (mainly Cohen's) and the realized dynamic marginality (mainly Mishkin's), that describe the same concept of the stability of the gradient-based optimization in a different aspect (direction).

Our contribution centers around the introduced quantity of the realized edge of stability. This is defined as a running median ζ^t=Med(ζ)\hat\zeta_t = \mathrm{Med}(\zeta) of the online curvature-to-alignment ratio ζt:=Bt/At=ut⊤Hˉtut/gt⊤ut\zeta_t := B_t / A_t = u_t^\top \bar H_t u_t / g_t^\top u_t. We call the instantaneous quantity ζt\zeta_t a learning impedance, and the statistical quantity ζ^t\hat\zeta_t a realized edge of stability. Our theoretical analysis, now strengthened with a new Section 6, says that

  1. (Section 4) The learning impedance ζt\zeta_t is interpretably decomposable into the original edge of stability (in our normalized form of χt\chi_t) and the spatial and temporal calibration factors.
  2. (Section 5) The learning impedance ζt\zeta_t and the worst-case marginality χt\chi_t can be the same quantity only under specific conditions, and by exploiting this idea, we can make diagnostics on the running learning systems, understanding how much of the potential loss decrease is actually converted into the real loss drop.
  3. (Section 6) By taking a running median of the learning impedance ζt\zeta_t, we get a robust estimator of the dynamic stability threshold, which we name the realized edge of stability ζ^t:=Med(ζt)\hat \zeta_t := \mathrm{Med}(\zeta_t).

We hope our answers appropriately address the reviewer's concerns.


We, again, appreciate the valuable feedback. Please feel free to share any additional concern. We would like to resolve them as best as we can.


[References]

  • Cohen, et al. Gradient descent on neural networks typically occurs at the edge of stability, in ICLR, 2021.
  • Cohen, et al. Adaptive gradient methods at the edge of stability, in NeurIPS Workshop, 2023.
  • Andreyev et al. Momentum further constrains sharpness at the edge of stochastic stability, in ICML, 2026.
  • Mishkin et al., Directional smoothness and gradient methods: Convergence and adaptivity, in NeurIPS, 2024.
  • Islamov et al., Non-euclidean gradient descent operates at the edge of stability. In ICML, 2026.
Reviewer_VuQ7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates optimizer-dependent deviations from conventional Edge of Stability (EoS) thresholds. The authors first develop a unified stability formulation for gradient-based optimizers with temporal filtering and spatial preconditioning, using root-locus analysis to characterize the stability threshold of frozen quadratic systems.

Motivated by empirical observations that the conventional EoS measure χ=ηSΓ/2\chi=\eta S\Gamma/2 can substantially exceed one, the authors introduce a new quantity called the realized Edge of Stability, defined as ζ=B/A\zeta=B/A, where AA measures gradient-update alignment and BB represents the curvature contribution along the executed update. They further establish the decomposition ζ=χς/τ\zeta=\chi\varsigma/\tau, interpreting ς\varsigma and τ\tau as spatial participation and temporal calibration factors, respectively. Experiments on CIFAR-10 with various full-batch optimizers suggest that the proposed measure exhibits smaller optimizer-dependent offsets and can be computed more efficiently than Hessian-based diagnostics.

Critical strengths
  1. A unified control-theoretic treatment of different optimizer families.

The extension of quadratic stability analysis to general temporal gradient filters is interesting. In particular, the root-locus formulation accommodates more complicated filters whose stability boundary does not necessarily correspond to period-two oscillations. This provides a useful theoretical perspective on the differences between optimizer families.

  1. An interpretable and computationally inexpensive diagnostic framework.

The decomposition of the proposed realized EoS into spatial and temporal factors offers a way to examine how update directions and optimizer states contribute to observed training behavior. The broad comparison across optimizer families and the reduced computational cost of evaluating the proposed metric are also useful aspects of the work.

Critical weaknesses
  1. The proposed realized EoS does not convincingly characterize dynamical stability beyond one-step loss behavior.

My primary concern is the interpretation and significance of the proposed metric ζ\zeta. From Equation (11), the realized EoS can be expressed exactly as

ζ=1+ΔLηg⊤u.\zeta=1+\frac{\Delta L}{\eta g^\top u}.

When g⊤u>0g^\top u>0, the condition ζ≤1\zeta\leq1 is equivalent to ΔL≤0\Delta L\leq0. Therefore, the observation that ζ\zeta remains below one for loss-decreasing updates follows directly from its definition.

This raises a fundamental distinction between one-step descent and dynamical stability. The conventional EoS threshold concerns the stability of optimization dynamics, whereas the proposed ζ\zeta primarily measures the discrepancy between predicted and realized loss decrease along an executed update. Although the latter is a meaningful diagnostic, the paper does not sufficiently establish why ζ≈1\zeta\approx1 should characterize marginal dynamical stability or explain the EoS phenomenon.

Furthermore, the decomposition ζ=χς/τ\zeta=\chi\varsigma/\tau is an exact algebraic identity. While the individual factors may provide useful descriptive information, their decomposition alone does not establish a mechanism explaining why optimizers select particular directions or maintain stability. I find the connection between the proposed diagnostic and the claimed stability interpretation insufficiently substantiated.

  1. The novelty and empirical significance of the proposed metric are not sufficiently demonstrated.

The curvature load is derived from the integral remainder of Taylor's theorem, and its relationship to curvature along the actual optimization direction is closely connected to existing directional-smoothness analyses, including Mishkin et al. (2024). Although the paper introduces an optimizer-dependent decomposition, the distinction between the proposed metric and existing measures of directional smoothness or realized optimization progress is not sufficiently clarified.

The experiments also do not convincingly establish that ζ\zeta provides new information about stability beyond the observed loss trajectory. Figures 1 and 3 primarily demonstrate that the proposed quantity exhibits smaller deviations from the threshold of one. However, this property is partly expected from its definition for loss-decreasing steps. Moreover, the reported behavior is not universal: some optimizer configurations remain substantially below ζ=1\zeta=1, rather than approaching the proposed edge.

The experiments cover a useful range of optimizers, but the main neural-network results are obtained using one architecture and dataset in the full-batch setting. More importantly, the paper does not demonstrate that ζ\zeta predicts an independent stability-related outcome better than existing diagnostics. Since the proposed reformulation of EoS is the central contribution, this substantially limits the demonstrated significance of the work.

Questions for authors
  1. Relationship to existing directional-smoothness measures.

Could the authors clarify the essential difference between the proposed realized EoS ζ\zeta and previously studied directional-smoothness or point-wise smoothness measures, particularly those of Mishkin et al. (2024)? What additional theoretical conclusion or diagnostic capability is uniquely enabled by the proposed formulation?

  1. Connection between realized descent and dynamical stability.

Since ζ≤1\zeta\leq1 is equivalent to a non-increasing loss for positive gradient-update alignment, what justifies interpreting ζ=1\zeta=1 as a dynamical stability boundary rather than a one-step descent boundary? Is there an independent theoretical or empirical result connecting ζ\zeta to sustained oscillations, marginal stability, or future instability?

  1. Interpretation of the experimental observations.

The reported values of ζ\zeta do not consistently concentrate near one across all optimizer families, particularly for some two-pole filters. How should these cases be interpreted within the proposed EoS framework? Additionally, could the authors clarify how the conclusions depend on the 51-step smoothing and the selection of the edge-residence windows?

  1. Non-positive gradient-update alignment.

For stateful optimizers, g⊤ug^\top u may be zero or negative. How frequently does this occur in the reported experiments, and how are such cases handled when computing ζ\zeta and its logarithmic decomposition? Does the proposed interpretation remain meaningful outside the positive-alignment regime?

5 comments

We deeply appreciate the reviewer for the considerate and meticulous review. We are greatly encouraged by the reviewer's acknowledgement of the theoretical and practical usefulness, which was the most important thing we cared for. We answer each of the raised concerns below:


[W1,Q2] Significance of the proposed metric ζ\zeta: Connection between realized descent and dynamical stability

We believe that distinguishing the instantaneous metric for realized descent ζ\zeta and the measure of dynamical stability ζ^:=Med(ζ)\hat \zeta := \mathrm{Med}(\zeta) gives a clearer view. As the reviewer noted, the instantaneous metric ζ\zeta does not by itself give the marginal dynamical stability, but its running median ζ^\hat \zeta does. To this end, we decided to separate the terminology for these two quantities. The former, instantaneous curvature-to-alignment ratio ζt:=Bt/At=ut⊤Hˉtut/gt⊤ut\zeta_t := B_t / A_t = u_t^\top \bar H_t u_t / g_t^\top u_t, is now called only the learning impedance, which was suggested as an alias for this quantity in the previous version of the manuscript. And, as a consequence, we exclusively name its running median ζ^\hat \zeta the realized edge of stability, responsible for estimating the marginal dynamical stability threshold.

To further clarify the significance of the latter quantity, we additionally add a dedicated Section 6: Realized Edge of Stability that discusses how this running median ζ^\hat \zeta measures dynamical stability through three theorems (Theorem 6.1, Proposition 6.2, and Proposition E.1). We can summarize the added part as follows:

  • Theorem 6.1 proves that under mode-locked dynamics, the learning impedance ζ\zeta follows a (heavy-tailed) Cauchy distribution.
  • This heavy-tailed Cauchy distribution of ζ\zeta incorporates the negative gradient-update alignment ζ<0\zeta < 0 as a natural tail of this distribution, rather than an optimization pathology (this answers [Q4] Non-positive alignment).
  • Theorem 6.1 additionally gives that a combined statistical value R(ζ):=(1−2,Med(ζ))2+IQR(ζ)2\mathcal R(\zeta) := \sqrt{(1 - 2\\, \mathrm{Med}(\zeta))^2 + \mathrm{IQR}(\zeta)^2} recovers the spectral marginality under general frequency.
  • For Nyquist frequency (ω=π\omega = \pi), Theorem 6.1 implies that a median of ζ\zeta recovers a spectral marginality. This corresponds to most of the optimizers except for the two-pole case (please also see our answer to [W2,Q3] Interpretation...).
  • For non-Nyquist frequency, this median still produces the spectral marginality with a phase-dependent factor sin⁡2(ω/2)\sin^2(\omega / 2), hence giving a sub-unity threshold with a simple, analytically tractable factor for this two-pole case (this answers [W2,Q3] Interpretation...).
  • Proposition 6.2 proves that a finite-window running median converges to this ideal statistic. At Nyquist frequency, the same median recovers the population value of the spectral marginality under mode-locked learning dynamics exactly whenever fewer than half of the observations are corrupted.
  • Proposition E.1 further proves this finite-window robustness for the IQR-supplemented value R(ζ)\mathcal{R}(\zeta).

In practice, we find that a long-windowed running median of ζ\zeta produces this theoretically expected range of [0,1][0, 1] for spectral marginality (Figure 3), in contrast to the worst-case available marginality χ\chi from the original edge of stability. Also, through the root locus analysis (Figure 7) and analytic gauge calculation in Appendix C, we find that a non-Nyquist edge is a rare phenomenon, and most of the optimizers do not correspond to one. Since we can analytically calculate the root locus's unit circle escape angle ω⋆\omega_\star, the sub-unity factor sin⁡2(ω⋆/2)\sin^2 (\omega_\star / 2) from Theorem 6.1 is a tractable boundary. Therefore, using a running median is sufficient for measuring spectral marginality at root-locus unit-circle escape, validating the median of ζ\zeta, and not the raw ζ\zeta, as a measure of the marginal dynamical stability boundary.

We hope this addresses the concern. We have updated the manuscript with these sharpened definitions. This does not change our experimental settings, results, and their interpretations, which have already been done on the running median values. The change is therefore to clarify, rather than to replace the existing results. Please also take a look at the general response above.


(Continued)

(Continued)


[Q4] Non-positive gradient-update alignment

We have answered this question above in our response to [W1,Q2] Significance.... As the reviewer might already have known, this non-positive gradient-update alignment has been treated as a pathological, undesired behavior at the edge of stability that needs to be smoothened out (Cohen et al., 2025) or rejected (Liang et al., 2026, Chang and Yuan, 2025). However, our new theoretical analysis in Section 6, Theorem 6.1 shows that this negative alignment is a natural consequence of the heavy-tailed Cauchy distribution, which our instantaneous metric of learning impedance follows under mode-locked dynamics. Moreover, by specifying the Cauchy distribution it follows, its (running) median gains an explanatory power that even incorporates this non-positive alignment. That said, the median of ζ\zeta recovers the spectral marginality under Nyquist frequency, and recovers a factored version of this value under non-Nyquist frequency, with an analytically tractable factor sin⁡2(ω/2)\sin^2 (\omega/2) depending on the phase of the oscillation. We, therefore, call this median ζ^\hat\zeta the realized edge of stability, separate from the (instantaneous) learning impedance ζ\zeta.


[References]

  • Cohen et al., Understanding Optimization in Deep Learning with Central Flows, in ICLR, 2025.
  • Liang et al., Cautious optimizers: Improving training with one line of code, in ICLR, 2026.
  • Chang and Yuan, MGUP: A momentum-gradient alignment update policy for stochastic optimization, in NeurIPS, 2025.

[W2,Q3] Interpretation of the experimental observations: some optimizer configurations remain substantially below ζ=1\zeta = 1

We thank the reviewer for this meticulous catch. As elaborated in Section 6, Theorem 6.1, this sub-unity concentration of the running median of ζ\zeta is an expected behavior for this two-pole optimizer. This specific type of optimizer is provided as an example case of non-Nyquist frequency root locus escape, which is typically not observable in widely-used optimizers. As shown in Figure 7, only this type of optimizer among our experimental suite exhibits such behavior. Even in these exotic non-Nyquist scenarios, Theorem 6.1 proves that the median of the learning impedance ζ\zeta recovers the spectral marginality with a simple, analytically tractable factor of sin⁡2(ω/2)\sin^2 (\omega/2). We additionally prove in Theorem 6.1 that the phase-independent spectral marginality can be recovered for general non-Nyquist cases by using the interquartile range (IQR) as a supplement for the statistics: R(ζ)=(1−2,Med(ζ))2+IQR(ζ)2\mathcal{R}(\zeta) = \sqrt{(1-2\\,\mathrm{Med}(\zeta))^2 + \mathrm{IQR}(\zeta)^2}. Proposition E.1 further proves that this statistic can be robustly estimated by the finite-window running statistics. However, we choose to resort to the running median, because

  1. it is simpler,
  2. it is less noisy than R(ζ)\mathcal{R}(\zeta) involving the IQR (the subtraction),
  3. it gives us an additional statistical interpretability by: "a running median of the learning impedance attains a dynamic stability threshold", and
  4. it additionally estimates the realized oscillation phase ω\omega through ζ^=sin⁡2(ω/2)\hat \zeta = \sin^2(\omega / 2), since the radius is unity r=1r = 1 at the spectral marginality (hence the unit circle escape), together with the negative alignment probability Pr(At<0)=(π−ω)/(2π)\mathrm{Pr}(A_t < 0) = (\pi - \omega) / (2\pi).

We hope this clarifies the concern.


[Q3] Interpretation of the experimental observations: dependence on 51-step smoothing

As a follow-up to our answer from [W1,Q2] Significance..., the 51-step smoothing is the key ingredient of the realized edge of stability making this value sharper rather than a smoothened instantaneous metric ζ\zeta. By Proposition 6.2 and Proposition E.1, the longer the window, the closer the (running) realized edge of stability ζ^N:=MedN(ζ)\hat \zeta^N := \mathrm{Med}_N(\zeta) estimates the true statistic ζ^\hat \zeta. We additionally provide a multi-window comparison in Appendix E. From this, we choose the window size of 51 that balances between the delay and the noise.


(Continued)

(Continued)


[W2,Q1] Novelty: Relationship to existing directional-smoothness measures

We appreciate this question. An essential difference between the proposed realized edge of stability ζ^\hat\zeta and the previously studied directional smoothness (Mishkin et al., 2024 and Islamov et al., 2026) or point-wise smoothness measures lies in its running median statistics, which is central to its ability to estimate the dynamic stability threshold, as we have elaborated in the above answers.

Additionally, we respectfully note that Islamov et al. (2026), the most recent work to utilize Mishkin's directional smoothness, explicitly wrote in their Conclusion and Future Work section that extending the framework to momentum-based and adaptive optimizers is beyond their scope. Our work fills this gap.


[References]

  • Mishkin et al., Directional smoothness and gradient methods: Convergence and adaptivity, in NeurIPS, 2024.
  • Islamov et al., Non-euclidean gradient descent operates at the edge of stability. In ICML, 2026.

We would like to ask the reviewer to take a look at the revised manuscript as well as the general responses above summarizing the updates, which were greatly influenced by the reviewer's comment. Again, we appreciate your thoughtful concerns.

Reviewer_wcXYOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper identifies systematic optimizer-dependent discrepancies in conventional edge-of-stability (EoS) measurements. It extends the stability analysis to general stateful gradient filters and introduces a realized EoS formulation based on the actual optimizer update. The resulting spatial–temporal decomposition is evaluated across various full-batch optimizers.

Critical strengths

S1. Clear presentation and well-motivated problem. The paper effectively introduces prior results and relevant preliminaries, identifies a meaningful limitation of conventional worst-case EoS formulations, and supports its motivation with systematic experiments across optimizer families.

S2. A useful balance between analytical simplicity and explanatory power. Rather than attempting to model the full complexity of nonlinear optimizer dynamics, the paper introduces an intermediate description through the executed update direction. This allows it to retain analytical tractability while accounting for optimizer-dependent behavior overlooked by worst-case spectral analyses. The paper also effectively adapts established tools from dynamical systems and control theory, particularly root-locus analysis, bringing a richer analytical perspective to EoS research. The theoretical results, interpretable decomposition, and extensive empirical evaluation together make a coherent contribution.

Critical weaknesses

W1. Primarily diagnostic explanation. The realized EoS decomposition characterizes optimizer-dependent behavior using observed updates rather than independently predicting how those updates evolve. This limits the predictive scope of the methodology, although it reflects a reasonable tradeoff between analytical tractability and explanatory power rather than a substantive shortcoming of the work.

W2. Limited empirical coverage across loss geometries. Although the paper systematically evaluates a broad range of optimizers, its nonlinear experiments are confined to a single architecture and dataset. This leaves some uncertainty about how consistently the observed EoS behavior extends across different loss geometries. Nevertheless, the experimental setting seems to follow established EoS benchmarks, and this limitation appears more reflective of the current scope of empirical EoS research than a substantial shortcoming of this particular work.

Questions for authors

Q1. Since ζ=1\zeta=1 corresponds exactly to one-step loss neutrality, what evidence supports its interpretation as a marginal dynamical stability boundary for general stateful optimizers, rather than primarily a measure of normalized loss progress?

3 comments

We greatly appreciate the reviewer for overall positive and insightful comment. We are very much encouraged by the reviewer's acknowledgement of our clear presentation, motivation, and usefulness of our approach. We will answer each raised concern below.


[W1] Primarily diagnostic explanation

We agree with this insightful comment. Although our measures of the learning impedance ζ\zeta and the realized edge of stability are online diagnostics that can be inexpensively computed at runtime, they are not predictive in their primary form. Achieving predictive capability will surely enhance the practical value of the measures dramatically, and we hope to address this in future work.

Aligned to this concern, our new Theorem 6.1 shows that, for a mode-locked, possibly oscillatory learning system at its stability boundary, our quantity, learning impedance ζt=Bt/At\zeta_t = B_t / A_t, converges to a Cauchy distribution (a heavy-tailed distribution). This not only characterizes its running median ζ^t=Med(ζt)\hat \zeta_t = \mathrm{Med}(\zeta_t) (now exclusively called the realized edge of stability) as a robust measure for dynamic stability, but also allows us to study this time-varying learning dynamics as an evolution over a Cauchy distribution.

This may not finalize our understanding of how learning will evolve, but we believe that this can unlock more predictive measures to eventually control learning dynamics. We hope this addresses your concern.


[W2] Limited empirical coverage across loss geometries

We also agree with this concern. A deep learning system consists of many inseparable components, e.g., model, data, sampling, optimizer, lr scheduler, etc., which complicates our understanding of learning process as a system, and hampers our attempts to control this for our needs. We believe that carefully separating these interfering factors from the entangled learning process can be a stepping stone for improving our understanding and controllability, and that the current research on EoS can take a role in this decomposition job. For example, in our work, by fixing the learning problem, we could separate the learning rate scheduling ηQ(1)\eta Q(1), optimizer preconditioning PP, and optimizer state Q(z)/Q(1)Q(z)/Q(1) from the first-order optimization algorithm. This distinction allows us to focus on the stability margin prescribed by the optimizer's temporal structure, eventually leading us to an online diagnostic of ζ\zeta and ζ^\hat \zeta. We agree with the reviewer that scaling up the experiments is a necessary step towards practical value. We believe that this will be available once the roles of these interfering components are fully factorized and the complication in our understanding of them is settled.


[Q1] Connection of the one-step loss neutrality ζ\zeta with a marginal dynamical stability boundary

We thank you for this comment. After a careful consideration, we believe that separating the definition of the one-step loss neutrality ζ\zeta and its running median ζ^=Med(ζ)\hat \zeta = \mathrm{Med}(\zeta) as a marginal dynamical stability boundary can build a cleaner connection between these roles. In our revised manuscript, we added a dedicated Section 6 that justifies the use of finite-window running median to estimate the spectral marginality of a dynamic learning system. To summarize in plain language what this section does:

  • Prove that under mode-locked dynamics, the one-step loss neutrality ζ\zeta follows a (heavy-tailed) Cauchy distribution.
  • Prove that a median of ζ\zeta recovers a spectral marginality.
  • Prove that a finite-window running median converges to this ideal statistic.

This consolidates the median of ζ\zeta as a measure of the marginal dynamical stability boundary.

We now exclusively call the running median ζ^\hat \zeta the realized edge of stability and call ζ\zeta only the learning impedance (we promoted the existing alias in the previous version), and updated our manuscript accordingly. This does not change our experimental settings, results, and their interpretations, which have already been done on the running median values. The change is therefore to clarify, rather than to replace the existing results. We hope this addresses the question.


We would like to ask the reviewer to check the revised manuscript as well as our summary in the general remarks above of the updates. Thank you again for the helpful feedback.

Thank you for the detailed response and clarifications. I maintain my positive assessment of the paper and my recommendation for acceptance.

Thank you again for the encouraging review. We truly appreciate your time and support.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Reviewers,

We appreciate the reviewers for taking the time to review our work and providing valuable feedback. We are greatly encouraged by the reviewers' interest and acknowledgement of our systematic approach to studying optimizer behaviors.

After careful reading, we find that a majority of the reviewers share the same concern about how the instantaneous measure of learning inefficiency ζ\zeta can be interpreted as a marginal dynamical stability boundary. We believe that distinguishing the instantaneous quantity ζ\zeta from its running median ζ^:=Med(ζ)\hat\zeta := \mathrm{Med}(\zeta) clarifies this.

As the reviewers have noted, the instantaneous curvature-to-alignment ratio ζ=B/A=u⊤Hˉu/g⊤u\zeta = B/A = u^\top \bar H u / g^\top u is primarily a measure of normalized loss progress [wcXY, Q1] or a measure of discrepancy between predicted and realized loss decrease [VuQ7, W1]. However, its running median, now separately marked as ζ^\hat \zeta, gives a phase-specific dynamic stability threshold for oscillatory learning dynamics (new Theorem 6.1). Under the Nyquist frequency (ω=π\omega = \pi), i.e., a period-2 oscillation, corresponding to all the optimizers except for an exotic two-pole optimizer in our experimental pool, this running median ζ^\hat \zeta correctly recovers the spectral marginality (new Proposition 6.2).

To better deliver this connection, we revised the manuscript to lay a clear distinction between the instantaneous quantity ζ\zeta and its running median ζ^\hat \zeta. The former, curvature-to-alignment ratio ζ\zeta, is now called the learning impedance, which was an auxiliary name for this quantity suggested in the previous version that reflects its primary role of measuring instantaneous learning inefficiency. The latter, a running median ζ^\hat \zeta, is now exclusively called the realized edge of stability. We renamed Section 4: Learning Impedance as so and instead added a dedicated Section 6: Realized Edge of Stability for this new definition backed by two theorems (new Theorem 6.1 and Proposition 6.2). Experiments already show the running median of ζ\zeta, so the change is more of a sharpening of the definition of existing quantities, rather than a full revision. We hope this answers this major concern of the reviewers.

As a final note, we would like to emphasize that our new quantities, the learning impedance ζ\zeta and the realized EoS, do not replace the original definition of edge of stability from Cohen et al. With our full respect, we would rather say that our new quantities serve as complementary measures to this original EoS (χ\chi in our manuscript), which gives an available worst-case marginality, apart from our realized dynamic marginality. We hope this addresses the reviewers' concern [ziea, Q3].

We thank the reviewers again for their considerate reviews, which have helped us improve this manuscript. We would love to have an opportunity to communicate with the reviewers in this discussion phase. Please feel free to share any unresolved concerns. We will remain attentive to the very last minute.

Also, it would be very kind of the reviewers to check the manuscript updates, especially the new Section 6.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.