acceptodds
Under review as a conference paper at ICLR 2027

From the Edge of Stability to Convergence Theory

15 views

Abstract

Gradient descent is usually analyzed under smoothness conditions that guarantee a decrease in the objective at every step. In modern neural-network training, however, gradient descent often operates at the Edge of Stability (EoS), where the largest Hessian eigenvalue oscillates around or above the classical stability threshold and the loss is not monotone. We study what controls convergence in this regime. Our analysis uses an averaged directional sharpness that takes into account the Hessian spectrum along the gradient-descent update. We show that a squared-gradient-weighted stability margin built from this quantity exactly determines the cumulative loss decrease along the trajectory. Lower bounds on this margin directly give gradient-stationarity rates, as well as convergence rates under PL condition. To control the margin when the loss oscillates, we introduce a sufficient condition based on comparing iterates steps apart. The condition is derived using classical smoothness, but can still hold when individual gradient-descent steps increase the loss. Importantly, we show that it is satisfied by various theoretical neural-network models. Experimentally, we verify lower margin bounds in MLPs and Vision Transformers. Across full-batch experiments on CIFAR-10 and Tiny ImageNet, the cumulative margins is shown to control the dynamics, and we further characterize how mini-batching affects averaged sharpness.

Primary area:
optimization
Keywords:
Edge of Stability, Gradient Descent, Non-smooth Optimisation, Non-convex Optimization, Sharpness

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_oxEZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies convergence guarantees for gradient descent operating at the Edge of Stability (EoS), where the largest Hessian eigenvalue may exceed the classical stability threshold and the training loss is not necessarily monotone. The authors introduce an averaged directional sharpness measure and define a cumulative margin weighted by squared gradient norms. Using an exact loss-change identity, they derive best-iterate gradient-stationarity bounds and convergence rates under a trajectory-local Polyak–Łojasiewicz (PL) condition.

The main theoretical contribution is a multi-step sufficient condition that guarantees positive cumulative progress despite individual loss-increasing updates. The authors establish corresponding convergence rates and construct explicit neural-network examples exhibiting non-monotone loss and sharpness oscillations while satisfying the proposed condition. The experimental analysis includes full-batch gradient descent on MLPs, CNNs, and Vision Transformers, together with supplementary investigations of mini-batch optimization and an OPEN-1B language-model training trajectory.

Critical strengths
  1. A meaningful theoretical framework connecting non-monotone optimization dynamics to convergence guarantees.

The paper provides a clear and intuitive approach to studying convergence at the Edge of Stability. In particular, the cumulative margin formulation connects the actual curvature encountered along gradient-descent steps with gradient-stationarity guarantees. More importantly, Theorem 2 introduces a quantitative multi-step sufficient condition that can accommodate loss-increasing iterations while ensuring longer-term optimization progress.

  1. Nontrivial theoretical constructions supported by a carefully designed empirical study.

A particularly valuable aspect of the paper is that the authors do not stop at deriving conditional convergence statements. Propositions 4 and 5 provide explicit neural-network constructions exhibiting EoS-type behavior while satisfying the proposed sufficient condition. The experimental study further supports the relevance of the proposed quantities across different architectures and datasets. I also appreciate the additional investigation of mini-batch optimization and the reproducible analysis of OPEN-1B training checkpoints. Although these experiments do not directly establish the main convergence guarantees, they provide useful empirical context for the theoretical results.

Critical weaknesses
  1. The applicability of the proposed multi-step condition to practical neural-network training remains insufficiently established.

The convergence guarantees depend on a quantitative multi-step sufficient condition involving accumulated gradients and local smoothness bounds. While the authors rigorously verify this condition for specially constructed neural-network objectives, its applicability to general neural-network loss landscapes remains uncertain. The practical experiments primarily measure cumulative margins and fit their observed scaling over finite training windows. These measurements support the qualitative motivation, but they do not directly verify the sufficient condition underlying Theorem 2 or establish uniform lower bounds over arbitrarily long trajectories.

  1. The novelty of the general convergence formulation should be distinguished more explicitly from existing directional-smoothness and non-monotone descent theory.

The averaged directional sharpness and its associated loss-change identity are closely related to previously established results, including Mishkin et al. (2024) and Litman (2026), as the authors acknowledge. Moreover, the convergence bounds in Theorem 1 follow relatively directly once a positive cumulative margin is assumed.

The more substantial novelty appears to reside in the multi-step sufficient condition of Theorem 2 and the explicit constructions in Propositions 4 and 5. I would appreciate a sharper explanation of how this condition advances beyond established multi-step or non-monotone sufficient-decrease analyses.

This does not undermine the validity of the proposed framework, but a clearer separation between existing mathematical ingredients and the new theoretical contribution would strengthen the paper's significance.

Questions for authors

See in weaknesses.

3 comments

Thank you for your work and for the positive evaluation of our framework.

W1:

Indeed, the multi-step condition can be challenging to verify in practice. The reason is that, in practical neural networks, the oscillations that occur at EoS can exhibit irregular patterns (see Figure 1 for instance), whereas the multi-step condition (Cmk)(C_m^k) requires progress over every block of a fixed length kk. Another consideration is that an accurate estimate of the smoothness bound LL along the trajectory is not typically known a priori. We therefore view (Cmk)(C_m^k) as a middle ground between the monotone one-step descent guarantees based on LL-smoothness or (L0,L1)(L_0,L_1)-smoothness, and the long-term cumulative-margin approach of Section 2. We have added a short discussion of these practical considerations in Section 6.

W2:

We agree that our work builds on these results and we cited them again in lines 134-136 of our revised version.

As you pointed out, the main novelty is indeed in Theorem 2 as well as Propositions 4–5, while Section 5 features a non-tivial adaptation of our main idea in stochastic GD. Unlike classical one-step sufficient-decrease conditions, our condition (Cmk)(C_m^k) allows individual GD steps to increase the loss while still guaranteeing convergence over multiple steps. For k=1k=1 and m=0m=0, our condition recovers an instance of the classic descent lemma for gradient descent under LL-smoothness (we added this remark in line 263 of our revised version). For k=2k=2, our condition recovers many existent conditions and a full comparison is made in Appendix E. As shown in Propositions 9 to 13, our condition is the most general among the considered ones, as it is non asymptotic, incorporates kk cycles and allows a general power on Bk(x)B_k(x).

Dear reviewer,

as the discussion window closes soon, we wonder whether we have addressed your concerns adequately. We would be happy to elaborate more in case you have further concerns.

I appreciate the response from the authors. I got most of my questions clear after reading the response and the revised paper. I will maintain my positive evaluation.

Reviewer_vTLCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the convergence theory on the Edge of Stability, where the largest Hessian eigenvalue oscillates around the classical stability threshold. The authors investigate the source of the loss decrease to control it. The authors derive a squared-gradient-weighted stability margin from the averaged directional sharpness and show that the margin exactly determines the cumulative loss decrease. Based on this identity, they show that a lower bound on the margin guarantees convergence in the long run, even when individual steps increase the loss or have large curvature.

The authors further propose a k-step sufficient condition for such a lower bound. They construct small neural networks that satisfy the condition while the loss and the sharpness oscillate. Finally, they evaluate a mini-batch analog of the margin in Transformer training.

Critical strengths

The authors provide technically sound theory to explain the convergence on the edge of stability.

Critical weaknesses

The convergence result is circular (informativeness 1)

The authors present the identity between the margin and the loss decrease, and the resulting convergence rates, as the main contribution. However, the identity is not new, and the rates follow from it without additional insight.

The identity and the averaged directional sharpness were derived by Litman (2026), as acknowledged in the related work. Given the identity, assuming a lower bound on the cumulative margin is equivalent to assuming that the loss decreases in proportion to the sum of squared gradient norms. Theorem 1 restates this assumption as a convergence rate. Trajectory-dependent rates based on directional curvature were also studied by Mishkin et al. (2024).

The convergence guarantee does not explain why the loss decreases on the edge of stability; it assumes that it does.

Questions for authors

Are there any stronger, more meaningful sufficient conditions on the network structure or optimization problem that guarantee convergence?

What is the useful insight or takeaway from the result in this paper?

3 comments

Thank you for your work and for finding our theory technically sound.

W:

We thank the reviewer for raising this point. We agree that the identity relating averaged directional sharpness to the loss decrease was previously derived by Litman (2026), and that Theorem 1 follows from this identity once a positive lower bound on the cumulative margin is assumed. We have therefore made the connections to both Litman (2026) and Mishkin et al. (2024) explicit in the revised main text, where we introduce averaged directional sharpness and the associated loss-change identity (line 134). However, our main theoretical contribution is not Theorem 1 in isolation, but rather establishing realistic conditions under which its assumptions can be guaranteed in the EOS regime. This is the purpose of Definition 1 which introduces a new condition, Theorem 2 which shows that this condition implies lower bounds for the cumulative margin and Propositions 4–5 which show that this condition holds in certain neural networks during the EoS regime.

Q1:

Stronger conditions that guarantee cumulative margin lower bounds (thus also convergence) exist and one of them is the condition (Cmk)(C_m^k) we introduce in Definition 1. We show that this condition is satisfied in small neural networks that operate at the edge of stability in Section 4.

Q2:

As mentioned in our response in (W), our paper shows the following:

i) Convergence at the EoS is governed by the curvature encountered along the actual gradient descent trajectory, rather than by the largest Hessian eigenvalue,

ii) Theoretical (through the (Cmk)(C_m^k) condition) and experimental lower bounds on a cumulative margin made by a weighted sum of Hessian eigenvalues combined with the positive PL constant guarantees convergence (see Table 1 for empirically fitted constants),

iii) The (Cmk)(C_m^k) condition is realistic for simple networks and can co-exist with cyclic/oscillatory GD dynamics.

iv) Averaged sharpness is the main ingredient for analyzing the convergence of optimizers such as SGD in mini-batch settings (Section 5).

Thank you for the response to my review. Although I appreciate the authors' clarification and the updates to the manuscript, my previous concern remains unaddressed, so I will keep my score unchanged. Here are the details of my remaining concerns:

  1. The "Covergence at the EoS is governed by the curvature encountered along the actual gradient descent trajectory" part

Given Litman (2026) and Mishkin et al. (2024), I still think that this insight was already available in the literature. Standard convergence analysis has the curvature encountered along the actual gradient descent trajectory as an intermediate term, which is further bounded when the curvature of the function is globally bounded (or by other assumptions on the regularity of the function). Thus, I disagree that this should be counted as the contribution of this paper.

  1. The "C_m^k condition" part

This condition still feels like a condition that (1) just bounds an intractable term in the convergence analysis with some other term that are easier to handle for deriving convergence and (2) does not provide useful fundamental insights into what kind of architecture has better convergence properties or how different optimizers or hyperparameters lead to convergence.

Thanks for your response and for digging deeper into what our contributions really mean.

  1. We agree that our paper is not the first to show the connection between directional sharpness and convergence. The difference of Theorem 1 with Mishkin et al. is that we consider nonconvex stationarity and convergence under a PL condition allowing a horizon-dependent vanishing margin. The difference to Litman is that they use the directional sharpness identity mainly to understand self-stabilization and curvature concentration around the EoS threshold. This is the reason we mentioned directional sharpness in our previous response; we made this point clearer in lines 137-140 and 208-212 of our revised paper. Beyond that, Theorem 1 is a relatively direct consequence of the very definition of directional sharpness. Instead, the central contribution of our work is the introduction of the (Cmk)(C_m^k) condition and its verification in specific neural networks that operate at EoS, while less central contributions include empirical scaling that governs the cumulative margin and the PL constant of practical networks at EoS (see Table 1) and an application to stochastic GD.

  2. (1) We kindly disagree with the reviewer that the (Cmk)(C_m^k) condition "just bounds an intractable term in the convergence analysis with some other term that are easier to handle". This would be largely fair, if we had stopped only on defining the condition, or if we could only show that this condition holds for a very specific GD trajectory. In Propositions 4 and 5, we show that the condition holds in explicit forward-invariant regions with nonempty interior (on which the same certificate inequality holds uniformly), i.e. it is a verifiable condition concerning the loss landscape of neural networks and the guarantees apply to every initialization in the specified sets. For k=1,m=0k=1,m=0, the condition recovers the classic descent lemma for GD under LL-smoothness (as noted in line 272), while it generalizes many conditions from the EoS-related literature, as analyzed in Appendix E. (2) The condition and its mm and kk parameters give also specific insights. Proposition 4 identifies different leading orders 2m+12m+1 for residual alternation for three small neural network objectives. These produce the corresponding powers B2m+1B_2^{m+1} and different mm-dependent convergence rates. Proposition 5 features a competition between a linearly accumulating damping and a geometrically decaying mismatch, allowing the first loss improvement to occur at any prescribed even step k≥4k\ge4, with every shorter prefix remaining above the initial loss. Thus mm and kk reflect concrete properties of the constructed dynamics, rather than merely convenient parameters in a bound. An interesting architectural observation comes from comparing the networks of cases (i) and (iii) of Proposition 4, which can be both written as $ F(w)=\frac{1}{2}\tau(w_1w_2x-1)^2, $ with activation functions τ(z)=z\tau(z)=z in case (i) and τ(z)=z1+z2/2\tau(z)=z \sqrt{1+z^2/2} in case (iii). The choice of the activation function generates specific one-step residual alternation dynamics of the form r+=(1+r)(1−τ(r)τ(r)′)2−1r^+ = (1+r)(1-\tau(r)\tau(r)')^2-1, which create two-step residual dynamics of the form r++=r−cr2m+1+O(∣r∣2m+2)r^{++} = r-cr^{2m+1}+O(|r|^{2m+2}) for some mm; the cancellation that yields the term −cr2m+1-cr^{2m+1} being the first after rr depends on τ\tau. For τ(z)=z\tau(z)=z we have m=1m=1, while for τ(z)=z1+z2/2\tau(z)=z \sqrt{1+z^2/2} we have m=3m=3. Thus, at the same step size, this change of output nonlinearity changes the asymptotic loss decay from Θ(t−1)\Theta(t^{-1}) to Θ(t−1/3)\Theta(t^{-1/3}), i.e. it produces a concrete convergence speed difference. Lastly, the reviewer is right that our analysis concerns only full-batch GD at constant learning rate. It does not claim a general rule for choosing learning rates or comparing optimizers.

Reviewer_FBjyOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper theoretically studies how to model the final convergence stage in the EoS regime through the lens of "averaged directional sharpness" (the average sharpness along a GD step). In particular, they extend the analytical framework in [1] and proved bounded convergence rate based on how a k-step descent condition.

Then the authors constructed a family of 2-variable toy tasks satisfying the condition with k=2k=2, and showed that the model can predict the convergence rate well, and further extend the example to a matrix-valued model (though the dynamics is still 2-dimensional, which is not uncommon for analysis in the EoS regime).

Given the careful analysis and the concrete theoretical contributions on top of existing works, I lean toward acceptance.


[1] Litman, Elon. "The Origin of Edge of Stability." arXiv preprint arXiv:2604.20446 (2026).

Critical strengths
  1. The paper provides a clean convergence analysis based on a general condition of multi-step dynamics. I find the analysis rigorous and believe it is a concrete contribution toward quantitative convergence theory in the EoS regime.
Critical weaknesses
  1. Among all analysis of the toy models, the studied set of initialization are all of measure 0, which limits the generality of the analysis.

  2. I believe the authors shall more explicitly state the paper's relation with [1] in the main text when presenting the general analytical framework. Now it is only cited in the appendix. In addition, [2] also studies EoS convergence beyond worst-case sharpness by modeling the time-averaged dynamics, yet it is not cited and discussed against in this paper.


[1] Litman, Elon. "The Origin of Edge of Stability." arXiv preprint arXiv:2604.20446 (2026). [2] Cohen, Jeremy M., et al. "Understanding optimization in deep learning with central flows." arXiv preprint arXiv:2410.24206 (2024).

Questions for authors
  1. Among all toy models discussed in the paper, the valid regime of initialization are all of measure zero. Can any of these analysis be extended to an initialization set that has positive measure and not asymptotically small? If not, what is the main obstruction?
3 comments

Thank you for your time and positive evaluation of our work.

W1:

Indeed, when designing our examples, we chose the specific zero-measure initializations for simplicity. We can generalize them to non-zero measure sets. See our response to Q1 for more details.

W2:

We agree with the reviewer that the connections to Litman (and also Mishkin et al.) should be stated more clearly when introducing averaged directional sharpness. We have therefore added this discussion to the main text in line 134. The Central Flow paper is also closely related to our work as it models the time-averaged path of oscillatory gradient descent through continuous-time dynamics, without formal convergence guarantees. In contrast, since optimizers used in practice update network parameters in discrete steps, we analyze the GD iterates directly, deriving rigorous convergence guarantees. In Section 6, we acknowledge, that extending our theory to gradient flows is a promising direction. We have now expanded that paragraph to cite Central Flows and other continuous-time dynamics works. We have also cited some of these works in Appendix A.

Q1:

Thank you for raising this point. Indeed, extensions to initialization sets of non-zero measure are possible for all cases of Proposition 4 and Proposition 5 and you can see the changes in our revised version. The simplest extension is for (i) of Proposition 4 to the box (7/8,1)2(7/8,1)^2, which is of measure 1/641/64 (original version was operating only on the lines w1=1,w2=1w_1=1,w_2=1). For cases (ii) and (iii) of Proposition 4 that were both operating in the balanced line w1=w2w_1=w_2, one can allow for some imbalance that must be controlled by an mm power of the residual (the same mm that appears in the (Cmk)(C_m^k) condition), thus creating a cusp that vanishes as the loss vanishes, but with measure that can be lower bounded by a positive number. In Appendix F.1 (lines 1828-1832), we construct a GD-invariant region by taking the union of a compact set containing the initializations with its image after one GD step.

Similarly, in Proposition 5, the specific previous initialization based on P=eeTP = ee^T was only used to simplify the calculations and can be enlarged to a nonempty open initialisation set J\mathcal{J} of positive 2d22d^2-dimensional Lebesgue measure. The invariant region Ω\Omega stays the same. J\mathcal{J} also looks like a cusp as ∣U∣F\\|U\\|_F becomes small, but has a strictly positive Lebesgue measure, since it is an open subset of Rd×d×Rd×d\mathbb{R}^{d \times d} \times \mathbb{R}^{d \times d}.

We are not totally sure what the reviewer means by "non-asymptotic". While the box (7/8,1)2(7/8,1)^2 of the first case is certainly non-asymptotic, in all other cases the initialization sets have the form of cusps, which become asymptotically small with the residual, but have strictly positive measures as a whole. There are specific obstructions that do not allow the construction of non-vanishing cusps in these cases while keeping the oscillation result in exactly the same form and we would be happy to discuss further if this is what the reviewer means by asymptotic.

Dear reviewer,

as the discussion window closes soon, we wonder whether we have addressed your concerns adequately. We would be happy to elaborate more in case you have further concerns.

I would like to thank the authors for their response and for properly attributing the theoretical framework to previous works. The revised Propositions 4 and 5 address my primary concern on initialization, and I also find the clarification regarding the asymptotically narrowing cusp regions reasonable. I will maintain my positive score.

Comments on the paper

1 comment

We thank the reviewers for their precious time and for being constructive. We would also like to thank them for the generally positive evaluation of our work. The main concerns are mainly:

i) Relationship to previous literature, mainly Litman (2026), Mishkin et al. (2024) and Cohen et al. (2024).

ii) The generality of initialization in our examples of Propositions 4 and 5.

iii) How our (Cmk)(C_m^k) condition advances beyond established multi-step or non-monotone sufficient-decrease analyses.

To address these concerns, we have uploaded a revised version of our paper with changes marked in blue. More precisely,

i) we cited papers related to continuous dynamics for EoS and clarified the connection to Litman and Mishkin et al., in Sections 2, 6 and Appendix A.

ii) We extended all our examples to initialization sets of non-zero measure.

iii) We clarified the connection of our (Cmk)(C_m^k) condition to LL-smoothness in Section 3 and note that a thorough comparison to other existing conditions involving oscillatory dynamics is included in Appendix E.

We respond with further details to each reviewer separately. Thank you again for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.