- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies convergence guarantees for gradient descent operating at the Edge of Stability (EoS), where the largest Hessian eigenvalue may exceed the classical stability threshold and the training loss is not necessarily monotone. The authors introduce an averaged directional sharpness measure and define a cumulative margin weighted by squared gradient norms. Using an exact loss-change identity, they derive best-iterate gradient-stationarity bounds and convergence rates under a trajectory-local Polyak–Łojasiewicz (PL) condition.
The main theoretical contribution is a multi-step sufficient condition that guarantees positive cumulative progress despite individual loss-increasing updates. The authors establish corresponding convergence rates and construct explicit neural-network examples exhibiting non-monotone loss and sharpness oscillations while satisfying the proposed condition. The experimental analysis includes full-batch gradient descent on MLPs, CNNs, and Vision Transformers, together with supplementary investigations of mini-batch optimization and an OPEN-1B language-model training trajectory.
Critical strengths
- A meaningful theoretical framework connecting non-monotone optimization dynamics to convergence guarantees.
The paper provides a clear and intuitive approach to studying convergence at the Edge of Stability. In particular, the cumulative margin formulation connects the actual curvature encountered along gradient-descent steps with gradient-stationarity guarantees. More importantly, Theorem 2 introduces a quantitative multi-step sufficient condition that can accommodate loss-increasing iterations while ensuring longer-term optimization progress.
- Nontrivial theoretical constructions supported by a carefully designed empirical study.
A particularly valuable aspect of the paper is that the authors do not stop at deriving conditional convergence statements. Propositions 4 and 5 provide explicit neural-network constructions exhibiting EoS-type behavior while satisfying the proposed sufficient condition. The experimental study further supports the relevance of the proposed quantities across different architectures and datasets. I also appreciate the additional investigation of mini-batch optimization and the reproducible analysis of OPEN-1B training checkpoints. Although these experiments do not directly establish the main convergence guarantees, they provide useful empirical context for the theoretical results.
Critical weaknesses
- The applicability of the proposed multi-step condition to practical neural-network training remains insufficiently established.
The convergence guarantees depend on a quantitative multi-step sufficient condition involving accumulated gradients and local smoothness bounds. While the authors rigorously verify this condition for specially constructed neural-network objectives, its applicability to general neural-network loss landscapes remains uncertain. The practical experiments primarily measure cumulative margins and fit their observed scaling over finite training windows. These measurements support the qualitative motivation, but they do not directly verify the sufficient condition underlying Theorem 2 or establish uniform lower bounds over arbitrarily long trajectories.
- The novelty of the general convergence formulation should be distinguished more explicitly from existing directional-smoothness and non-monotone descent theory.
The averaged directional sharpness and its associated loss-change identity are closely related to previously established results, including Mishkin et al. (2024) and Litman (2026), as the authors acknowledge. Moreover, the convergence bounds in Theorem 1 follow relatively directly once a positive cumulative margin is assumed.
The more substantial novelty appears to reside in the multi-step sufficient condition of Theorem 2 and the explicit constructions in Propositions 4 and 5. I would appreciate a sharper explanation of how this condition advances beyond established multi-step or non-monotone sufficient-decrease analyses.
This does not undermine the validity of the proposed framework, but a clearer separation between existing mathematical ingredients and the new theoretical contribution would strengthen the paper's significance.
Questions for authors
See in weaknesses.
3 comments
Dear reviewer,
as the discussion window closes soon, we wonder whether we have addressed your concerns adequately. We would be happy to elaborate more in case you have further concerns.
I appreciate the response from the authors. I got most of my questions clear after reading the response and the revised paper. I will maintain my positive evaluation.
Thank you for your work and for the positive evaluation of our framework.
W1:
Indeed, the multi-step condition can be challenging to verify in practice. The reason is that, in practical neural networks, the oscillations that occur at EoS can exhibit irregular patterns (see Figure 1 for instance), whereas the multi-step condition (Cmk) requires progress over every block of a fixed length k. Another consideration is that an accurate estimate of the smoothness bound L along the trajectory is not typically known a priori. We therefore view (Cmk) as a middle ground between the monotone one-step descent guarantees based on L-smoothness or (L0,L1)-smoothness, and the long-term cumulative-margin approach of Section 2. We have added a short discussion of these practical considerations in Section 6.
W2:
We agree that our work builds on these results and we cited them again in lines 134-136 of our revised version.
As you pointed out, the main novelty is indeed in Theorem 2 as well as Propositions 4–5, while Section 5 features a non-tivial adaptation of our main idea in stochastic GD. Unlike classical one-step sufficient-decrease conditions, our condition (Cmk) allows individual GD steps to increase the loss while still guaranteeing convergence over multiple steps. For k=1 and m=0, our condition recovers an instance of the classic descent lemma for gradient descent under L-smoothness (we added this remark in line 263 of our revised version). For k=2, our condition recovers many existent conditions and a full comparison is made in Appendix E. As shown in Propositions 9 to 13, our condition is the most general among the considered ones, as it is non asymptotic, incorporates k cycles and allows a general power on Bk(x).