acceptodds
Under review as a conference paper at ICLR 2027

Does LLM Pre-Training Typically Occur at the Edge of Stability?

Abstract

Quadratic approximations provide a common lens for studying neural network optimization, but recent evidence challenges their predictive validity. In full-batch gradient descent with learning rate (LR) , Cohen et al. (2021) observed the Edge of Stability (EoS), where the largest Hessian eigenvalue concentrates near , seemingly at odds with classical stability conditions predicted by quadratic approximations. In this work, we revisit the fidelity of quadratic approximations as a model of neural network training dynamics, with a particular focus on their failure modes in LLM training. We first highlight a distinct failure mode of the quadratic approximation, where persistent negative curvature causes the quadratic dynamics to diverge regardless of the LR, even though the real dynamics remain stable. We term this phenomenon the Edge of Convexity (EoC). After removing the effect of this LR-independent instability, we then propose a generalized definition of Edge of Stability (EoS) for stochastic training with adaptive optimizers, based on whether increasing the LR induces significant divergence in the quadratic dynamics while leaving the real dynamics largely unchanged. Across different LLM pretraining runs with various model sizes up to M, we find that (1) EoC is always observed across LLM pretraining, and (2) EoS is also prevalent but not universal: it disappears when the LR becomes sufficiently small (e.g., after decay) or when the batch size falls below a critical threshold that has a positive linear correlation with the critical batch size. Together, these findings characterize when and how quadratic approximations fail and provide a foundation for future work on understanding training dynamics.

Primary area:
optimization
Keywords:
Optimization, Edge of Stability, Large Language Model, Pre-training

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_dtrzOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper characterizes failure modes of the quadratic approximation in analyzing neural network training dynamics. Specifically, the authors identify Edge of Convexity (EoC): the dynamics do not diverge even though negative curvature persists. The authors also generalize Edge of Stability (EoS, the dynamics keeping the curvature near the threshold required for the descent lemma) to stochastic training with adaptive optimizers. The generalization defines EoS as the training learning rate being close to the mean-square stability threshold on the local stochastic quadratic.

The authors evaluate EoC and EoS in 3000 checkpoints from more than 100 pretraining runs. Across LLM pretraining, they observe EoC consistently and EoS whenever the learning rate and the batch size are not too small. They also report that the batch size below which EoS disappears correlates positively with the critical batch size.

Critical strengths

The experiments cover a larger scale than prior studies of EoS.

Critical weaknesses

The stability notion does not make sense for a nonconvex objective (soundness 1)

The authors define stability as boundedness of the distance to the stationary point of the local quadratic and argues that negative curvature is problematic because it makes the dynamics unstable in this definition. However, this definition of stability is not a useful notion in nonconvex optimization, and negative curvature does not break the optimization.

This notion of stability is not desired in nonconvex optimization. Stationary points with negative curvature directions are saddles. Near the stationary point that is a saddle, it is desired to move away from the point (be unstable in the defined notion of stability) to decrease the loss. So it is not problematic at all to be “unstable.” Refer to the literature on escaping saddles.

Optimization still works with negative curvature. A nonconvex function can be decreased even if the iterate moves in the opposite direction to (or away from) a stationary or target point. Therefore, when analyzing nonconvex optimization, it is customary to track the objective value rather than the distance to a specific point. Refer to the literature on smooth nonconvex optimization. The descent lemma works (so SGD decreases the objective in expectation) even with negative curvature as long as smoothness (spectral radius of the Hessian) is bounded.

Taken together, the main empirical analysis focuses on the wrong problem: EoC labels expected dynamics in nonconvex optimization as a failure.

The paper re-iterates well-known results (informativeness 1)

All analysis in the paper builds on SGD dynamics for local quadratic approximation of the objective function. The main conclusions are based on the observation that deep learning dynamics deviate from the quadratic optimization dynamics. However, all these analyses are elementary results from classic optimization theory, and the paper offers no new explanation of why the real dynamics deviate from the quadratic optimization dynamics.

Refer to the analysis of GD/SGD dynamics for quadratic functions:

https://www.damtp.cam.ac.uk/user/hf323/M21-II-NA/Lect18.pdf

https://www.cs.toronto.edu/~rgrosse/courses/csc2541_2021/slides/lec07.pdf

https://ernestryu.com/courses/MLT/chapter4.pdf

Zhang, G., et al., "Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model," in Advances in Neural Information Processing Systems, 2019.

Stability of SGD on quadratic approximations has been studied extensively:

L. Wu, C. Ma, W. E, "How SGD Selects the Global Minima in Over-parameterized Learning: A Dynamical Stability Perspective," in Advances in Neural Information Processing Systems, 2018.

L. Wu, W. Su, "The Implicit Regularization of Dynamical Stability in Stochastic Gradient Descent," in Proceedings of the 40th International Conference on Machine Learning, 2023, pp. 37656–37684.

R. Mulayoff, T. Michaeli, "Exact Mean Square Linear Stability Analysis for SGD," in Proceedings of Thirty Seventh Conference on Learning Theory, 2024, pp. 3915–3969.

Questions for authors

Doesn’t quadratic approximation still provide useful quantitative estimates locally, even with negative curvature directions?

What new result or insight does the paper provide beyond the existing stability analyses of SGD on quadratics and the original EoS observation?

Reviewer_3i49OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks when the local quadratic model at a checkpoint, with the Adam preconditioner frozen, stops predicting LLM pre-training: because of a small negative population curvature (Edge of Convexity) or because the training learning rate sits at the local model's stochastic stability threshold (Edge of Stability). A stochastic power iteration and a compensated local simulation serve as detectors. On GPT models from 43M to 787M parameters, EoC is reported throughout, while EoS depends on the learning rate, batch size and schedule, and very large peak learning rates go beyond the edge.

Critical strengths

The question is worth asking, and the paper goes about it the right way: rather than tracking one sharpness number, it fixes a checkpoint, runs the local quadratic model forward, and asks when that model stops matching training. Two findings stand on their own: learning-rate decay moves training out of EoS and it may not return (Fig. 3, 11, 12), and at the largest peak learning rates the local model explodes well below the training learning rate while training still converges (Fig. 5, 14).

Critical weaknesses
  1. The stochastic detector is never compared with the deterministic Adam threshold, and the batch-size effect is confounded with the learning rate. The paper's own Appendix G shows noise can only lower the critical learning rate (eq.(73)), so the deterministic threshold is the natural reference, and Cohen et al. (2022) already studied it under minibatch Adam, including a Transformer. Without a comparison at the same checkpoints I cannot tell what the stochastic detector adds. The batch-size sweep also changes the learning rate, weight decay and second-moment decay together. Since Section 5.4 shows that lowering the learning rate alone moves training out of EoS, the independent effect of batch size remains unclear. The EBZ–CBZ correlation rests on five datasets without reported uncertainty, with the small-batch values censored at the top of the grid (Fig. 4b).

  2. The diagnostics need independent validation. Algorithm 1 returns a negative minimum eigenvalue by construction, so a negative output cannot by itself establish negative population curvature. The paper notes this but offers no independent calibration, and the EoC finding rests on it. Nothing marks the "edge" either: saddleness simply drifts down through training (Fig. 2, C.5), and ν_tol = 0.1 is not motivated. For EoS, the theory bounds second moments averaged over mini-batches, whereas the implementation runs one short trajectory per learning-rate factor. The paper admits the two thresholds need not coincide, but its bounds are never evaluated at the T and M actually used, and neither is varied. That matters when checkpoints are classified with a ±10% tolerance. The only check against real training is the two checkpoints of Fig. 1.

Questions for authors
  1. How do the estimated critical learning rates compare with the deterministic Adam threshold at the same checkpoints? If they agree, what does the stochastic detector add?
  2. Does the batch-size trend persist when the other optimiser parameters are held fixed? And how do censored checkpoints enter the EBZ fit?
  3. Can you validate the minimum-eigenvalue estimate independently (and report Λ, N and η_p), and check whether the EoS calls are stable to the simulation length and explosion cutoff?
  4. At the best-performing learning rate in Fig. 5, what fraction of checkpoints is within the EoS band? Fig. 2 is at half that learning rate.
7 comments

We thank the reviewer for a careful report. The revised paper makes these changes. New text is marked in blue.

  1. The eigenvalue panels of Figures 2, 8, 9, 11, 12 and 14–16 show the deterministic Adam threshold 38/η38/\eta as a dashed line, which the caption of Figure 2 explains, and a new Section 5.5 compares our measurements with Cohen et al. (2022) (Q1).
  2. Section 5.3 states that the batch-size results hold under the square-root scaling rule, and makes the censoring and the sample size of the EBZ–CBZ relation explicit (Q2).
  3. A new Appendix C.7 reports a held-out validation of the negative curvature (Table 2), the search settings, and the sensitivity to TT and MM (Table 3) (Q3).


W1: The stochastic detector is never compared with the deterministic Adam threshold, and the batch-size effect is confounded with the learning rate. The paper's own Appendix G shows noise can only lower the critical learning rate (eq.(73)), so the deterministic threshold is the natural reference, and Cohen et al. (2022) already studied it under minibatch Adam, including a Transformer. Without a comparison at the same checkpoints I cannot tell what the stochastic detector adds. The batch-size sweep also changes the learning rate, weight decay and second-moment decay together. Since Section 5.4 shows that lowering the learning rate alone moves training out of EoS, the independent effect of batch size remains unclear. The EBZ–CBZ correlation rests on five datasets without reported uncertainty, with the small-batch values censored at the top of the grid (Fig. 4b).

A: We agree with all three points. We answer them in Q1 and Q2, and summarise here.

  1. We now compare the detector with 38/η38/\eta at all 1,280 analysed checkpoints. At the 749 checkpoints near the threshold, the two agree to within one grid step at 92%. Q1 gives the numbers, the cases where they differ, and what the detector adds.
  2. Our batch-size sweep follows the square-root scaling rule (Malladi et al., 2024; Marek et al., 2025), which is the standard way to change the batch size in LLM pretraining. The reviewer is right that the sweep changes the learning rate, weight decay and β2\beta_2 together with the batch size. The paper states that the sweep follows this rule (Section 5.3), and the revision adds that the batch-size results hold under it. We have not run a sweep with the other parameters fixed (Q2).
  3. We present the EBZ–CBZ relation as a positive correlation observed on five datasets, not as a law. The correlation rests on five datasets, and we reported it without uncertainty. With five points the uncertainty is large: the 95% interval of the correlation is roughly 0.4 to 1. The data therefore support a positive relation. The small-batch values are censored at the top of the grid, as the reviewer notes, so those means are lower bounds (Q2).

W2: The diagnostics need independent validation. Algorithm 1 returns a negative minimum eigenvalue by construction, so a negative output cannot by itself establish negative population curvature. The paper notes this but offers no independent calibration, and the EoC finding rests on it. Nothing marks the "edge" either: saddleness simply drifts down through training (Fig. 2, C.5), and ν_tol = 0.1 is not motivated. For EoS, the theory bounds second moments averaged over mini-batches, whereas the implementation runs one short trajectory per learning-rate factor. The paper admits the two thresholds need not coincide, but its bounds are never evaluated at the T and M actually used, and neither is varied. That matters when checkpoints are classified with a ±10% tolerance. The only check against real training is the two checkpoints of Fig. 1.

A: We agree that the diagnostics needed independent checks, and we added two. Three of the reviewer's points stand as stated.

  1. On held-out batches the curvature is negative at all ten checkpoints we tested (new experiment, Q3). A negative output of the search does not by itself establish negative mean curvature, as the reviewer says. We now measure the curvature of the direction along which the uncompensated local model escapes, on batches that neither the training run nor the simulation used. At ten checkpoints of GPT-small, GPT-medium and GPT-large all 80 held-out values are negative, with means between −703 and −1738 against λP\lambda_P between 831 and 2791.
  2. The detector's output is stable when the simulation length and the cutoff are changed (Q3). The detector runs each local simulation for TT steps and counts it as exploding if the displacement norm reaches the cutoff MM. From the per-step logs of 384 checkpoints, all run for 50 steps, we recomputed its output for shorter simulations and for other cutoffs. The critical LR factor moves by at most one grid step at 98–99% of the checkpoints for T=30T=30 and T=40T=40 and at 94–99% for MM between 10310^3 and 10810^8. The EoS call is unchanged at 91% to 97% of the checkpoints.
  3. Three points stand as the reviewer states. (a) Nothing marks a sharp edge for EoC. The saddleness declines gradually through training, and νtol=0.1\nu_{\rm tol}=0.1 is a chosen tolerance that we do not derive. (b) The theory bounds second moments averaged over mini-batches, and the implementation runs one trajectory per learning-rate factor. We have not evaluated the bounds at the TT and MM used. The sensitivity numbers above are the evidence we have. (c) The only comparison with real training is the two checkpoints of Figure 1. We have not added one.

Q1: How do the estimated critical learning rates compare with the deterministic Adam threshold at the same checkpoints? If they agree, what does the stochastic detector add?

A: The two agree at most checkpoints, so our experiments validate 38/η38/\eta as the stability threshold in LLM pretraining. We ran this comparison at every analysed checkpoint of the runs behind Figures 2, 3, 8, 9, 11, 12 and 14–16 (46 runs, 1,280 checkpoints). The detector does not assume the threshold: it simulates the local quadratic model with mini-batch Hessians and negative curvature, a setting for which 38/η38/\eta was not derived. That it arrives at the same boundary is what the detector adds.

  1. Near the threshold, the detector's critical LR factor is within one grid step of the threshold's prediction at 92% of the checkpoints. For β1=0.9\beta_1=0.9 the noise-free threshold of Appendix G is λmax⁡=38/η\lambda_{\max}=38/\eta, up to a weight-decay term of about 0.1%. It predicts a critical LR factor of 38/(ηλmax⁡)38/(\eta\lambda_{\max}), which is 1 when λmax⁡\lambda_{\max} is at the threshold and smaller when λmax⁡\lambda_{\max} is above it. We call this the predicted factor. At the 749 checkpoints where λmax⁡\lambda_{\max} is within a factor 1.25 of the threshold, the detector's critical LR factor is within one grid step (0.1) of the predicted factor at 92%. The detector calls EoS at 1 of the 527 checkpoints where λmax⁡\lambda_{\max} is more than 10% below the threshold, at 299 of the 433 where it is within 10% of the threshold, and at 68 of the 97 where it is 10% to 25% above.
  2. The agreement is weaker for larger models. In the Figure 2 runs the median λmax⁡\lambda_{\max} at the peak LR is 0.99 (104M), 1.13 (344M) and 1.27 (787M) times the threshold. The single-batch Lanczos estimate of λmax⁡\lambda_{\max} is more than 25% above the threshold at 0 of 39, 12 of 39 and 19 of 37 checkpoints. The detector does not follow these high values: at all 31 of these checkpoints the predicted factor is below 0.8, and the detector's critical LR factor is between 0.9 and 1.2.
  3. Our training setting differs from Cohen et al. (2022). They compare the preconditioned sharpness with 38/η38/\eta. Under minibatch Adam they report it "below the full-batch stability threshold" and "smaller when the batch size is smaller" for ResNet-50, and at the threshold or "a value just short of" it for an encoder-decoder Transformer on WMT translation. Our setting is decoder-only LLM pretraining with AdamW, in 46 runs that vary the model size (43M to 787M), the peak LR, the batch size and the LR schedule.
  4. Our object of study is more fundamental: we test the instability of the local quadratic model itself, of which 38/η38/\eta is a consequence. We define EoS as the instability of the local quadratic model with mini-batch Hessians at the training LR, and the detector measures it directly by running the optimizer's dynamics on that model. In the full-batch case this definition reduces to theirs. The threshold 38/η38/\eta follows from the same model only when there is no noise and no negative curvature. It is a convenient indicator in practice, and our experiments validate it, while the detector tests the instability that the threshold stands for. The same model gives the second failure mode, negative curvature (EoC, see Q3), which a sharpness threshold does not see.
  5. We accept the reviewer's point that the two are close, and the revised figures show the threshold. The eigenvalue panels of Figures 2, 8, 9, 11, 12 and 14–16 now show the 38/η38/\eta line, and a new Section 5.5 compares our measurements with the minibatch results of Cohen et al. (2022).
Reviewer_rXGhOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the curvature along the training trajectory in deep neural networks. Specifically, it proposes algorithmic methods to determine whether the training dynamics are at the edge of stability with respect to their local quadratic approximation in stochastic optimization. Using these methods, they compute the local stability threshold via a grid search over the step size and show that training with adaptive optimizers typically occurs at the edge of stability. This means that the curvature of the landscape along the trajectory is coupled with the optimizer's hyperparameters. They also introduce a new concept, the Edge of Convexity (EoC), in which negative eigenvalues persist during training. However, these negative eigenvalues are small and dominated by the maximal one.

Critical strengths

The algorithmic approach to compute the local stability threshold and to show edge of stability phenomenon is interesting and effective.

Critical weaknesses

I find it hard to understand the contribution behind the edge of convexity. It has already been shown that negative Hessian eigenvalues persist during training. The fact that the positive eigenvalues dominate the negative ones along the trajectory in minimization is not surprising. The objective function is bounded from below but not from above. Moreover, as the iterations approach minima, the negative eigenvalues vanish.

Furthermore, it has been shown that stable behavior can still emerge even if the first-order approximation is unstable; higher-order terms can stabilize the dynamics. See, e.g., [R1, R2, R3].

References:
[R1] - Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability
[R2] - Beyond the Edge of Stability via Two-step Gradient Updates
[R3] - On the Stability of Nonlinear Dynamics in GD and SGD: Beyond Quadratic Potentials

Questions for authors
  1. What evidence shows that the practical detector approximates the mean-square stability threshold η⋆\eta_\star?

  2. What substantive conclusion does EoC add beyond prior observations of small negative Hessian eigenvalues and the limitations of quadratic approximations in negative-curvature directions? These phenomena were already investigated, for example, by Alain et al. (2019). Please distinguish the contributions of the EoC terminology, the proposed stochastic estimator, the extension to LLM pretraining, and any additional insight into training dynamics. In particular, please identify which findings establish an advance beyond reproducing a known spectral pattern at a larger scale.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.