acceptodds
Under review as a conference paper at ICLR 2027

Adam at the Edge of Stability: Adaptive Feedback, Provable Oscillation, and Gradient Reversal

10 views

Abstract

The edge-of-stability (EoS) phenomenon of full-batch Adam has been widely observed, yet its underlying dynamical mechanism remains poorly understood. In this paper, we identify Adam's second-moment adaptation as a negative-feedback mechanism that drives the dynamics toward the stability boundary. We characterize this mechanism through the *active curvature*, namely, the preconditioned curvature along the preconditioned gradient direction, and establish rigorous characterizations in progressively richer settings: rank-one quadratics with momentum, diagonal quadratics, on which the active curvature separates from the sharpness, and general objectives. Importantly, the mechanism predicts *gradient reversal* of full-batch Adam near the edge: consecutive gradients repeatedly point in nearly opposite directions, as we observe across fully connected networks, ResNets, ViTs, LSTMs, GPT-2 medium, and Adam-family optimizers. Consistent with this picture, averaging iterates suppresses these fast oscillations and produces smoother and lower loss curves. Together, these results provide an important first step towards fully understanding the dynamical behavior of Adam's EoS through active curvature and gradient reversal.

Primary area:
optimization
Keywords:
edge of stability, adam, optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Xa2tOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the edge-of-stability behavior of full-batch Adam through active curvature, a gradient-weighted measure of preconditioned curvature. It analyzes how second-moment adaptation provides negative feedback on quadratic objectives, with an extension to general objectives without momentum. Experiments across several neural architectures connect this perspective to gradient reversal and show that parameter averaging suppresses oscillations.

Critical strengths
  • Active curvature offers a useful explanation for why preconditioned sharpness can exceed its nominal threshold while the observed trajectory remains controlled. The examples in Section 3.2 make this distinction concrete.

  • The momentum analysis goes beyond the simpler RMSProp setting. In particular, the aligned and misaligned regimes in Theorem 3.3 reveal behaviors that a frozen-preconditioner analysis alone would miss.

Critical weaknesses

-The theoretical results do not fully establish the sustained near-boundary behavior emphasized in the paper. Theorems 3.2–3.5 primarily provide threshold-exit or liminf/limsup statements, which still permit large recurrent excursions. Likewise, the existence of the two-cycle in Proposition 3.1 does not establish its attraction. The paper should distinguish these guarantees more clearly from the empirical observation that active curvature remains close to 2.

  • The empirical scope limits the broader significance of the proposed picture. The full-batch experiments are informative, but Appendix C.6, Table 9 shows that gradient reversal largely disappears with smaller batches and standard momentum: for GPT-2, the reported full-gradient reversal frequency falls from 65% to 0% at batch fraction 1/8. Although full-batch training is the stated focus, the paper leaves unclear which parts of the explanation remain useful for stochastic Adam.
Questions for authors
  • Do any existing results bound the amplitude of repeated excursions from the threshold, or the fraction of time spent near it? Please clarify what supports sustained concentration near 2 beyond threshold crossings and the existence of a two-cycle.

  • Given the batch-size results in Table 9, which conclusions about active-curvature regulation and gradient reversal do you expect to retain under stochastic Adam with momentum? Please distinguish conclusions supported by the current analysis from empirical conjectures.

Reviewer_vxgSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the edge-of-stability in Adam, i.e., optimization dynamics operate near the stability boundary. The authors introduce active curvature to explain how Adam’s adaptive second-moment estimation regulates optimization dynamics, and analyze the connection between active curvature and gradient reversal.

Critical strengths
  1. The paper introduces active curvature, characterizes Adam’s optimization dynamics, explaining why the largest eigenvalue of the preconditioned Hessian alone may not accurately reflect training stability.

  2. The theoretical analysis on quadratic objectives is complemented by experiments across diverse neural network architectures, providing empirical evidence for the prevalence of gradient reversal.

Critical weaknesses

The main theoretical analysis is based on full-batch gradients, whereas modern deep networks are typically trained with mini-batch stochastic gradients. It remains unclear to what extent the theoretical results extend to practical training settings

Questions for authors

The active curvature provides a more informative characterization of Adam's training dynamics than the largest eigenvalue of the preconditioned Hessian. Can the authors provide empirical evidence that active curvature is more effective at predicting training instability or loss spikes in practice, rather than merely showing that it approaches the theoretical stability threshold of 2?

2 comments

Q3. Is active curvature more effective than the sharpness at predicting training instability or loss spikes, beyond approaching the threshold 2?

New experiment with momentum (β1=0.9\beta_1=0.9). Eq. (14) is exact only at β1=0\beta_1=0. To test the reviewer's question in the setting used in practice, we ran full-batch Adam with β1=0.9\beta_1=0.9 on two networks and compared how well Wact>2W_{\mathrm{act}}\gt 2 and λmax⁡(Pt)>2\lambda_{\max}(P_t)\gt 2 signal that the loss is rising.

Setting.

  • MLP. A 30723072-256256-256256-1010 tanh network with the squared loss on 5,0005{,}000 CIFAR-10 images, η=10−3\eta=10^{-3}, β2∈{0.99,0.999}\beta_2\in\lbrace 0.99,0.999\rbrace, 5,0005{,}000 steps.
  • Transformer. A two-layer causal transformer (width 128128, 44 heads, context 6464) trained with cross-entropy on 256256 FineWeb sequences.

Measurement.

  • Readings. At every iterate xtx_t, before the update, we compute WactW_{\mathrm{act}} (Eq. (10)) with one exact Hessian-vector product on the full training set, and λmax⁡(Pt)\lambda_{\max}(P_t) with a Lanczos iteration warm-started from the previous step's top eigenvector (2020 iterations for the MLP, 1010 for the transformer; at the snapshots we checked, these readings are within 1.31.3\\% of a 150150-iteration reference, and any remaining error underestimates λmax⁡(Pt)\lambda_{\max}(P_t), which only lowers its false-alarm rate).
  • What is predicted. Each reading at xtx_t is compared with the change of the training loss over the same step, L(xt+1)−L(xt)L(x_{t+1})-L(x_t). We score the last 8080\\% of each run (steps 1,0001{,}000–5,0005{,}000 for the MLP, 600600–3,0003{,}000 for the transformer), by which point each run has reached the edge.
  • Events and scores. A spike is a rise of the training loss by at least a factor of 22 (0.30.3 decades) within 100100 steps. A false alarm is a step outside the rise of every spike (from the preceding minimum to the peak) on which the reading exceeds 22. The false-alarm rate is the fraction of the steps outside every rise that are false alarms.

Results. Each cell gives WactW_{\mathrm{act}} / λmax⁡(Pt)\lambda_{\max}(P_t).

Setting (spikes)Steps that raise the lossSteps with reading >2\gt 2Loss rises, given reading >2\gt 2False-alarm rate
MLP, β2=0.999\beta_2=0.999 (9)0.540.13 / 0.661.00 / 0.620.11 / 0.64
MLP, β2=0.99\beta_2=0.99 (61)0.540.03 / 0.751.00 / 0.580.00 / 0.64
Transformer, β2=0.95\beta_2=0.95 (9)0.310.03 / 0.530.85 / 0.490.03 / 0.43
Transformer, β2=0.999\beta_2=0.999 (3)0.360.12 / 0.730.68 / 0.380.12 / 0.73

When Wact>2W_{\mathrm{act}}\gt 2, the loss is usually rising. On the MLP, at least 9999\\% of the steps with Wact>2W_{\mathrm{act}}\gt 2 raise the loss, against a base rate of 5454\\%. On the transformer the figure is 6868–8585\\%, against base rates of 3131–3636\\%. For λmax⁡(Pt)>2\lambda_{\max}(P_t)\gt 2 the corresponding fractions are 3838–6262\\%, only 0.020.02–0.180.18 above the base rates.

These results concern the precision of the signal, not its coverage or timing. WactW_{\mathrm{act}} exceeds 22 on only 33–1313\\% of the steps, and in the MLP with β2=0.99\beta_2=0.99 it stays below 22 throughout the rise of 3636\\% of the spikes. λmax⁡(Pt)\lambda_{\max}(P_t) exceeds 22 during almost every spike, but also on 4343–7373\\% of the steps outside spikes. Under momentum, WactW_{\mathrm{act}} exceeds 22 at the first step of only 00–3333\\% of the spikes, so it does not warn of a spike earlier than λmax⁡(Pt)\lambda_{\max}(P_t), and neither reading forecasts a spike tens of steps ahead.

Computational cost. WactW_{\mathrm{act}} is a closed-form Rayleigh quotient (Eq. (10)): one Hessian-vector product with Dt−1gtD_t^{-1}g_t and two inner products, at a fixed cost and with no approximation error. λmax⁡(Pt)\lambda_{\max}(P_t) requires an iterative eigensolver whose iteration count depends on the spectral gap and the starting vector. Timed on one A100 GPU (medians over six timed steps per network, each reading including its full-batch gradient), WactW_{\mathrm{act}} takes 3.33.3 ms on the MLP and 2727 ms on the transformer, i.e. one to two gradient evaluations beyond the gradient itself. λmax⁡(Pt)\lambda_{\max}(P_t) with the Lanczos iterations used in our runs takes 4444 ms and 191191 ms, 1313 and 77 times as much. A cold start needs 55–4141 iterations to reach 0.10.1\\% accuracy; a warm start needs 22–1313, but only if λmax⁡(Pt)\lambda_{\max}(P_t) is tracked at every step. To our knowledge, λmax⁡(Pt)\lambda_{\max}(P_t) is still rarely monitored in practical training, partly because of this cost, and its practical use as a signal of instability remains to be explored.

I agree with D24P that prior work already recognized second-moment feedback. The main contribution here is showing how this feedback operates in discrete Adam with momentum on quadratic objectives. Its importance relative to changes in the Hessian during neural-network training remains open.

The new experiments address my original question. Active curvature is informative about whether the next update raises the loss, though it often misses the onset of a spike. Table 9 also shows that gradient reversal can disappear with smaller batches. Within the full-batch setting, I find the theory sufficient to maintain my rating 3.

Reviewer_D24POpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The manuscript studies EoS for Adam through second-moment feedback. Concretely, the authors analyze discrete dynamics, including momentum, and present experiments showing active curvature tracking the stability threshold and gradient reversal across successive steps.

Critical strengths
  • Thorough mathematical treatment of the discrete dynamics of Adam, giving concrete results on oscillatory behavior and second-moment feedback.
  • The mathematical treatment of momentum in Adam is novel and substantial, going beyond adaptive analyses that omit momentum.
  • Thorough experiments showcasing EoS for Adam across a range of architectures and training settings - significantly increasin the evidence behind AEoS.
  • The systematic measurements of gradient reversal provide useful empirical characterization of Adam’s oscillatory behavior, even though the phenomenon itself has precedents.
Critical weaknesses

Overall, my biggest concern is with novelty and the framing relative to existing work:

  • The biggest weakness is the lack of a substantive comparison with the self-stabilization mechanism of Damian et al. (Self-Stabilization) [1] and its adaptive extension in Cohen et al. (Central Flow) [3]. To start with, in my opinion, a work proposing an explanation of EoS for Adam needs to discuss these accounts thoroughly, alongside the Adam observations of Cohen et al. [2]. Concretely, the geomertic self-stabilization of those works is a fairly estbalished mechanism, including for adaptive methods without momentum. Now, this manuscripts instead relies solely on the second-moment feedback mechanism to explain EoS. It is unclear when this mechanism dominates geometric self-stabilization, or how the two interact. Without that comparison, the manuscript has not established that this second-moment feedback mechanism is the one that controls EoS for Adam.

  • Moreover, this second-moment feedback mechanism was already known in prior optimization work by Ma et al. ("A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithms"), Lyu et al. ("Understanding the Generalization Benefit of Normalization...") and in the adaptive-optimization accounts of Cohen et al. [2, 3], with Central Flows explicitly acknowledging that this mechanism could coexist with geometric self-stabilization. Central Flows even studies this empirically by ablating the curvature-regularizing terms and showing that the resulting flow enters sharper regions (although this is a heuristic intervention on the flow model, not a direct intervention on discrete Adam). Therefore the mechanism itself is not novel, even from the point of view of both earlier work and specifically the picture of EoS. Now the novel (and good!) part is a more concrete mathematical treatment of the discreet dynamics. However, these results concern relatively restricted settings, and their explanatory reach for neural-network training remains unclear, especially given that it is not established that second-moment feedback is the controlling mechanism.

  • The fact about the "active curvature" stabilizing better at the threshold is also not novel and was also known from Cohen et al. (EoS) [3]. In particular the reason why it's not the quantity that they look at, is that this quantity crossing the threshold is not what causes the destabilization and therefore the geometric self-stabilization, but it is rather the eigenvalue crossing that threshold. Therefore, the fact that active curvature is more closely following the threshold is interesting, but is not important for establishing an regime of instability.

  • Moreover, the related-work discussion also misses several directly relevant comparisons. Kalra et al. (2026), A Scalable Measure of Loss Landscape..., define a very similar active-curvature quantity up to normalization. Roulet et al. (2024), Stepping on the Edge: Curvature..., already explicitly write this preconditioned directional-curvature quotient; it also measures successive updates becoming opposed in full-batch GD, providing a precedent for the gradient-reversal discussion. Andreyev et al. (2026), Momentum Further Constrains Sharpness..., also includes Adam experiments and discussion showing that difference of effect of momentum on the edge behavior in small-batch compared to large-batch settings.

  • Gradient reversal is not an entirely new phenomenon: it comes out naturally from the picture in Cohen et al. [2, 3], and exact two-cycles in earlier Adam analyses already exhibit it [5]. The manuscript’s systematic measurements for Adam are a novel contribution, but should be distinguished from discovering gradient reversal as a phenomenon.

  • Concerning the experiments, they unfortunately do not establish whether the observed EoS behavior is indeed controlled by the mechanism emphasized in the manuscript. Measurements along the lines of Damian et al. [1], tracking the relevant quantities simultaneously, seem necessary here to confirm the validity of the theoretical picture.

With all of the above, I have to say that this paper has a lot of merit, yet requires substantial repositioning.

[1] Damian et al., Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability.

[2] Cohen et al., Adaptive Gradient Methods at the Edge of Stability.

[3] Cohen et al., Understanding Optimization in Deep Learning with Central Flows,

[5] Bock and Weiß, Non-Convergence and Limit Cycles in the Adam Optimizer.

Questions for authors
  • When does second-moment feedback dominate geometric self-stabilization, and how do the two mechanisms interact?
  • Can the authors track both mechanisms empirically to establish which controls the observed EoS behavior?
  • How do the active-curvature and gradient-reversal observations relate to prior work, and what is new beyond their systematic measurement for Adam?
  • Is there a reason to believe that active curvature is the quantity of Adam EoS as opposed to EoS?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.