- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies the edge-of-stability behavior of full-batch Adam through active curvature, a gradient-weighted measure of preconditioned curvature. It analyzes how second-moment adaptation provides negative feedback on quadratic objectives, with an extension to general objectives without momentum. Experiments across several neural architectures connect this perspective to gradient reversal and show that parameter averaging suppresses oscillations.
Critical strengths
-
Active curvature offers a useful explanation for why preconditioned sharpness can exceed its nominal threshold while the observed trajectory remains controlled. The examples in Section 3.2 make this distinction concrete.
-
The momentum analysis goes beyond the simpler RMSProp setting. In particular, the aligned and misaligned regimes in Theorem 3.3 reveal behaviors that a frozen-preconditioner analysis alone would miss.
Critical weaknesses
-The theoretical results do not fully establish the sustained near-boundary behavior emphasized in the paper. Theorems 3.2–3.5 primarily provide threshold-exit or liminf/limsup statements, which still permit large recurrent excursions. Likewise, the existence of the two-cycle in Proposition 3.1 does not establish its attraction. The paper should distinguish these guarantees more clearly from the empirical observation that active curvature remains close to 2.
- The empirical scope limits the broader significance of the proposed picture. The full-batch experiments are informative, but Appendix C.6, Table 9 shows that gradient reversal largely disappears with smaller batches and standard momentum: for GPT-2, the reported full-gradient reversal frequency falls from 65% to 0% at batch fraction 1/8. Although full-batch training is the stated focus, the paper leaves unclear which parts of the explanation remain useful for stochastic Adam.
Questions for authors
-
Do any existing results bound the amplitude of repeated excursions from the threshold, or the fraction of time spent near it? Please clarify what supports sustained concentration near 2 beyond threshold crossings and the existence of a two-cycle.
-
Given the batch-size results in Table 9, which conclusions about active-curvature regulation and gradient reversal do you expect to retain under stochastic Adam with momentum? Please distinguish conclusions supported by the current analysis from empirical conjectures.
New experiment with momentum (β1=0.9). Eq. (14) is exact only at β1=0. To test the reviewer's question in the setting used in practice, we ran full-batch Adam with β1=0.9 on two networks and compared how well Wact>2 and λmax(Pt)>2 signal that the loss is rising.
Setting.
Measurement.
Results. Each cell gives Wact / λmax(Pt).
When Wact>2, the loss is usually rising. On the MLP, at least 99 of the steps with Wact>2 raise the loss, against a base rate of 54. On the transformer the figure is 68–85, against base rates of 31–36. For λmax(Pt)>2 the corresponding fractions are 38–62, only 0.02–0.18 above the base rates.
These results concern the precision of the signal, not its coverage or timing. Wact exceeds 2 on only 3–13 of the steps, and in the MLP with β2=0.99 it stays below 2 throughout the rise of 36 of the spikes. λmax(Pt) exceeds 2 during almost every spike, but also on 43–73 of the steps outside spikes. Under momentum, Wact exceeds 2 at the first step of only 0–33 of the spikes, so it does not warn of a spike earlier than λmax(Pt), and neither reading forecasts a spike tens of steps ahead.
Computational cost. Wact is a closed-form Rayleigh quotient (Eq. (10)): one Hessian-vector product with Dt−1gt and two inner products, at a fixed cost and with no approximation error. λmax(Pt) requires an iterative eigensolver whose iteration count depends on the spectral gap and the starting vector. Timed on one A100 GPU (medians over six timed steps per network, each reading including its full-batch gradient), Wact takes 3.3 ms on the MLP and 27 ms on the transformer, i.e. one to two gradient evaluations beyond the gradient itself. λmax(Pt) with the Lanczos iterations used in our runs takes 44 ms and 191 ms, 13 and 7 times as much. A cold start needs 5–41 iterations to reach 0.1 accuracy; a warm start needs 2–13, but only if λmax(Pt) is tracked at every step. To our knowledge, λmax(Pt) is still rarely monitored in practical training, partly because of this cost, and its practical use as a signal of instability remains to be explored.