acceptodds
Under review as a conference paper at ICLR 2027

The Multiple Timescales of Gradient Descent on the Edge of Stability: A Perturbative Derivation of the Central Flow

Abstract

The central flow of Cohen et al. (2025) is an empirically accurate model of gradient descent on the edge of stability in deep learning. However, its derivation is heuristic. We propose a perturbative regime in which the central flow is the limit of gradient descent: we assume that the loss decomposes as ; in the limit , the dynamics of gradient descent with learning rate converge to the gradient flow of constrained to the minimizers of of sharpness at most . Our approach is formal rather than rigorous; it treats gradient descent as a singularly perturbed dynamical system in . Three timescales emerge: a fast timescale of oscillations along the sharpest direction, an intermediate timescale of the self-stabilization at the edge of stability, and a slow timescale of the dynamics along the minimizers of —the central flow. Using the method of multiple scales, a classical formal method from singular perturbation theory, we derive the expansion of the dynamics in : the central flow emerges as the leading-order term in the expansion, while the self-stabilization mechanism appears in the next-order term. We study this mechanism beyond previous analyses: with a single eigenvalue at the edge of stability, we compute the slow drift of the energy of the fluctuations; with several eigenvalues at the edge of stability, we derive the self-stabilization system and explain why fluctuations persist.

Primary area:
optimization
Keywords:
edge of stability, self-stabilization, central flow, method of multiple scales, perturbation theory

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Rt6UOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies gradient descent at the edge of stability under a sharp-valley decomposition f=g+εh.f=g+\varepsilon h. Through a formal multiple-timescale expansion, it derives the central flow as the leading-order dynamics and recovers self-stabilization at the next order. It further computes the slow drift of oscillation energy in a two-dimensional example and extends the self-stabilization system to multiple critical directions.

Critical strengths
  • The multiple-timescale framework provides a systematic organization of fast oscillations, intermediate self-stabilization dynamics, and slow motion along the valley.
  • The paper makes the averaging underlying the central-flow approximation more explicit.
Critical weaknesses
  1. The relationship between the assumed setting and neural network training is insufficiently established. The decomposition f=g+εhf=g+\varepsilon h is a useful modeling assumption, but the paper does not identify a concrete small parameter or demonstrate how this decomposition arises in a neural network learning problem. The empirical observations discussed are consistent with a sharp-valley picture, but do not establish the particular asymptotic structure required by the analysis. Furthermore, the leading-order trajectory is assumed to remain in the stable valley. Thus, the framework conditionally describes EoS dynamics without adequately explaining when neural network training satisfies its assumptions.

  1. The illustrative self-stabilization mechanism is largely already present in prior work. Appendix B of Damian et al., Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability, provides a closely related toy loss whose coupled dynamics are two-dimensional, with an additional decoupled direction. Section 4.2 of that work already derives the oscillation–sharpness feedback system and its conserved quantity. The present paper acknowledges this connection, but the two-dimensional example adds limited mechanistic insight. Once the stability boundary is reached, its leading-order constrained trajectory is stationary, and the remaining dynamics primarily describe local fluctuations. The contribution here is therefore the asymptotic organization of an existing mechanism, rather than a new explanation of self-stabilization.

  1. The higher-order drift calculation is highly model-specific, and its broader implications remain unclear. Result 3.3 determines whether oscillation energy grows or decays through the coefficient of a prescribed cubic term. However, the paper does not explain why neural network losses should possess the structure needed for dissipation, nor does it provide an interpretable criterion for the drift’s sign in a general loss landscape. The calculation establishes that higher-order effects can modify the conserved leading-order cycles, but does not yet explain the observed damping in neural network training. Its significance currently appears closer to demonstrating the perturbative method than to identifying a broadly useful mechanism.
Questions for authors
  1. Can the authors identify a concrete learning model in which the small parameter ε\varepsilon, the sharp-valley decomposition, and the stable-leading-trajectory assumption can be justified?

  2. Beyond recovering the known self-stabilization system, what qualitative prediction of the multiple-timescale analysis distinguishes it from Damian et al.’s toy setting in the Appendix B?

2 comments
AuthorsOpenReview ↗

Individual response — Reviewer Rt6U

Thank you for your work on this review. We address your concerns separately below.

W1. The relationship between the assumed setting and neural network training is insufficiently established.

The decomposition f=g+εhf = g + \varepsilon h should not be understood as a realistic model of the loss landscape of neural networks, but only as a local approximation around the gradient descent trajectory. We modified a sentence in the introduction to make this more explicit. We expect this approximation to be faithful only after a certain progressive sharpening phase.

The first paragraph of Appendix A lists extensive empirical evidence for the local valley model. The asymptotic regime ε≪1\varepsilon \ll 1 used in this paper is motivated by the large ratio between the outlier eigenvalues and the eigenvalues of the bulk; we have added a sentence to Appendix A to clarify this.

For instance, consider Figure 3 of Ghorbani et al. (2019). We observe that during training, a few eigenvalues stick out of the bulk. These eigenvalues are an order of magnitude larger than most other eigenvalues (note the log scale on the y-axis: most eigenvalues are very close to zero).

W1. Furthermore, the leading-order trajectory is assumed to remain in the stable valley. Thus, the framework conditionally describes EoS dynamics without adequately explaining when neural network training satisfies its assumptions.

See the common answer to all reviewers.

Q1. Can the authors identify a concrete learning model in which the small parameter ε\varepsilon, the sharp-valley decomposition, and the stable-leading-trajectory assumption can be justified?

A concrete learning model for the sharp-valley decomposition has been proposed by Wen et al., 2025 (cited in Appendix A). The toy task is learning the distribution of the next token given one token. They show that the sharpness in different directions relates to the uncertainty of the next token depending on the given token. If this uncertainty varies on different scales (in machine learning, some predictions are easy, others are hard), this would create the small parameter ε\varepsilon required for our analysis.

That being said, extrapolating that different uncertainty levels in predictions create the asymptotic regime that we need in training neural networks is largely speculative. Overall, it is more accurate to say that our understanding of the neural network landscape and of training dynamics evolve jointly, and that neither is fully within the reach of theory for now.

W2. The illustrative self-stabilization mechanism is largely already present in prior work.

We definitely agree that there is a strong similarity between the setting of the minimal example and previous works, and that Result 3.1 was essentially already known. In fact, we made this connection on purpose so that the new techniques (the asymptotic regime in ε\varepsilon, the method of multiple scales) are illustrated on an example the community is already familiar with. The novel results of this paper are in Sections 3.3, 4 and 5 (see the answer to Q2 for details).

However, we had not noticed that Damian et al. (2023) had such a similar example in their appendix. We thank you for pointing that out. We added a sentence at the beginning of Section 3.1 to acknowledge this.

Q2. Beyond recovering the known self-stabilization system, what qualitative prediction of the multiple-timescale analysis distinguishes it from Damian et al.’s toy setting in the Appendix B?

The asymptotic regime in ε\varepsilon and the method of multiple scales enabled the following new results and qualitative predictions.

  • The conservative self-stabilization system (in t2t_2) is not incompatible with slow variation of the energy (in t3t_3). This is observed on a neural network in Figure 4 (former Figure 3), for example. The method of multiple scales provides a framework to study these variations, although we have done the analysis only in the minimal example, see more details about this below.
  • We have derived the self-stabilization system in a situation where the leading-order constrained trajectory is not stationary, see Result 4.1 and its illustration in Figure 5 (former Figure 4). Our approximation is valid on the t3t_3 timescale (see updated discussion in Appendix A).
  • We extend the self-stabilization system to the case of multiple eigenvalues at the edge of stability, left open by Damian et al. (2023), see Result 5.1 and its illustration in Figure 9 (former Figure 8). The resulting system generically does not have a fixed point (when dim⁡Y>dim⁡C\dim Y > \dim C); thus it predicts that we cannot observe damping of the self-stabilization fluctuations. This is observed in both Figure 9 (former Figure 8, a toy setting) and in Figure 4 (former Figure 3, on a neural network): when a second eigenvalue reaches the edge of stability, there is a significant and persistent increase of the self-stabilization fluctuations.
AuthorsOpenReview ↗

Individual response — Reviewer Rt6U (continued)

W3. The higher-order drift calculation is highly model-specific, and its broader implications remain unclear.

We agree with this criticism. Let us explain our choices.

First, we believe there was a mismatch in previous works: the self-stabilization system derived by Damian et al. (2023) preserves an energy, while energy variations are observed in practice (see Figure 4, former Figure 3, for instance). Here, the contribution of the method of multiple scales is to resolve this apparent contradiction: one can have a conservative system in t2t_2 and energy variations in t3t_3.

Further, we wanted to show that the method of multiple scales enables the computation of these energy variations, at least conceptually. As a "proof of concept", we provided Result 3.4 (former Result 3.3).

We wanted to generalize Result 3.4 (former Result 3.3) to the setting of Section 4. The proof of Result 3.4 is already 2-3 pages of tedious computations; we were unable to derive the generalization ourselves. We then asked AI: as the method is systematic, it directly gave the generalization to the setting of Section 4.

We did not include this generalization in the paper for two reasons. First, we somehow still have a preference for human-written mathematics. But more importantly, we found no explanation in the generalization of why neural networks systematically dissipate energy. From the formula, both signs of the energy variation seem possible, and thus there must be some additional structure in the landscape of neural networks, not included in our ansatz, that makes the variation of the energy systematically negative.

In the end, we decided to omit this technical, AI-generated and not-so-insightful generalization. We still believe that reconciling the conservative self-stabilization system with the observed energy variations, and opening a route to study them, is worth sharing.

Soundness grading. Your grading of the soundness indicates that you have "significant concerns about the paper soundness", but your review does not point to incorrect mathematical reasoning. Could you specify which parts you consider unsound, so that we can address them?

Reviewer_uzd8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether the central flow can be obtained as a limit of discrete gradient descent near the edge of stability. In a sharp valley model, the authors fix the learning rate and weaken the drift along the valley. The expansion separates oscillations, self-stabilization, and slow motion, and recovers the central flow at leading order. In toy experiments, the iterates track the central flow more closely as the perturbation shrinks.

Critical strengths
  1. The paper identifies a clear gap between the empirical success of the central flow and its derivation.
  2. The expansion explains how oscillations can average out in parameter space while still shaping the slow trajectory.
Critical weaknesses

The derivation assumes that the slow trajectory stays in the stable valley. Fig. 3 shows that weak transverse oscillations can delay self-stabilization enough for the iterates to overshoot the boundary.

Questions for authors

Can the authors identify a measurable condition that predicts whether self-stabilization begins in time to prevent such an overshoot in a neural network?

2 comments
AuthorsOpenReview ↗

Individual response - Reviewer uzd8

Thank you for your updated review.

Q1. Can the authors identify a measurable condition that predicts whether self-stabilization begins in time to prevent such an overshoot in a neural network?

In our simplified setting, we can directly monitor the deviation xx from the valley; the smaller it is when the sharpness reaches 2/η2/\eta, the larger the overshoot, see Figure 3.

In neural networks, a moderate overshoot is visible at the onset, but we never observe a long delay as in Figure 3 with h=−yh = -y; see Figure 4 and the numerous experiments of Cohen et al. (2025). As a consequence, it is not clear which failure cases a condition would aim to detect. We believe that the more important question is what prevents a long delay. As this question was raised by the other reviewers, we refer to the common response for some thoughts on this, which remain speculative.

Thank you for the detailed clarification. Your response addresses my concern.

Reviewer_7VJtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies gradient descent on a sharp-valley loss f=g+εhf=g+\varepsilon h, with fixed learning rate and a small perturbation inducing motion along the minimizers of gg. It separates iteration-scale oscillations, intermediate self-stabilization, and slow valley motion using a three-timescale expansion. Under the assumption that the leading trajectory remains in the stable valley, its formal solvability conditions recover the central flow of Cohen et al. The paper also computes a slow energy-drift law in a scalar minimal example and derives critical-subspace modulation equations when several Hessian eigenvalues reach the stability threshold. Numerical examples illustrate the separation of scales and the predictions. The results are explicitly presented as formal asymptotic derivations, without a convergence theorem or remainder bounds.

Critical strengths
  1. The separation of scales provides genuine explanatory value. Distinguishing the fast sign-alternating oscillation, its intermediate envelope dynamics, and the slow central trajectory clarifies what is averaged out by different effective models. Within the assumed stable branch, the constrained velocity emerges from solvability conditions. This determines more than feasibility alone, particularly the tangential motion in higher-dimensional valleys. It is a useful organization of the known central-flow and self-stabilization descriptions, although it does not establish why GD selects this branch.

  2. The slow energy-drift calculation is a substantive prediction beyond the leading-order picture. The scalar self-stabilization system and its conserved energy were already known from Damian et al. (2023). Result 3.3 goes further by computing how that energy evolves on the slow timescale, including a change between damping and growth as the cubic coefficient varies. The Appendix E algebra appears consistent. An independent numerical check integrating the autonomous averaged energy equation also supported the prediction in the tested, well-prepared toy regime. This is numerical support for the scalar result, not a validation of the general limiting claim.

  3. The treatment is transparent about its mathematical status and some failure modes. The authors explicitly distinguish formal derivations from theorems, expose a delayed-instability failure in Appendix C, and identify the breakdown of their expansion at large energy in Appendix D. Those qualifications are scientifically valuable. The matrix-valued formulation also gives a concrete route for investigating interactions between critical directions, even though its interpretation needs qualification below.

Critical weaknesses
  1. The conditions under which actual GD follows the proposed expansion remain unresolved. Results 3.1, 4.1 and 5.1 assume that the leading trajectory remains in the stable valley, while the derivation requires bounded, nonsecular corrections. The analysis therefore describes a selected asymptotic regime without establishing when GD enters or remains in it over the slow timescale. Appendix C shows that the sharp-valley decomposition alone is insufficient. This is an important applicability limitation, and the abstract should state the conditional scope more clearly. It is not complete circularity: feasibility alone does not determine the constrained velocity or higher-order dynamics. The comparison with Damian et al. also needs care: that work proves local tracking under substantial conditions, including a uniform lower bound on predicted oscillation amplitude. In the relevant sharp-valley scaling, its proved window does not already cover the longer slow-time averaging studied here. The useful missing ingredient is an operational characterization of the initialization/excitation regime, rather than merely another statement that the calculation is formal.
  2. Multiple critical directions alone do not imply the claimed generic absence of modulation equilibria. Section 5’s rank-one covariance argument requires conditions on the dimensions, the Hessian-differential map and the forcing. For example, take a two-dimensional valley and critical subspace, with
Ψ[y]=diag⁡(y[1],y[2]),∇yh=(−1,−1),a1=(2,2),U1=0.\Psi[y]=\operatorname{diag}(y[1],y[2]),\qquad \nabla_y h=(-1,-1),\qquad a_1=(\sqrt{2},\sqrt{2}),\qquad U_1=0.

Both right-hand sides of equation (5.1) vanish. The reduced amplitude constraints have a nonsingular Jacobian at this point, so equilibrium existence also persists under small perturbations of the map and forcing within these dimensions. Thus the issue is not confined to one exactly diagonal example. The fully coupled example in Appendix H can still exhibit the intended obstruction, but the general statement needs a precise rank/geometry hypothesis. Moreover, absence of an equilibrium alone does not establish bounded persistent fluctuations or validity of the asymptotic expansion. This concern targets the interpretation, rather than the derivation of equation (5.1). 3. The numerical validation could test the new predictive closure more directly. Figure 7 constructs the energy prediction by integrating a fluctuation statistic measured from the GD trajectory itself. This is a useful balance check, but an autonomous prediction from the initial energy and an independently computed orbit average would be stronger. Quantitative approximation errors over a stated slow-time interval, across perturbation sizes, would also help distinguish errors during entry from errors within an established edge regime. The neural-network illustration in Figure 3 does not directly validate the new scalar energy law or the matrix modulation equations. 4. The distinct contribution relative to existing covariance models could be sharper. Section 6 appropriately discusses competing approximations. However, Chodron de Courcel’s June 2026 model, equation (3.4), already evolves a center and a matrix covariance; its critical small-amplitude specialization is closely related to the covariance form of equation (5.1). The strongest distinction appears to be the systematic three-scale derivation, its averaging interpretation, and the additional slow-order calculation. Making that relationship explicit would strengthen the contribution statement. This is not a claim that the complete derivation or the scalar energy law is duplicated.

Questions for authors

the above plus:

  • Can the intended validity regime be stated in terms of initialization and excitation, distinguishing entry into the edge regime from evolution within an already established critical phase?
  • Under what dimensional and rank assumptions is the claimed generic absence of a rank-one modulation equilibrium valid? What additional assumptions justify persistent bounded fluctuations?
  • Can the slow energy law be tested as an autonomous forecast, with quantitative errors over a specified slow-time interval?
1 comment
AuthorsOpenReview ↗

Individual response - reviewer 7VJt

Thank you again for your precise review. We address your concerns separately below.

Q1. Can the intended validity regime be stated in terms of initialization and excitation, distinguishing entry into the edge regime from evolution within an already established critical phase?

See the common answer to all reviewers.

W1. The comparison with Damian et al. also needs care.

Definitely, thank you for pointing this out. We have included a detailed comparison in Appendix A, commenting on the time window and the assumption on the oscillation amplitude.

Q2. Under what dimensional and rank assumptions is the claimed generic absence of a rank-one modulation equilibrium valid?

We agree that our formulation was too approximate. We revised this paragraph. The new formulation specifies which objects should be thought of as generic (all Ψ:Y→SymC\Psi:Y \to {\text{Sym} C} and ∇yh(0,y0)\nabla_y h(0,y_0) outside a set of measure zero) and needs to assume that dim⁡Y>dim⁡C\dim Y > \dim C. Your counter-example takes dim⁡Y=2=dim⁡C\dim Y = 2 = \dim C, thus it is appropriately excluded from the formulation. The example of Appendix G takes dim⁡Y>dim⁡X≥dim⁡C\dim Y > \dim X \geq \dim C, thus it fits the formulation. In Figure 4 (former Figure 3), one has two eigenvalues at the edge of stability, while the network contains hundreds of thousands of parameters, thus the condition is satisfied. More generally, when training neural networks, we observe that the dimension of the critical space is much smaller than the remaining flat directions.

Q2. What additional assumptions justify persistent bounded fluctuations?

We are unable to show that the solutions of (5.1) remain bounded, although we expect it to be the case. The conserved quantity of Damian et al. (2023) in the case of a single eigenvalue at the edge of stability does not seem to generalize straightforwardly. Note that the method of multiple scales does not require to show that the solutions are bounded; it is an assumption of the method.

W3. The numerical validation could test the new predictive closure more directly. Figure 7 constructs the energy prediction by integrating a fluctuation statistic measured from the GD trajectory itself. This is a useful balance check, but an autonomous prediction from the initial energy and an independently computed orbit average would be stronger.

Agreed. We have updated Figure 7 following your advice. The results are qualitatively similar.

Q3. Can the slow energy law be tested as an autonomous forecast, with quantitative errors over a specified slow-time interval?

Thank you for the suggestion. We have provided quantitative errors in the new Figure 8. As expected from the expansion, we observe an ε1/2\varepsilon^{1/2} scaling of the error.

Note that the errors are measured uniformly over an interval that does not include the energy saturation (acknowledged in Figure 7), nor the entry into the edge of stability regime, where we expect our ansatz to fail (see global response).

W4. However, Chodron de Courcel’s June 2026 model, equation (3.4), already evolves a center and a matrix covariance; its critical small-amplitude specialization is closely related to the covariance form of equation (5.1). The strongest distinction appears to be the systematic three-scale derivation, its averaging interpretation, and the additional slow-order calculation. Making that relationship explicit would strengthen the contribution statement.

Thank you for pointing that out. We added a paragraph on the comparison in Section 6. A precise comparison was difficult due to the different settings, thus we kept the discussion at a qualitative level. Let us know if you would like any modification of the comparison.

Note that we refer to the latest arXiv version of the paper by Chodron de Courcel.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global response to all reviewers

We thank the reviewers for their work in evaluating our paper. Their constructive feedback has significantly improved the paper, including more detailed experiments (Appendix C, former Appendix D), a better discussion of the related work (Section 6, Appendix A), and an added assumption for the self-stabilization fluctuations to be persistent (Section 5).

We have uploaded a revised version of the paper, in which changes are shown in blue.

The reviewers shared a concern about the assumption that the leading trajectory remains in the stable valley. We answer this concern here and respond to their other comments in separate replies below.

Reviewer Rt6U: W1. Furthermore, the leading-order trajectory is assumed to remain in the stable valley. Thus, the framework conditionally describes EoS dynamics without adequately explaining when neural network training satisfies its assumptions.

Reviewer 7VJt: Q1. Can the intended validity regime be stated in terms of initialization and excitation, distinguishing entry into the edge regime from evolution within an already established critical phase?

We agree this is a limitation of our work. We found it difficult to replace this assumption by a more structural assumption for two reasons.

First, we found two ingredients that prevent metastability beyond the edge of stability (and thus the need for the assumption): adding a transversal term in hh / assuming that the valley VV is curved, or adding noise to the gradient descent iterates. Both work well in (variations of) the minimal example. However, it is unclear to us which of the two plays a role in training neural networks. Both are plausible: the typical valleys are curved in the landscape of neural networks, and the numerical errors create sufficient noise. We refrained from developing the theory further without sufficient progress on the modeling of the problem.

Second, understanding the entry into the edge of stability regime requires a significant extension of our formalism. Indeed, at the entrance, small perturbations from the valley grow to become self-stabilization fluctuations. Thus, we expect our ansatz to fail, and a specific ansatz for this phase to be required, combined with the method of matched asymptotic expansions (Holmes, 2012, Chapter 2).

Overall, in the face of both modeling and technical difficulties, we have chosen to keep the simplifying assumption, and gather our thoughts on this problem in an appendix. We agree with the reviewers on the importance of this discussion: we thus moved the contents of this appendix to the main text (Remark 3.3) and slightly extended the discussion below Result 3.1.

Reviewer 7VJt: W1. This is an important applicability limitation, and the abstract should state the conditional scope more clearly.

Agreed. We have revised the abstract accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.