acceptodds
Under review as a conference paper at ICLR 2027

Mean–Fluctuation Dynamics at the Edge of Stability

Abstract

We study the dynamics of gradient descent in the Edge of Stability regime, where the learning rate is large enough to induce persistent oscillations in the trajectory, which has been linked to better generalization performances. We introduce the mean-fluctuation dynamics, a tractable continuous-time model coupling the window-averaged trajectory to its fluctuation covariance. Among our contributions, we rigorously derive this model from gradient descent in a sharp valley framework, characterize its stationary states and their linear stability, and establish precise connections with other effective dynamics. Numerical experiments illustrate these predictions and their finite-time limitations. We also study our model in the overparametrized, fixed learning rate, regime of wide two-layer network, where we rigorously derive a kinetic equation describing weights and their fluctuations as a Wasserstein-2 gradient flow, for which we prove well-posedness, mean-field limit, and conditional convergence results.

Primary area:
optimization
Keywords:
Gradient Descent, Effective Model, Gradient Flow, Edge of Stability, Implicit Bias, Kinetic Theory

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aLqvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper develops a coarse-grained model of gradient descent (GD) dynamics in the edge of stability regime, where the largest eigenvalue(s) of the Hessian reach and hover around the threshold 2/η\eta, where η\eta is the learning rate. This regime is an intriguing feature of GD dynamics, but is challenging to theoretically describe and analyze because of the sharp oscillations that characterize its trajectories. The paper overcomes this problem by tracking coarse-grained variables: a "slow mean" θt\theta_t defined by trajectories averaged over a time window of length mm and a corresponding "slow covariance" Σt\Sigma_t describing fluctuations around that mean. The authors take the continuum time limit and derive a system of coupled differential equations for θ(t)\theta(t) and Σ(t)\Sigma(t), which they dub the mean-fluctuation (MF) dynamics model. They validate the model on three architectures trained on CIFAR-10, and show that it tracks GD observables more closely than earlier approximate descriptions of GD in the literature. Finally they apply their model to a two-layer network architecture, and show it can be understood via a hydrodynamic equation describing the time evolution of the density of weights, and that this evolution minimizes an effective free energy.

Critical strengths

The paper provides a rigorous derivation of coarse-grained GD dynamics, focusing not just on the mean flow path but also the fluctuations around that path. The resulting equations are simple in form, making them amenable for analysis.

The formalism is tested in a variety of real-world contexts (rather than toy models), validating it as an accurate description of GD observables (loss, unstable eigenvalues, etc.). The authors are careful to compare with earlier approaches in the literature in a similar vein, and find that their method generally provides better agreement with GD (though in some cases the improvement is fairly small).

Critical weaknesses

As the authors point out, this is not the first study to try to approximate GD dynamics, and the Rod flow approach in particular is a close precursor (roughly corresponding to the current method with a window time of m=2m=2). The numerical results (Figure 3-5) show some improvements relative to these earlier models, but in the end many of these models were already reasonably good (for example exhibiting fractions of a percent for center error). The overall impression is that authors provide an elegant model, but there is no real conceptual payoff from introducing this model: do we learn anything significant about edge of stability GD dynamics from the model? The rigorous derivations may be nice for a mathematical audience, but the authors do not really explain the importance of any of the theorems, or how the added rigor provides any novel insights into the underlying phenomena.

Questions for authors

Following up on the comments above, there are several questions that the authors could answer to help motivate the significance of their results:

  1. What is the practical advantage of having a slightly better approximate description of GD dynamics? Are there contexts in which the numerical improvements over earlier methods are meaningful?

  2. Are there broad lessons or insights that we can draw from the model about edge of stability dynamics, and are there any practical implications for training neural networks via GD?

  3. The final main text results section, on the kinetic theory for two-layer networks, is not strongly motivated. What do we learn from having such a kinetic description, or from showing the flow in this specific system minimizes a free energy? Are there ramifications for other machine learning architectures?

4 comments
AuthorsOpenReview ↗

Answer to Questions 1 and 2

First of all, we would like to thank the reviewer for the time taken reading our manuscript! Should the following answers to his/her questions be satisfying, we will be more than happy to update the article as proposed, if the reviewer thinks accordingly.

Question 1

Our model is currently the only one we are aware of that is able to follow GD both at the onset of EoS and when there are multiple unstable eigenvalues. For instance, the Rod flow is unable to settle at the EoS when initialized before oscillations have grown (it then follows the gradient flow, see fig. 5), while the Edge flow is unable to capture the EoS dynamics when several eigenvalues are unstable (as reported in the figures). These are important failure modes that our model addresses. Therefore, we don't see our model as "closer to GD" but as genuinely capturing scenarios that are not captured by these other flows. Furthermore, our model has the advantage of addressing these failure modes while being minimally complex compared to gradient flow. In particular, we have demonstrated that our model is amenable to analysis and we are therefore allowed to draw conclusions about GD from our model, at least in the scaling limit we have exposed (the derivation of this scaling limit, and therefore the domain of validity of the model, being itself a contribution left behind in Edge and Rod flows). Finally, we would like to make a small remark which is that our model was initially developed at around the same time as Rod and Edge flows. Therefore, we see our model as an independently developed framework, and wouldn't call Rod and Edge as "precursors" or "earlier methods". Of course, this can't be proved so we don't see this claim as a major argument for our model.

Question 2

First, the stationarity condition is ∇E(θ∗)+12∇H(θ∗):Σ∗=0,\nabla E(\theta_*)+\frac12\nabla H(\theta_*):\Sigma_*=0, so it need not be a critical point of the original loss. Therefore, convergence of averaged GD does not mean convergence to the minimizers of the loss. We also analyze the stability of such configurations (Theorem 2.3). For practical GD, computing the empirical H:ΣH:\Sigma contribution can be a first test to understand whether the average GD is roughly around the minimum value of EE, or if the discrepancy between the loss value and its minimum state is not only explained by fluctuations. Second, although we acknowledge that several qualitative behaviors of GD at EoS were already known, we provide a quantitative and justified understanding of such behaviors, which may be insightful for training diagnostics of GD at EoS.

For instance, a local, single-mode approximation gives an explicit estimate of the self-stabilization period: take Σ≃suuT\Sigma \simeq s uu^T with ∣u∣=1|u|=1, Hu=λuH u = \lambda u for a simple eigenvalue λ\lambda, and u˙≃0\dot u \simeq 0. One obtains (in algorithmic time) λ˙(θ)=−η∇λ(θ)⋅∇E(θ)−η2∣∇λ(θ)∣2s,\dot \lambda(\theta) = -\eta \nabla \lambda(\theta) \cdot \nabla E(\theta) - \frac{\eta}{2} |\nabla \lambda(\theta)|^2 s, and s˙=−ηsλ(θ)(2−ηλ(θ))\dot s = -\eta s \lambda (\theta) (2- \eta\lambda(\theta)). Freezing the coefficients −η∇λ(θ)⋅∇E(θ)-\eta\nabla\lambda(\theta)\cdot\nabla E(\theta) and ∣∇λ(θ)∣2|\nabla\lambda(\theta)|^2 at a reference point θ∗\theta_* such that ηλ(θ∗)=2\eta\lambda(\theta_*)=2, and linearizing the resulting (λ,s)(\lambda,s) system around (2/η,s∗)(2/\eta,s_*) with s∗=−2∇λ(θ∗)⋅∇E(θ∗)/∣∇λ(θ∗)∣2s_* = -2{\nabla\lambda(\theta_*)\cdot \nabla E(\theta_*)}/{|\nabla\lambda(\theta_*)|^2}, one obtains, with δ=ηλ(θ)−2\delta = \eta\lambda(\theta) - 2 and S=−η∇λ(θ∗)⋅∇E(θ∗)>0\mathcal{S} = - \eta \nabla \lambda(\theta_*)\cdot \nabla E(\theta_*) >0 (progressive sharpening rate), d2dt2δ+2ηSδ=0,\frac{d^2}{dt^2} \delta + 2 \eta \mathcal{S}\delta=0, which gives the small-oscillation period of this reduced model around 2/η2/\eta: Tself−stabilization=2π2ηS.T_{\rm self-stabilization} = \frac{2\pi}{\sqrt{2 \eta \mathcal{S}}}.

AuthorsOpenReview ↗

Answer to Question 3

Question 3

The restriction to the mean-field scaling of the one hidden layer architecture is standard in the literature we rely on. Although the mean-field framework has been extended to deeper architectures (see e.g. https://arxiv.org/pdf/2001.11443), it is outside the scope of this article to try to mix both fluctuations and depths effects.
Nevertheless, there are direct ramifications for other machine learning tasks. As explained in the text (around line 420), we can derive such a kinetic PDE for generic losses, tasks E and data structures. Our exposition was supposed to keep it simple by focusing on a specific example which is the MMD flow. There is however a second way to motivate this PDE, which goes through generative modeling. Indeed, given a classical Wasserstein gradient flow (or more general continuity equations) mapping a prior distribution to a target, one can try to approximate its vector field to generate new samples. This is done e.g. in https://arxiv.org/html/2305.11463v4 for MMD flows and in https://arxiv.org/abs/2602.04770 for another type of (renormalised) vector field. In this context, our PDE transports both samples and their fluctuation factors. We can include this motivation in the relevant section. Overall, the kinetic framework makes explicit how fluctuations modify classical transport. Its Wasserstein gradient-flow structure supplies an effective free energy and a dissipation identity, while our convergence theorem quantifies how (i.e. quantitatively how) decay of the fluctuations allows convergence toward the target under the stated additional assumptions.

Thank you for the clarifications on those questions. I think they help position the contributions of the paper better with respect to the existing literature, and I'm happy to raise my score.

AuthorsOpenReview ↗

Uploaded version (just made plots larger)

Thank you very much! We have uploaded a new version in which we only changed the first figure in the intro to make it cleaner, and have made figures 3 and 4 more readable (hopefully). Please do not hesitate to tell if you would like other changes!

Reviewer_uGVfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes mean–fluctuation (MF) dynamics, a continuous-time approximation of gradient descent that couples the window-averaged parameters and their fluctuation covariance. It establishes a finite-time approximation theorem under sharp-valley assumptions, characterizes stationary states and their linear stability, and compares MF with other effective dynamics. Experiments examine its ability to track GD near the edge of stability. A further contribution derives a kinetic limit for wide two-layer networks.

Critical strengths
  • The explicit covariance evolution provides an interpretable description of the feedback between fluctuation growth and curvature-dependent drift.
  • The approximation theorem and stationary-state analysis give the model a mathematical foundation beyond empirical agreement.
  • The experiments compare mean trajectories, losses, and Hessian eigenvalues across several architectures.
Critical weaknesses
  • The paper provides a tractable formulation of the feedback between progressive sharpening and curvature-related drift. However, it is less clear what new understanding of GD follows from this formulation beyond the known self-stabilization mechanism.

  • Theorem 2.1 assumes a sharp valley with curvature already near the stability threshold and sharp directions orthogonal to the drift. The paper also states that covariance is rank one at leading order in this regime. Consequently, the approximation theorem does not directly establish the claimed advantages in genuinely multidirectional fluctuation regimes.

  • MF tracks GD well in the reported single-mode settings, and the onset experiment provides a concrete advantage over Rod flow. However, results with multiple unstable modes are mixed, with Rod performing better in some CNN and ResNet experiments. These outcomes leave unclear when its covariance closure offers a decisive benefit.

  • The figures could be presented more clearly. The unnumbered, hand-drawn schematic in the introduction would benefit from a cleaner rendering and replacing “(E)” with “loss.” In Figures 3 and 4, the gray shaded regions in the middle column are not explained in the legends. Although readers may infer that they relate to (\eta\lambda_1) and (\eta\lambda_2), explicitly identifying their meaning would avoid ambiguity.

Questions for authors
  • What is the most important conclusion about GD near EoS enabled by the MF analysis, beyond reproducing the known stabilization feedback?
  • What window length (m) is used to initialize MF in the experiments, and how sensitive are the results to this choice? Could the authors explain why Rod flow outperforms MF in the two-mode CNN and ResNet experiments?
2 comments

Thank you very much for the time you have taken to review this manuscript! We are more than happy to upload an updated version taking into account your propositions, such as detailing some parts as is done below. In any case, we will improve the clarity of the figures as you pointed out. Thank you for this remark!

We agree that Theorem 2.1 concerns established oscillations with leading-order rank-one covariance and does not establish approximation in a general multidirectional fluctuation regime. The benefit of retaining smaller covariance components is demonstrated separately in our onset experiment (figure 5), where a small critical component grows before becoming dominant.

Question 1

The stationary condition of the Mean-fluctuation equation is ∇E(θ∗)+12∇H(θ∗):Σ∗=0,\nabla E(\theta_*)+\frac12\nabla H(\theta_*):\Sigma_*=0, so the mean θ∗\theta_* need not be a critical point of the original loss. Therefore, convergence of averaged GD does not mean convergence to the minimizers of the loss. We also analyze the linear stability of such configurations (Theorem 2.3). For practical GD trained in the regime of validity of our model (defined in Theorem 2.1), comparing the empirical correction H:ΣH:\Sigma with EE can tell you the energy level of the valley, and therefore help diagnose whether the averaged GD has reached the global minimum of EE or not if its value is known. Second, although we acknowledge that several qualitative behaviors of GD at EoS were already known, we provide a quantitative and justified understanding of such behaviors, which may be insightful. For example, the self-stabilization mechanism was indeed already known but a local, single-mode approximation gives an explicit estimate of its feedback period: take Σ≃suuT\Sigma \simeq s uu^T with ∣u∣=1|u|=1, Hu=λuH u = \lambda u for a simple eigenvalue λ\lambda, and u˙≃0\dot u \simeq 0. One obtains (in algorithmic time) λ˙(θ)=−η∇λ(θ)⋅∇E(θ)−η2∣∇λ(θ)∣2s,\dot \lambda(\theta) = -\eta \nabla \lambda(\theta) \cdot \nabla E(\theta) - \frac{\eta}{2} |\nabla \lambda(\theta)|^2 s, and s˙=−ηsλ(θ)(2−ηλ(θ))\dot s = -\eta s \lambda (\theta) (2- \eta\lambda(\theta)). Freezing the coefficients −η∇λ(θ)⋅∇E(θ)-\eta\nabla\lambda(\theta)\cdot\nabla E(\theta) and ∣∇λ(θ)∣2|\nabla\lambda(\theta)|^2 at a reference point θ∗\theta_* such that ηλ(θ∗)=2\eta\lambda(\theta_*)=2, and linearizing the resulting (λ,s)(\lambda,s) system around (2/η,s∗)(2/\eta,s_*) with s∗=−2∇λ(θ∗)⋅∇E(θ∗)/∣∇λ(θ∗)∣2s_* = -2{\nabla\lambda(\theta_*)\cdot \nabla E(\theta_*)}/{|\nabla\lambda(\theta_*)|^2}, one obtains, with δ=ηλ(θ)−2\delta = \eta\lambda(\theta) - 2 and S=−η∇λ(θ∗)⋅∇E(θ∗)>0\mathcal{S} = - \eta \nabla \lambda(\theta_*)\cdot \nabla E(\theta_*) >0 (progressive sharpening rate), d2dt2δ+2ηSδ=0,\frac{d^2}{dt^2} \delta + 2 \eta \mathcal{S}\delta=0, which gives the small-oscillation period of this reduced model around 2/η2/\eta: Tself−stabilization=2π2ηS.T_{\rm self-stabilization} = \frac{2\pi}{\sqrt{2 \eta \mathcal{S}}}.

Question 2

As reported in the experimental-summary table in the appendix, we use m=16m=16 consecutive GD iterates for MLP, including the onset experiment, m=4m=4 for CNN, and m=8m=8 for ResNet. We found that the size of the window was not critical for the experiments, so we can include a small table in the appendix on this point, if you feel it should be. However, taking significantly larger windows can introduce additional covariance due to motion of the mean, rather than EoS oscillations alone. The window should average the fast oscillations while remaining short compared with the evolution of the mean and oscillation amplitude. In the sharp-valley setting of our theorem, this is expressed by an even mm satisfying m(η+a2/η)≪1m(\eta+a^2/\eta)\ll1, where aa denotes the fluctuation amplitude.

Our model should be thought of as a second-order description of fluctuations that extends gradient flow to account for EoS behavior. In particular, it uses a Taylor closure, while Rod retains nonlinear gradient evaluations at its endpoints.

For this reason, we chose to coarsen the Rod flow equations by Taylor expanding in the Rod extension variable ξ\xi, which leads to what we have called the “coarse Rod flow” and allows a direct comparison of the vector fields. The terms discarded by MF may contribute to Rod's better accuracy in the reported two-mode CNN and ResNet runs.

We therefore chose to go further in comparing these two models by showing that the covariance drifts of “coarse Rod” and MF are asymptotically equivalent when the covariance is nearly rank one and supported near the EoS threshold (Proposition 2.4). This theoretical result quantifies their agreement in that regime.

We are happy to clarify the window choice in the experimental section.

AuthorsOpenReview ↗

Uploaded version (just made plots larger)

Since the end of rebuttal is approaching, we have uploaded a new version in which we only changed the first figure in the intro to make it cleaner, and have made figures 3 and 4 more readable (hopefully). Please do not hesitate to tell if these changes do not address your clarity concerns!

Reviewer_zq4qOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper derives a new model for EoS training, compares it carefully to existing approaches, proves an approximation guarantee and validates it on non-toy experiments.

Critical strengths
  • First model with provable approximation guarantee
  • This work also provide a very informative comparison with existing models, both for theory and experiments
Critical weaknesses
  • I am a bit confused about the mean-field part: I fail to see the link with EoS. The authors themselves mention that the landscape is flat in this regime.
Questions for authors
  • Could the authors clarify the link between the mean-field theory and EoS?
4 comments

Thank you very much for the time you have spent reviewing the manuscript!

Indeed, the flat landscape limit kills the quadratic Hessian term ηNH(θ)ΣH(θ)\eta N H(\theta)\Sigma H(\theta), which is responsible for sustained oscillations at the EoS. A full (overparametrized + EoS) theory would require taking η∝N\eta\propto N (in the notations of the article).

Nevertheless, this kinetic equation is still meaningful for GD as a second-order description of fluctuations induced by window averaging. Fix a small ε>0\varepsilon>0, independent of NN, take η∝Nα\eta\propto N^\alpha for any fixed α∈[0,1)\alpha\in[0,1), and choose a window of size m∝εN/ηm\propto\varepsilon N/\eta. In macroscopic time τ=kη/N\tau=k\eta/N, this window has duration asymptotic to ε\varepsilon.

Under the uniform smoothness assumptions of the MMD model, the displacement per neuron during a window is O(mη/N)=O(ε)O(m\eta/N)=O(\varepsilon). The initial covariance therefore satisfies ∥Σ0∥=O((mη/N)2)=O(ε2)\|\Sigma_0\|=O((m\eta/N)^2)=O(\varepsilon^2), where ∥Σ∥=max⁡i,j∥Σij∥\|\Sigma\|=\max_{i,j}\|\Sigma_{ij}\| denotes the covariance block norm. Meanwhile, in macroscopic time, the quadratic Hessian term ηNHΣH\eta N H \Sigma H is O((η/N)ε2)O((\eta/N)\varepsilon^2) per block and vanishes as N→∞N\to\infty.

The kinetic equation therefore gives a second-order approximation of window-averaged GD. The covariance correction is of order ε2\varepsilon^2, while the expansion neglects higher-order terms in ε\varepsilon. Here, the fluctuations describe the spread of the iterates across the averaging window, which can be nonzero even along a smooth, non oscillating GD trajectory.

This result can also be read the other way around, giving a practical information for GD in this overparametrized regime: if we do not want to see fluctuations that are due to large window effects, then one should take windows of size m≪N/η.m \ll N/\eta.

Of course, we are happy to update our manuscript and add this remark if you think this helps in the understanding of this kinetic Wasserstein gradient-flow framework we introduced!

AuthorsOpenReview ↗

Uploaded new version (just made plots larger)

We only changed the first figure in the intro to make it cleaner, and have made figures 3 and 4 more readable (hopefully). Please do not hesitate to tell if you would like other changes!

Thank you for your answer and this useful explanation. I would be grateful if it could be reflected in the future revision.

Sure, we will add the paragraph below in the main text (around line 457) and put the details in a new appendix C.3. Since the rebuttal is ending in a few hours and we want to write a clean appendix subsection, we will probably upload the new version after the AC-reviewer discussion (if this is indeed what you meant by "future revision"). Thank you very much, again, for your time!

"As we will show in \cref{thm:kinetic-mean-field-limit}, the quadratic term in the covariance equation is of order O(N−1)O(N^{-1}) when η\eta is fixed. This term thus disappears as N→∞N\to\infty, which is the flat landscape regime (\cref{prop:mf-limits}). This is important, since the quadratic Hessian term is responsible for sustained EoS oscillations. Therefore, for η≪N\eta \ll N, the large NN description of our model does not describe GD at EoS anymore (to do this, one should take η∝N\eta\propto N). Nevertheless, if the window size mm of averaging is large (of order N/ηN/\eta), then fluctuations can appear as a consequence of a non negligible move of GD during a window time. The kinetic description we derive in this section can be understood as a second order description of GD taking into account this fluctuation effect, even in non EoS regimes. For more details, we refer to appendix C.3."

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.