- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper develops a coarse-grained model of gradient descent (GD) dynamics in the edge of stability regime, where the largest eigenvalue(s) of the Hessian reach and hover around the threshold 2/, where is the learning rate. This regime is an intriguing feature of GD dynamics, but is challenging to theoretically describe and analyze because of the sharp oscillations that characterize its trajectories. The paper overcomes this problem by tracking coarse-grained variables: a "slow mean" defined by trajectories averaged over a time window of length and a corresponding "slow covariance" describing fluctuations around that mean. The authors take the continuum time limit and derive a system of coupled differential equations for and , which they dub the mean-fluctuation (MF) dynamics model. They validate the model on three architectures trained on CIFAR-10, and show that it tracks GD observables more closely than earlier approximate descriptions of GD in the literature. Finally they apply their model to a two-layer network architecture, and show it can be understood via a hydrodynamic equation describing the time evolution of the density of weights, and that this evolution minimizes an effective free energy.
Critical strengths
The paper provides a rigorous derivation of coarse-grained GD dynamics, focusing not just on the mean flow path but also the fluctuations around that path. The resulting equations are simple in form, making them amenable for analysis.
The formalism is tested in a variety of real-world contexts (rather than toy models), validating it as an accurate description of GD observables (loss, unstable eigenvalues, etc.). The authors are careful to compare with earlier approaches in the literature in a similar vein, and find that their method generally provides better agreement with GD (though in some cases the improvement is fairly small).
Critical weaknesses
As the authors point out, this is not the first study to try to approximate GD dynamics, and the Rod flow approach in particular is a close precursor (roughly corresponding to the current method with a window time of ). The numerical results (Figure 3-5) show some improvements relative to these earlier models, but in the end many of these models were already reasonably good (for example exhibiting fractions of a percent for center error). The overall impression is that authors provide an elegant model, but there is no real conceptual payoff from introducing this model: do we learn anything significant about edge of stability GD dynamics from the model? The rigorous derivations may be nice for a mathematical audience, but the authors do not really explain the importance of any of the theorems, or how the added rigor provides any novel insights into the underlying phenomena.
Questions for authors
Following up on the comments above, there are several questions that the authors could answer to help motivate the significance of their results:
-
What is the practical advantage of having a slightly better approximate description of GD dynamics? Are there contexts in which the numerical improvements over earlier methods are meaningful?
-
Are there broad lessons or insights that we can draw from the model about edge of stability dynamics, and are there any practical implications for training neural networks via GD?
-
The final main text results section, on the kinetic theory for two-layer networks, is not strongly motivated. What do we learn from having such a kinetic description, or from showing the flow in this specific system minimizes a free energy? Are there ramifications for other machine learning architectures?
4 comments
Answer to Question 3
Question 3
The restriction to the mean-field scaling of the one hidden layer architecture is standard in the literature we rely on. Although the mean-field framework has been extended to deeper architectures (see e.g. https://arxiv.org/pdf/2001.11443), it is outside the scope of this article to try to mix both fluctuations and depths effects.
Nevertheless, there are direct ramifications for other machine learning tasks. As explained in the text (around line 420), we can derive such a kinetic PDE for generic losses, tasks E and data structures. Our exposition was supposed to keep it simple by focusing on a specific example which is the MMD flow.
There is however a second way to motivate this PDE, which goes through generative modeling. Indeed, given a classical Wasserstein gradient flow (or more general continuity equations) mapping a prior distribution to a target, one can try to approximate its vector field to generate new samples. This is done e.g. in https://arxiv.org/html/2305.11463v4 for MMD flows and in https://arxiv.org/abs/2602.04770 for another type of (renormalised) vector field. In this context, our PDE transports both samples and their fluctuation factors. We can include this motivation in the relevant section.
Overall, the kinetic framework makes explicit how fluctuations modify classical transport. Its Wasserstein gradient-flow structure supplies an effective free energy and a dissipation identity, while our convergence theorem quantifies how (i.e. quantitatively how) decay of the fluctuations allows convergence toward the target under the stated additional assumptions.
Thank you for the clarifications on those questions. I think they help position the contributions of the paper better with respect to the existing literature, and I'm happy to raise my score.
Uploaded version (just made plots larger)
Thank you very much! We have uploaded a new version in which we only changed the first figure in the intro to make it cleaner, and have made figures 3 and 4 more readable (hopefully). Please do not hesitate to tell if you would like other changes!
Answer to Questions 1 and 2
First of all, we would like to thank the reviewer for the time taken reading our manuscript! Should the following answers to his/her questions be satisfying, we will be more than happy to update the article as proposed, if the reviewer thinks accordingly.
Question 1
Our model is currently the only one we are aware of that is able to follow GD both at the onset of EoS and when there are multiple unstable eigenvalues. For instance, the Rod flow is unable to settle at the EoS when initialized before oscillations have grown (it then follows the gradient flow, see fig. 5), while the Edge flow is unable to capture the EoS dynamics when several eigenvalues are unstable (as reported in the figures). These are important failure modes that our model addresses. Therefore, we don't see our model as "closer to GD" but as genuinely capturing scenarios that are not captured by these other flows. Furthermore, our model has the advantage of addressing these failure modes while being minimally complex compared to gradient flow. In particular, we have demonstrated that our model is amenable to analysis and we are therefore allowed to draw conclusions about GD from our model, at least in the scaling limit we have exposed (the derivation of this scaling limit, and therefore the domain of validity of the model, being itself a contribution left behind in Edge and Rod flows). Finally, we would like to make a small remark which is that our model was initially developed at around the same time as Rod and Edge flows. Therefore, we see our model as an independently developed framework, and wouldn't call Rod and Edge as "precursors" or "earlier methods". Of course, this can't be proved so we don't see this claim as a major argument for our model.
Question 2
First, the stationarity condition is ∇E(θ∗)+21∇H(θ∗):Σ∗=0, so it need not be a critical point of the original loss. Therefore, convergence of averaged GD does not mean convergence to the minimizers of the loss. We also analyze the stability of such configurations (Theorem 2.3). For practical GD, computing the empirical H:Σ contribution can be a first test to understand whether the average GD is roughly around the minimum value of E, or if the discrepancy between the loss value and its minimum state is not only explained by fluctuations. Second, although we acknowledge that several qualitative behaviors of GD at EoS were already known, we provide a quantitative and justified understanding of such behaviors, which may be insightful for training diagnostics of GD at EoS.
For instance, a local, single-mode approximation gives an explicit estimate of the self-stabilization period: take Σ≃suuT with ∣u∣=1, Hu=λu for a simple eigenvalue λ, and u˙≃0. One obtains (in algorithmic time) λ˙(θ)=−η∇λ(θ)⋅∇E(θ)−2η∣∇λ(θ)∣2s, and s˙=−ηsλ(θ)(2−ηλ(θ)). Freezing the coefficients −η∇λ(θ)⋅∇E(θ) and ∣∇λ(θ)∣2 at a reference point θ∗ such that ηλ(θ∗)=2, and linearizing the resulting (λ,s) system around (2/η,s∗) with s∗=−2∇λ(θ∗)⋅∇E(θ∗)/∣∇λ(θ∗)∣2, one obtains, with δ=ηλ(θ)−2 and S=−η∇λ(θ∗)⋅∇E(θ∗)>0 (progressive sharpening rate), dt2d2δ+2ηSδ=0, which gives the small-oscillation period of this reduced model around 2/η: Tself−stabilization=2ηS2π.