- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies gradient descent at the edge of stability under a sharp-valley decomposition Through a formal multiple-timescale expansion, it derives the central flow as the leading-order dynamics and recovers self-stabilization at the next order. It further computes the slow drift of oscillation energy in a two-dimensional example and extends the self-stabilization system to multiple critical directions.
Critical strengths
- The multiple-timescale framework provides a systematic organization of fast oscillations, intermediate self-stabilization dynamics, and slow motion along the valley.
- The paper makes the averaging underlying the central-flow approximation more explicit.
Critical weaknesses
- The relationship between the assumed setting and neural network training is insufficiently established. The decomposition is a useful modeling assumption, but the paper does not identify a concrete small parameter or demonstrate how this decomposition arises in a neural network learning problem. The empirical observations discussed are consistent with a sharp-valley picture, but do not establish the particular asymptotic structure required by the analysis. Furthermore, the leading-order trajectory is assumed to remain in the stable valley. Thus, the framework conditionally describes EoS dynamics without adequately explaining when neural network training satisfies its assumptions.
- The illustrative self-stabilization mechanism is largely already present in prior work. Appendix B of Damian et al., Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability, provides a closely related toy loss whose coupled dynamics are two-dimensional, with an additional decoupled direction. Section 4.2 of that work already derives the oscillation–sharpness feedback system and its conserved quantity. The present paper acknowledges this connection, but the two-dimensional example adds limited mechanistic insight. Once the stability boundary is reached, its leading-order constrained trajectory is stationary, and the remaining dynamics primarily describe local fluctuations. The contribution here is therefore the asymptotic organization of an existing mechanism, rather than a new explanation of self-stabilization.
- The higher-order drift calculation is highly model-specific, and its broader implications remain unclear. Result 3.3 determines whether oscillation energy grows or decays through the coefficient of a prescribed cubic term. However, the paper does not explain why neural network losses should possess the structure needed for dissipation, nor does it provide an interpretable criterion for the drift’s sign in a general loss landscape. The calculation establishes that higher-order effects can modify the conserved leading-order cycles, but does not yet explain the observed damping in neural network training. Its significance currently appears closer to demonstrating the perturbative method than to identifying a broadly useful mechanism.
Questions for authors
-
Can the authors identify a concrete learning model in which the small parameter , the sharp-valley decomposition, and the stable-leading-trajectory assumption can be justified?
-
Beyond recovering the known self-stabilization system, what qualitative prediction of the multiple-timescale analysis distinguishes it from Damian et al.’s toy setting in the Appendix B?
2 comments
Individual response — Reviewer Rt6U (continued)
W3. The higher-order drift calculation is highly model-specific, and its broader implications remain unclear.
We agree with this criticism. Let us explain our choices.
First, we believe there was a mismatch in previous works: the self-stabilization system derived by Damian et al. (2023) preserves an energy, while energy variations are observed in practice (see Figure 4, former Figure 3, for instance). Here, the contribution of the method of multiple scales is to resolve this apparent contradiction: one can have a conservative system in and energy variations in .
Further, we wanted to show that the method of multiple scales enables the computation of these energy variations, at least conceptually. As a "proof of concept", we provided Result 3.4 (former Result 3.3).
We wanted to generalize Result 3.4 (former Result 3.3) to the setting of Section 4. The proof of Result 3.4 is already 2-3 pages of tedious computations; we were unable to derive the generalization ourselves. We then asked AI: as the method is systematic, it directly gave the generalization to the setting of Section 4.
We did not include this generalization in the paper for two reasons. First, we somehow still have a preference for human-written mathematics. But more importantly, we found no explanation in the generalization of why neural networks systematically dissipate energy. From the formula, both signs of the energy variation seem possible, and thus there must be some additional structure in the landscape of neural networks, not included in our ansatz, that makes the variation of the energy systematically negative.
In the end, we decided to omit this technical, AI-generated and not-so-insightful generalization. We still believe that reconciling the conservative self-stabilization system with the observed energy variations, and opening a route to study them, is worth sharing.
Soundness grading. Your grading of the soundness indicates that you have "significant concerns about the paper soundness", but your review does not point to incorrect mathematical reasoning. Could you specify which parts you consider unsound, so that we can address them?
Individual response — Reviewer Rt6U
Thank you for your work on this review. We address your concerns separately below.
The decomposition f=g+εh should not be understood as a realistic model of the loss landscape of neural networks, but only as a local approximation around the gradient descent trajectory. We modified a sentence in the introduction to make this more explicit. We expect this approximation to be faithful only after a certain progressive sharpening phase.
The first paragraph of Appendix A lists extensive empirical evidence for the local valley model. The asymptotic regime ε≪1 used in this paper is motivated by the large ratio between the outlier eigenvalues and the eigenvalues of the bulk; we have added a sentence to Appendix A to clarify this.
For instance, consider Figure 3 of Ghorbani et al. (2019). We observe that during training, a few eigenvalues stick out of the bulk. These eigenvalues are an order of magnitude larger than most other eigenvalues (note the log scale on the y-axis: most eigenvalues are very close to zero).
See the common answer to all reviewers.
A concrete learning model for the sharp-valley decomposition has been proposed by Wen et al., 2025 (cited in Appendix A). The toy task is learning the distribution of the next token given one token. They show that the sharpness in different directions relates to the uncertainty of the next token depending on the given token. If this uncertainty varies on different scales (in machine learning, some predictions are easy, others are hard), this would create the small parameter ε required for our analysis.
That being said, extrapolating that different uncertainty levels in predictions create the asymptotic regime that we need in training neural networks is largely speculative. Overall, it is more accurate to say that our understanding of the neural network landscape and of training dynamics evolve jointly, and that neither is fully within the reach of theory for now.
We definitely agree that there is a strong similarity between the setting of the minimal example and previous works, and that Result 3.1 was essentially already known. In fact, we made this connection on purpose so that the new techniques (the asymptotic regime in ε, the method of multiple scales) are illustrated on an example the community is already familiar with. The novel results of this paper are in Sections 3.3, 4 and 5 (see the answer to Q2 for details).
However, we had not noticed that Damian et al. (2023) had such a similar example in their appendix. We thank you for pointing that out. We added a sentence at the beginning of Section 3.1 to acknowledge this.
The asymptotic regime in ε and the method of multiple scales enabled the following new results and qualitative predictions.