acceptodds
Under review as a conference paper at ICLR 2027

Dynamics of Large-Step Gradient Descent Near a Manifold of Flat Minima

Abstract

An important quantity in the theory of gradient descent (GD) is the sharpness, defined as the largest eigenvalue of the objective Hessian. Classical analyses typically require the step size to be uniformly smaller than twice the reciprocal of the sharpness, but this condition is frequently violated in the training of deep neural networks. Recent work bridges this gap in the setting of overparametrised least-squares with a single scalar output, providing a normal form for large-step GD in a neighbourhood of an isolated flat minimum and establishing three corresponding convergence results. In this paper, we extend this theory in two directions: (1) to overparametrised least-squares with vector-valued outputs (including regression with arbitrarily many observations), and (2) to a neighbourhood of a manifold of flat minima (which we show is essential for applications such as matrix factorisation). We generalise both the normal form and all three convergence theorems of prior work to this broader setting, overcoming several technical challenges. We further show that our framework applies to deep matrix factorisation under mild assumptions, yielding several new structural results. In particular, we prove that the set of flat minima forms a fibre bundle over a product of spheres, and that the sharpness is Morse-Bott along this manifold.

Primary area:
optimization
Keywords:
gradient descent, edge of stability, large step size, optimization

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_eUbROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper mainly goes a step further on the local dynamical systems analysis of large step gradient descent in MacDonald et al. (2025) from q=1q=1 to q>1q>1, and also from an isolated sharpness minimizer to a manifold of flat minima. Leveraging new theoretical tools arising and some non-trivial assumptions, it derives a normal form separating sharpness descent along the solution manifold, a period doubling direction, and contracting normal directions, and extends the three regimes---subcritical, critical, and supercritical convergence results in MacDonald et al. (2025). The authors also establish geometric properties of the flat minimum set in deep matrix factorization and provide numerical illustrations to verify theory.

Critical strengths
  1. Generalizing the normal form analysis to higher codimension and non-isolated flat minima requires substantial work, including a center manifold reduction. The resulting decomposition offers a useful account of local large step GD dynamics.
  2. The fibre bundle characterization and Morse–Bott analysis for deep matrix factorization provide meaningful structural results and demonstrate that the abstract framework applies beyond scalar factorization.
Critical weaknesses
  1. The geometric assumptions are restrictive, and their applicability to realistic networks remains unclear. In particular, Assumption 3.5 requires αη>0\alpha_\eta>0 and Dαη=0D\alpha_\eta=0 on F\mathcal F, where αη\alpha_\eta is defined through a complicated combination of higher-order derivatives (Eqs. (6)–(7)). The paper provides little intuition for why these conditions should hold or how to verify them. Together with the simple top eigenvalue requirement (Assumption 3.3) and the invariant eigendirection condition (Assumption 3.4(2)), it is unclear how broadly the framework applies beyond the matrix factorization examples.
  2. The experiments focus on small three-layer, 2×22\times2 matrix factorization problems. They illustrate the predicted regimes but provide limited evidence for the broader high codimension framework (especially for this paper where extension to q>1q>1 is one of its major contribution) or the normal form decomposition itself.
  3. Overparameterized least squares is a reasonable theoretical model for this literature (and common for EoS studies), but the guarantees remain local to a neighborhood of flat minima. They do not address the broader dynamics of practical neural network training, limiting the paper's significance for deep learning. And in my opinion this is one of reasons why the score could not go higher.
Questions for authors
  1. Could the authors recheck the sign of the D4ℓ/3D^4\ell/3 term in Assumption 3.5 against Eq. (122)–(126) and Proposition E.7? As written, setting L=2L=2 and ηλ1∣F=2\eta\lambda_1|_{\mathcal F}=2 in Eq. (355) gives 1010, whereas Eq. (356) gives 88. Since positivity of this coefficient underpins the normal form and convergence theorems, please clarify the correct formula and whether the stated assumptions and results remain valid.
  2. Lemma B.1 (Eq. (19)) appears to omit a quadratic term involving the second fundamental form of M\mathcal M, namely −12ν(x)⊤IIx(vx,vx)-\frac12\nu(x)^\top\mathrm{II}x(v_x,v_x). Even when vy=0v_y=0, the normal displacement is generally O(∣vx∣2)O(|v_x|^2) rather than O(∣vx∣3)O(|v_x|^3). Similarly, Eq. (34) in Proposition B.2 also appears to omit terms of order ∥y2:q∥2\|y_{2:q}\|^2 in the tangential update. Could the authors clarify these expansions and show that any corrections are compatible with the remainder estimates in Theorem 4.1?
3 comments

Thank you for your time and consideration. We have updated our manuscript to address your critiques.

Weakness 1 (strong assumptions): We are ourselves unclear on exactly how far our theory extends; investigation of its limits strikes us as fertile ground for future research. As evidence of this, we have included numerical demonstrations in our new Appendix G suggesting that the invariant eigendirection assumption might be relaxed to an invariant curve assumption in nonlinear settings, and that similar bifurcating dynamics to those predicted by our theory hold in these nonlinear settings. Rigorous theoretical proof of these observations seems nontrivial so we leave it to future work.

Weakness 2 (few experiments): We have ablated over further widths and spectral gaps in our matrix factorisation experiments in our new Appendix F, with all results being exactly as predicted by our theory.

Weakness 3 (strong assumptions): While the coincidence of theory with practice is the ultimate ideal, we would argue that it is unrealistic to use this ideal as a criterion for rejection. Deep learning theory remains far behind practice, and this is unlikely to change with a single paper. We ask that our contribution be evaluated relative to existing work on the topic, whose assumptions are typically stronger, and results typically weaker, than our own.

Question 1: We have checked these equations and cannot find any errors. In particular, according to our calculations, both Eqs 355 and 356 (eqs 354 and 355 in our new version) give 8 when substituting L=2L=2 and η=2/λ1∣F\eta=2/\lambda_1|_F.

Question 2: Thank you for bringing this to our attention. You are correct that we missed the correction coming from the second fundamental form in Lemma B.1, and that we missed the additional O(∣y2:q∣2)O(|y_{2:q}|^2) term in Proposition B.2. We have updated our proofs of Lemmas B.1 and Proposition B.2 to include these additional terms. The statement of Proposition B.2 (and all following results) remains unchanged by the O(∣vx∣2)O(|v_x|^2) term since it is sufficiently high order that it can be absorbed in the remainders; the O(∣y2:q∣2)O(|y_{2:q}|^2) term is absorbed by the invariance of the centre manifold and stable foliation in Proposition B.4.

Thank you for the detailed rebuttal and revisions. My technical questions have been addressed, and I will raise my soundness score to 3.

I also read the other reviewers' comments and your responses. I understand your point that the contribution should be evaluated relative to MacDonald et al. (2025), and I appreciate the theoretical extensions and technical challenges overcome in this paper. However, I personally place more weight on whether the theory provides new insights into what actually happens in practical training, as opposed to new techniques beyond MacDonald et al. (2025). So far, I am not fully convinced, as much of the new insight (that arises from new proof) relies on geometric assumptions that may be too restrictive---and that the rebuttal fails to address them well. For example, your explanation of Dαη∣F=0D\alpha_\eta|_{\mathcal F}=0 clarifies why it is needed for the proof, but not why we should expect it to hold in realistic models.

The additional experiments in Appendix F are helpful, and I find the nonlinear examples in Appendix G interesting. Still, the latter are limited to two-parameter, scalar-output models, and the extension to invariant curves remains unproven. Unless the authors can provide stronger evidence that the key assumptions hold or can be relaxed in more realistic settings, I am inclined to keep my overall rating at 2.

While we share your desire for practically relevant theory, it is not just MacDonald et al (2025) over which our paper improves. Even if our assumptions extended no further than matrix factorisation, our results would still represent a substantial advance over the state of the art, both in the generality of the models considered and in the strength of the conclusions obtained.

To give several concrete examples:

  1. Ghosh et al. (ICLR 2025) study deep matrix factorisation beyond the edge of stability, but their analysis relies on highly structured initialisations and singular-vector alignment, with the explicitly prescribed initialisations belonging to measure-zero, one-parameter families. In contrast, our convergence theorems apply to an open, positive-measure set of initialisations in the full parameter space, without requiring singular-vector alignment at initialisation.

  2. Kreisler et al. (ICML 2023) establish monotonic decrease of gradient-flow-solution sharpness for scalar neural networks and characterize convergence to the edge of stability under particular conditions. Our results instead establish quantitative convergence guarantees in a substantially broader, vector-valued setting, including convergence toward an entire manifold of flat minima.

  3. Kalra et al. (ICLR 2025) develop their theoretical analysis principally through a two-layer linear network trained on a single example, without establishing comparable quantitative convergence guarantees.

  4. Wu et al. (NeurIPS 2023, COLT 2024, NeurIPS 2025) obtain important large-step convergence and optimisation results primarily in the setting of linear logistic regression.

If theoretical guarantees were required to have already been verified in realistic neural networks to merit acceptance, the same objection would apply to all of these published works. Yet these contributions have been essential to developing our current theoretical understanding of large-step optimisation. We believe our work should be evaluated by the same standard, particularly given that our assumptions permit substantially more general models and stronger convergence guarantees than comparable precedents.

Reviewer_cDSvOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This article analyses the dynamics of gradient descent with fixed stepsize η\eta in least-squares regression. It assumes an overparametrized setting where there is a manifold of minimizers of the least-squares loss. Among this manifold of minimizers, the submanifold of points that minimize the sharpness of the loss is of special importance. The article proposes a local analysis of gradient descent around this submanifold.

The main contribution of the article is a normal form for gradient descent: this normal form shows that gradient descent performs a gradient descent on the sharpness in the direction of the manifold and provides the bifurcating dynamical system in the orthogonal directions. As a consequence, convergence results can be derived. The qualitative behavior depends on the comparison between the minimal sharpness and 2/η2/\eta. In particular, if the minimal sharpness is above 2/η2/\eta, the paper shows convergence to a period-2 cycle centered around a point with minimal sharpness.

Critical strengths

Relevance. There is an intense interest around gradient descent with large stepsizes in the literature. Along with MacDonald et al. (2025), this paper provides a new perspective through the theory of dynamical systems. This level of rigor is uncommon to the line of work. The imported machinery is likely to be useful in follow-up works on the edge of stability regime. Generalizing the results of MacDonald et al. (2025) to problems of codimension >1 is crucial to reasonable applications.

Disclaimer. My technical understanding of the paper is too superficial to assess the technical difficulty of generalizing the results of MacDonald et al. (2025).

Clarity. The paper is very well-written. Mathematical writing is concise and pedagogical. The matrix factorization example provides a convincing application of the general framework.

Critical weaknesses

None

Questions for authors

Minor comments:

  • Could you clarify what Assumption 3.4, 2 means? I'm unsure of the meaning of "an open neighborhood of zero in the span of ν1∣F\nu_1\vert_F".
  • l.255: Does Dαη(x)=0D\alpha_\eta(x) = 0 mean that αη\alpha_\eta is constant? How strong is this assumption?
  • Given the importance in your setting, I suggest explaining what a "flip bifurcation" is.
  • l.375: 'the iterates eventually reach a point where η<2/λ1∣M\eta < 2/\lambda_1\vert_M' -> do you mean 'the iterates eventually reach a point xx where η<2/λ1(x)\eta < 2/\lambda_1(x)'?
  • Thm. 5.3: Do you have convergence of xtx_t to an element of FF and the convergence of yty_t to a cycle? Or is it more complicated than this?
  • Fig. 4: The axes do not show clearly that this is log scale. Add ticks.
2 comments

Thank you for your time and consideration. We have updated our manuscript with your comments in mind.

Question 1: We have rewritten that assumption. It now reads "For each x∈Fx\in F, there is a neighbourhood II of zero in R\mathbb{R} such that {x+tν1(x):t∈I}\{x + t\nu_1(x):t\in I\} is invariant under gradient descent on ℓ\ell." This is actually stronger than what we previously had, but it is what we originally intended is what we checked for our matrix factorisation example.

Question 2: That Dαη(x)=0D\alpha_{\eta}(x) = 0 for all x∈Fx\in F implies that αη\alpha_{\eta} is constant on F, but it may still vary away from FF. We are unsure of how strong an assumption this is: it is nontrivial to verify for matrix factorisation, but this does not necessarily rule out its being satisfied in nonlinear settings. Indeed, our numerical results in our new Appendix G seem to suggest that similar bifurcating dynamics to those predicted by our theory hold even in simple nonlinear settings. We leave investigation of this question to future work.

Question 3: Thank you for the suggestion. We have added a description of the flip bifurcation and its dynamics Lines 337-350.

Question 4: Yes, that is exactly what we meant, thank you for asking. We have made this precise in the new version.

Question 5: Yes, we intended convergence of xtx_t to some x∞∈Fx_{\infty}\in F and convergence of yty_t to a cycle on the span of ν(x∞)\nu(x_{\infty}); we have made this clearer in the new version.

Question 6: Thank you for the suggestion, we have added ticks.

I thank you for your reply. My concerns have been addressed. I am still uncomfortable with Assumption 3.5, as are all of the reviewers. It can only be checked on the matrix factorization example, and it is unclear how realistic it is for other problems.

That being said, I think this paper should not be evaluated for its contribution to practical ML (as suggested by other reviewers), and rather for its contribution to building firm foundations for the theory. The tools used in this paper combined with this level of rigor are unmatched in the current works on the topic. As a consequence, I believe that even partial progress on a rigorous understanding of large-step size GD are a solid contribution.

Reviewer_ri6HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+2)
Read review
Paper summary

This paper addreses the following question: what large-step gradient descent does once it is near a manifold, rather than a single point, of flat minima. Building on MacDonald et al. (2025), it extends the analysis from scalar- to vector-valued least squares. The main normal form separates motion along the solution manifold (driven by sharpness), a flip bifurcation in the leading normal direction, and contraction in the remaining normal directions. It then establishes convergence behavior in three step-size regimes and works out the geometry for deep matrix factorization

Critical strengths

The geometric picture is compelling. What could have been a difficult collection of case-specific calculations is organized into an intuitive description of how the tangential and normal dynamics interact. Moving beyond codimension one is meaningful here, since the extra stable directions are genuinely part of the problem rather than a cosmetic generalization. I also value the care taken with the singular PDE in Appendices B–C, especially the authors’ discussion of a gap in the earlier normal-form argument. The fibre-bundle and Morse–Bott results for matrix factorization are interesting contributions beyond the convergence theorems themselves.

Critical weaknesses

My main concern is about the assumptions. In particular, the invariant leading-normal line in Assumption 3.4 and the conditions on αη in Assumption 3.5 seem quite restrictive. They are checked for the chosen factorization setting, but I am less sure how often a mildly perturbed or more realistic network would satisfy them. I would have liked a clearer discussion of which conclusions survive when these geometric properties fail.

Figures 3–5 are helpful sanity checks, but five trajectories on small three-layer 2 × 2 factorizations do not tell us much about robustness to width, depth, spectral-gap changes, or other codimensions. I do not view the local nature of the theorems as a flaw by itself; it is simply important not to read them as a theory of how training reaches this neighborhood in the first plac

Questions for authors
  1. Is the invariant-line condition in Assumption 3.4 stable under small perturbations of the factorization model? If not, which part of the normal form breaks first?

  2. Could you show the predicted rates for a few larger factorization problems, ideally while varying the gap between the top two Hessian eigenvalues?

2 comments

Thank you for your time and consideration. We have updated our manuscript in response to your critiques. Since the weaknesses you identified are all reflected in your questions, we will proceed directly to answering your questions.

Question 1 (stability of assumptions): We have added numerical demonstrations in our new Appendix G indicating that in nonlinear settings, invariance of the normal line could potentially be generalised to invariance of a normal curve through the flat minimum, with similarly bifurcating dynamics to what is predicted by our theory. We have also included a brief discussion of this point in the main body of the paper, Lines 237-245. We thus believe it likely that the theory extends to nonlinear settings, but feel that the details would be sufficiently nontrivial that this extension is best left to future work.

Question 2 (experimental ablation): We have ablated over width and spectral gap in our new Appendix F. The dynamics in all settings we examined are exactly as predicted by our theory.

Thank you for the response which answers my questions and concerns. After reading the paper again in light of your response, I think my rating is fair.

Reviewer_j3DhOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The study extends MacDonald et al. (2025) from scalar-output least squares near an isolated flat minimum to vector-valued outputs near a manifold of flat minima. Under geometric assumptions on the solution manifold, the sharpness, and the top Hessian eigendirection, the authors derive a normal form separating GD into sharpness descent along the solution manifold, oscillation in the top normal direction, and contraction in the remaining normal directions. This yields three local regimes: geometric convergence to a generally suboptimally flat solution, convergence to a flat minimum at rate (\Theta(t^{-1/2})), and geometric convergence. The authors also verify the assumptions for deep matrix factorization and establish structural properties of its flat-minimum set.

Critical strengths
  • Extending the analysis to vector-valued outputs and manifolds of sharpness minimizers removes substantial restrictions of the previous work. Separating the additional contracting normal directions and constructing a sufficiently regular normalizing coordinate change are nontrivial technical steps. The Lyapunov argument for the critical rate also provides a useful simplification of the earlier proof.

  • The matrix-factorization analysis characterizes the sharpness-minimizing manifold and establishes the Morse–Bott property of sharpness along it. These structural results support the relevance of the manifold setting and verify substantive geometric assumptions in a concrete example.

Critical weaknesses

-------- Following rebuttal -------

Thank you for the response and revisions. To the best of my understanding, the corrections resolve the two technical issues raised in my review. The additional nonlinear examples are a good addition, and the explanation of the enlarged supercritical convergence neighborhood makes the contribution clearer.

The scope of the assumptions remains a limitation: the new examples illustrate similar dynamics but do not verify the assumptions. Also, it remains unclear whether MacDonald et al.’s results are recovered under their original hypotheses, given the different Assumption 3.5.

I raised soundness and rating by 1.

-------- First cycle review --------

  • The scope of the assumptions limits the conclusions and their connection to the motivation. Beyond a regular solution manifold, Assumptions 3.4–3.5 require quadratic restoring geometry for the sharpness, exact normal-direction invariance, and higher-order conditions (\alpha_\eta>0) and (D\alpha_\eta|_F=0). These properties do not follow from overparameterization alone, and their relevance beyond matrix factorization remains unclear. Moreover, the analysis starts near the flat-minimum manifold and concerns deterministic, constant-step GD. It does not explain how this neighborhood is reached or establish how the conclusions extend to stochastic training.

  • The normal-form derivation needs clarification and a correction. If I understand correctly, Assumption 3.4(2) should explicitly distinguish invariance of each individual normal line from invariance of their union: the argument immediately before (38) uses the stronger property, which Proposition E.6 verifies for matrix factorization. Also, if I am correct, the tangential update in Proposition B.2 appears to omit an (O(|y_{2}|^2)) contribution; the stated remainder cannot capture this when (y_1=0). This term appears absorbable after the centre-manifold reduction, but the authors should correct the expansion and verify that the subsequent estimates remain valid.

Questions for authors
  1. Which assumptions are essential to the dynamical conclusions, and which are imposed to obtain the particular normal form? In particular, what behavior can occur if (D\alpha_\eta|_F=0) or the exact normal-direction invariance is relaxed? Can the authors provide a nonlinear example beyond matrix factorization, or evidence that these properties approximately hold in a relevant training problem?

  2. Does Assumption 3.4(2) require invariance of every individual affine line (x+\operatorname{span}{\nu_1(x)}), for (x\in F)? As written, it appears to require invariance of their union, which permits drift along (F). However, immediately before (38), the proof uses
    [
    P_{T_xM}\nabla\ell(x+y_1\nu_1(x))=0.
    ]
    This does not follow from invariance of the union alone. Proposition E.6 establishes the stronger, individual-line invariance for matrix factorization. Please state explicitly which property is assumed in the general theory.

  3. Please clarify precisely which results of MacDonald et al. (2025) are recovered as special cases. Setting (q=1) and (F={x^\star}), do their assumptions imply yours, and do your theorems recover their conclusions? Since you also identify an error in their normal-form argument, please distinguish which earlier statements are recovered, strengthened, or corrected.

  4. What additional insight into stepsize choice or solution selection follows from the vector-output and manifold setting? In particular, please explain the significance of the enlarged supercritical convergence neighborhood and which conclusions go beyond the qualitative regimes established in the scalar-output setting.

1 comment

Thank you for your time and consideration. We have updated our manuscript in response to your critiques.

Weakness 1 (limited scope): We agree that the scope of our theory is limited relative to deep learning practice. However, we ask that the contribution be judged relative to the existing state of deep learning theory, wherein it is a significant step beyond prior work, the closest being MacDonald et al 2025. More generally, prior work on the topic typically has stronger assumptions and weaker results than ours.

Weakness 2 (missing term in normal form derivation): Thank you for bringing these to our attention; you are correct on all counts. We do indeed require invariance of each normal line rather than their union. Also, the tangential update in Proposition B.2 does require the extra O(∣y2:q∣2)O(|y_{2:q}|^2) term, but it is indeed absorbed by the centre manifold reduction that follows. The manuscript has been updated to reflect all points.

Question 1 (necessity of assumptions): Dαη=0D\alpha_{\eta}=0 is a sufficient condition for the C1C^1-solubility of the PDE that pops up in deriving our normal form. It is only required due to the singularity of the PDE, arising from our working in a neighbourhood of the flat minima manifold. Away from the flat minima manifold, the PDE is regular and can be solved straightforwardly using the method of characteristics without the Dαη=0D\alpha_{\eta}=0 hypothesis; we leave this extension to future work. We have added numerical demonstrations in our new Appendix G indicating that it may be possible to weaken invariance of the normal line to invariance of a curve in the normal direction while still retaining similar dynamics, but we defer rigorous theoretical investigation of this to future work.

Question 2: Please see our response to Weakness 2.

Question 3 (extension of MacDonald et al (2025)): All results of MacDonald et al (2025) are extended in our work. In their special q=1q=1 isolated flat minimum setting, our assumptions do specialise to theirs with the exception of our Assumption 3.5, which is genuinely different from what is in MacDonald et al due to their flawed proof of the normal form. Our C1C^1 normal form is lower regularity than their (incorrect) analytic normal form. The MacDonald et al critical regime convergence theorem therefore also has a flawed proof, since it used analyticity of the normal form (although the statement itself is correct). Our critical regime convergence theorem works even with our lower-regularity normal form.

Question 4: in MacDonald et al (2025), the supercritical convergence neighbourhood has radius going to zero as ηλ1∣F→2\eta\lambda_1|_F\rightarrow 2. Our work shows that this is unnecessary: convergence can be guaranteed from a neighbourhood of constant radius for all η\eta sufficiently close to 2/λ1∣F2/\lambda_1|_F. Our results demonstrate that the bifurcation studied in MacDonald et al is not merely a fluke of their toy setting, but a low-dimensional manifestation of much more general geometric and dynamical principles. Beyond that, our work offers new technical tools for analysing large-step dynamics in high dimensions that have not previously been utilised in the deep learning optimisation literature. We believe these tools have substantial promise for bringing theory closer to practice in future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.