- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper mainly goes a step further on the local dynamical systems analysis of large step gradient descent in MacDonald et al. (2025) from to , and also from an isolated sharpness minimizer to a manifold of flat minima. Leveraging new theoretical tools arising and some non-trivial assumptions, it derives a normal form separating sharpness descent along the solution manifold, a period doubling direction, and contracting normal directions, and extends the three regimes---subcritical, critical, and supercritical convergence results in MacDonald et al. (2025). The authors also establish geometric properties of the flat minimum set in deep matrix factorization and provide numerical illustrations to verify theory.
Critical strengths
- Generalizing the normal form analysis to higher codimension and non-isolated flat minima requires substantial work, including a center manifold reduction. The resulting decomposition offers a useful account of local large step GD dynamics.
- The fibre bundle characterization and Morse–Bott analysis for deep matrix factorization provide meaningful structural results and demonstrate that the abstract framework applies beyond scalar factorization.
Critical weaknesses
- The geometric assumptions are restrictive, and their applicability to realistic networks remains unclear. In particular, Assumption 3.5 requires and on , where is defined through a complicated combination of higher-order derivatives (Eqs. (6)–(7)). The paper provides little intuition for why these conditions should hold or how to verify them. Together with the simple top eigenvalue requirement (Assumption 3.3) and the invariant eigendirection condition (Assumption 3.4(2)), it is unclear how broadly the framework applies beyond the matrix factorization examples.
- The experiments focus on small three-layer, matrix factorization problems. They illustrate the predicted regimes but provide limited evidence for the broader high codimension framework (especially for this paper where extension to is one of its major contribution) or the normal form decomposition itself.
- Overparameterized least squares is a reasonable theoretical model for this literature (and common for EoS studies), but the guarantees remain local to a neighborhood of flat minima. They do not address the broader dynamics of practical neural network training, limiting the paper's significance for deep learning. And in my opinion this is one of reasons why the score could not go higher.
Questions for authors
- Could the authors recheck the sign of the term in Assumption 3.5 against Eq. (122)–(126) and Proposition E.7? As written, setting and in Eq. (355) gives , whereas Eq. (356) gives . Since positivity of this coefficient underpins the normal form and convergence theorems, please clarify the correct formula and whether the stated assumptions and results remain valid.
- Lemma B.1 (Eq. (19)) appears to omit a quadratic term involving the second fundamental form of , namely . Even when , the normal displacement is generally rather than . Similarly, Eq. (34) in Proposition B.2 also appears to omit terms of order in the tangential update. Could the authors clarify these expansions and show that any corrections are compatible with the remainder estimates in Theorem 4.1?
3 comments
Thank you for the detailed rebuttal and revisions. My technical questions have been addressed, and I will raise my soundness score to 3.
I also read the other reviewers' comments and your responses. I understand your point that the contribution should be evaluated relative to MacDonald et al. (2025), and I appreciate the theoretical extensions and technical challenges overcome in this paper. However, I personally place more weight on whether the theory provides new insights into what actually happens in practical training, as opposed to new techniques beyond MacDonald et al. (2025). So far, I am not fully convinced, as much of the new insight (that arises from new proof) relies on geometric assumptions that may be too restrictive---and that the rebuttal fails to address them well. For example, your explanation of clarifies why it is needed for the proof, but not why we should expect it to hold in realistic models.
The additional experiments in Appendix F are helpful, and I find the nonlinear examples in Appendix G interesting. Still, the latter are limited to two-parameter, scalar-output models, and the extension to invariant curves remains unproven. Unless the authors can provide stronger evidence that the key assumptions hold or can be relaxed in more realistic settings, I am inclined to keep my overall rating at 2.
While we share your desire for practically relevant theory, it is not just MacDonald et al (2025) over which our paper improves. Even if our assumptions extended no further than matrix factorisation, our results would still represent a substantial advance over the state of the art, both in the generality of the models considered and in the strength of the conclusions obtained.
To give several concrete examples:
-
Ghosh et al. (ICLR 2025) study deep matrix factorisation beyond the edge of stability, but their analysis relies on highly structured initialisations and singular-vector alignment, with the explicitly prescribed initialisations belonging to measure-zero, one-parameter families. In contrast, our convergence theorems apply to an open, positive-measure set of initialisations in the full parameter space, without requiring singular-vector alignment at initialisation.
-
Kreisler et al. (ICML 2023) establish monotonic decrease of gradient-flow-solution sharpness for scalar neural networks and characterize convergence to the edge of stability under particular conditions. Our results instead establish quantitative convergence guarantees in a substantially broader, vector-valued setting, including convergence toward an entire manifold of flat minima.
-
Kalra et al. (ICLR 2025) develop their theoretical analysis principally through a two-layer linear network trained on a single example, without establishing comparable quantitative convergence guarantees.
-
Wu et al. (NeurIPS 2023, COLT 2024, NeurIPS 2025) obtain important large-step convergence and optimisation results primarily in the setting of linear logistic regression.
If theoretical guarantees were required to have already been verified in realistic neural networks to merit acceptance, the same objection would apply to all of these published works. Yet these contributions have been essential to developing our current theoretical understanding of large-step optimisation. We believe our work should be evaluated by the same standard, particularly given that our assumptions permit substantially more general models and stronger convergence guarantees than comparable precedents.
Rebuttal
Thank you for your time and consideration. We have updated our manuscript to address your critiques.
Weakness 1 (strong assumptions): We are ourselves unclear on exactly how far our theory extends; investigation of its limits strikes us as fertile ground for future research. As evidence of this, we have included numerical demonstrations in our new Appendix G suggesting that the invariant eigendirection assumption might be relaxed to an invariant curve assumption in nonlinear settings, and that similar bifurcating dynamics to those predicted by our theory hold in these nonlinear settings. Rigorous theoretical proof of these observations seems nontrivial so we leave it to future work.
Weakness 2 (few experiments): We have ablated over further widths and spectral gaps in our matrix factorisation experiments in our new Appendix F, with all results being exactly as predicted by our theory.
Weakness 3 (strong assumptions): While the coincidence of theory with practice is the ultimate ideal, we would argue that it is unrealistic to use this ideal as a criterion for rejection. Deep learning theory remains far behind practice, and this is unlikely to change with a single paper. We ask that our contribution be evaluated relative to existing work on the topic, whose assumptions are typically stronger, and results typically weaker, than our own.
Question 1: We have checked these equations and cannot find any errors. In particular, according to our calculations, both Eqs 355 and 356 (eqs 354 and 355 in our new version) give 8 when substituting L=2 and η=2/λ1∣F.
Question 2: Thank you for bringing this to our attention. You are correct that we missed the correction coming from the second fundamental form in Lemma B.1, and that we missed the additional O(∣y2:q∣2) term in Proposition B.2. We have updated our proofs of Lemmas B.1 and Proposition B.2 to include these additional terms. The statement of Proposition B.2 (and all following results) remains unchanged by the O(∣vx∣2) term since it is sufficiently high order that it can be absorbed in the remainders; the O(∣y2:q∣2) term is absorbed by the invariance of the centre manifold and stable foliation in Proposition B.4.