- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies constant-step gradient descent (GD) near a non-isolated minimum, which extends Chen et al.(2026) from isolated minima to the more general setting of smooth manifold of minima. Under some assumptions on the regularity of minima set M and function class, the paper establishes two main results when restricting step size : (i) Theorem 4.1 shows there exists a neighborhood around 0 (which is assumed to be a minima) such that the GD iterates never escape once reaching it, in particular if GD is initialized in this region; also the funcion is strictly decreasing along the GD trajectory. (ii) Theorem 4.2 dicusses convergence direction of GD. It shows that as long as initialized within the region given in Theorem 4.1 and (although the current statement excludes the choice ), the GD iterates converges to a minima in M. Additionally, the convergence direction also converges if , and alternative converges otherwise. A key novelty in the analysis is introduing the mapping in Eq. (27), which uses coordinates correpsond to motion along and orthogonal to M, respectively. The theoretical results are validated empirically on both simple 2-dim functions and Neural Networks.
Critical strengths
-
The question is clean and natural. In real applications such as overparameterized models, minima are typically non-isolated. Therefore, the extension from isolated minima (established in Chen et al. 2026) to non-isolated ones is highly motivated and meaningful. The two main results also provide interesting insights on how GD iterates evolve near the minima set, which seem new in the literature.
-
The presentation is well-organized and easy to follow. I particularly like the organization that the authors start with a simple 2-d example before moving into the main theoretical results. This significantly helps understand the motivation and main results visually to readers not familiar with the topic.
Critical weaknesses
-
Regarding Assumptions. The assumptions seem relatively strong, or at least lack sufficient explanations. In particular, I an concerned with (17) as it needs . This matters because in the motivating example and also the first part of experiments, the rank of Hessian at a minima seems only have rank equals 1, which contradicts (17) directly. Additionally, some others such as Eq (18) and the assumption that the GD map pulls back Lebesgue null sets to null sets also need to be justified or relaxed.
-
Scope of the main results. Theorems 4.1 and 4.2 focus on the stable region . Therefore the behavior of larger step sizes or at least around is missing. This somewhat contradicts the motivation regarding the edge of stability (as stated in the introduction), because EoS considers rather large step sizes and the loss is non-monotone, while within , Theorem 4.1 explicitly shows that the loss is monotonically decreasing. Also, the convergence direction result in Theorem 4.2 excludes the choice , which should be included or explained if this choice can not be handled easily.
-
Comparison with Chen et al. (2026). This paper extend to non-isolated cases, which separates from the isolated mimima considered in Chen et al. (2026). Therefore, it would be better if a discussion of comparison with Chen et al. (2026) is included after the main results, which should state precisely how their analysis fail in the non-isolated case and how they are repaired in this work. This would help highlight the technical contributions.
Questions for authors
See my comments in the weakness discussion. In particular, I would like to ask the following main questions:
-
Does the proof extend to in (17), so that the motivating example in Section 3 and experiments in Section 5.1 better fit into the scope of the main results?
-
Can you justify some of the assumptions properly? e.g., relax or explain its fairness.
-
Can the results or discussions of the choice be included, as complement to existing Theorems 4.1 and 4.2? Please also include the result for in Theorem 4.2 if possible.
-
For the NN tests in Section 5.2 or some other settings, can you also test for and to see what the behavior looks like?
5 comments
I appreciate the response and revision from the authors. Remark 8.16 reasonably addresses the case away from the zero-multiplier rate, and I accept the explanations for excluding the transition and stability-boundary rates. These clarifications are helpful, but some of my main concerns remain unsolved.
In particular, the response does not explain why Eq. (18) requires all positive eigenvalues to be distinct, or clarify when the structural assumptions hold in the neural-network examples. The non-singularity references are relevant, but their applicability still lacks explanation. Additionally, the comparison with Chen et al. seems only high level. As a theory-focused paper, it is important and helpful to identify the precise technical novelties, e.g., key proof steps that are tailed for non-isolated minima but not included in Chen et al. setting.
For experiments, I understand and appreciate the difficulty of certifying an exact critical rate in a large network. Nevertheless, a small controlled example with , known Hessian eigenvalues, and measurements of the predicted eigendirection would still be interesting.
Follow-up Response with Technical Clarifications and Additional Controlled Experiments
We thank the reviewer for the constructive follow-up and for accepting our previous clarifications concerning the rank-one case and the excluded learning rates. We appreciate the opportunity to address the remaining questions.
1. Distinct positive Hessian eigenvalues and the neural-network examples.
The distinctness assumption in Eq. (18) is mainly required by our proof. As shown in Section 8.2 and Lemma 8.3, distinct positive Hessian eigenvalues guarantee a smooth local frame of normal eigenvectors along the minimum manifold. This frame is used to construct the coordinate transformation. Its smoothness is also needed for the derivative estimates in Lemma 8.7. Moreover, the separation of positive eigenvalues allows uniform contraction estimates as the Hessian varies along the manifold. Thus, the assumption has a specific technical role in our analysis. It is a sufficient condition for the present proof, but we do not claim it is necessary; similar directional behavior may hold under weaker spectral assumptions.
Regarding neural networks, several empirical studies provide examples of separated positive Hessian eigenvalues. Ghorbani et al. (ICML 2019), Papyan (JMLR 2020), and Granziol et al. (JMLR 2022) investigate Hessian spectra and report distinct positive spectral outliers in neural-network training. These observations support the relevance of our spectral setting, although they do not establish that all positive Hessian eigenvalues are distinct in arbitrary neural networks. We consider it plausible that the spectral condition also holds for the networks studied here, but to our knowledge, rigorously checking every positive Hessian eigenvalue at their limiting minimizers would be difficult and has not been attempted.
The three neural-network architectures in our experiments were selected to illustrate typical directional behavior consistent with our results. We do not claim that they satisfy every assumption of the theorem. The experiments provide qualitative illustrations, while the theoretical conclusions are established under the stated assumptions.
2. The null-set preimage assumption.
We thank the reviewer for the continued discussion of this assumption. As explained in our previous response, it is an established non-singularity condition in GD theory.
Ahn et al. (ICML 2022, Assumption 1) already use this condition in their analysis of GD. Chemnitz and Engel (JMLR 2025, Hypothesis H3) also employ it when studying dynamical stability in overparameterized learning, including non-isolated minima. Furthermore, Crăciun and Ghoshdastidar (NeurIPS 2025, Theorem 1) prove the non-singularity property for specified classes of neural networks with piecewise analytic activations, under their stated assumptions and for almost all learning rates.
These works show that the assumption has been used in related theoretical studies and has been verified in important settings. It is not specific to our analysis. We state it explicitly because it is needed to ensure that the exceptional initializations form a Lebesgue-null set.
3. Technical comparison with Chen et al. (2026).
We thank the reviewer for requesting a more specific comparison. The main difference is that Chen et al. consider an isolated minimum, whereas our work considers a manifold of minima. In our setting, the Hessian eigenvalues and normal eigenvectors may change as the iterates move along the manifold. This requires several additional estimates.
First, Section 8.3 introduces the constants , , and . They are used to obtain uniform contraction bounds when the positive Hessian eigenvalues vary along the minimum manifold. In the isolated-minimum setting of Chen et al., the linearization is fixed at the considered minimum.
Second, the forward-invariance proof must account for both tangential and normal motion. Lemma 8.5 provides the necessary gradient estimates, while Lemma 8.7 establishes the remainder and derivative bounds in the local coordinates. These estimates are used to control the effect of tangential motion on the normal dynamics.
Third, Lemma 8.9 requires additional induction arguments for the derivatives of the iterates with respect to their initial coordinates. The main issue is that the normal-coordinate recurrence depends on the changing tangential coordinates. Therefore, the derivative estimates involve both components. This issue does not arise in the same form for an isolated minimum. These estimates are subsequently used in the measure-zero argument.
These are three specific differences between our proof and that of Chen et al. They explain why the isolated-minimum argument cannot be applied directly to our setting.
Learning-Rate Regimes, Non-Isolated Minima, and Experimental Scope
We thank the reviewer for the constructive comments. We clarify the assumptions, learning-rate boundaries, and experimental scope.
1. The case m=1 and the null-set assumption. The example f(x,y)=21(xy−1)2 has Hessian rank one along its minimum manifold, whereas the general theorem treats m>1. There is no contradiction: with m=1, only one positive normal direction exists, so competition among normal directions disappears and the exceptional set S in Equation (275) is empty. The same local estimates give the convergence direction or alternative convergence direction directly, excluding a zero normal coefficient and finite-time arrival at the manifold. A brief clarification has been included in the revised appendix.
The null-set preimage condition is an established non-singularity assumption for GD. Ahn et al. (ICML 2022, Assumption 1 and Theorem 1) use it to prove that exceptional initializations form a null set. Crăciun and Ghoshdastidar (NeurIPS 2025, Theorem 1) prove non-singularity for specified neural networks with piecewise analytic activations, under their hypotheses for almost all learning rates; their Lemma 10 provides a sufficient Jacobian-determinant condition. Our assumption has the same theoretical purpose: excluding a Lebesgue-null set without changing the directional conclusions elsewhere.
2. Learning-rate regimes and EoS. Theorem 4.2 already covers the larger stable-step regime
λ1(0)+λm(0)2<η<λm(0)2,
establishing the alternative convergence direction near the stability boundary from below. For η>2/λm(0), the linearized map has an unstable normal direction, so generic local convergence to the considered minimum is not guaranteed; at equality its multiplier is −1, and the strict contraction used in our proof fails. These cases lie outside our theorem. EoS motivates this research, but we do not claim to explain EoS: alternating asymptotic directions within the local stable regime do not imply EoS and do not contradict its existing theory.
3. Excluded transition value. At η=2/(λ1(0)+λm(0)), the extreme normal factors satisfy
1−ηλ1(0)=−(1−ηλm(0)).
They have equal contraction magnitudes but opposite signs: one component contracts with preserved sign, while the other contracts with alternating sign. A convergence direction may therefore fail to exist in the usual sense, yet the behavior remains intuitively tractable from its two components. This distinct boundary case is readily seen in a quadratic example; excluding it is not an obstacle to the two open-regime theorems.
4. Comparison with Chen et al. (2026). We thank the reviewer for suggesting a clearer comparison. Chen et al. study isolated minima, as acknowledged in Related Work. The extension is nontrivial: non-isolated minima entail zero tangential Hessian eigenvalues, an initialization-dependent limiting minimizer, and varying normal eigenvectors. Our local coordinates, rigorous forward invariance, and tangential/normal estimates address these new difficulties and establish both directional regimes. The result is not a restatement of Chen et al.
5. Additional NN experiments. We appreciate the suggestion, but exact-critical-rate experiments cannot reliably test the theorem's boundary cases in large networks. The critical rates depend on Hessian eigenvalues at the limiting minimizer, unknown before training; even if a candidate minimizer is obtained, numerical eigenvalue errors prevent certifying equality with a critical rate. Approximate-boundary experiments are possible but would not verify an exact-boundary claim. Our contribution is theoretical; existing low-dimensional and neural-network experiments already show representative fixed and alternating behavior and are intended as qualitative illustrations, not verification of every hypothesis.
References:
K. Ahn, J. Zhang, S. Sra, ICML 2022, Understanding the Unstable Convergence of Gradient Descent.
A. Crăciun, D. Ghoshdastidar, NeurIPS 2025, Non-Singularity of the Gradient Descent Map for Neural Networks with Piecewise Analytic Activations.
About the revised paper:
All newly added remarks and comparisons in the Related Work section are highlighted in blue for ease of identification, while minor typographical and notational errors have been corrected directly.
To avoid confusion with the original submission, the revised PDF has been uploaded to the supplementary materials; we kindly ask the reviewers to refer to this version rather than the previously submitted PDF.