acceptodds
Under review as a conference paper at ICLR 2027

On the Asymptotic Behavior of Gradient Descent

Abstract

Recent work has studied the asymptotic behavior of gradient descent (GD) near isolated minima, including the existence of convergence directions. In this work, it is investigated whether analogous directional behavior persists near non-isolated minima, where the minimum set locally forms a regular surface. This setting is particularly relevant to modern overparameterized learning models, where non-isolated minima arise naturally and the structure of the solution set can shape the local optimization dynamics. The non-isolated setting introduces an additional difficulty because the limiting minimizer is not fixed in advance, and the relevant Hessian eigendirections may vary along the minimum set. Under the stated local regularity assumptions, a forward-invariant neighborhood is first established, within which GD converges to a specific minimizer and satisfies a quantitative descent estimate. The asymptotic direction of approach is then characterized. Depending on the learning rate, the GD trajectory admits either a limiting convergence direction or an alternating convergence direction. These results extend the directional convergence phenomenon from isolated minima to non-isolated minima and provide a local description of the long-time behavior of GD near a regular surface of minima.

Primary area:
optimization
Keywords:
Optimization, Gradient Descent, Convergence Direction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6gf7OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies constant-step gradient descent (GD) near a non-isolated minimum, which extends Chen et al.(2026) from isolated minima to the more general setting of smooth manifold of minima. Under some assumptions on the regularity of minima set M and function class, the paper establishes two main results when restricting step size η<2/λm(0)\eta<2/\lambda_m(0): (i) Theorem 4.1 shows there exists a neighborhood around 0 (which is assumed to be a minima) such that the GD iterates never escape once reaching it, in particular if GD is initialized in this region; also the funcion is strictly decreasing along the GD trajectory. (ii) Theorem 4.2 dicusses convergence direction of GD. It shows that as long as initialized within the region given in Theorem 4.1 and η<2/λm(0)\eta<2/\lambda_m(0) (although the current statement excludes the choice 2/(λ1(0)+λm(0))2/(\lambda_1(0)+\lambda_m(0))), the GD iterates converges to a minima in M. Additionally, the convergence direction dkd_k also converges if η<2/(λ1(0)+λm(0))\eta<2/(\lambda_1(0)+\lambda_m(0)), and alternative (−1)kdk(-1)^k d_k converges otherwise. A key novelty in the analysis is introduing the mapping G(u,v)G(u, v) in Eq. (27), which uses coordinates (u,v)(u, v) correpsond to motion along and orthogonal to M, respectively. The theoretical results are validated empirically on both simple 2-dim functions and Neural Networks.

Critical strengths
  1. The question is clean and natural. In real applications such as overparameterized models, minima are typically non-isolated. Therefore, the extension from isolated minima (established in Chen et al. 2026) to non-isolated ones is highly motivated and meaningful. The two main results also provide interesting insights on how GD iterates evolve near the minima set, which seem new in the literature.

  2. The presentation is well-organized and easy to follow. I particularly like the organization that the authors start with a simple 2-d example before moving into the main theoretical results. This significantly helps understand the motivation and main results visually to readers not familiar with the topic.

Critical weaknesses
  1. Regarding Assumptions. The assumptions seem relatively strong, or at least lack sufficient explanations. In particular, I an concerned with (17) as it needs rank>1rank>1. This matters because in the motivating example 0.5(xy−1)20.5(xy-1)^2 and also the first part of experiments, the rank of Hessian at a minima xy=1xy=1 seems only have rank equals 1, which contradicts (17) directly. Additionally, some others such as Eq (18) and the assumption that the GD map pulls back Lebesgue null sets to null sets also need to be justified or relaxed.

  2. Scope of the main results. Theorems 4.1 and 4.2 focus on the stable region η<2/λm\eta<2/\lambda_m. Therefore the behavior of larger step sizes η≥2/λm\eta\ge 2/\lambda_m or at least around 2/λm2/\lambda_m is missing. This somewhat contradicts the motivation regarding the edge of stability (as stated in the introduction), because EoS considers rather large step sizes and the loss is non-monotone, while within η<2/λm\eta<2/\lambda_m, Theorem 4.1 explicitly shows that the loss is monotonically decreasing. Also, the convergence direction result in Theorem 4.2 excludes the choice 2/(λ1(0)+λm(0))2/(\lambda_1(0)+\lambda_m(0)), which should be included or explained if this choice can not be handled easily.

  3. Comparison with Chen et al. (2026). This paper extend to non-isolated cases, which separates from the isolated mimima considered in Chen et al. (2026). Therefore, it would be better if a discussion of comparison with Chen et al. (2026) is included after the main results, which should state precisely how their analysis fail in the non-isolated case and how they are repaired in this work. This would help highlight the technical contributions.

Questions for authors

See my comments in the weakness discussion. In particular, I would like to ask the following main questions:

  1. Does the proof extend to m=1m=1 in (17), so that the motivating example in Section 3 and experiments in Section 5.1 better fit into the scope of the main results?

  2. Can you justify some of the assumptions properly? e.g., relax or explain its fairness.

  3. Can the results or discussions of the choice η=2/λm(0)\eta=2/\lambda_m(0) be included, as complement to existing Theorems 4.1 and 4.2? Please also include the result for η=2/(λ1(0)+λm(0))\eta=2/(\lambda_1(0)+\lambda_m(0)) in Theorem 4.2 if possible.

  4. For the NN tests in Section 5.2 or some other settings, can you also test for η=2/λm(0)\eta=2/\lambda_m(0) and η=2/(λ1(0)+λm(0))\eta=2/(\lambda_1(0)+\lambda_m(0)) to see what the behavior looks like?

5 comments
AuthorsOpenReview ↗

Learning-Rate Regimes, Non-Isolated Minima, and Experimental Scope

We thank the reviewer for the constructive comments. We clarify the assumptions, learning-rate boundaries, and experimental scope.

1. The case m=1m=1 and the null-set assumption. The example f(x,y)=12(xy−1)2f(x,y)=\frac12(xy-1)^2 has Hessian rank one along its minimum manifold, whereas the general theorem treats m>1m > 1. There is no contradiction: with m=1m=1, only one positive normal direction exists, so competition among normal directions disappears and the exceptional set S~\widetilde S in Equation (275) is empty. The same local estimates give the convergence direction or alternative convergence direction directly, excluding a zero normal coefficient and finite-time arrival at the manifold. A brief clarification has been included in the revised appendix.

The null-set preimage condition is an established non-singularity assumption for GD. Ahn et al. (ICML 2022, Assumption 1 and Theorem 1) use it to prove that exceptional initializations form a null set. Crăciun and Ghoshdastidar (NeurIPS 2025, Theorem 1) prove non-singularity for specified neural networks with piecewise analytic activations, under their hypotheses for almost all learning rates; their Lemma 10 provides a sufficient Jacobian-determinant condition. Our assumption has the same theoretical purpose: excluding a Lebesgue-null set without changing the directional conclusions elsewhere.

2. Learning-rate regimes and EoS. Theorem 4.2 already covers the larger stable-step regime

2λ1(0)+λm(0)<η<2λm(0)\frac{2}{\lambda_1(\boldsymbol{0})+\lambda_m(\boldsymbol{0})} < \eta < \frac{2}{\lambda_m(\boldsymbol{0})},

establishing the alternative convergence direction near the stability boundary from below. For η>2/λm(0)\eta > 2/\lambda_m(\boldsymbol{0}), the linearized map has an unstable normal direction, so generic local convergence to the considered minimum is not guaranteed; at equality its multiplier is −1-1, and the strict contraction used in our proof fails. These cases lie outside our theorem. EoS motivates this research, but we do not claim to explain EoS: alternating asymptotic directions within the local stable regime do not imply EoS and do not contradict its existing theory.

3. Excluded transition value. At η=2/(λ1(0)+λm(0))\eta=2/(\lambda_1(\boldsymbol{0})+\lambda_m(\boldsymbol{0})), the extreme normal factors satisfy

1−ηλ1(0)=−(1−ηλm(0))1-\eta\lambda_1(\boldsymbol{0})=-(1-\eta\lambda_m(\boldsymbol{0})).

They have equal contraction magnitudes but opposite signs: one component contracts with preserved sign, while the other contracts with alternating sign. A convergence direction may therefore fail to exist in the usual sense, yet the behavior remains intuitively tractable from its two components. This distinct boundary case is readily seen in a quadratic example; excluding it is not an obstacle to the two open-regime theorems.

4. Comparison with Chen et al. (2026). We thank the reviewer for suggesting a clearer comparison. Chen et al. study isolated minima, as acknowledged in Related Work. The extension is nontrivial: non-isolated minima entail zero tangential Hessian eigenvalues, an initialization-dependent limiting minimizer, and varying normal eigenvectors. Our local coordinates, rigorous forward invariance, and tangential/normal estimates address these new difficulties and establish both directional regimes. The result is not a restatement of Chen et al.

5. Additional NN experiments. We appreciate the suggestion, but exact-critical-rate experiments cannot reliably test the theorem's boundary cases in large networks. The critical rates depend on Hessian eigenvalues at the limiting minimizer, unknown before training; even if a candidate minimizer is obtained, numerical eigenvalue errors prevent certifying equality with a critical rate. Approximate-boundary experiments are possible but would not verify an exact-boundary claim. Our contribution is theoretical; existing low-dimensional and neural-network experiments already show representative fixed and alternating behavior and are intended as qualitative illustrations, not verification of every hypothesis.

References:

K. Ahn, J. Zhang, S. Sra, ICML 2022, Understanding the Unstable Convergence of Gradient Descent.

A. Crăciun, D. Ghoshdastidar, NeurIPS 2025, Non-Singularity of the Gradient Descent Map for Neural Networks with Piecewise Analytic Activations.

About the revised paper:

All newly added remarks and comparisons in the Related Work section are highlighted in blue for ease of identification, while minor typographical and notational errors have been corrected directly.

To avoid confusion with the original submission, the revised PDF has been uploaded to the supplementary materials; we kindly ask the reviewers to refer to this version rather than the previously submitted PDF.

I appreciate the response and revision from the authors. Remark 8.16 reasonably addresses the m=1m=1 case away from the zero-multiplier rate, and I accept the explanations for excluding the transition and stability-boundary rates. These clarifications are helpful, but some of my main concerns remain unsolved.

In particular, the response does not explain why Eq. (18) requires all positive eigenvalues to be distinct, or clarify when the structural assumptions hold in the neural-network examples. The non-singularity references are relevant, but their applicability still lacks explanation. Additionally, the comparison with Chen et al. seems only high level. As a theory-focused paper, it is important and helpful to identify the precise technical novelties, e.g., key proof steps that are tailed for non-isolated minima but not included in Chen et al. setting.

For experiments, I understand and appreciate the difficulty of certifying an exact critical rate in a large network. Nevertheless, a small controlled example with m>1m>1, known Hessian eigenvalues, and measurements of the predicted eigendirection would still be interesting.

AuthorsOpenReview ↗

Follow-up Response with Technical Clarifications and Additional Controlled Experiments

We thank the reviewer for the constructive follow-up and for accepting our previous clarifications concerning the rank-one case and the excluded learning rates. We appreciate the opportunity to address the remaining questions.

1. Distinct positive Hessian eigenvalues and the neural-network examples.

The distinctness assumption in Eq. (18) is mainly required by our proof. As shown in Section 8.2 and Lemma 8.3, distinct positive Hessian eigenvalues guarantee a smooth local frame of normal eigenvectors along the minimum manifold. This frame is used to construct the coordinate transformation. Its smoothness is also needed for the derivative estimates in Lemma 8.7. Moreover, the separation of positive eigenvalues allows uniform contraction estimates as the Hessian varies along the manifold. Thus, the assumption has a specific technical role in our analysis. It is a sufficient condition for the present proof, but we do not claim it is necessary; similar directional behavior may hold under weaker spectral assumptions.

Regarding neural networks, several empirical studies provide examples of separated positive Hessian eigenvalues. Ghorbani et al. (ICML 2019), Papyan (JMLR 2020), and Granziol et al. (JMLR 2022) investigate Hessian spectra and report distinct positive spectral outliers in neural-network training. These observations support the relevance of our spectral setting, although they do not establish that all positive Hessian eigenvalues are distinct in arbitrary neural networks. We consider it plausible that the spectral condition also holds for the networks studied here, but to our knowledge, rigorously checking every positive Hessian eigenvalue at their limiting minimizers would be difficult and has not been attempted.

The three neural-network architectures in our experiments were selected to illustrate typical directional behavior consistent with our results. We do not claim that they satisfy every assumption of the theorem. The experiments provide qualitative illustrations, while the theoretical conclusions are established under the stated assumptions.

2. The null-set preimage assumption.

We thank the reviewer for the continued discussion of this assumption. As explained in our previous response, it is an established non-singularity condition in GD theory.

Ahn et al. (ICML 2022, Assumption 1) already use this condition in their analysis of GD. Chemnitz and Engel (JMLR 2025, Hypothesis H3) also employ it when studying dynamical stability in overparameterized learning, including non-isolated minima. Furthermore, Crăciun and Ghoshdastidar (NeurIPS 2025, Theorem 1) prove the non-singularity property for specified classes of neural networks with piecewise analytic activations, under their stated assumptions and for almost all learning rates.

These works show that the assumption has been used in related theoretical studies and has been verified in important settings. It is not specific to our analysis. We state it explicitly because it is needed to ensure that the exceptional initializations form a Lebesgue-null set.

3. Technical comparison with Chen et al. (2026).

We thank the reviewer for requesting a more specific comparison. The main difference is that Chen et al. consider an isolated minimum, whereas our work considers a manifold of minima. In our setting, the Hessian eigenvalues and normal eigenvectors may change as the iterates move along the manifold. This requires several additional estimates.

First, Section 8.3 introduces the constants a−a_-, a+a_+, and bb. They are used to obtain uniform contraction bounds when the positive Hessian eigenvalues vary along the minimum manifold. In the isolated-minimum setting of Chen et al., the linearization is fixed at the considered minimum.

Second, the forward-invariance proof must account for both tangential and normal motion. Lemma 8.5 provides the necessary gradient estimates, while Lemma 8.7 establishes the remainder and derivative bounds in the local coordinates. These estimates are used to control the effect of tangential motion on the normal dynamics.

Third, Lemma 8.9 requires additional induction arguments for the derivatives of the iterates with respect to their initial coordinates. The main issue is that the normal-coordinate recurrence depends on the changing tangential coordinates. Therefore, the derivative estimates involve both components. This issue does not arise in the same form for an isolated minimum. These estimates are subsequently used in the measure-zero argument.

These are three specific differences between our proof and that of Chen et al. They explain why the isolated-minimum argument cannot be applied directly to our setting.

Reviewer_fYdHOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper extends the convergence-direction results of Chen et al. (2026) from isolated minima to a smooth manifold of minima. Under local Morse–Bott geometry and additional regularity assumptions, it establishes convergence of constant-step GD to a single minimizer and characterizes the asymptotic direction for almost every nearby initialization. Smaller steps yield alignment with the least-curved normal direction; larger steps yield alternating alignment with the most-curved direction. Both regimes remain below the local stability threshold.

Critical strengths
  • The derived extension to non-isolated minima requires additional analysis: the limiting minimizer depends on initialization, eigendirections vary along the manifold, and tangential drift must be controlled.

  • The theorem gives a precise asymptotic description of the trajectory near the minimum manifold, identifying the dominant normal eigendirection and the step-size threshold separating fixed and alternating directional convergence.

Critical weaknesses

---------- After rebuttal -----------

Thank you for the response and revisions, which address my proof concerns and clarify the assumptions. I am still deliberating whether the additional contribution beyond existing local convergence theory and Chen et al. result is sufficient for publication. I am raising my soundness, informativeness scores and my overall rating, while retaining reservations about the significance of the contribution.

---------- Before rebuttal -----------

  • The advance over existing theory needs clearer justification. Morse–Bott geometry implies local PL, and Rebjock and Boumal  already establish local linear convergence of constant-step GD to a single minimum. The additional content of Theorem 4.1 is therefore unclear. Theorem 4.2 extends Chen et al.’s directional result to non-isolated minima, which requires additional analysis, but assumes (C^\infty) smoothness and simplicity of the entire positive spectrum; Chen et al. require (C^3) smoothness and simplicity only of the extreme eigenvalues. The significance of this extension should be assessed against these additional restrictions. The comparison with MacDonald et al. should also distinguish weaker assumptions from the narrower conclusions retained, since the present results remain strictly below the local stability threshold.

  • The significance of the directional refinement is insufficiently developed. Local geometry is useful for understanding convergence near non-isolated minima, but it is unclear what the additional directional characterization enables: an algorithmic improvement, a conclusion about the selected minimizer, or an explanation of training behavior beyond the deterministic local model. The additional spectral and null-set assumptions are also not established for the motivating neural-network models. The experiments provide limited support.

Questions for authors
  1. What does Theorem 4.1 add to existing local PL convergence results? What is the main advance over Chen et al. in Theorem 4.2, and which conclusions of MacDonald et al. are retained?

  2. Why are (C^\infty) smoothness and simplicity of the entire positive spectrum needed, compared with Chen et al.’s (C^3) and simple extreme eigenvalues? Are these essential to the conclusion or requirements of the proof? Can the null-set preimage assumption be verified for the motivating models?

  3. Does the directional characterization have an algorithmic consequence or explain an observed training phenomenon? Can its eigendirection and transition threshold be tested on a nontrivial example where the assumptions are verified?

  4. In Appendix 8.9, where is differentiability of (\sigma) with respect to (u) established? The differentiated-series estimates appear to address only the initial normal variables. Also, should (296) require ((u,0,z)\in h(B_n(0,\varepsilon_2))), so that (h^{-1}) in (298) is defined? I think that the appendix presentation and writing should be improved - since the paper’s contribution is theoretical, the proofs should be spotless as possible; also it seems that there are some index issues there.

2 comments
AuthorsOpenReview ↗

Theoretical Significance, Assumptions, and Proof Clarifications

We thank the reviewer for the detailed assessment. We clarify the contributions, assumptions, and scope.

1. Contributions and related work. Local convergence near Morse–Bott minima has been studied, including by Rebjock and Boumal. Nevertheless, Theorem 4.1 rigorously constructs a forward-invariant neighborhood for GD under our assumptions and bounds tangential and normal motion, ensuring that the local estimates remain valid along the trajectory. This explicit construction is an important contribution and provides the foundation for the directional results.

Theorem 4.2 characterizes the asymptotic geometry of GD, rather than merely proving convergence: smaller learning rates select the normal direction of the smallest positive Hessian eigenvalue, whereas larger learning rates within the stable regime yield an alternative convergence direction associated with the largest positive eigenvalue. This identifies a learning-rate-dependent transition between two distinct forms of directional behavior.

Unlike Chen et al. (2026), our theorem concerns non-isolated minima: the Hessian has neutral tangential directions, the limiting minimizer depends on initialization, and the normal eigenvectors vary along the minimum manifold. These effects require new estimates controlling tangential drift and normal contraction; the isolated-minimum proof does not directly apply.

MacDonald et al. (2025) study convergence rates and EoS-related dynamics for specialized overparameterized least squares under structural assumptions, including orthogonal stability. Our objective and conclusions differ. In particular, our forward-invariance result does not require the invariance of a prescribed normal line, offering a less restrictive alternative in that geometric respect. The results are complementary, not contradictory.

2. Assumptions. The C∞C^\infty assumption is for notational convenience; C5C^5 suffices for the derivatives used. This is explained in the revised appendix. Simplicity of the positive Hessian eigenvalues provides a smoothly varying local eigenvector frame for componentwise estimates; we do not claim it is necessary for the phenomenon.

The null-set preimage assumption is an established GD-map non-singularity condition, not peculiar to our analysis. Ahn et al. (ICML 2022, Assumption 1 and Theorem 1) use it to exclude exceptional initializations of Lebesgue measure zero. Crăciun and Ghoshdastidar (NeurIPS 2025, Theorem 1) prove it for neural networks with specified fully connected, convolutional, or softmax attention layers and piecewise analytic activations, under their stated hypotheses for almost all learning rates. Their Lemma 10 also gives the sufficient condition that the Jacobian determinant vanishes only on a null set. Thus, the assumption is justified in important settings; here it ensures that the exceptional initializations in the directional analysis are Lebesgue-null, without affecting conclusions outside that set.

3. Significance and experiments. The goal is a rigorous qualitative characterization of GD trajectories, not an optimizer design. Learning-rate-dependent selection of fixed versus alternative convergence directions is meaningful in itself and may inform later studies or algorithms. The experiments are supporting illustrations: low-dimensional examples exhibit normal contraction with small tangential drift, and neural-network experiments show representative aligned and alternating updates. They are not claimed to verify all hypotheses in arbitrary networks. Since the principal contribution is theoretical, no additional algorithmic development or experiments are needed to establish it.

4. Appendix 8.9. We thank the reviewer for these technical observations. The differentiability of σ\sigma in initial tangential variables follows by the same method used for normal variables in Lemmas 8.8–8.10: tangential derivatives remain uniformly bounded, normal derivatives decay geometrically, and a~2<a−\widetilde a^2 < a_- guarantees uniform convergence of the differentiated series. Hence σ∈C1\sigma\in C^1. A short justification appears in the revised appendix; no new assumption or proof strategy is needed.

The revised appendix also corrects Equation (298)'s domain by requiring (u,0,z)∈h(Bn(0,ε2))(\boldsymbol{u},0,\boldsymbol{z})\in h(B_n(\boldsymbol{0},\varepsilon_2)), making h−1h^{-1} well-defined, and fixes the identified indexing and notation errors. These local corrections preserve the estimates and main conclusions.

References:

K. Ahn, J. Zhang, S. Sra, ICML 2022, Understanding the Unstable Convergence of Gradient Descent.

A. Crăciun, D. Ghoshdastidar, NeurIPS 2025, Non-Singularity of the Gradient Descent Map for Neural Networks with Piecewise Analytic Activations.

About the revised paper:

All newly added remarks and comparisons in the Related Work section are highlighted in blue for ease of identification, while minor typographical and notational errors have been corrected directly.

To avoid confusion with the original submission, the revised PDF has been uploaded to the supplementary materials; we kindly ask the reviewers to refer to this version rather than the previously submitted PDF.

Reviewer_ZBwjOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the eventual alignment of the gradient descent iterates near a manifold of minimizer. It show that either one direction or one alternating direction emerges, depending on the magnitude of the step-size. Then the findings are validated on real-world examples.

Critical strengths
  • Interesting results
Critical weaknesses

Missing literature

This work is missing this important related paper: [1] https://arxiv.org/pdf/2604.11213

[1] studies the same phenomenon as the paper under review: [1] also shows that either one direction or one alternating direction emerges, with the same criterion the magnitude of the step-size. The only difference is that [1] only considers one isolated minimum while the paper under review considers a manifold. Could the authors compare their proof techniques to [1] and explain how the manifold case differs?

Correctness

There seems to be a few issues with the proof, in particular:

  • Equation (298): the inverse of hh is applied to WW but there is not guarantees that h−1h^{-1} is defined on WW, only hh itself is guaranteed to be defined on WW by construction.

Proof length

I am concerned that the proof is somewhat reinventing some standard dynamical systems tools, I would encourage the author to explore whether stable manifold theorems would allow them to shorten their proofs.

Questions for authors

See weaknesses above

2 comments
AuthorsOpenReview ↗

Clarification of Related Work, Analytical Contributions, and Technical Corrections

We thank the reviewer for the comments and for identifying relevant related work.

1. Relationship to Bégout et al. (2026) [1]. The differences in setting, results, and proof strategy are substantial.

First, the mathematical settings differ. Assumptions 2.1–2.2 of [1] require an isolated nondegenerate critical point with simple Hessian spectrum and local positive definiteness at a minimizer. Our paper concerns a regular manifold of non-isolated minima, where the Hessian has a nontrivial kernel in tangential directions. Consequently, the limiting minimizer depends on initialization, positive Hessian eigenvectors may vary along the manifold, and tangential motion must be controlled. These features are excluded by [1]; extending its analysis to our setting is nontrivial.

Second, the formal directional results differ. Theorem 3.2 of [1] proves directional convergence for gradient flow near isolated minima. For discrete GD, Lemma 4.2 treats the linearized system under a small-step condition, while Theorem 4.4(iii) proves nonlinear directional convergence for γ<2/(λd−i+1(x∗)+λd(x∗))\gamma < 2/(\lambda_{d-i+1}(x^{\ast})+\lambda_d(x^{\ast})). Its generic fixed-direction result thus corresponds to γ<2/(λ1(x∗)+λd(x∗))\gamma < 2/(\lambda_1(x^{\ast})+\lambda_d(x^{\ast})). Remark 4.5 discusses larger learning rates, but its explicit asymptotic calculations concern quadratic dynamics, followed by a suggestion that analogous arguments may apply to nonlinear dynamics. No separate nonlinear alternating-direction theorem is stated. Our Theorem 4.2 formally establishes both the convergence direction and the alternative convergence direction near a manifold of minima in the two distinct stable learning-rate regimes.

Moreover, Theorem 4.4(i) of [1] claims a local C1C^1 conjugacy for all γ∈(0,2/λd(x∗))\gamma\in(0,2/\lambda_d(x^{\ast})), including singular choices where 1−γλj(x∗)=01-\gamma\lambda_j(x^{\ast})=0. Such a conjugacy does not hold in general at these values without additional assumptions. We recognize [1]'s distinct contributions concerning gradient extremals, talwegs, and alignment rates, but its stated results do not cover our non-isolated setting.

Third, the proof strategies differ. The analysis in [1] employs local conjugacy near hyperbolic fixed points. Our work develops a direct analytical proof using a local coordinate transformation, a rigorously constructed forward-invariant neighborhood, and explicit estimates controlling tangential and normal dynamics. These are crucial for a manifold of fixed points with neutral tangential directions. Therefore, [1] does not subsume our assumptions, conclusions, or analytical framework.

2. Technical issue concerning Equation (298). We thank the reviewer for identifying this domain issue. In the revised supplementary material, WW is defined by (u,0,z)∈h(Bn(0,ε2))(\boldsymbol{u},0,\boldsymbol{z})\in h(B_n(\boldsymbol{0},\varepsilon_2)), so h−1h^{-1} is well-defined on the required domain. The correction preserves the subsequent Sard-based measure-zero argument and does not affect the statements or conclusions of the main theorems.

3. Proof length and stable manifold theory. We appreciate the suggestion and agree that dynamical systems tools may provide alternative approaches to local convergence. Nevertheless, the GD map has neutral tangential directions along a non-isolated minimum manifold, so standard results for hyperbolic fixed points are not directly applicable. More specialized invariant-manifold techniques may help with local convergence, but obtaining the precise asymptotic convergence direction and excluding exceptional initializations still requires additional arguments. To our knowledge, these directional conclusions do not follow directly from a standard stable manifold theorem.

Our self-contained analytical approach gives explicit estimates for tangential and normal dynamics, forward invariance, and the measure-zero argument. Such a direct proof is worthwhile both for the present directional characterization and for understanding the GD dynamics. We thank the reviewer for suggesting alternative techniques, while maintaining that our approach makes an independent mathematical contribution.

About the revised paper: All newly added remarks and comparisons in the Related Work section are highlighted in blue for ease of identification, while minor typographical and notational errors have been corrected directly.

To avoid confusion with the original submission, the revised PDF has been uploaded to the supplementary materials; we kindly ask the reviewers to refer to this version rather than the previously submitted PDF.

Thank you for your answers, Begout et al. should be cited as well.

My wider concerns about proof quality, which is paramount for such theoretical work, remain. I am nevertheless look forward to discussing with the other reviewers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.