On the Asymptotic Behavior of Gradient Descent

Recent work has studied the asymptotic behavior of gradient descent (GD) near isolated minima, including the existence of convergence directions. In this work, it is investigated whether analogous directional behavior persists near non-isolated minima, where the minimum set locally forms a regular surface. This setting is particularly relevant to modern overparameterized learning models, where non-isolated minima arise naturally and the structure of the solution set can shape the local optimization dynamics. The non-isolated setting introduces an additional difficulty because the limiting minimizer is not fixed in advance, and the relevant Hessian eigendirections may vary along the minimum set. Under the stated local regularity assumptions, a forward-invariant neighborhood is first established, within which GD converges to a specific minimizer and satisfies a quantitative descent estimate. The asymptotic direction of approach is then characterized. Depending on the learning rate, the GD trajectory admits either a limiting convergence direction or an alternating convergence direction. These results extend the directional convergence phenomenon from isolated minima to non-isolated minima and provide a local description of the long-time behavior of GD near a regular surface of minima.