acceptodds
Under review as a conference paper at ICLR 2027

Understanding Edge of Stability in Rank-1 Linear Models for Binary Classification

Abstract

Recent research in deep learning optimization reveals that many neural network architectures trained using gradient descent with practical step sizes, , exhibit an interesting phenomenon where the top eigenvalue of the Hessian of the loss function, , increases to and oscillates about the stability threshold, . The two parts of the trajectory are referred to as Progressive Sharpening and Edge of Stability. The oscillation in is accompanied by a non-monotonically decreasing training loss. In this work, we study the Edge of Stability phenomenon in a two-layer rank- linear model for the binary classification task with linearly separable data to minimize logistic loss. By capturing the core training dynamics of our model as a low-dimensional system, we rigorously prove that Edge of Stability behavior is not possible in the simplest one datapoint setting. With two datapoints, empirically we observe that, when the margin between the two points is small, Edge of Stability may occur. The oscillation in loss and sharpness becomes perpetual in the limit where the margin converges to 0. Using bifurcation and normal form analysis of the dynamical system in this asymptotic setting, we provide a sufficient condition under which the dynamical system exhibits perpetual oscillation in both loss and sharpness.

Primary area:
optimization
Keywords:
gradient descent, optimization, edge of stability, training dynamics, linear networks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_oT1jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In this paper, the authors analyze a 2-layer rank-1 linear model with logistic loss, proving that one needs at least two 2 datapoints to create a minimal example that would display the edge-of-stability phenomenon observed by Cohen et al. (2021) for real neural networks. The primary motivation for doing this seems to be in connection to Kalra et al. (2025) which showed that 1 data point is sufficient to reproduce the phenomena in the regression setting. In other words, this paper shows you need 2 data points for the 2-class classification setting. In the zero-margin limit, they also make a connection to Neimark-Sacker bifurcation to show perpetual loss and sharpness oscillations for their model.

Critical strengths

The paper is clearly written, the assumptions are presented honestly, and the exposition is easy to follow.

Critical weaknesses

I am not sure about the theoretical value of this paper on such a simplistic setting beyond "it is a sequel to Kalra et al. (2025) but now for classification and it shows that you need at least two datapoints for classification."

Questions for authors

Since my main concern for this paper is it's scope and relevance (rather than content or clarity), I ask the following two questions:

  • What insights from this theory could generalize to used-in-practice neural networks once the assumptions of the toy model are nolonger true? (That is not already covered by the current EoS literature.)
  • Alternatively, what practical recommendations about neural network practice that would not proposed by the current literature does this suggest?
2 comments

Thank you for your thoughtful comments.

Critical Weakness 1: I am not sure about the theoretical value of this paper on such a simplistic setting beyond "it is a sequel to Kalra et al. (2025) but now for classification and it shows that you need at least two datapoints for classification."

While our work does use a similar dynamical systems approach as Kalra et al., we note that an the sufficient condition we show for the two datapoint asymptotic setting shows loss and (approximate) sharpness oscillations that can be quasi-periodic or periodic with a period greater than 2. Compared to many prior regression works that focus on period-2 oscillations, we provide an analysis of a setting where looking at a two-step update would be insufficient.

Question 1: What insights from this theory could generalize to used-in-practice neural networks once the assumptions of the toy model are no longer true? (That is not already covered by the current EoS literature.)

Consider Figure 1 from our paper, where we show Edge of Stability for a 4-layer MLP. As the dynamics do not reflect a period-2 orbit, our analysis provides a potential explanation for the source of such intricate dynamics via the Neimark-Sacker bifurcation, where dynamics on the invariant curve can be aperiodic or periodic.

Question 2: Alternatively, what practical recommendations about neural network practice that would not proposed by the current literature does this suggest?

Our work does have immediate practical implications.

I will retain my score of 2. For reason that the authors' responses come down to their opinion is different than mine, but none of the concrete concerns I have are misunderstandings. In particular, my original concern is that this model is highly stylized and holds limited relevance to used-in-practice networks. The authors rebuttal states that the authors disagrees and the authors' opinion is that 4-layer MLP is sufficient to prove relevance to practical networks. I do not agree with this opinion.

Similarly, for the comparison with Kalra et al. (2025) my understanding is that the authors start with the premise Kalra et al. (2025), change the setting slightly, and prove new phenomenon in that setting. The authors rebuttal lists some of the new phenomenon they prove. Thus, we are again at simply a difference in opinion on whether this constitutes a significant/relevant contribution to the literature.

Thus, I will retain my score and leave the final decision to the AC after making my premises clear above.

Reviewer_3Ub7OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies gradient descent on a rank-one linear network with logistic loss. Under matched initialization, it proves that the training loss has at most one peak for a single data point. For two nearly aligned points with opposite labels, it observes oscillations in loss and sharpness. It then analyzes a zero-margin limiting system and gives conditions for persistent oscillations.

Critical strengths
  1. The low-dimensional formulation makes the training dynamics easier to analyze.

  2. Studying logistic loss complements existing analyses based on squared loss.

Critical weaknesses
  1. Chen and Bruna (ICML 2023) already analyze oscillations for general nonlinear objectives, explicitly including squared-loss models with sigmoid nonlinearities. The paper should explain precisely what new mechanism its logistic-loss analysis establishes beyond this framework.

  2. Ghosh et al. (ICLR 2025) analyze balancing-gap decay and characterize the subspace containing oscillations in deep matrix factorization. A mechanistic comparison with these results would help establish the contribution.

  3. The analysis imposes u=qu=q, preserving equality throughout training. It consequently does not establish whether unequal factors approach this configuration or how their imbalance affects oscillations.

  4. The appendix assumes a negative first Lyapunov coefficient and additional rotation-angle conditions. The coefficient’s sign is supported numerically. These dependencies should be explicit in Theorem 5.4.

  5. Multiple loss peaks provide only a necessary-condition. The theoretical results do not quantitatively establish that the exact largest Hessian eigenvalue remains near 2/η2/\eta; the sharpness analysis also relies on an approximation.

  6. The paper studies separable datasets but provides no results for training on fixed non-separable datasets. Whether the proposed mechanism persists in non-separable settings with finite minimizers remains unexplored.

Questions for authors
  1. What new insight does the analysis provide beyond prior work on oscillations in factorized models?

  2. Can the numerically supported assumptions behind Theorem 5.4 be proved?

  3. Do the conclusions extend to unequal factor initialization and non-separable datasets?

2 comments

Thank you for your thoughtful comments.

Question 1: What new insight does the analysis provide beyond prior work on oscillations in factorized models?

Compared to the result about sigmoid activation from Chen and Bruna (ICML 2023), we note that their work relies on an analysis of two-step updates of a system in the regression setting. In our classification setting, mainly the asymptotic result, we find that analysis of the two step updates is insufficient, especially for trajectories that can be quasi-periodic, which our sufficient condition for loss and (approximate) sharpness oscillations considers.

Compared to Ghosh et al. (ICLR 2025), we also note that their analysis focuses on period-2 orbits, despite the potential for higher period orbits introduced by the period-doubling route to chaos. We also note that the balancing gap analysis focuses on the square loss, where our setting for rank 1 models under logistic loss is different.

As already mentioned in the paper, compared to Kalra et al. (2025) (which also studies period-2 orbit in the regression setting), our setting requires two datapoints to observe Edge of Stability for logistic loss.

Compared to many prior works that study (period-2) oscillations in factorized models, we study oscillations in our asymptotic setting that can be quasi-periodic or periodic with period greater than 2. We will add this discussion to the paper.

Question 2: Can the numerically supported assumptions behind Theorem 5.4 be proved?

For assumption 1 about the first Lyapunov coefficient in our setting, we observe that the expression is rather complicated, making analysis of it unclear. Even with substitution of the conditions satisfied by the fixed point of interest, the expression for the first Lyapunov coefficient does not simplify sufficiently. Furthermore, the first Lyapunov coefficient relies on having a closed form expression for the left and right eigenvectors of the Jacobian matrix evaluated at the fixed point, which is difficult, since finding a closed form for the fixed point of interest is already intractable. That is why it is challenging to analytically bound the first Lyapunov coefficient, so we made it an assumption in the paper.

For assumption 2 regarding the rotation angle, we do have a proof strategy to show that the rotation angle is bounded between 0.115π0.115\pi and 0.205π0.205\pi. Since the derivations required for the proof are quite involved, we defer the proof of the assumption to future work.

Question 3: Do the conclusions extend to unequal factor initialization and non-separable datasets?

We ran some experiments on the rank 1 model with unequal factor initialization on an non-separable version of the Iris dataset consisting of classes versicolour and virginica (we verified the dataset is non-separable using hard-margin svm). From our experiments with different initialization and widths for the rank 1 hidden layer, we observed the loss and sharpness oscillate initially followed by convergence, exhibit chaotic dynamics, or diverge. As our results apply to sustained oscillations that are quasi-periodic or periodic, we do not see an immediate extension of our results to the unequal factor initialization and non-separable datasets setting.

Critical Weakness 4: The appendix assumes a negative first Lyapunov coefficient and additional rotation-angle conditions. The coefficient’s sign is supported numerically. These dependencies should be explicit in Theorem 5.4.

We will revise Theorem 5.4 to reflect these dependencies explicitly.

" we find that analysis of the two step updates is insufficient, especially for trajectories that can be quasi-periodic": I hardly believe this is very relevant and seen in practice. Two period orbiticity occurs most of the time and only for very small learning rate ranges, the periodicity can be higher order or chaotic. So differentiating this work and saying going beyond two periodicity really do not distinguish this work.

Furthermore, if you really look at Chen et al paper, they also show bifurcation for sigmoid activation and tanh activations. i think this is in the appendix in Chen et al for a large class of non linear activations which satifies a specific curvature property. So saying you differ from Ghosh et al because this paper analyzes logistic instead of square, is a weak argument. We already know how to tackle different non linear activations from the Chen et al paper.

Furtermore, assuming balanced quantiites make the theorem very weak, since how balancing happens as a consequence of Eos is one of the most interesting phenomenon and studing in presence of logistic loss makes it more interesting. I maintain my original score.

Reviewer_jSFHOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies Edge of Stability in a rank-1 linear classification model trained with logistic loss. It proves that repeated loss oscillations cannot occur with one datapoint and empirically demonstrates them with two highly correlated, oppositely labeled datapoints. In the zero-margin limit, bifurcation analysis provides sufficient conditions for sustained loss and sharpness oscillations. The results combine theoretical analysis with numerical simulations, with some key assumptions supported empirically.

Critical strengths
  1. A clear negative result for logistic-loss dynamics. The proof that the single-datapoint loss has at most one local maximum rules out repeated oscillations under the model’s assumptions. This provides a useful contrast with squared-loss models and clarifies how the loss function affects training dynamics.

  2. A mechanistic analysis of sustained oscillations. The low-dimensional reduction connects two-datapoint oscillations to interactions between model scale and alignment. In the zero-margin limit, bifurcation analysis strengthens this explanation through conditional theoretical guarantees, moving beyond empirical observation.

Critical weaknesses
  1. Sustained oscillations do not establish the defining sharpness behavior of EOS. The main positive result concerns oscillations in loss and sharpness, without quantitatively establishing that sharpness remains near 2/η2/\eta. Although the definition correctly treats repeated loss maxima as a necessary condition, the gap between oscillatory dynamics and EOS remains central to assessing the claimed contribution.

  2. The positive guarantee is restrictive and depends on empirically supported assumptions. The analysis applies to a zero-margin limiting system under symmetric initialization and local parameter conditions. Moreover, the Lyapunov-coefficient sign and rotation-angle assumptions underlying the sustained-oscillation result are supported numerically rather than proved. These limitations leave both the completeness of the theoretical explanation and its applicability to ordinary positive-margin classification unresolved.

Questions for authors
  1. Do your results imply a quantitative bound relating sharpness to 2/η2/\eta along the invariant curve? If not, what distinguishes the proved oscillatory regime from sustained oscillations more generally?

  2. Can Assumptions 1 and 2 in Appendix B.5.4 be established analytically or through certified numerical bounds over the stated parameter range? Please clarify which parts of the sustained-oscillation guarantee remain conditional on unverified assumptions.

4 comments

Thank you for your thoughtful comments.

Question 1: Do your results imply a quantitative bound relating sharpness to 2/\eta along the invariant curve? If not, what distinguishes the proved oscillatory regime from sustained oscillations more generally?

We are currently working on proving that the distance of the (approximate) sharpness to the stability threshold is between 0.351 and 1.305 and will add this result to the paper.

Question 2: Can Assumptions 1 and 2 in Appendix B.5.4 be established analytically or through certified numerical bounds over the stated parameter range? Please clarify which parts of the sustained-oscillation guarantee remain conditional on unverified assumptions.

For assumption 1 about the first Lyapunov coefficient in our setting, we observe that the expression is rather complicated, making analysis of it unclear. Even with substitution of the conditions satisfied by the fixed point of interest, the expression for the first Lyapunov coefficient does not simplify sufficiently. Furthermore, the first Lyapunov coefficient relies on having a closed form expression for the left and right eigenvectors of the Jacobian matrix evaluated at the fixed point, which is difficult, since finding a closed form for the fixed point of interest is already intractable. That is why it is challenging to analytically bound the first Lyapunov coefficient, so we made it an assumption in the paper.

For assumption 2 regarding the rotation angle, we do have a proof strategy to show that the rotation angle is bounded between 0.115π0.115\pi and 0.205π0.205\pi. Since the derivations required for the proof are quite involved, we defer the proof of the assumption to future work.

For the sustained oscillation, assumption 1 serves to show that the invariant curve generated from Neimark-Sacker bifurcation is attracting, so if we initialize sufficiently close to it, the dynamical system will converge to the invariant curve. We then show sustained oscillation once the dynamical system converges to the invariant curve. Assumption 2 serves to show how the error for our approximation of the dynamics on the invariant curve as a rotation by some fixed rotation angle to show the sustained oscillation is quite small.

Thank you for the thoughtful response and your efforts to strengthen the theoretical results. The proposed sharpness bound is encouraging, and I appreciate the clarification of the assumptions. However, the bound and verification of the key assumptions remain pending, so my main concerns are not yet fully addressed. I will therefore maintain my current score.

Thank you for engaging with our submission. We wanted to update you on the bound for the distance between the stability threshold and our sharpness approximation mentioned in our earlier response. We have completed the proof, showing that the distance is bounded between 0.351 and 1.305 for the sufficient condition described in Theorem 5.4 of the paper. We are adding the full derivation to the appendix of the paper.

In short, since the invariant curve for our sufficient condition is contained within the fixed point rectangle described in Theorem 5.2, we prove the distance between sharpness approximation and stability threshold is bounded for any point in the rectangle, so it applies to points on the invariant curve.

Thank you for the update and your efforts to complete the proof. Establishing a uniform bound over the rectangle containing the invariant curve is a meaningful improvement and addresses part of my first concern. However, the connection to the true sharpness and the verification of Assumptions 1 and 2 remain unresolved. While I appreciate this progress, I will maintain my current score based on these remaining concerns.

Reviewer_bBfgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the Edge of Stability phenomenon in two-layer rank-1 linear neural networks trained with gradient descent and logistic loss. The authors reduce the training dynamics to low-dimensional discrete-time systems and analyze their stability properties. For a single training datapoint, they prove that the loss trajectory cannot exhibit multiple local maxima. For two datapoints, numerical experiments demonstrate oscillatory behavior in loss and sharpness. The authors further consider an asymptotic setting where the classification margin approaches zero and use bifurcation and normal form analysis to derive conditions under which perpetual oscillations occur.

Critical strengths

The paper provides an interesting theoretical perspective on the Edge of Stability phenomenon for classification problems, where the existing literature is less developed than for regression. Reducing the gradient descent dynamics to low-dimensional systems enables a more explicit analysis of the mechanisms responsible for oscillatory behavior.

The distinction between the one-datapoint and two-datapoint settings is a useful contribution. The single-datapoint result is supported by a phase-space argument, while the asymptotic two-datapoint analysis connects the observed oscillations to a Neimark-Sacker bifurcation. This provides a potentially useful connection between neural network optimization and discrete-time dynamical systems.

Critical weaknesses

The main theoretical result relies on restrictive assumptions that are not fully established for the proposed model. In particular, the perpetual oscillation result requires a negative first Lyapunov coefficient, which is assumed and supported numerically rather than proved over the considered parameter range. The result also depends on additional assumptions regarding the local coordinate transformation and rotation dynamics. Consequently, the theoretical guarantees are more conditional than the main presentation suggests. Furthermore, the proof primarily analyzes an approximation of the Hessian sharpness, and it is unclear whether the established oscillations necessarily extend to the exact largest Hessian eigenvalue.

The connection to Edge of Stability in more general classification settings remains limited. The principal bifurcation result applies to the limiting case where the margin approaches zero, rather than to a fixed dataset with a positive classification margin. The analysis also assumes identical initialization of two parameter vectors and focuses on one or two training examples. Moreover, the formal condition used to identify Edge of Stability is based on multiple local maxima of the loss, without explicitly requiring the sharpness to remain near the stability threshold. Therefore, the theoretical results do not fully characterize the Edge of Stability phenomenon as commonly understood.

Questions for authors

Can the authors establish the negativity of the first Lyapunov coefficient analytically, at least for a nontrivial subset of the considered parameter range? Since this assumption is central to the existence of the stable invariant curve, establishing it would substantially strengthen the main result.

Does Theorem 5.4 guarantee perpetual oscillations of the exact Hessian sharpness, or only of the approximation used in the proof? Can the authors additionally establish that the sharpness remains close to the stability threshold (2/\eta)?

To what extent do the asymptotic results extend to datasets with a fixed positive margin? In particular, is there a theoretical or numerical characterization of how the oscillations change as the margin increases away from zero?

1 comment

Thank you for your thoughtful comments.

Question 1: Can the authors establish the negativity of the first Lyapunov coefficient analytically, at least for a nontrivial subset of the considered parameter range? Since this assumption is central to the existence of the stable invariant curve, establishing it would substantially strengthen the main result.

For the first Lyapunov coefficient in our setting, we observe that the expression is rather complicated, making analysis of it unclear. Even with substitution of the conditions satisfied by the fixed point of interest, the expression for the first Lyapunov coefficient does not simplify sufficiently. Furthermore, the first Lyapunov coefficient relies on having a closed form expression for the left and right eigenvectors of the Jacobian matrix evaluated at the fixed point, which is difficult, since finding a closed form for the fixed point of interest is already intractable. That is why it is challenging to analytically bound the first Lyapunov coefficient, so we made it an assumption in the paper.

Question 2: Does Theorem 5.4 guarantee perpetual oscillations of the exact Hessian sharpness, or only of the approximation used in the proof? Can the authors additionally establish that the sharpness remains close to the stability threshold (2/η2 / \eta) ?

Our claim about the oscillations applies to the approximation of the sharpness used in the proof. We are currently working on proving that the distance of the (approximate) sharpness to the stability threshold is between 0.351 and 1.305 and will add this result to the paper.

Question 3:To what extent do the asymptotic results extend to datasets with a fixed positive margin? In particular, is there a theoretical or numerical characterization of how the oscillations change as the margin increases away from zero?

The asymptotic result is used to illustrate how oscillations become more prominent for datasets with margin quite small. From our empirical observations of the two datapoint setting with positive margin, we note many cases where the dynamical system escapes the Edge of Stability regime in finite time. We attribute this to the m_t component of the two-datapoint system, which grows monotonically. A theoretical characterization of escaping the Edge of Stability regime in finite time is already provided by Cai et al. (2024).

[1] Yuhang Cai, Jingfeng Wu, Song Mei, Michael Lindsey, and Peter L. Bartlett. Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimization. NeurIPS 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.