acceptodds
Under review as a conference paper at ICLR 2027

Neural Flip Laws and Tangent Stabilization

Abstract

A positive flip coefficient supports a local period-two branch of projected gradient descent (GD), while tangent motion can suppress oscillations in full GD. We quantify this distinction in wide additive networks. Exact empirical functionals determine the projected flip coefficient and the tangent-drift coefficient; two Softplus initialization laws share the same population tangent kernel but produce opposite flip signs. For one residual, balanced i.i.d. initialization with positive tangent drift, and a near-critical step calibrated by the initial kernel, a single GD trajectory tracks an explicit amplitude–sharpness system with error over steps, converges to its own interpolant, and becomes subcritical under either flip sign. The theorem covers unbounded initializations through a finite-eighth-moment derivative envelope. Experiments from random initialization verify the trajectory law, while endpoint interventions identify the tangent sharpness direction that produces damping.

Primary area:
learning theory
Keywords:
edge of stability, gradient descent dynamics, neural tangent kernel, flip bifurcation, tangent stabilization, implicit regularization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_adfTOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the relationship between local flip bifurcations and tangent self-stabilization in gradient descent near the Edge of Stability (EoS). The authors derive exact neural functionals characterizing the flip coefficient and tangent sharpness drift, and construct two Softplus initialization distributions with identical population neural tangent kernels but opposite flip signs. Their main theoretical result establishes an amplitude–sharpness approximation for near-critical discrete GD in wide additive networks with a single residual, including a quantitative tracking error and convergence to a subcritical interpolating solution. Experiments on synthetic networks and deeper models are presented to validate the predicted stabilization mechanism.

Critical strengths
  1. Quantitative theoretical analysis. The paper derives exact expressions for the flip and tangent-drift coefficients and provides a width-dependent approximation of discrete GD dynamics. The convergence analysis goes beyond a purely heuristic continuous-time model. 

  2. Mechanism-oriented experimental validation. The projected-versus-full GD comparisons and tangent-direction interventions provide useful evidence for the proposed stabilization mechanism. The experiments are designed to investigate the theoretical predictions rather than merely demonstrate empirical performance.

Critical weaknesses
  1. Limited generality of the main theoretical result. Theorem 6.1 is restricted to wide additive networks with a single training residual, squared loss, balanced initialization, positive tangent drift, and a specially calibrated near-critical learning rate. In particular, the single-residual setting yields a rank-one Hessian at interpolation, substantially simplifying the dynamics compared with general neural-network optimization. These restrictions limit the broader significance of the theoretical guarantee. 

  2. A substantial gap between the theoretical setting and practical neural-network training. Although the authors conduct experiments with multiple residuals and deeper networks, these experiments primarily investigate dynamics around numerically selected interpolating solutions. They do not establish that the main initialization-to-interpolation trajectory theorem extends to these more general settings. Consequently, the extent to which the theoretical results explain EoS behavior in practical deep learning remains unclear.

Questions for authors
  1. Which aspects of the proof of Theorem 6.1 fundamentally depend on the single-residual assumption? Can the authors clarify whether an analogous stabilization guarantee is expected to hold with multiple interacting residuals and nontrivial Hessian spectra?

  2. How essential is the balance condition E[h⊤Ah]=0\mathbb{E}[h^\top Ah]=0 to the predicted stabilization dynamics? What qualitative behavior should be expected when this condition is not satisfied?

  3. Given that the deep-network experiments are conducted around selected interpolating endpoints, what evidence supports extending the proposed mechanism to complete training trajectories from random initialization?

2 comments
AuthorsOpenReview ↗

Response to Reviewer adfT

W1: restrictions of Theorem 6.1. We agree they are real. Section 6 and Appendix F say what each is for.

  • One residual gives the closed two-variable state and the rank-one Gauss–Newton part you mention (Q1).
  • Balance: see Q2.
  • Positive tangent drift cannot simply be dropped. Proposition F.6 gives a SiLU initialization with zero drift for which near-critical GD with positive initial detuning diverges; its proof checks that the other hypotheses hold.
  • The calibrated step defines the near-critical regime. The proof of Lemma F.4 shows that the population rate 2/λ∞2/\lambda_\infty is also admissible, with a random initial detuning.
  • The squared loss gives the decomposition D2L=gg⊤+rHD^2L=gg^\top+rH.

W2 and Q3: theory versus practice. We agree, and we have narrowed the claims. The abstract, Section 1, Section 7 and the new Section 8 describe the multi-residual and deep experiments as endpoint interventions, with no theorem. The only evidence from random initialization concerns one residual: Theorem 6.1 and 54 runs (36 in Figure 2, all in Appendix G.3). We do not claim that the mechanism governs deep training from random initialization.

Q1: dependence on one residual (Section 6, p. 8; Appendix H.1). The proof uses one residual at four points: the residual and the kernel form a closed two-variable state; the amplitude is a scalar; the comparison of the loss sharpness with the kernel is a one-dimensional Schur complement; the tail is a scalar contraction. With nn residuals the other n−1n-1 modes are forced at every pair and relax slowly when the kernel is ill-conditioned, the top eigenvector rotates, and the constants degrade with the condition number. Whether an analogous guarantee holds is open. We expect the sharpness comparison to extend under a kernel gap; keeping the other modes small over O(m)O(\sqrt m) pairs when the kernel is ill-conditioned is the step we do not know how to do (Appendix H.1). Table 1 and Appendix G.4.1 are consistent with the scalar law for the top mode for n≤256n\le256 at small detuning, with an error constant that grows with nn.

Q2: balance (Section 6, pp. 7–8; Appendix H.2). Balance keeps g⊤Hg=OP(m−1)g^\top Hg=O_P(m^{-1}). The proof uses this three times: to keep the quadratic term of the residual map below the detuning, to give even and odd iterates one common detuning, and to compare the loss sharpness with the kernel. For readout blocks, balance follows from m1=m3=0m_1=m_3=0 (Theorem 5.1). Without it these terms are of the order of the detuning, and Theorem 6.1 does not apply. A formal expansion, which is not rigorous and is labelled as such, predicts the same sech-type damping to a subcritical endpoint with a modified tangent coefficient. It also predicts a step-to-step oscillation of the sharpness of the order of the detuning. A new probe agrees (one residual, Softplus, m3=±2m_3=\pm2, widths 1024–65536; Table 3). All 99 runs converge, with final normalized sharpness at most −0.29-0.29. The amplitude error of the formal system falls to about 10−310^{-3} at m=65536m=65536, while that of the system of Theorem 6.1, which assumes balance, levels off near 0.03–0.04 for the skewed laws; at m=65536m=65536 consecutive detunings still differ by 0.17–0.19. This is evidence in one model, not a proof.

I appreciate the response from the authors. I got most things clear and I will update my rating.

Reviewer_FGudOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors investigate the oscillatory behaviour of gradient descent near the EoS and how tangential motion along the manifold of interpolating solutions can suppress such oscillations. The authors introduce two computable neural network functions that characterise the flip coefficient of projected gradient descent and the tangential sharpness-drift coefficient, respectively. They further construct two Softplus initialisation distributions with the same overall tangential kernel but opposite signs of the flip coefficient. Under a specific additive network architecture, initialisation balance conditions and a positive tangential drift condition, the paper proves that a single-residual gradient descent trajectory starting from random initialisation can be approximated by an explicit amplitude–sharpness dynamical system over a width-dependent timescale, eventually converging to its own interpolating solution. The paper is technically sound with solid theoretical foundation.

Critical strengths

S1: The paper represents the local flip coefficient and tangential sharpness drift as computable neural network functions and establishes their connections to higher-order derivatives at interpolating solutions. S2: Instead of studying only the local dynamics near a given interpolating solution, the paper also attempts to analyse GD trajectories starting from random initialisation, providing an approximation by an amplitude–sharpness system and a subsequent convergence argument. S3: The authors compare full GD with projected GD and intervene along a specific tangential sharpness direction to investigate the source of the additional damping. This provides more direct mechanistic evidence than solely observing correlations between training curves.

Critical weaknesses

W1: Although the paper does investigate more complex models experimentally, these experiments do not establish theoretical guarantees equivalent to those provided by the single-residual theorem. W2: The experiments are insufficient to establish that the same mechanism generally dominates in deep networks and under standard training trajectories. The authors should clarify the scope of this conclusion or provide further evidence across different architectures, datasets, and training conditions.

Questions for authors

Q1: In the multi-residual setting, could the authors further clarify the main technical obstacles that prevent the existing scalar-trajectory theorem from being directly extended, and identify which conclusions are currently supported only empirically? Q2: The amplitude prediction error increases with the number of residuals. How do the authors explain this phenomenon? Within the small-detuning regime, what range of stabilisation behaviour can be supported by the current experiments? Q3: The directional intervention experiments are conducted near successfully identified interpolating endpoints. To what extent do the authors believe these results generalise to standard deep network training from random initialisation? If the current evidence primarily supports a local mechanism near interpolating endpoints, could the overall claims of the paper be narrowed accordingly?

1 comment
AuthorsOpenReview ↗

Response to Reviewer FGud

Scope (W1, W2, Q3). We agree, and we have narrowed the claims rather than adding experiments on other architectures or datasets. The abstract, Section 1 (p. 2), the opening of Section 7 and the new Section 8 say which claims are theorems, which are tested from random initialization (one residual only: Theorem 6.1 and 54 runs, 36 of them in Figure 2, all in Appendix G.3), and which rest on endpoint interventions (several residuals, the deep network). The evidence supports a local mechanism near interpolating endpoints. The deep-network intervention shows that near eight reached interpolants, over 1,000 steps, one fixed tangent direction accounts for the extra damping of full GD. We do not claim that the mechanism governs standard deep training from random initialization.

Q1: obstacles to a multi-residual theorem (Section 6, p. 8; Appendix H.1). The main obstacle is closure. With one residual every GD update is a multiple of g=∇fmg=\nabla f_m, so the residual and the kernel form a closed two-variable state. With nn residuals only the top mode has multiplier near −1-1. The other n−1n-1 modes are forced by nonlinear terms at every pair and relax slowly when the kernel is ill-conditioned, and a proof must keep them small over O(m)O(\sqrt m) pairs. Three further steps use the scalar structure: the amplitude (with nn residuals the top eigenvector rotates), the comparison of the loss sharpness with the kernel (a one-dimensional Schur complement), and the final contraction, whose constants would degrade with the condition number. Supported only empirically: every statement about multi-residual GD dynamics, and the dynamics of the deep network.

Q2: growth of the amplitude error with nn (Section 7, pp. 9–10 and Table 1; Appendix G.4.1). At fixed nn the error falls with width, but its constant grows with nn. The median of m\sqrt m times the error is 0.05 at n=16n=16, and 0.44–0.51 (Softplus) or 0.10–0.19 (tanh) at n=256n=256. The scalar system has no counterpart of the other n−1n-1 normal modes. For Softplus, the relaxation time of the weakest of them grows from about 10210^2 pairs at n=16n=16 to 2.4×1052.4\times10^5 at n=256n=256, far beyond the horizons of 68–272 pairs. We regard this coupling as the likely cause; the records do not isolate it, and the paper says so.

The experiments cover n≤256n\le256, widths 512–8192, initial amplitude 0.2, a slow-time horizon T=3T=3, and detuning δ=0.3/m\delta=0.3/\sqrt m or coefficient-based δ≤0.102\delta\le0.102. In all 130 completed comparisons, full GD ends with a smaller amplitude than projected GD. Runs with δ≥1.29\delta\ge1.29 escape. Longer horizons, larger amplitudes and multi-residual trajectories from initialization are untested.

Reviewer_NngKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper provides a theoretical analysis of an interesting phenomenon in machine learning optimization. Even when a large learning rate causes local oscillations, gradient descent can still remain stable and eventually converge. The authors explain this phenomenon by showing that tangent motion can help stabilize the training process, while motion in the normal direction may lead to oscillations. To support this explanation, the authors derive two functionals that characterize the local flip bifurcation behavior and tangent sharpness drift. Furthermore, the authors construct an example showing that two networks with the same neural tangent kernel can have opposite flip signs. The paper also provides a theoretical analysis of the full gradient descent stabilization process using wide additive networks. Specifically, the oscillation amplitude and normalized sharpness deviation are modeled by a system of ordinary differential equations (ODEs). Finally, the authors conduct several experiments and ablation studies to validate their theoretical findings.

Critical strengths

1: The paper provides an interesting perspective on understanding the training dynamics of neural networks. In particular, the analysis of full gradient descent dynamics is valuable.

2: The paper provides a comprehensive theoretical analysis of the proposed mechanism, with detailed mathematical derivations and convergence guarantees.

3: The observation that networks with the same neural tangent kernel can have different flip coefficients and exhibit different nonlinear dynamics is interesting and provides useful insights into the limitations of kernel-based analysis.

Critical weaknesses

1:The authors cites several relevant previous studies, but the distinction between their work and existing approaches is not clearly explained. The novelty of the proposed theoretical analysis needs further clarification.

2: The theoretical analysis mainly focuses on simplified neural network settings, while real-world neural networks can be much more complex and involve different optimization algorithms. It is unclear whether the theoretical findings can be generalized to these settings.

3: The experiments are conducted on relatively small-scale neural networks, which limits the validation of the proposed theory. Although the authors report some failed cases, they do not provide sufficient analysis of these failures or explain their implications for the robustness of the proposed framework.

Questions for authors

1: What is the main contribution of this work compared with previous work? 2: Can the authors further clarify whether the proposed theoretical analysis can be extended to more general neural network architectures? 3: Can the authors provide further clarification on the experimental cases that failed to reach interpolation, including the possible reasons for these failures?

2 comments
AuthorsOpenReview ↗

Response to Reviewer NngK

Novelty (W1/Q1). The paper contributes three results.

  1. Exact neural functionals for the flip coefficient and the tangent drift at every regular interpolant of an additive network, for any number of residuals (Theorem 4.1, Proposition 4.2), with their limits under random initialization (Theorem 4.4, under Assumption 4.3).
  2. Two Softplus laws with the same population tangent kernel and, at one training input, opposite flip signs (Corollaries 5.3 and F.5).
  3. For one residual, a rigorous trajectory theorem for constant-step GD on a wide nonlinear additive network started at random initialization, with both coefficients computed from the initialization law (Theorem 6.1).

The stabilizing mechanism behind the third result is known: the cubic term of the loss moves an oscillating iterate along −∇S-\nabla S (self-stabilization; Damian, Nichani and Lee, 2023), and two-step dynamics were studied by Chen and Bruna (2023). Theorem 6.1 proves it in a new setting. The closest rigorous trajectory results concern scalar, two-layer linear or simplified two-layer models (Zhu et al., 2023; Song and Yun, 2023; Li, Wang and Li, 2022; Ahn et al., 2023), assume progressive sharpening (Damian et al.), or track GD near a sharp valley of curvature 2/η+O(1)2/\eta+O(1) (Chodron de Courcel, 2026). The one-example network of Lewkowycz et al. (2020) satisfies the hypotheses of Theorem 6.1, and near ηλ0=2\eta\lambda_0=2 its exact recursion reduces to our system; its catapult phase, at detuning of order one, is outside the theorem (Appendix H.3). Section 2 gives the full comparison.

Generality (W2/Q2). Part of the analysis applies to any network. The flip coefficient and the tangent sharpness gradient are defined and computable at interpolants of any C4C^4 residual map that satisfy a rank and gap condition (Propositions 3.1 and A.1; Appendix D.1 for ∇S\nabla S), and Section 7 uses the tangent direction in a three-hidden-layer network. The closed-form functionals of Theorem 4.1 and Proposition 4.2 need additive blocks and the squared loss, with any number of residuals, which includes two-layer networks with trainable readout and bias. Theorem 6.1 needs one residual, the squared loss, C4C^4 blocks with balanced i.i.d. initialization (Eψ=E[h⊤Ah]=0\mathbb E\psi=\mathbb E[h^\top Ah]=0) and an eighth-moment derivative envelope, positive tangent drift, and a near-critical step. Section 6 and Appendices H.1–H.2 say where its proof uses one residual and balance, and Proposition F.6 shows why positive drift is needed. Deeper networks and other optimizers are not covered by the theorems, and Section 8 says so.

Failures (W3/Q3). We did not add large-scale experiments: Theorem 6.1 concerns one residual, which larger models would not test, so we narrowed the claims to what the current tests support. The revision analyzes each failure category (Section 7; Appendices G.1, G.4.1, G.4.3).

  • Deep network, 5 of 13 endpoint attempts. All five failed while the endpoint was being computed, before any intervention ran, when the solver budget ran out (four with damped Newton corrections; the fifth had just begun full Newton steps). They are solver failures, not observations of the dynamics. We have not diagnosed why all three tanh attempts failed. The intervention results are therefore conditional on reached endpoints, which cover GELU and SiLU only.
  • Multiple residuals, 10 of 140 comparisons. All ten are Softplus runs at n=256n=256 under coefficient-based detuning. There mc1≥1.6×104mc_1\ge1.6\times10^4, so δ≥1.29\delta\ge1.29, far outside the small-detuning regime, and the iterates escape within three pairs. We attribute the large coefficients to ill-conditioned kernels (condition numbers 6.6×1056.6\times10^5 to 1.8×1061.8\times10^6) but have not isolated this. Under δ=0.3/m\delta=0.3/\sqrt m every run at n=256n=256 completes.
  • Earlier gradient-flow runs. Five of 72 stopped at the time limit. Two sign mismatches at width 512 have the predicted sign at width 2048. The six escapes at negative c1c_1 are what local theory predicts.

Thanks for detailed reply from author. I think it address my concerns. And I wnat to keep the positive rating I have for this paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers. We have uploaded a revision in which new or changed text is blue; the main text has 10 pages. The main changes:

  1. Positioning (Section 2, pp. 2–3). Section 2 compares this paper by mechanism with catapult dynamics, self-stabilization, low-dimensional analyses of the edge of stability and flow frameworks, and it adds six works: Zhu et al. (2023), Li, Wang and Li (2022), Ahn et al. (2023), Gess and Heydecker (2026), Regis and Chewi (2026) and Chodron de Courcel (2026). Section 1 relates the paired update to self-stabilization (Damian et al., 2023). New Appendix H.3 shows that the one-example network of Lewkowycz et al. (2020) satisfies Theorem 6.1.
  2. Scope (abstract, Section 1, new Section 8 on p. 10). Each claim now states its support. The identities of Theorem 4.1 and Proposition 4.2 hold for any number of residuals. The trajectory theorem covers one residual. The multi-residual and deep-network experiments are endpoint interventions, and no theorem covers them.
  3. One residual and balance (Section 6, pp. 7–8; Appendix H.1–H.2). These parts say where the proof uses each assumption. A small probe without balance (Table 3) is the only new experiment.
  4. Failure analyses (Section 7, pp. 9–10; Appendices G.1, G.4.1, G.4.3).
  5. Presentation. Theorem 5.1 states its hypotheses, terms are defined at first use, and the introduction states the core idea of the proof. Proposition 3.2 moved unchanged to Appendix A (now Proposition A.1). Automatic renumbering is not colored.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.