- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates the relationship between local flip bifurcations and tangent self-stabilization in gradient descent near the Edge of Stability (EoS). The authors derive exact neural functionals characterizing the flip coefficient and tangent sharpness drift, and construct two Softplus initialization distributions with identical population neural tangent kernels but opposite flip signs. Their main theoretical result establishes an amplitude–sharpness approximation for near-critical discrete GD in wide additive networks with a single residual, including a quantitative tracking error and convergence to a subcritical interpolating solution. Experiments on synthetic networks and deeper models are presented to validate the predicted stabilization mechanism.
Critical strengths
-
Quantitative theoretical analysis. The paper derives exact expressions for the flip and tangent-drift coefficients and provides a width-dependent approximation of discrete GD dynamics. The convergence analysis goes beyond a purely heuristic continuous-time model.
-
Mechanism-oriented experimental validation. The projected-versus-full GD comparisons and tangent-direction interventions provide useful evidence for the proposed stabilization mechanism. The experiments are designed to investigate the theoretical predictions rather than merely demonstrate empirical performance.
Critical weaknesses
-
Limited generality of the main theoretical result. Theorem 6.1 is restricted to wide additive networks with a single training residual, squared loss, balanced initialization, positive tangent drift, and a specially calibrated near-critical learning rate. In particular, the single-residual setting yields a rank-one Hessian at interpolation, substantially simplifying the dynamics compared with general neural-network optimization. These restrictions limit the broader significance of the theoretical guarantee.
-
A substantial gap between the theoretical setting and practical neural-network training. Although the authors conduct experiments with multiple residuals and deeper networks, these experiments primarily investigate dynamics around numerically selected interpolating solutions. They do not establish that the main initialization-to-interpolation trajectory theorem extends to these more general settings. Consequently, the extent to which the theoretical results explain EoS behavior in practical deep learning remains unclear.
Questions for authors
-
Which aspects of the proof of Theorem 6.1 fundamentally depend on the single-residual assumption? Can the authors clarify whether an analogous stabilization guarantee is expected to hold with multiple interacting residuals and nontrivial Hessian spectra?
-
How essential is the balance condition to the predicted stabilization dynamics? What qualitative behavior should be expected when this condition is not satisfied?
-
Given that the deep-network experiments are conducted around selected interpolating endpoints, what evidence supports extending the proposed mechanism to complete training trajectories from random initialization?
2 comments
I appreciate the response from the authors. I got most things clear and I will update my rating.
Response to Reviewer adfT
W1: restrictions of Theorem 6.1. We agree they are real. Section 6 and Appendix F say what each is for.
W2 and Q3: theory versus practice. We agree, and we have narrowed the claims. The abstract, Section 1, Section 7 and the new Section 8 describe the multi-residual and deep experiments as endpoint interventions, with no theorem. The only evidence from random initialization concerns one residual: Theorem 6.1 and 54 runs (36 in Figure 2, all in Appendix G.3). We do not claim that the mechanism governs deep training from random initialization.
Q1: dependence on one residual (Section 6, p. 8; Appendix H.1). The proof uses one residual at four points: the residual and the kernel form a closed two-variable state; the amplitude is a scalar; the comparison of the loss sharpness with the kernel is a one-dimensional Schur complement; the tail is a scalar contraction. With n residuals the other n−1 modes are forced at every pair and relax slowly when the kernel is ill-conditioned, the top eigenvector rotates, and the constants degrade with the condition number. Whether an analogous guarantee holds is open. We expect the sharpness comparison to extend under a kernel gap; keeping the other modes small over O(m) pairs when the kernel is ill-conditioned is the step we do not know how to do (Appendix H.1). Table 1 and Appendix G.4.1 are consistent with the scalar law for the top mode for n≤256 at small detuning, with an error constant that grows with n.
Q2: balance (Section 6, pp. 7–8; Appendix H.2). Balance keeps g⊤Hg=OP(m−1). The proof uses this three times: to keep the quadratic term of the residual map below the detuning, to give even and odd iterates one common detuning, and to compare the loss sharpness with the kernel. For readout blocks, balance follows from m1=m3=0 (Theorem 5.1). Without it these terms are of the order of the detuning, and Theorem 6.1 does not apply. A formal expansion, which is not rigorous and is labelled as such, predicts the same sech-type damping to a subcritical endpoint with a modified tangent coefficient. It also predicts a step-to-step oscillation of the sharpness of the order of the detuning. A new probe agrees (one residual, Softplus, m3=±2, widths 1024–65536; Table 3). All 99 runs converge, with final normalized sharpness at most −0.29. The amplitude error of the formal system falls to about 10−3 at m=65536, while that of the system of Theorem 6.1, which assumes balance, levels off near 0.03–0.04 for the skewed laws; at m=65536 consecutive detunings still differ by 0.17–0.19. This is evidence in one model, not a proof.