acceptodds
Under review as a conference paper at ICLR 2027

Gradient Clipping at and Beyond the Edge of Stability

Abstract

When training deep neural networks with full-batch gradient descent, the curvature of the loss stays at the threshold of instability, hovering around where is the learning rate. We study how clipping changes this Edge-of-Stability (EoS) phenomenon in first-order (FO) and zeroth-order (ZO) optimization under batch and per-sample clipping. Our central finding is that clipping stabilizes FO and ZO training in qualitatively different ways. For FO, clipping shifts the relevant stability edge, and the trajectory remains near this clipping-aware edge; stronger clipping permits stability at larger curvature. For batch-clipped ZO, training remains bounded at curvatures well beyond the mean-square stability threshold, even though standard stability analysis predicts instability in this regime. For both batch- and per-sample clipped ZO, we characterize nonzero fixed points through the covariance of the parameter displacement and show local attraction within the corresponding closed covariance dynamics under explicit conditions. Experiments show that FO trajectories track clipping-aware stability edges, while batch-clipped ZO approaches its nonzero covariance balance beyond the predicted stability limit and per-sample ZO moves toward the predicted matrix-valued covariance balance along the leading-curvature directions.

Primary area:
optimization
Keywords:
edge of stability, gradient clipping, zeroth-order optimization, training dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rgaiOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This work studies the role of clipping in first-order (FO) and zeroth-order (ZO) optimization regarding stability. It uses usual GD/SGD as the FO setting. It argues that clipping affects FO and ZO stability through distinct mechanisms: (i) for FO, clipping modifies the stability boundary; (ii) for ZO, clipping enables bounded dynamics beyond the linear stability threshold through a nonlinear covariance balance.

Depsite the clipping role in optimization stability is interesting, i think the theoretical significance seems limited since the practical setting nowadays is Adam family instead of GD/SGD; the novelty seems marginal since the main results substantialy overlap with existing literature, which has extensively explored this problem. The main results in this manuscript appears a summarization with incremental results. however it doesn't cite the prior work that included the related results.

Additionally. i feel some results seem wrong such as Proposition 3.2 which is false (I'll state in detail in the questions-to-authors).

[1]. Arora et al. ICML22. Understanding Gradient Descent on the Edge of Stability in Deep Learning. [2]. Revisiting Gradient Clipping: Stochastic bias and tight convergence guarantees. https://proceedings.mlr.press/v202/koloskova23a/koloskova23a.pdf [3]. Song et al. ICML26. Zeroth-Order Optimization at the Edge of Stability. https://openreview.net/forum?id=s87tQaKAER

I apologise that there are more literature I can't list one by one.

Critical strengths
  1. The presentation is easy to follow.
  2. Notations appear good in theoretical community so I can easily follow the mathematics without burden.
Critical weaknesses
  1. There lacks sufficient experiments for verifying its lemmas/theorems/propositions numerically. This can't convince me to believe the results are sound. In a sound theoretical work, I won't blindly believe the results before i see numerical verifications for the major theorems/propositions.

I believe this is the improvement room that the work can be improved by including sufficient experiments to show that the theorems/propositions numerically hold. but before that, the manuscript needs significant efforts to fix the wrong mathematical deductions.

  1. This work appears collecting results existing in literature however it doesn't properly cite the related work. This is why the novelty seems marginal.

for example. [1] ICML22. Arora et al. Understanding Gradient Descent on the Edge of Stability in Deep Learning. https://proceedings.mlr.press/v162/arora22a.html [2] ICML26. Song et al. Zeroth-Order Optimization at the Edge of Stability. [3] ICLR26. Mulayoff et al. https://openreview.net/forum?id=hBNC8w9pd7 [4] ICML26. https://proceedings.mlr.press/v306/islamov26a.html

  1. Some theorems/propositions are technically wrong.

eg Proposition 3.2. sec4.1.1 singular-Hessian qualification. theorem 4.1 and appendix c.12. theorems 4.3 and 4.4. there may exist more wrong results, i haven't checked all results yet.

Questions for authors
  1. Regarding Proposition 3.2.

I think this proposition needs effort to fix since it's wrong.

for dynamics et+1=(I−ηJps)ete_{t+1} = (I - \eta J_{ps})e_t

Conditions ∣1−ηλ∣≤1,∀λ∈spec(Jps)|1-\eta\lambda | \leq 1, \forall \lambda \in \mathrm{spec}(J_{ps}) can't guarantee the claims when eigenvalues are degenerate.

Can you further clarify and present the refined results?

4 comments

Thank you for the careful review. We have expanded the numerical checks, clarified the contributions, and corrected the stability qualifications.

W1. Numerical Validation of the Main Theoretical Results

The original submission included numerical checks of FO stability and the linear batch-ZO boundary. In this revision, we supplement these with direct perturbation tests for batch FO and numerical tests of the nonzero covariance balance and local attraction within the closed covariance model.

  • FO stability (Theorem 3.1 and Proposition 3.2). The original submission already reported numerical checks of the stability formulas and proxy accuracy. For batch clipping, the inequalities in Eq. (9) hold at all six active checkpoints. For per-sample clipping, the symmetric proxy S=λ1(A−Rs)S=\lambda_1(\boldsymbol A-\boldsymbol R_s) approximates 2/ηps∗2/\eta^*_\text{ps} with relative error at most 0.015%, and all six checkpoints satisfy the required cluster-separation condition 4∥Ra∥2≤Γc4\lVert \boldsymbol{R}_a\lVert_2\le\Gamma_c. These existing results are summarized in Lines 182–185 and 233–237, with details in Appendix G.1, Figure 5 and Tables 4 and 6.

    In this revision, we added independent one-step perturbation tests showing that perturbations along the tested eigenvector shrink below and grow above the predicted critical learning rate, consistent with Theorem 3.1 (Lines 485–487; Appendix G.2.1, Figure 6).

  • Linear batch-ZO boundary (Theorem 4.1). The following checks were already included in the original submission. Six synthetic settings satisfying the alignment assumption confirm that the numerically computed scalar thresholds obey the trace-approximation bound (Appendix H.5, Table 13). At 24 neural-network checkpoints, the scalar and trace approximations differ from the reduced-operator boundary by 0.042%–0.440% and 3.05%–7.04%, respectively (Lines 321–323; Appendix H.3, Table 12).

  • Nonzero covariance balance and local attraction (Theorems 4.3–4.4). The original submission already included neural-network trajectory experiments assessing the balance predicted by Theorem 4.3 (Figure 2c in the revised manuscript). Specifically, Eq. (21) predicts ητ(κ⋆)=2\eta\tau(\kappa^\star)=2, and Eq. (23) bounds the error of approximating τ\tau by γ,tr(H)\gamma \\,\mathrm{tr}(\boldsymbol H). The existing experiments therefore compare γ,tr(H)\gamma\\,\mathrm{tr}(\boldsymbol H) with 2/η2/\eta along training trajectories. At 10610^6 updates, ηγ^,tr(H)/2\eta\widehat\gamma\\,\mathrm{tr}(\boldsymbol H)/2 lies between 0.927 and 1.022 for five of the six clipping thresholds; the strongest-clipping setting remains at 0.685 and is still increasing (Table 19). These observations support approximate agreement with the predicted balance in five settings, while the strongest-clipping setting has not reached it.

    In this revision, we added checks of the trace-approximation bound in Eq. (23) and the local-attraction prediction in Eq. (24) within the closed covariance model. Across six quadratic settings, the trace-approximation errors satisfy Eq. (23). The full covariance-map Jacobian has spectral radius 0.9352–0.9533, and nearby perturbations decay under the map, consistent with Eq. (24) (Appendix H.8, Table 15). We additionally evaluate Theorem 4.4’s sufficient conditions in frozen models constructed from existing neural-network checkpoints (Appendix H.9, Table 16).

W2. Novelty and Positioning Relative to Prior Work

Our contributions are clipping-specific stability and covariance results that are distinct from the cited analyses. Four of the five works mentioned—Arora et al. (2022), Koloskova et al. (2023), Song et al. (2026), and Mulayoff and Stich (2026)—were already cited and discussed in the original submission, Lines 64–73. We have added Islamov et al. (2026) in revised Lines 69–71 and sharpened the distinctions in Lines 66–78 and Appendix J.1–J.4:

  • FO: Beyond normalized-GD EoS (Arora et al., 2022) and generalized sharpness (Islamov et al. (2026)), we derive the generally nonsymmetric per-sample clipped Jacobian and quantify the error of its tractable symmetric stability proxy (Proposition 3.2).
  • ZO: Beyond clipping convergence and bias (Koloskova et al., 2023) and the unclipped mean-square boundary (Song et al., 2026), we characterize a nonzero clipped covariance balance and its local attraction under explicit closure assumptions (Theorems 4.3–4.4), and derive a distinct matrix-valued per-sample balance (Proposition 4.5 and Theorem 4.6).
  • Nonlinear stability: Mulayoff and Stich (2026) study nonlinear stability beyond the quadratic approximation. Here, retaining clipping makes the update nonlinear even on a fixed quadratic, allowing explicit clipping-dependent covariance fixed points and attraction conditions (Appendix J.2).

W3. Correctness of Proposition 3.2, the Singular-Hessian Treatment, and Theorems 4.1, 4.3, and 4.4

We have corrected the missing stability qualifications of Proposition 3.2 and made the fixed-point and attraction arguments explicit.

  • Proposition 3.2: The missing zero-mode and Jordan qualification is now stated explicitly; see Q1 below.
  • Theorem 4.1 and singular Hessians: The main theorem now assumes H≻0\boldsymbol H\succ0 and explicitly states the conditions for the scalar threshold equation and trace bound. Appendix C.5 (formerly Appendix C.12) treats H⪰0\boldsymbol H\succeq0 through the projected process zt=P+et\boldsymbol z_t=\boldsymbol P_+\boldsymbol e_t and distinguishes its stability from full-space covariance boundedness.
  • Theorem 4.3: The revised proof establishes existence and uniqueness of the nonzero PSD fixed point for η>η0⋆\eta>\eta_0^\star. It also proves that the zero covariance is locally attracting for η<η0⋆\eta<\eta_0^\star and unstable for η>η0⋆\eta>\eta_0^\star (Appendix D.2). At equality, we assert only that zero is the unique PSD fixed point.
  • Theorem 4.4: We explicitly require η>η0⋆\eta>\eta_0^\star and control the full symmetric-matrix Jacobian, including off-diagonal modes (Appendix D.3).

The balance equation and trace-error bound are unchanged. If a specific step in the proofs of Theorems 4.3–4.4 remains problematic, we would appreciate your identifying it so that we can address it directly.

W4. Relevance of GD/SGD Analysis to Modern Optimization

Our results provide a foundation for analyzing clipping in modern optimizers. GD and random-direction ZO isolate clipping’s effects from momentum and adaptive preconditioning (Lines 46–48), allowing us to characterize how clipping changes the local stability boundary and produces a nonzero covariance balance. These effects must be accounted for when extending stability analysis to clipped Adam or ZO-Adam.

Such an extension requires incorporating the clipped-update derivatives and moment calculations developed here into the optimizer’s joint parameter and internal-state dynamics. Frozen preconditioning provides a starting point; evolving momentum and preconditioners require further analysis of their interaction with clipping. We now discuss existing Adam EoS analyses (Lines 74–76) and identify this extension in the Conclusion (Lines 535–536).

Q1. Stability Conditions for Degenerate Eigenvalues in Proposition 3.2

We agree that eigenvalues alone do not guarantee bounded iterates. We have revised Lines 206–215 and Proposition 3.2 to explicitly require ker⁡(J2)=ker⁡(J)\ker(\boldsymbol J^2)=\ker(\boldsymbol J), excluding Jordan blocks of size greater than one at the zero eigenvalue of J\boldsymbol J. If this condition fails, linearized perturbations can grow without bound for every positive step size.

Under the revised assumptions, Eq. (12) gives the supremum of stable step sizes: every strictly smaller positive step is stable. At the endpoint, boundedness additionally requires all Jordan blocks associated with eigenvalues of I−ηJ\boldsymbol I-\eta \boldsymbol J having absolute value one to have size one. This distinction is explicit in Lines 214–215 and Appendix B.4. Appendix B.4 provides the proof and a counterexample. The spectral proxy bounds remain unchanged.

Reviewer_rgaiOpenReview ↗

Thanks for the replies.

I thank the replies from the authors.

Thank you for taking the time to read our responses. If any of your concerns remain, we’d appreciate knowing which points you think we could address more fully in the paper.

Reviewer_L1tJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how gradient clipping changes edge-of-stability behavior in first- and zeroth-order optimization. It derives clipping-aware local stability criteria for FO methods and nonlinear covariance balances for ZO methods.

Critical strengths

The distinction between local linear instability and bounded nonlinear fluctuations is useful. The treatment of batch versus per-sample clipping is technically detailed, and the experiments support several proposed diagnostics.

Critical weaknesses

The FO analysis characterizes a modified local stability boundary but does not explain the trajectory dynamics that motivate the paper. The central question is why unclipped training remains near the conventional EoS, whereas clipped training continues into higher-curvature regions and eventually approaches a different boundary.

A frozen local quadratic analysis cannot explain this change in curvature, since its Hessian is fixed. Moreover, in simple isotropic quadratic examples, the clipping-aware edge relation can follow directly from the amplitude of a symmetric clipped two-cycle, independently of sharpness. Thus, observing the proposed diagnostic near 2/η2/\eta does not by itself establish a mechanism for boundary tracking.

I would like to see an analysis or targeted experiment showing how clipping modifies the feedback between oscillations and curvature evolution, including the observed nonmonotonic gradient norm. The ZO covariance results are more mechanistic, although their scope is limited by the closure assumptions and restrictive attraction conditions.

Questions for authors

What mechanism allows clipped FO training to cross the conventional EoS and approach a higher-curvature boundary, while unclipped training remains near the original edge? The frozen local quadratic analysis characterizes a stability threshold but does not explain this curvature evolution.

4 comments

Thank you for the constructive questions. We have added targeted experiments and direct checks of the ZO attraction conditions.

W1, W3 & Q1. FO Trajectory Dynamics and Oscillation–Curvature Feedback

Our experiments support the interpretation that stronger clipping limits oscillations and weakens their curvature-reducing effect, thereby allowing further sharpening. In unclipped GD, oscillations along sharp directions can counteract progressive sharpening through the self-stabilization mechanism (Damian et al., 2023). Active clipping bounds the step norm by ηC\eta C, restricting the realized oscillation amplitude and potentially changing this balance. We investigate this interpretation through the following trajectory experiments.

  • Same-checkpoint comparison: With identical starting parameters and matched first updates, the clipped continuation reaches estimated sharpness 626.91626.91 after 5,000 updates, while unclipped GD reaches 573.73573.73, near its conventional threshold 573.99573.99 (Appendix G.2.2, Figure 7). These results show that the subsequent update rules lead to different curvature paths despite the identical starting state and first update.
  • Controlled clipping-threshold intervention: To test how clipping affects the curvature-reducing effect, we vary CC while holding the learning rate and starting parameters fixed. Reducing CC by 20% reduces the realized leading-eigenvector oscillation amplitude by approximately 20% and accelerates sharpening; increasing CC by 20% increases the amplitude by approximately 20% and reverses sharpening into an actual sharpness decrease over 500 updates. This pattern holds at all three checkpoints from the same training run. These results are consistent with stronger clipping weakening the curvature-reducing effect of oscillations, thereby allowing further sharpening in the tested regime. Full experimental details are included in Appendix G.2.3-G.2.4.
  • Nonmonotonic gradient norms: In the added three-seed, 100,000-step experiments, the gradient norm decreases late in training while sharpness also decreases. Under active clipping, this gradient-norm decrease raises ηeff=ηC/∣g∣\eta_{\mathrm{eff}}=\eta C/\\|\boldsymbol g\\| and lowers the corresponding reference 2/ηeff2/\eta_{\mathrm{eff}}, consistent with the observed decrease in sharpness (Appendix G.2.5, Figure 9). These observations characterize the late-stage behavior.

W2. Clipped Two-Cycles and the Edge Diagnostic

Our use of a frozen local analysis follows the conventional EoS framework. Cohen et al. (2021) use the stability threshold 2/η2/\eta on a fixed quadratic and compare it with the Hessian sharpness measured along neural-network training trajectories. We apply the same approach to the clipped update. Here, “frozen” refers to the local stability calculation at each checkpoint; the diagnostic is evaluated using the current gradient and Hessian as training progresses.

We agree that the two-cycle example highlights a limitation of diagnostic agreement. On an isotropic quadratic, a symmetric clipped two-cycle can satisfy the edge relation through its oscillation amplitude, without any curvature evolution. Thus, observing the diagnostic near 2/η2/\eta alone does not establish a mechanism for boundary tracking.

To address the trajectory dynamics, we added the experiments described in W1, W3 & Q1 above. The clipping-threshold interventions show that stronger clipping reduces realized oscillations and permits further sharpening in the tested regime, consistent with a weaker curvature-reducing effect of oscillations.

W4. Closure Assumptions and Attraction Conditions

Although the sufficient attraction conditions in Theorem 4.4 may appear restrictive, our numerical checks show that they hold in every tested closed batch covariance model with a nonzero balance, including models constructed from neural-network checkpoints.

Among the 24 checkpoint-derived frozen models, all 18 with a nonzero balance satisfy both conditions: ηλ1≤0.252\eta\lambda_1\le0.252 and e(κ⋆)≤0.395\mathfrak e(\kappa^\star)\le0.395. The remaining six models are at or below the existence threshold (Lines 408–412; Appendix H.9, Table 16). All six synthetic quadratic configurations also satisfy the conditions, with full-Jacobian spectral radii 0.9352–0.95330.9352–0.9533 and decaying perturbations (Lines 502–506; Appendix H.8, Table 15). These checks establish applicability within the tested closed batch models. The separate numerical assessment of Assumption 4.2 is provided in the general response above.

Dear Reviewer L1tJ,

Thank you for your question about how clipping affects the interaction between oscillations and curvature. We added an experiment showing that, across three checkpoints, lowering the clipping threshold reduced oscillations along the leading Hessian eigenvector and led to faster sharpening (Appendix G.2). If you have a chance to look over the changes, we’d be grateful to hear whether any of your concerns remain.

I thank the authors for the additional explanations and experiments. To me, it is expected that clipping weakens the self-stabilization effect, allowing the trajectory to enter sharper regions. The mechanism described in Damian et al. (2023) already seems sufficient to explain this phenomenon. I therefore remain skeptical about the novelty of this work.

Thank you for clarifying. We agree that the mechanism of Damian et al. gives a natural explanation for why limiting oscillations can let batch-clipped gradient descent reach sharper regions.

Our first-order results, however, go beyond this observation about sharpening. We derive the step size at which a perturbation along the critical direction begins to grow under the clipped update. For active batch clipping, this threshold depends on the curvature projected orthogonally to the gradient and on the clipping factor. We observed perturbations shrink below the predicted threshold and grow above it (Figure 6). For per-sample clipping, the threshold depends on the individual sample gradient directions, which enter the update Jacobian separately (Proposition 3.2).

Our zeroth-order analysis goes a step further by predicting what happens beyond the stability threshold. For a quadratic loss under Assumption 4.2, our batch-clipped covariance model predicts how the size of stationary fluctuations depends on the clipping threshold once the origin becomes mean-square unstable (Theorem 4.3).

We see these clipping-specific predictions as the central contribution of our work, and we hope you will take them into account when assessing the paper’s novelty.

Reviewer_Au5AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The submission analyzes the edge of stability for first-order and zeroth-order optimizers with gradient clipping. For the first-order methods, they show that clipping shifts the EoS threshold. For the zeroth-order methods, they derive the conditions using a balancing condition (which is affected by clipping). Finally, they validate their EoS threshold on a CIFAR image classification setup.

Critical strengths
  • The problem of the EoS threshold for first- and zeroth-order optimizers is well motivated
  • The main text is easy to read and understand
Critical weaknesses
  • Assumption 4.2 is not validated.
  • The manuscript includes unedited text, such as "the older text says,” “the earlier draft,” and “the requested codimension test”. The paper analyzes the experiments in a very restricted setup: a small CNN on 1,000 CIFAR-10 images and MSE loss. Moreover, they do not test whether the results hold at large learning rates or for Transformers. The effect of loss function is also not studied.
Questions for authors
  1. In neural network training, one cannot assume that the curvature is fixed. How does the changing curvature affect the balancing condition in Section 4?
  2. Can you numerically test Assumption 4.2?
  3. How is γ\gamma computed in practice as it depends on the minima x∗x^*.
  4. Can you analyze the results for different experimental setups (transformer training), learning rates, and loss functions?
2 comments

Thank you for the suggestions. We have added numerical tests and experiments and clarified the estimator and evolving-curvature interpretation.

W1 & Q2. Numerical Validation of Assumption 4.2

We have added direct tests of all three moment identities, together with one-step and multi-step covariance comparisons. The tests cover 27 quadratic configurations and complementary neural-network continuations (Lines 353–356 and 508–513; Appendix H.10). Please see the general response for the shared response on the protocol, results, and scope.

W2. Editorial Corrections

We have removed the cited editorial remnants, including “the older text says,” “the earlier draft,” and “the requested codimension test.” We also reorganized the appendices into theoretical derivations (B–E), measurement protocols (F), FO/ZO experiments (G–H), and additional architectures, losses, and learning rates (I).

W3 & Q4. Architectures, Learning Rates, and Loss Functions

We have added all three requested experimental comparisons, each covering FO/ZO and batch/per-sample clipping with three clipping thresholds. Lines 518–522 summarize the findings; Appendix I gives the protocols and full results.

  • Doubled learning rates: In CNN–MSE, the measured FO sharpness proxies remain within 6% of 2/η2/\eta over the final 20% of training across all tested clipping thresholds. The ZO results also show approximate agreement with the predicted batch and per-sample covariance balances (Figure 25).
  • Transformer architecture: The small ViT–MSE experiments show agreement with the predicted FO stability edges and ZO covariance balances, with the degree of agreement varying across clipping thresholds (Figure 26).
  • Different loss: For CNN–CE, Figure 27 shows the settings in which the measurements closely match the predicted FO stability edges and ZO covariance balances. Subsequent curvature decreases are also observed, consistent with the cross-entropy behavior reported by Cohen et al. (2021).

The extensions retain the 1,000-example, four-class subset and one initialization seed.

Q1. Effects of Evolving Curvature on the Covariance Balance

Section 4 derives the covariance balance for a fixed quadratic model. In neural-network experiments, we evaluate the balance conditions using the curvature measured along the training trajectory. This follows the conventional EoS approach: Cohen et al. (2021) use the stability threshold of a fixed quadratic and compare it with the evolving Hessian sharpness during neural-network training.

For batch ZO, Figure 2c compares γ^ktr⁡(H)\widehat{\gamma}_k\operatorname{tr}(\boldsymbol H) with 2/η2/\eta, using the current Hessian trace and a local estimate of the gradient amplitude. Thus, both curvature and the amplitude-dependent clipping factor are updated as training progresses. At one million steps, the ratio γ^ktr⁡(H)/(2/η)\widehat{\gamma}_k\operatorname{tr}(\boldsymbol H)/(2/\eta) lies between 0.927 and 1.022 for five of the six clipping settings, close to the balance reference of 1. For per-sample ZO, Figure 2d evaluates the directional balance using the weighted curvature measured at successive checkpoints. These results support the empirical relevance of the balance conditions along evolving neural-network trajectories.

Q3. Computing γ\gamma without Knowing the Minimizer

No estimate of the minimizer is required. Although the theoretical representation g=H(x−x⋆)\boldsymbol g=\boldsymbol H(\boldsymbol x-\boldsymbol x^\star) involves the unknown minimizer, the quantity needed for γ\gamma can be estimated from scalar directional derivatives already recorded during ZO training.

For a Gaussian direction ut∼N(0,I)\boldsymbol u_t\sim\mathcal N(0,\boldsymbol I), the exact directional derivative dt=ut⊤gt\boldsymbol d_t=\boldsymbol u_t^\top \boldsymbol g_t satisfies

Eut[dt2∣gt]=∣gt∣2.\mathbb E_{\boldsymbol u_t}[d_t^2\mid \boldsymbol g_t]=\\|\boldsymbol g_t\\|^2.

Thus squared directional derivatives estimate the squared gradient norm without reconstructing gt\boldsymbol g_t from x⋆\boldsymbol x^\star or computing the full gradient. We use the recorded finite-difference estimates dtμ≈dtd_t^\mu\approx d_t, obtained from two loss evaluations before clipping. Averaging their squares estimates the mean squared gradient norm, which equals T(W)=tr⁡(H2W)=E∣g∣2T(\boldsymbol W)=\operatorname{tr}(\boldsymbol H^2 \boldsymbol W)=\mathbb E\\|\boldsymbol g\\|^2 under the local quadratic model. We compute

\widehat\kappa_k=\frac{C}{\sqrt{\widehat T_k}},\qquad \widehat\gamma_k=\frac{\mu_2(\widehat\kappa_k)}{p(\widehat\kappa_k)}.$$ The functions $p(\kappa)=\Pr(|Z|<\kappa)$ and $\mu_2(\kappa)=\mathbb E[\min(Z^2,\kappa^2)]$, for $Z\sim\mathcal N(0,1)$, are evaluated analytically. Figure 2c plots $\widehat\gamma_k\operatorname{tr}(\boldsymbol H)$. The estimator is stated in Lines 387–390; Appendix F.3 details the procedure and its finite-difference, sampling, and approximate-stationarity qualifications.

Dear Reviewer Au5A,

Thank you for suggesting a direct test of Assumption 4.2. We now report how closely its three moment relations hold across 27 quadratic settings and on CNN training trajectories (Appendix H.10). If you have a chance to look over the changes, we’d be grateful to hear whether any of your concerns remain.

Comments on the paper

1 comment

Numerical Verification of Assumption 4.2

We provide a shared response to Reviewer L1tJ’s W4 and Reviewer Au5A’s W1/Q2.

Thank you for requesting a direct assessment. We measure the discrepancies between the two sides of the three moment relations in Eq. (56) to assess the accuracy of Assumption 4.2. The numerical assessment is introduced in Lines 353–356 and summarized in Lines 508–513 in the revised manuscript.

Quadratic tests. We evaluate 27 configurations spanning three dimensions, three spectra, and three relative learning rates (Appendix H.10, Table 17 and Figure 21). The approximation replaces the random gradient norm by its root-mean-square value. When many Hessian eigendirections contribute comparably to the norm, their fluctuations can average out, making this replacement more accurate. A few dominant Hessian eigenvalues can reduce this averaging effect. At step 8,192, all three moment discrepancies are below 4% in every flat-spectrum configuration. However, the third moment discrepancy reaches 229.54% in the single-dominant-eigenvalue case with d=16d=16 and η=2η0⋆\eta=2\eta_0^\star, the largest moment discrepancy in the sweep. These results support the accuracy of the approximation in the tested flat-spectrum regime and show its sensitivity to spectral anisotropy.

NN moment measurements. We evaluated the three moment relations using the measured neural-network gradients on CNN continuations (Appendix H.10, Table 18b). Across two checkpoints, three score rules, and four continuation times, the maximum measured discrepancies are 5.73%, 0.75%, and 9.27%, respectively, including reference-point sensitivity checks for the first relation. These results show that all three moment relations hold approximately in the tested CNN continuations.

Numerical checks within the closed covariance model also support the fixed-point, trace-bound, and local-attraction predictions derived under Assumption 4.2 (Theorems 4.3–4.4). See our response to Reviewer rgai, W1, for details (Appendix H.8, Table 15).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.