- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This work studies the role of clipping in first-order (FO) and zeroth-order (ZO) optimization regarding stability. It uses usual GD/SGD as the FO setting. It argues that clipping affects FO and ZO stability through distinct mechanisms: (i) for FO, clipping modifies the stability boundary; (ii) for ZO, clipping enables bounded dynamics beyond the linear stability threshold through a nonlinear covariance balance.
Depsite the clipping role in optimization stability is interesting, i think the theoretical significance seems limited since the practical setting nowadays is Adam family instead of GD/SGD; the novelty seems marginal since the main results substantialy overlap with existing literature, which has extensively explored this problem. The main results in this manuscript appears a summarization with incremental results. however it doesn't cite the prior work that included the related results.
Additionally. i feel some results seem wrong such as Proposition 3.2 which is false (I'll state in detail in the questions-to-authors).
[1]. Arora et al. ICML22. Understanding Gradient Descent on the Edge of Stability in Deep Learning. [2]. Revisiting Gradient Clipping: Stochastic bias and tight convergence guarantees. https://proceedings.mlr.press/v202/koloskova23a/koloskova23a.pdf [3]. Song et al. ICML26. Zeroth-Order Optimization at the Edge of Stability. https://openreview.net/forum?id=s87tQaKAER
I apologise that there are more literature I can't list one by one.
Critical strengths
- The presentation is easy to follow.
- Notations appear good in theoretical community so I can easily follow the mathematics without burden.
Critical weaknesses
- There lacks sufficient experiments for verifying its lemmas/theorems/propositions numerically. This can't convince me to believe the results are sound. In a sound theoretical work, I won't blindly believe the results before i see numerical verifications for the major theorems/propositions.
I believe this is the improvement room that the work can be improved by including sufficient experiments to show that the theorems/propositions numerically hold. but before that, the manuscript needs significant efforts to fix the wrong mathematical deductions.
- This work appears collecting results existing in literature however it doesn't properly cite the related work. This is why the novelty seems marginal.
for example. [1] ICML22. Arora et al. Understanding Gradient Descent on the Edge of Stability in Deep Learning. https://proceedings.mlr.press/v162/arora22a.html [2] ICML26. Song et al. Zeroth-Order Optimization at the Edge of Stability. [3] ICLR26. Mulayoff et al. https://openreview.net/forum?id=hBNC8w9pd7 [4] ICML26. https://proceedings.mlr.press/v306/islamov26a.html
- Some theorems/propositions are technically wrong.
eg Proposition 3.2. sec4.1.1 singular-Hessian qualification. theorem 4.1 and appendix c.12. theorems 4.3 and 4.4. there may exist more wrong results, i haven't checked all results yet.
Questions for authors
- Regarding Proposition 3.2.
I think this proposition needs effort to fix since it's wrong.
for dynamics
Conditions can't guarantee the claims when eigenvalues are degenerate.
Can you further clarify and present the refined results?
4 comments
W3. Correctness of Proposition 3.2, the Singular-Hessian Treatment, and Theorems 4.1, 4.3, and 4.4
We have corrected the missing stability qualifications of Proposition 3.2 and made the fixed-point and attraction arguments explicit.
- Proposition 3.2: The missing zero-mode and Jordan qualification is now stated explicitly; see Q1 below.
- Theorem 4.1 and singular Hessians: The main theorem now assumes and explicitly states the conditions for the scalar threshold equation and trace bound. Appendix C.5 (formerly Appendix C.12) treats through the projected process and distinguishes its stability from full-space covariance boundedness.
- Theorem 4.3: The revised proof establishes existence and uniqueness of the nonzero PSD fixed point for . It also proves that the zero covariance is locally attracting for and unstable for (Appendix D.2). At equality, we assert only that zero is the unique PSD fixed point.
- Theorem 4.4: We explicitly require and control the full symmetric-matrix Jacobian, including off-diagonal modes (Appendix D.3).
The balance equation and trace-error bound are unchanged. If a specific step in the proofs of Theorems 4.3–4.4 remains problematic, we would appreciate your identifying it so that we can address it directly.
W4. Relevance of GD/SGD Analysis to Modern Optimization
Our results provide a foundation for analyzing clipping in modern optimizers. GD and random-direction ZO isolate clipping’s effects from momentum and adaptive preconditioning (Lines 46–48), allowing us to characterize how clipping changes the local stability boundary and produces a nonzero covariance balance. These effects must be accounted for when extending stability analysis to clipped Adam or ZO-Adam.
Such an extension requires incorporating the clipped-update derivatives and moment calculations developed here into the optimizer’s joint parameter and internal-state dynamics. Frozen preconditioning provides a starting point; evolving momentum and preconditioners require further analysis of their interaction with clipping. We now discuss existing Adam EoS analyses (Lines 74–76) and identify this extension in the Conclusion (Lines 535–536).
Q1. Stability Conditions for Degenerate Eigenvalues in Proposition 3.2
We agree that eigenvalues alone do not guarantee bounded iterates. We have revised Lines 206–215 and Proposition 3.2 to explicitly require , excluding Jordan blocks of size greater than one at the zero eigenvalue of . If this condition fails, linearized perturbations can grow without bound for every positive step size.
Under the revised assumptions, Eq. (12) gives the supremum of stable step sizes: every strictly smaller positive step is stable. At the endpoint, boundedness additionally requires all Jordan blocks associated with eigenvalues of having absolute value one to have size one. This distinction is explicit in Lines 214–215 and Appendix B.4. Appendix B.4 provides the proof and a counterexample. The spectral proxy bounds remain unchanged.
Thanks for the replies.
I thank the replies from the authors.
Thank you for taking the time to read our responses. If any of your concerns remain, we’d appreciate knowing which points you think we could address more fully in the paper.
Thank you for the careful review. We have expanded the numerical checks, clarified the contributions, and corrected the stability qualifications.
W1. Numerical Validation of the Main Theoretical Results
The original submission included numerical checks of FO stability and the linear batch-ZO boundary. In this revision, we supplement these with direct perturbation tests for batch FO and numerical tests of the nonzero covariance balance and local attraction within the closed covariance model.
FO stability (Theorem 3.1 and Proposition 3.2). The original submission already reported numerical checks of the stability formulas and proxy accuracy. For batch clipping, the inequalities in Eq. (9) hold at all six active checkpoints. For per-sample clipping, the symmetric proxy S=λ1(A−Rs) approximates 2/ηps∗ with relative error at most 0.015%, and all six checkpoints satisfy the required cluster-separation condition 4∥Ra∥2≤Γc. These existing results are summarized in Lines 182–185 and 233–237, with details in Appendix G.1, Figure 5 and Tables 4 and 6.
In this revision, we added independent one-step perturbation tests showing that perturbations along the tested eigenvector shrink below and grow above the predicted critical learning rate, consistent with Theorem 3.1 (Lines 485–487; Appendix G.2.1, Figure 6).
Linear batch-ZO boundary (Theorem 4.1). The following checks were already included in the original submission. Six synthetic settings satisfying the alignment assumption confirm that the numerically computed scalar thresholds obey the trace-approximation bound (Appendix H.5, Table 13). At 24 neural-network checkpoints, the scalar and trace approximations differ from the reduced-operator boundary by 0.042%–0.440% and 3.05%–7.04%, respectively (Lines 321–323; Appendix H.3, Table 12).
Nonzero covariance balance and local attraction (Theorems 4.3–4.4). The original submission already included neural-network trajectory experiments assessing the balance predicted by Theorem 4.3 (Figure 2c in the revised manuscript). Specifically, Eq. (21) predicts ητ(κ⋆)=2, and Eq. (23) bounds the error of approximating τ by γ,tr(H). The existing experiments therefore compare γ,tr(H) with 2/η along training trajectories. At 106 updates, ηγ,tr(H)/2 lies between 0.927 and 1.022 for five of the six clipping thresholds; the strongest-clipping setting remains at 0.685 and is still increasing (Table 19). These observations support approximate agreement with the predicted balance in five settings, while the strongest-clipping setting has not reached it.
In this revision, we added checks of the trace-approximation bound in Eq. (23) and the local-attraction prediction in Eq. (24) within the closed covariance model. Across six quadratic settings, the trace-approximation errors satisfy Eq. (23). The full covariance-map Jacobian has spectral radius 0.9352–0.9533, and nearby perturbations decay under the map, consistent with Eq. (24) (Appendix H.8, Table 15). We additionally evaluate Theorem 4.4’s sufficient conditions in frozen models constructed from existing neural-network checkpoints (Appendix H.9, Table 16).
W2. Novelty and Positioning Relative to Prior Work
Our contributions are clipping-specific stability and covariance results that are distinct from the cited analyses. Four of the five works mentioned—Arora et al. (2022), Koloskova et al. (2023), Song et al. (2026), and Mulayoff and Stich (2026)—were already cited and discussed in the original submission, Lines 64–73. We have added Islamov et al. (2026) in revised Lines 69–71 and sharpened the distinctions in Lines 66–78 and Appendix J.1–J.4: