acceptodds
Under review as a conference paper at ICLR 2027

Convergence of Large-Stepsize SGD for the Cross-entropy Loss via Stochastic Lyapunov Stability

Abstract

Modern deep learning has been shown to operate using learning rates far larger than those justified by classical optimization theory. Most prior analyses of this phenomenon focus on deterministic gradient descent, leaving the stochastic setting largely unexplored. In this work, we provide sharp convergence guarantees for Stochastic Gradient Descent (SGD) applied to the multiclass cross-entropy loss, for both linear classifiers and two-layer neural networks. We show that the stochasticity of SGD may cause the dynamics to alternate between an edge-of-stability regime that is dominated by curvature-driven oscillations, and a stable regime in which the expected loss decreases at a controlled rate. Despite that, we prove that SGD stabilizes the dynamics, ensuring that the iterates return to stability in a fixed number of iterations and allowing convergence in the best-iterate sense even with large learning rates. Experiments validate our theoretical findings and illustrate the benefits of SGD in the large-stepsize regime.

Primary area:
optimization
Keywords:
edge of stability, large learning rates, stochastic optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_hkqmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates stochastic gradient descent (SGD) with a large constant step size under the multiclass cross-entropy loss. Rather than labeling every increase in loss as "instability," the authors define a "stable region"—a regime where the loss exhibits a downward trend in terms of conditional expectation. For linear classifiers, the authors provide a "best-iterate" loss bound for any positive step size, and they explore how SGD enters this stable region, occasionally exits it due to minibatch noise, and subsequently returns. Furthermore, the paper provides a "last-iterate" loss bound for cases where the training trajectory remains within this region. This analysis is extended to two-layer neural networks, with experiments illustrating these distinct phases of the training process.

Overall, I find the stochastic stability perspective promising, but I cannot recommend acceptance in the paper’s current form. The activation lemma appears to be incorrect, and the proofs rely on a zero-initialization assumption that is missing from the theorem statements. If the authors can correct these issues without weakening the main results, I would be glad to reconsider my assessment.

Critical strengths

What I find most interesting is the attempt to make edge-of-stability behavior precise for SGD. Defining stability through conditional expected loss decrease makes sense when individual updates are noisy. The paper also goes beyond a best-iterate guarantee by asking what happens after the trajectory enters the stable region, including whether noise can push it out and how it returns. Extending the analysis from binary logistic regression to multiclass cross-entropy and two-layer networks broadens its relevance.

Critical weaknesses

There are two issues in the theoretical formulation that warrant attention. Lemma 3.2 identifies Leaky GELU and SiLU as activation functions that satisfy Assumption 3.1; however, their derivatives do not adhere to the bounds required for the proof of this lemma. For instance, the derivative of GELU is approximately -0.083 at x=−1x = -1 and 1.083 at x=1x = 1. If gg is the GELU function, the derivative of the hybrid function proposed in the paper—given by c+(1−c)g′(x)c + (1-c)g'(x)—could fall below cc or exceed 1. This appears to limit the applicability of the conclusions drawn for two-layer models, although these results might still hold for other activation functions.

Additionally, the statement of Theorem 2.2 likely requires an explicit initialization condition. The bound provided by the theorem does not explicitly depend on the initial weights W0W_0; however, the proof relies on the assumption that W0=0W_0=0. Without this condition, a poorly initialized classifier could serve as a counterexample at t=1t=1. I suggest that the authors either clarify the initialization scheme used or incorporate the influence of initialization into the bound formula.

While the experimental results offer useful insights, they do not directly validate the concept of stability proposed in the paper. In particular, when a sharp spike in loss is observed, it is unclear whether this is due to minibatch noise or an excessively large step size—distinguishing between these two factors is, in fact, one of the paper's core motivations. Measuring the "conditional loss drift" and the extent to which the loss deviates from the established stability threshold would significantly strengthen the empirical evidence.

Questions for authors
  1. Which of the activation function examples mentioned in Lemma 3.2 satisfy the conditions of Assumption 3.1? Could this lemma—and the corresponding conclusions for two-layer networks—be revised to better reflect their scope of applicability?
  2. Does Theorem 2.2 (and the related results for two-layer networks) implicitly assume "zero initialization"? If not, how should the bounds account for the dependence on the initial weights?
  3. Is it possible to directly measure the proposed stability criterion—for instance, by quantifying how frequently stochastic gradient descent (SGD) exits the stable region, or the time required for it to return?
1 comment
AuthorsOpenReview ↗

Response to Reviewer hkqm

We thank the reviewer for the thoughtful and constructive review, and for finding the stochastic-stability perspective promising. We particularly appreciate the recognition that conditional expected descent provides a meaningful notion of stability in stochastic optimization, as well as the importance of distinguishing entrance into, retention in, and return to the stable region.

Q1 — Lemma 3.2. We thank the reviewer for the careful reading. We have updated Lemma 3.2 accordingly, refining the construction for GELU and SiLU. With d=1+2c3d=\frac{1+2c}{3} and ϕ~(x)=d x+1−d4ϕ(x)\tilde\phi(x)=d\,x+\frac{1-d}{4}\phi(x), the bound ∣ϕ′∣≤2|\phi'|\le2 gives ϕ~′∈[c,2+c3]⊆[c,1]\tilde\phi'\in[c,\frac{2+c}{3}]\subseteq[c,1]. Therefore all activations listed in Lemma 3.2 (GELU, Softplus, SiLU, Huberized ReLU, tanh, sigmoid) satisfy Assumption 3.1; we verified the constants both analytically and numerically. Moreover, the neural-network results are formulated under Assumption 3.1 and do not depend on any specific activation, so they hold for every activation satisfying it.

Q2 — Initialization. Thank you for the remark. Our analysis considers the zero initialization W0=0W_0=0; the results remain valid for a general initialization, up to an additive term depending on the norm of W0W_0. We have updated the manuscript accordingly:

  • Two-layer network. Theorem 3.3 is now stated for an arbitrary W0W_0, with an explicit additive term γ~2∥W0∥F2/m\tilde\gamma^2\|W_0\|_F^2/m in the numerator of the bound: the original constants are recovered for W0=0W_0=0. This covers standard random initializations with ∥W0∥F2=O(m)\|W_0\|_F^2=O(m).
  • Linear classifiers. A remark after Theorem 2.2 states that the zero initialization is assumed without loss of generality, and that for a nonzero initialization a term depending on ∥W0∥F\|W_0\|_F is added to the bounds.

Q3 — Direct experimental validation of stochastic stability. Thank you for the suggestion! One useful feature of the proposed notion of stability is that one can measure whether the loss exceeds the stability threshold and thereby decide whether the dynamics are stable or unstable. We plan to add plots showing exactly when the iterates cross the stability threshold. These will make apparent when the dynamics are stable, and which spikes are attributable to minibatch noise and which to large stepsizes.

We sincerely thank the reviewer for the detailed feedback.
We believe that we have addressed the reviewer's concerns and hope for a positive re-evaluation of our work. We remain available for any further questions.


Reviewer_UHfYOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the convergence of SGD in a large stepsize regime, in order to improve our theoretical understanding of the Edge of Stability phenomenon. The cross entropy loss is considered, for linear classifiers and two-layer neural networks.

Critical strengths

None

Critical weaknesses

The critical weakness of the paper is that claims are not adequately justified, and this seems not easy to fix.

  • Maybe the most important issue is that in case of two-layer neural networks, convexity of the loss LL is used several times, e.g. to prove Lemma D.10, D.13 and D.16. I do not see any reason why LL should be convex in this setting, and this seriously questions the soundness of the results of Section 3.
  • There is an important issue in the proof of Lemma C.11. In (86), an inequality of the form A0<A1A_0 < A_1 is established. (87)-(89) establishes A2<A1A_2 < A_1. Then, (90) deduces that A0<A2A_0 < A_2, which is clearly not true in general. As a more minor remark about this Lemma, I note that the formulation "for the event Et\mathcal{E}_t to hold, it suffices that" is misleading (if I understood well your proof). The same remarks holds for Lemma D.15.
  • Lemma 3.2 is also problematic, as for some choices of link functions the leaky variant does not verify Assumption 3.1. This is the case for GELU for instance, where we have ϕ~′(1)>1\tilde \phi'(1) > 1.
  • There are improper uses of stopping times. Precisely in Lemma C.12, between (104) and (105) Lemma C.7 is invoked, which provides a recursion for times t≥1t \ge 1, from which a result involving the random time toutt_{out} is claimed. Yet, we cannot deduce directly from a result on E[Vt]\mathbb{E}[V_t] a result for E[Vτ]\mathbb{E}[V_{\tau}] where τ\tau is a random time. Refined arguments are needed such as stopping time theorems.
Questions for authors

Please see my weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer UHfY (1/2)

We thank the reviewer for the careful examination of our theoretical arguments. We address each comment below.

Q1 — Lemmas D.10, D.13, D.16. We thank the reviewer for this comment. We have updated the manuscript accordingly: the two-layer analysis now relies only on the convexity of the cross-entropy with respect to the logits, applied to the linearized logits z^i(W;U)=zi(W)+Ji(W)(U−W)\hat z_i(W;U)=z_i(W)+J_i(W)(U-W). Lemma D.10 establishes ⟨∇L(W),U−W⟩F≤L^W(U)−L(W)for all W,U.\langle\nabla L(W),U-W\rangle_F\le\hat L_W(U)-L(W)\quad\text{for all }W,U. For the comparator U=ln⁡(γ~2ηT)+2κγ~W‾∗U=\frac{\ln(\tilde\gamma^2\eta T)+2\kappa}{\tilde\gamma}\overline W_*, near-homogeneity gives L^W(U)≤K−1γ~2ηT\hat L_W(U)\le\frac{K-1}{\tilde\gamma^2\eta T} uniformly in WW, which is the bound required by the subsequent arguments. Lemmas D.5, D.11, D.13 and D.17 build on this inequality, and Theorems 3.3–3.5 keep their statements.

Q2 — Logical implication in Lemma C.11. We thank the reviewer for the comment and have updated the manuscript accordingly. Let us know if there are any still any remaining concerns.

AuthorsOpenReview ↗

Response to Reviewer UHfY (2/2)

Q3 — Lemma 3.2 and GELU. We thank the reviewer for this remark. We have updated Lemma 3.2 accordingly, refining the leaky construction so that GELU, Softplus and SiLU remain covered. With d=1+2c3d=\frac{1+2c}{3}, we define ϕ~(x):=d x+1−d4 ϕ(x).\tilde\phi(x):=d\,x+\tfrac{1-d}{4}\,\phi(x). Since ∣ϕ′∣≤2|\phi'|\le2 for GELU, Softplus and SiLU (their ranges are [−0.13,1.13][-0.13,1.13], [0,1][0,1] and [−0.10,1.10][-0.10,1.10]), we obtain ϕ~′∈[d−1−d2, d+1−d2]=[c,2+c3]⊆[c,1]\tilde\phi'\in[d-\frac{1-d}{2},\,d+\frac{1-d}{2}]=[c,\frac{2+c}{3}]\subseteq[c,1]. The smoothness and near-homogeneity constants follow from sup⁡∣ϕ′′∣≤0.8\sup|\phi''|\le0.8 and sup⁡∣ϕ−zϕ′∣≤ln⁡2\sup|\phi-z\phi'|\le\ln2. We verified these constants both analytically and numerically.

Q4 — Random times in Lemma C.12. We thank the reviewer for this observation. We clarify that toutt_{\mathrm{out}} in Lemma C.12 denotes a deterministic iteration fixed a priori, rather than a random stopping time defined as the first exit from the stable region. Under this interpretation, Lemma C.7 can be applied directly at toutt_{\mathrm{out}}, and the transition from (104) to (105) is justified without an optional-stopping argument. We will revise Lemma C.12 to explicitly fix a deterministic iteration toutt_{\mathrm{out}} and state the result as a high-probability entrance guarantee within the subsequent tret_{\mathrm{re}} iterations.

We thank the reviewer for the careful reading of our paper. We believe that we have addressed the reviewer's concerns and hope for a positive evaluation of our work. We remain available for any further questions.


Reviewer_UhCVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the question of convergence of SGD for multi-class cross-entropy. Authors cover the particular examples of linear classifiers and shallow neural network. They identify the stable descent region (ie where descent lemma works), and provide entrance, retention and return guarantees.

Critical strengths

Good mathematical framework, and detailed derivations. Good organization of the proof with separation into the stable region, and guarantees of entering/staying in that region. The focus on mini-batch SGD and multi-class cross-entropy is relevant, and the explicit dependence on step size, batch size and separation margin makes the bounds easier to interpret. The distinction between expected descent and staying inside the stable region is useful in the stochastic setting.

Critical weaknesses

One is about the significance. Essentially, the qualitative idea behind the stable-region part for separable cross-entropy was established before (see e.g. Cohen's EoS paper [1]) with the back-of-the-napkin calculation shows that the Hessian dies, and one can thus apply the descent lemma (it was also empirically known/established). This paper provides exact bounds, but the bounds are potentially very conservative. For the linear case, the stable set requires L(W)≤L~=1/[8η(1+n/(bmin⁡{γ2,1}))]L(W)\leq \widetilde L =1/[8\eta(1+n/(b\min\{\gamma^2,1\}))], or approximately bγ2/(8ηn)b\gamma^2/(8\eta n) when γ≤1\gamma\leq1 and n≫bγ2n\gg b\gamma^2. Thus the certificate can require impractically small loss for stabilty. Moreover, this region is often reached only after all the examples are classified correctly. The certificate then describes late-stage margin growth rather than the earlier classification-learning phase, which limits its relevance to the training dynamics motivating the paper. For a concrete scale check, consider MNIST-experiemnt-style n=60000n=60000, b=128b=128, η=1\eta=1. The threshold is then about 3×10−63\times10^{-6} for γ=0.1\gamma=0.1, or 3×10−83\times10^{-8} for γ=0.01\gamma=0.01. A single misclassified example contributes at least log⁡2\log 2 to the total cross-entropy, so average loss below log⁡2/n≈1.2∗10−5\log 2/n\approx1.2*10^{-5} already implies 100% training accuracy. Moreover, the stability certificate presented in the paper creates a conservative region of small full-loss curvature. It does not identify the actual stochastic instability boundary - with the latter is what we want for an EoS-like regime (with more comments about EoS below)

The other concern about the manuscricpt is about novelty - and the parts that are truly novel I have concern about. In my opinion, the advance over Wu et al. COLT '24 [2] is limited, and needs more clear isolation. A lot of the arguments presented in the paper are essentially extensions of what has been done in Wu et al. The manuscript acknowledge that for the multi-class vs two class CE. Yet, this is the case for the entrance and return arguments, where the convergence machinery used is very similar to the ones of Wu et al. I think a more precise attribution should be made - which would allow to also evaluate the novelty of the contribution. As it stands right now, this seems more of an extension of those proofs. Additionally, in the manuscript there is the proof of convergence for a fixed-sign two-layer neural network - but, the proof and the technique there seems quite similar to the one in Cai et al 2024 [3]. The major differences are again the multi-class extension and SGD. I am not certain if the treatment of stochasticity constitutes sufficient difference. The multiclass is achieved through introduction of a lower bound on the derivative of the non-linearity. Yet, this bound is very restrictive, and with γ=0.1\gamma = 0.1, the lower bound needs to be α≈0.9\alpha \approx 0.9 and γ=0.01→α≈0.99\gamma=0.01 \rightarrow \alpha\approx0.99, which forces the activation very close to an affine function. The restrictiveness of the setting is exacerbated by the fact that the data is already assumed linearly separable - that is similar to an assumption of Cai et al., although their restriction on the model doesn't involve the linear separable gap. On top of that, it seems like there is an error in the proof, using convexity of the loss in the proof for the two-layer NN in Lemma D.10. This might be fixable (eg along the lines of Cai et al.), but non-trivially, as there is a reason that in Wu et al. they restrict to working in the NTK regime, essentially relying on convexity-like arguments.

Separately, there is an issue with positioning around EoS - I do not think the connection to EoS is demonstrated sufficiently; yet, the paper uses the EoS framing extensively. In particular, the EoS is a phenomena that is essentially progressive-sharpening-driven (and therefore neutral-network specific) - and it is specifically about staying near the instability boundary, as there are two factors that keep the training dynamics near that boundary - PS from below, and self-stabilization through third-order interactions from above. Progressive sharpening is particularly important to distinguishing an initially stable learning rate that becomes destabilizing during training from a learning rate that is already aggressive at initialization and eventually becomes benign because we got into a stability region. The present paper doesn't measure progressive sharpening, neither does the theory rely on it explicitly (although a-priori there could be an EoS-like regime without PS) - but more importantly, does not establish tracking of a instability boundary. The authors mention that their mechanism is different from Damian et al. self-stabilization, but their interpretation as a characterization of EoS dynamics remains unsupported.

Moreover, the discussion of references for stochastic EoS-like regime is incomplete. There is a missing reference for Andreyev and Beneventano (2024) [5] - although their work concerns mostly an EoS-like regime for SGD under MSE, it is directly relevant to the paper's motivation in discussion of instability of SGD.

Lastly, there is an issue with experiemebnts not testing the distinctive theoretical claims. The experiments primarily show that, for the selected learning rates, training loss decreases over the displayed horizon, sometimes faster for larger rates, and that loss spikes become less pronounced with larger batches or later training; the rescaled-loss curves provide some evidence consistent with the rate of convergence. Importantly, these observations do not validate the proposed stochastic stabilization mechanism. Neither there is an established source of loss spikes (which would otherwise require additional measurements), yet the the experimental discussion largely uses spikes and subsequent smoothing to identify unstable and stable regimes.

With all of the above, although the work has its merits, I don't think the work warrant acceptance in its current form.

References

[1] Jeremy Cohen, Simran Kaur, Yuanzhi Li, J. Zico Kolter, and Ameet Talwalkar. Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability. ICLR, 2021.

[2] Jingfeng Wu, Peter L. Bartlett, Matus Telgarsky, and Bin Yu. Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiency. COLT, 2024.

[3] Yuhang Cai, Jingfeng Wu, Song Mei, Michael Lindsey, and Peter L. Bartlett. Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimization. NeurIPS, 2024.

[4] Alex Damian, Eshaan Nichani, and Jason D. Lee. Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability. ICLR, 2023.

[5] Arseniy Andreyev and Pierfrancesco Beneventano. Edge of Stochastic Stability: Revisiting the Edge of Stability for SGD. arXiv:2412.20553, 2024.

Questions for authors
  • Lemma D.10: The proof appears to use convexity of the loss in the neural-network parameters, which does not hold generally. Can you justify this step under your assumptions or provide a corrected argument? Does the correction preserve the stated guarantees?
  • Relevance of the stable region: For the parameters in your experiments, what is the certified loss threshold, and when—if ever—does training reach it? Does the certificate explain the observed stabilization, or only a much later phase after training accuracy reaches 100%?
  • Nvelty: Which technical arguments are genuinely new relative to Wu et al. (2024) and Cai et al. (2024)? In particular, what obstacle introduced by stochasticity requires a substantially new argument rather than an adaptation of their proofs?
  • Nonlinear setting: For small separation margins, the derivative lower bound forces the activation to be nearly affine. What meaningful nonlinear behavior does the result cover, and can this restriction be relaxed while retaining the guarantees?
  • EoS and experimental evidence: Can you identify the relevant stochastic instability boundary and show that the dynamics hover near it? Beyond loss spikes and their eventual smoothing, what measurements support the proposed stabilization mechanism
3 comments

Response to Reviewer UhCV (1/3)

We thank the reviewer for the detailed and constructive review, and for appreciating the mathematical framework, the organization of the proofs, and the explicit dependence of our bounds on η\eta, bb and γ\gamma.

W/Q1 — Lemma D.10. We thank the reviewer for this remark. We have updated the manuscript accordingly: the comparator argument for the two-layer network now relies only on the convexity of the cross-entropy with respect to its logits, combined with a linearization argument in the spirit of Cai et al. (2024). Let z^i(W;U):=zi(W)+Ji(W)(U−W)\hat z_i(W;U):=z_i(W)+J_i(W)(U-W) denote the logits linearized at WW. The chain rule and convexity in the logits give, for all W,UW,U, ⟨∇L(W), U−W⟩F  ≤  L^W(U)−L(W),L^W(U):=1n∑iℓ(z^i(W;U),yi).\langle\nabla L(W),\,U-W\rangle_F\;\le\;\hat L_W(U)-L(W),\qquad \hat L_W(U):=\tfrac1n\textstyle\sum_i\ell(\hat z_i(W;U),y_i). For U=ln⁡(γ~2ηT)+2κγ~W‾∗U=\frac{\ln(\tilde\gamma^2\eta T)+2\kappa}{\tilde\gamma}\overline W_*, near-homogeneity (∣ϕ(s)−sϕ′(s)∣≤κ|\phi(s)-s\phi'(s)|\le\kappa) and ϕ′∈[α,1]\phi'\in[\alpha,1] give linearized margins of at least ln⁡(γ~2ηT)\ln(\tilde\gamma^2\eta T) uniformly in WW. Hence L^W(U)≤K−1γ~2ηT\hat L_W(U)\le\frac{K-1}{\tilde\gamma^2\eta T}, which is the bound required by the subsequent arguments.

This inequality (Lemma D.10) is used in Lemmas D.5, D.11, D.13 and D.17. The statements of Theorems 3.3–3.5 remain unchanged.

W/Q3 — Novelty relative to Wu et al. (2024) and Cai et al. (2024). To make the attribution precise, we have added a paragraph "Comparison with prior analyses" in Section 1:

  • Building on prior work. The best-iterate bounds (Thm 2.2, 3.3) build on the split-comparator (perceptron-type) argument of Wu et al. and on the near-homogeneity and linearization arguments of Cai et al. Extending these arguments to SGD and to the multi-class setting requires two new ingredients:
    • a variance bound that scales with the loss, E∥gt−∇L(Wt)∥2≤2bG(Wt)\mathbb E\|g_t-\nabla L(W_t)\|^2\le\frac2b G(W_t) (Lemmas C.6, D.9), which allows the stochastic noise to be absorbed into the comparator without any stepsize restriction;
    • a cross-class comparison of the activation derivatives at different pre-activations, which yields the effective margin γ~\tilde\gamma.
  • New results. The results on stability in the stochastic sense have no counterpart for gradient descent:
    • the region of conditional expected descent and its explicit dependence on the batch size (Lemma C.8);
    • the exit probabilities from this region, which decay as the loss decreases because the minibatch noise scales with the loss;
    • the return guarantees after stochastic excursions (Thm 2.4, 3.5);
    • the last-iterate rates inside the stable region (Thm 2.3, 3.4). The obstacle in all of these is that a single SGD step may increase the loss even below the deterministic stability threshold, so the monotonicity argument available for GD cannot be used.

Response to Reviewer UhCV (2/3)

W/Q2 — Relevance and conservativeness of the stable region. We thank the reviewer for this question. We have added a remark after the definition of S\mathcal S clarifying the regime it describes. Since L~≤1/(2n)\tilde L\le 1/(2n) and a misclassified sample contributes at least ln⁡2/n\ln2/n to the loss, the stable set corresponds to the late phase of training, in which all training samples are correctly classified. The overall picture is as follows:

  • Early phase. The earlier phase of training is covered by the best-iterate bounds (Thm 2.2, 3.3), which hold from initialization for any η>0\eta>0 and do not require stability.
  • Late phase. Achieving perfect training accuracy does not imply convergence of the cross-entropy loss, which keeps decreasing as the classification margins grow. Characterizing the late-stage dynamics is therefore relevant, and in the stochastic setting it is nontrivial, because the iterates may still exit the stable region.
  • Conservativeness. The threshold is a sufficient condition. It defines a region in which stochastic descent can be certified using explicit, verifiable inequalities. We will report the numerical stability thresholds corresponding to our experiments and compare them with the observed training trajectories.

W/Q4 — Neural-network assumptions and nonlinearity. We thank the reviewer for raising this point. Our analysis considers nonlinear activations satisfying explicit regularity and derivative conditions. These conditions ensure that the network retains sufficient alignment with a separating direction, which is essential to the convergence analysis. The conditions become more demanding as the separation margin decreases.

Requiring a derivative bounded away from zero does not, by itself, force an activation to be affine. For example, the function ϕ(u)=u−atanh⁡(u),0<a<1,\phi(u)=u-a\tanh(u),\qquad 0<a<1, is nonlinear and satisfies 1−a≤ϕ′(u)≤11-a\le\phi'(u)\le1. Moreover, its deviation from positive homogeneity is bounded uniformly: a ∣tanh⁡(u)−u sech2(u)∣≤Caa\,|\tanh(u)-u\,\mathrm{sech}^2(u)|\le Ca for a universal constant CC. Smooth nonlinear activations with uniformly positive derivatives can therefore satisfy assumptions of this type; whether a particular activation satisfies the full quantitative requirements depends on the corresponding constants and on the separation margin.

Our framework does not aim to cover arbitrary nonlinear activations, and identifying weaker sufficient conditions is an interesting direction for future work. We will make the scope of the assumptions more explicit in the manuscript.


Response to Reviewer UhCV (3/3)

W/Q5 — Connection to the edge of stability. We thank the reviewer for this insightful discussion. We have added a paragraph to Appendix A ("Loss-based versus sharpness-based stability") that clarifies the scope of our results:

  • Different notion of stability. Our stability notion differs from the sharpness-based characterization of the edge of stability in Cohen et al. (2021) and Damian et al. (2023). Our analysis does not rely on progressive sharpening, and our results characterize large-stepsize SGD through loss-based stochastic stability, rather than through the sharpness dynamics.
  • Why the criterion is loss-based. For separable data and the cross-entropy loss, the curvature is controlled by the loss itself, since ∥∇2L(W)∥op≤2L(W)\|\nabla^2L(W)\|_{\mathrm{op}}\le2L(W) (Lemma C.3). In the late phase of training, stability is therefore determined by the level of the loss. This motivates a loss-based criterion, in line with the deterministic analyses of Wu et al. (2024) and Cai et al. (2024). We also thank the reviewer for pointing out the work of Andreyev and Beneventano (2024), which we now cite and discuss in the related work.

W/Q5 — Experimental validation. We will update the plots to indicate the stability threshold, so that it is apparent which spikes occur inside the stable regime and which occur in the large-stepsize phase. Regarding the stabilization mechanism, the plots show smaller spikes as the loss decreases, which indicates a smaller chance of the dynamics exiting the stable regime because of a spike. We will add plots measuring the probability that an iterate becomes unstable because of the noise of the sampled minibatch gradient, showing that this probability decreases as the loss decreases. A useful feature of our notion of stability is that it can be checked directly: one can measure whether the loss exceeds the stability threshold and thereby decide whether the dynamics are stable.

We thank the reviewer again for the constructive suggestions. We believe the distinction between deterministic descent and stochastic stability is central to understanding large-stepsize SGD, and we hope that the revised manuscript addresses the reviewer's concerns.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.