- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates stochastic gradient descent (SGD) with a large constant step size under the multiclass cross-entropy loss. Rather than labeling every increase in loss as "instability," the authors define a "stable region"—a regime where the loss exhibits a downward trend in terms of conditional expectation. For linear classifiers, the authors provide a "best-iterate" loss bound for any positive step size, and they explore how SGD enters this stable region, occasionally exits it due to minibatch noise, and subsequently returns. Furthermore, the paper provides a "last-iterate" loss bound for cases where the training trajectory remains within this region. This analysis is extended to two-layer neural networks, with experiments illustrating these distinct phases of the training process.
Overall, I find the stochastic stability perspective promising, but I cannot recommend acceptance in the paper’s current form. The activation lemma appears to be incorrect, and the proofs rely on a zero-initialization assumption that is missing from the theorem statements. If the authors can correct these issues without weakening the main results, I would be glad to reconsider my assessment.
Critical strengths
What I find most interesting is the attempt to make edge-of-stability behavior precise for SGD. Defining stability through conditional expected loss decrease makes sense when individual updates are noisy. The paper also goes beyond a best-iterate guarantee by asking what happens after the trajectory enters the stable region, including whether noise can push it out and how it returns. Extending the analysis from binary logistic regression to multiclass cross-entropy and two-layer networks broadens its relevance.
Critical weaknesses
There are two issues in the theoretical formulation that warrant attention. Lemma 3.2 identifies Leaky GELU and SiLU as activation functions that satisfy Assumption 3.1; however, their derivatives do not adhere to the bounds required for the proof of this lemma. For instance, the derivative of GELU is approximately -0.083 at and 1.083 at . If is the GELU function, the derivative of the hybrid function proposed in the paper—given by —could fall below or exceed 1. This appears to limit the applicability of the conclusions drawn for two-layer models, although these results might still hold for other activation functions.
Additionally, the statement of Theorem 2.2 likely requires an explicit initialization condition. The bound provided by the theorem does not explicitly depend on the initial weights ; however, the proof relies on the assumption that . Without this condition, a poorly initialized classifier could serve as a counterexample at . I suggest that the authors either clarify the initialization scheme used or incorporate the influence of initialization into the bound formula.
While the experimental results offer useful insights, they do not directly validate the concept of stability proposed in the paper. In particular, when a sharp spike in loss is observed, it is unclear whether this is due to minibatch noise or an excessively large step size—distinguishing between these two factors is, in fact, one of the paper's core motivations. Measuring the "conditional loss drift" and the extent to which the loss deviates from the established stability threshold would significantly strengthen the empirical evidence.
Questions for authors
- Which of the activation function examples mentioned in Lemma 3.2 satisfy the conditions of Assumption 3.1? Could this lemma—and the corresponding conclusions for two-layer networks—be revised to better reflect their scope of applicability?
- Does Theorem 2.2 (and the related results for two-layer networks) implicitly assume "zero initialization"? If not, how should the bounds account for the dependence on the initial weights?
- Is it possible to directly measure the proposed stability criterion—for instance, by quantifying how frequently stochastic gradient descent (SGD) exits the stable region, or the time required for it to return?
Response to Reviewer hkqm
We thank the reviewer for the thoughtful and constructive review, and for finding the stochastic-stability perspective promising. We particularly appreciate the recognition that conditional expected descent provides a meaningful notion of stability in stochastic optimization, as well as the importance of distinguishing entrance into, retention in, and return to the stable region.
Q1 — Lemma 3.2. We thank the reviewer for the careful reading. We have updated Lemma 3.2 accordingly, refining the construction for GELU and SiLU. With d=31+2c and ϕ~(x)=dx+41−dϕ(x), the bound ∣ϕ′∣≤2 gives ϕ~′∈[c,32+c]⊆[c,1]. Therefore all activations listed in Lemma 3.2 (GELU, Softplus, SiLU, Huberized ReLU, tanh, sigmoid) satisfy Assumption 3.1; we verified the constants both analytically and numerically. Moreover, the neural-network results are formulated under Assumption 3.1 and do not depend on any specific activation, so they hold for every activation satisfying it.
Q2 — Initialization. Thank you for the remark. Our analysis considers the zero initialization W0=0; the results remain valid for a general initialization, up to an additive term depending on the norm of W0. We have updated the manuscript accordingly:
Q3 — Direct experimental validation of stochastic stability. Thank you for the suggestion! One useful feature of the proposed notion of stability is that one can measure whether the loss exceeds the stability threshold and thereby decide whether the dynamics are stable or unstable. We plan to add plots showing exactly when the iterates cross the stability threshold. These will make apparent when the dynamics are stable, and which spikes are attributable to minibatch noise and which to large stepsizes.
We sincerely thank the reviewer for the detailed feedback.
We believe that we have addressed the reviewer's concerns and hope for a positive re-evaluation of our work. We remain available for any further questions.