- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies average squared-gradient stationarity for ordinary stochastic gradient descent with momentum under finite-moment noise. Its main theorem separates a logarithmic variance contribution from a polynomial rare-shock contribution, with constants independent of dimension and momentum within an explicit step-size region. A scalar quadratic construction matches the polynomial term over a noise class whose law may depend on the horizon and confidence level.
Critical strengths
- The main proof isolates the effect of momentum cleanly. The virtual iterate reduces the update to a form suitable for descent, while the proof retains the filtered-noise energy rather than immediately replacing it by raw noise energy. Lifting the noise into a product Hilbert space gives the factor ; together with the step-size condition, this controls the apparently singular momentum coefficient.
- The lower bound and experiments help interpret the result without overstating it. Theorem 2.2 matches the polynomial component in horizon, confidence, and step-size dependence. The abstract explicitly allows a horizon- and confidence-dependent noise law, and the paper distinguishes that construction from its separate fixed-law simulations.
Critical weaknesses
- The extensions contribute less to the convergence rate than the main confidence split. At fixed momentum and confidence, the second-moment consequence of Theorem 3.1 yields the familiar stationarity order, with an energy allowance proportional to . Its distinctive value is the route through realized and predictable energy and the refined dependent-noise allowances. The paper could make that distinction more explicit relative to a Markov conversion of an expectation bound. 2, The demonstrated practical scope is limited. The matching construction establishes a worst-case moment-class obstruction, not a matching asymptotic result for one fixed noise law. The causal theorem treats additive stationary errors; the state-dependent application uses a bounded amplitude multiplying martingale noise. The experiments make these regimes concrete, including a vector learning problem, but use constructed or injected noise.
Questions for authors
- Could you summarize exactly what the second-moment consequence of Theorem 3.1 adds to an expectation-plus-Markov argument under the same assumptions, separating rate improvements from the usefulness of the energy formulation?
- Is the dependence on in the energy result necessary, or can additional control of filtered energy recover momentum-uniform constants for some random-volatility models?
- How does the rare-shock term compare with the variance term at experimentally relevant horizons for a fixed law with finite th moment?