- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors studied anytime-valid stopping for stochastic gradient descent. They proposed SOEP-π procedure which approximately minimizes average boundary height over a prescribed set of variance scales. Their Theorems 1 and 2 give simultaneous bounds for weighted suboptimality and weighted squared-gradient stationarity, respectively. Numerical studies are included.
Critical strengths
An important problem with a simple, validity-preserving construction.
Critical weaknesses
My biggest concern is that inequalties (20) and (26) in Theorems 1 and 2 are too conservative. The authors have not convincingly studied the sharpness of the final stopping inequalities (20) and (26). For example, at , the reported simulations imply empirical rates of 0% for both final stopping-failure events. Please directly estimate the probabilities in (20) and (26), separately for each problem and tolerance, with Monte Carlo confidence intervals and clearly stated horizons. Distinguish these estimates from weighted-average failure rates and underlying boundary-crossing probabilities. Since the present experiments report zero stopping failures at nominal level , they demonstrate empirical safety but do not establish how close the final guarantees are to the nominal risk level.
I also have concerns on the novelty of the paper. As mentioned in the paper under review, Aolaritei and Jordan (2025) already derived ; see their Remark 4.5 about geometric grids and scale matching. I am not convinced that the results in this paper are substantially novel. Formular (A51) in Section A.6 in Howard et al. (2021) is pretty similar as the manuscriptβs convex formula. Section 3.5 in the latter paper is also related.
In Appendix E, lines 1463β1465, claims that using Softplus and average pooling keeps the network globally with bounded Hessian. The bounded-Hessian assertion is false for an unrestricted multilayer parameterization.
The sin-mix example in (73) violates the gradient-bound assumption.
Questions for authors
Sharpness of (20) and (26). At πΌ = 0.05, please directly estimate, separately by problem and tolerance, the finite-horizon probabilities. Please provide Monte Carlo confidence intervals, independent replication counts, and the monitoring horizon. Distinguish these from weighted-average failures and from the boundary-crossing experiment, whose generating process and configuration also need specification. Use all trajectories as the denominator for these unconditional probabilities; a failure rate conditional on stopping is a different quantity. Repeated tolerances on the same path should not be treated as independent replications. The probabilities need not equal 0.05 in every example, but their distance from the nominal level is essential for assessing conservatism.
4 comments
Response to Reviewer hxXo Part 2/2
W2: Relation to prior work and novelty.
- We will narrow the contribution to weight allocation for a specified design distribution and its empirical tradeoffs. Thank you for identifying the precedents. Aolaritei and Jordan [1] provide trajectory-based SGD certificates and scale matching; Howard et al. [2] develop mixture confidence sequences. Matched experiments favor the tuned baseline on mean capped stopping time.
W3: The global bounded-Hessian assertion.
- We will withdraw the assertion and label the original CNN results as proxy evaluations. Thank you for identifying the error. Smooth activations and pooling alone leave parameter-space curvature potentially unbounded.
W4: The sin-mix gradient bound.
- The original sin-mix gradient is unbounded; we will remove it from theorem-valid results. Thank you for flagging this violation. Separately proved predictable-gradient-bound/guarded variants will be identified as new modifications.
References
[1] L. Aolaritei and M. I. Jordan (2025). Stopping Rules for Stochastic Gradient Descent via Anytime-Valid Confidence Sequences. arXiv:2512.13123.
[2] S. R. Howard, A. Ramdas, J. McAuliffe, and J. Sekhon (2021). Time-uniform, nonparametric, nonasymptotic confidence sequences. The Annals of Statistics, 49(2):1055-1080.
response to the rebutal
Thanks the authors for answering my main empirical question about (20) and (26) with additional experiments.
I agree with the authorsβ narrower novelty claim regarding weight allocation for a specified distribution of variance scales. Given the prior work on scale matching and mixture boundaries, the matched comparison carries significant weight in my assessment.
After I read other reviewers' comments, I keep my rating of 2 (weak rejection): the main reason is that the evidence that the proposed allocation improves stopping performance over a well-tuned baseline seems limited.
Dear Reviewer hxXo,
Thank you for your follow-up. To clarify the matched comparisons discussed with Reviewers hsQz and fQBD, the tuned baselines achieve 6.19% to 11.37% lower mean stopping time only by fitting hyperparameters on 64 target-task SGD pilot trajectories (264.27 s setup), which defeats single-run early stopping. Requiring zero SGD pilots (2.67 s setup), exact SOEP- outperforms both the closed-form baseline (ratio 1.6963) and the untuned numerical-inversion AJ mixture (ratio 1.0153, CI [1.0105, 1.0247]), while matching the pilot-tuned geometric mean (0.9897).
Since the calibration and assumption issues are resolved, could you also consider updating your score?
Thank you!
Response to Reviewer hxXo Part 1/2
Thank you for recognizing the problem and construction, and for identifying the calibration and assumption gaps.
Q1 & W1: How conservative are the final stopping guarantees?
Table 1. Distinct events, each counted over all 100,000 paths.