- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies the high-probability guarantees of the popular optimizer Adam under the general assumptions: smooth nonconvex function, unbiased gradient, and bounded variance. This work is motivated by studying the superiority of Adam over SGD through the facts that Adam achieving a better dependence on the failure probability , in terms of the convergence upper bound.
Building on existing study Jin et al. (2026), which established polynomial dependence on , this work refines the analysis through better choice of and learning rate, and achieve dependence.
Critical strengths
- This work is well-motivated, studying the superiority of Adam over SGD through convergence dependence on the failure probability (but sadly, the comparison is between upper bounds).
- The techniques is somewhat convincing, it uses the accumulation control rather than bounding the trajectory-wise terms, which make sense to avoid extra costs.
Critical weaknesses
Nontrivial technical issues
- The convergence result hide some nontrivial dependences:
- The polynomial dimension dependence of . It is unclear how to interpret this dependence, which usually appears for -norm criterion.
- The convergence result hide parameter . I believe is necessary here, and the refinement may come from a dependent tuning, a common strategy to achieve stronger result but is impractical.
- The constant is implicitly dependent on .
- The algorithm 1 is not plain Adam, it omits the bias correction terms.
Significance of the work
- Compared with existing study, the claimed improvement comes from the a different choice of and some analysis refinements, the technical contribution is somewhat weak and not well-presented as well.
- Meanwhile, the "established advantage" is all about comparison between upper bound; I don't see separation result to really support the claimed advantage over SGD.
Questions for authors
- how does the dimension enter the convergence bound?