- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies the convergence of Adam under heavy-tailed noise with bounded conditional -moments (). They show that when the objective is -smooth and bounded below and , one can choose the Adam parameters appropriately so that the expected average squared gradient norm decays at a polynomial rate. In addition, for each fixed moment bound and step size, with the parameter choice , they construct a function and a stochastic oracle for which the rate of Adam matches their upper bound up to a polylog factor.
Critical strengths
- The setting is clean. The analysis keeps Adam’s core updates and does not require clipping or additional normalization. The upper bound is also accompanied by nearly matching lower bounds.
- The chord cancellation method seems to be a useful generic approach to analyze the bias of transformed noisy gradients.
Critical weaknesses
The analysis relies heavily on the parameter choice . While this also appears in other papers in this field, it is still quite different from common practice and makes the lower bound somewhat restrictive.
Questions for authors
- Can the analysis be extended to setting where is a -independent constant, potentially under some additional assumptions?
- How generic is this chord cancellation method? The reduction to two-point chords part seem fairly general, but secant inequality part exploits Adam's specific normalization. Are there other situations where this method could be useful?