- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This papers studies the "Schedule-Free Stochastic Gradient Descent" algorithm, introduced by Defazio et al. in 2024. It studies the convergence to a critical point in expectation in the -smooth and bounded variance case, focusing on the constant averaging coefficient . Compared with previous works, there is a lower bound on the admissible value of , and there is an emphasize on the different results we can obtain on the sequences and .
Critical strengths
None
Critical weaknesses
- The contribution appears to be limited, there is only one convergence result in the -smooth regime. Critically, the paper argues a better upper bound for the gradient norm of compared with , from which the authors deduce some theoretical implications. Nonetheless, the bound for is deduced from the one of , using a formula of the form . It is thus trivial the the upper bound for is less favorable, and does not show that it is inherently worse.
- I think the writing could be significantly improved. There are several repetitions, for instance on the first page "In addition to the original choice of Defazio et al.[...] is the original choice of Defazio et al. [...]", or on page 2 "since the stochastic gradient is evaluated at [...] whereas is the current point where the stochastic gradient is evaluated". It is sometimes somewhat imprecise too. For instance, on page 4 "this condition is satisfied by taking as the cross entropy loss", I do not think that "condition" is the right term, and some lines below "Note that is independent of a generated by SF-SGD", it is an assumption, not something that should be noted. Overall I believe the writing is below the ICLR standards.
- It is hidden in the bigO notation a factor , which is not negligible for small value of .
Questions for authors
Please see my remarks in the weakness section.