- Rating
- 4
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper gives guarantees for constant stepsize SA under only assumptions of finite hyperbolic roots and second innovation moments. The previous known result on the distributional behavior of constant stepsize SGD was in the contractive setting, so this paper gives a generalization of that. The underlying is to argue that the distribution is characterized by a mixture of gaussians determined by fluctuations each root, which lines up with intuition on contractive SA.
Critical strengths
The paper gives a meaningful technical contribution to a important problem (constant stepsize SA). The previous best known work identified Gaussian fluctuations in the contractive setting, which is where the roots are unique. Here there are mutliple roots, so the technical ideas required to get around this are substantial and comprise of the bulk of the paper's contributions.
Critical weaknesses
The assumptions are quite technical and it is unclear why certain conditions are necessary and what role they play in the proof. Very little time is devoted to explaining why they are needed and outlining the proof itself, which made the paper a bit hard to read. Nevertheless, these are minor concerns and the paper itself is quite strong.
Questions for authors
Can the method be extended to other noise settings (e.g. Markovian or multiplicative noise)? Could you elaborate more on why certain assumptions are necessary? Where might such assumptions appear/not appear in practice?