- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper asserts that the expected optimization error can be a poor description of an accurate iterate, which is used to motivate the need for tail probabilities. For a sequence generated by stochastic gradient descent with heavy-tailed noise, the main result establishes a distributional limit for strongly convex ("dissipative") and smooth objectives. This result resembles the Levy-stable limits identified in prior works.
Critical strengths
I am unable to identify any critical strengths of the paper that support acceptance.
Critical weaknesses
Several critical errors are present already in the first few paragraphs.
The paper starts with a strong statement that "the expected optimization error can be a poor description of an accurate iterate". This directly conflicts with the results that this manuscript proves; the main result is the error's tail probability. Given that this is the only motivation stated for the results, it is unclear what problem/gap the authors are trying to close.
The paper states its novelty with respect to existing works as identifying "the exact shrinking-threshold tail probability ... including the sharp constant". How does this differ from the cited results, apart from differences in problem settings? What kind of established convergence guarantees and stable-limit behaviors are the authors referring to? It is written as if existing results do not recover tails as in Equation (2), which is by definition an asymptotic tail. Can the authors show why existing results do not suffice to obtain Equation (2)? For example, what are the authors claiming as the distinction with the [1]?
Theorem 2.5 is established assuming a condition on which is not defined.
The results are very hard to read. For example, Theorem 2.5 states a redudnatn condition on : and . It's also unclear whether can depend on and/or .
[1] Jose Blanchet, Aleksandar Mijatovic, and Wenhao Yang. "Limit theorems for stochastic gradient descent with infinite variance".
Minor comments
-
The is simply described as "the target". It should be defined as the minimizer of .
-
The is said to "mean that the ratio tends to one". The authors should specify the limiting variable, e.g., or .
-
Equation (2) uses which is not defined. It is used in Theorem 2.5 to specify the dependence of on with a specific requirement, but this is not clear in Section 1.
-
The statements in Corollaries 2.6 and 2.7 open with "Under Theorem 2.5". It should be "Under the assumptions of Theorem 2.5".
Questions for authors
Please see my comments under "Critical Weaknesses".