- Rating
- 4(+2)
- Confidence
- 4
- Informativeness
- 3(+1)
- Presentation
- 3
- Soundness
- 3(+2)
Read review
Paper summary
The paper studies the difference in generalization between SGD and guess & check (G&C), an algorithm where random interpolating solutions are chosen from a fixed prior. The goal is to use this analysis to better understand generalization in over-parameterized models. The paper first produces sample complexity results for both algorithms in a linear model setting. Then, the paper experimentally shows that the margin of SGD models is larger than G&C, and moreover SGD achieves better generalization.
Critical strengths
- The paper studies a very important problem, namely what is the source of good generalization in over-parameterized neural networks. The empirical approach of comparing SGD with G&C directly is promising as well and yields insights into the behavior of SGD.
Critical weaknesses
- l’m primarily concerned that “volume hypothesis” and “algorithmic bias” are not really two separate hypotheses, but they are significantly overlapping, and G&C is not experimentally set up in a way that helps meaningfully separate these hypotheses. Specifically, it’s not clear whether there exists a prior for G&C such that it’s solutions are the same (or nearly the same) as SGD. If so, then the results really compare two different priors for G&C and don’t really answer the question of whether algorithmic bias or volume hypothesis is more important for generalization. Put another way, perhaps the generalization benefits of SGD can be viewed as the volume hypothesis under a specific prior. Another way to state this is, how do we know that the prior chosen for G&C is the right one (maybe it wasn’t chosen to yield a fair comparison with SGD).
Questions for authors
- Does there exist a prior for G&C that matches the distribution of solutions for SGD?