- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper proposes to estimate a local learning coefficient along training curves by running a Langevin chain tethered to non-stationary points along the path, and computing the variance of the loss along it. This is based on a 7-year-old non-peer-reviewed arXiv paper, Imai 2019. From this variance the mini-batch evaluation noise is subtracted by empirical estimation of the batch noise (Theorem 1, the ANOVA within-group correction). It is validated on two toy energies with injected noise and on five MLPs on MNIST. On the latter, the curve shows a dip at epoch ~40 if the batch noise is accounted for, but that dip disappears if not. The significance of this observation is undiscussed
Critical strengths
None.
Critical weaknesses
- The paper is bearly readable, and presents no question. The abstract says a measurement can be fixed and describes a “turning structure” in a curve whose meaning is unclear and its discussion is deferred to future work. notation is used before it's defined and necessary contrived vocabulary is used which the generic ICLR reader is unfamiliar with. In addition the paper adds its own undefined and non standard terms (“local chart”, “Gaussian tether”, “retention rule”) to refer to standard and common concepts. Pages of formalism are devoted to bookkeeping (a burn-in window, a Bessel correction to the variance) while the conditions that would make the statistic an LLC (Eq. 15) are never checked.
- No contribution. The base estimator this paper claims to improve and treats as well known, is an unpublished 2019 preprint with 15 citations. The paper’s addition is the ANOVA within-group correction (Searle et al. 1992, cited by the authors) for a noise its own protocol creates: more than half the samples are evaluated per step, while evaluating on all would remove the noise identically for at less than twice the cost. Theorem 1 is just linearity of expectation.
Questions for authors
What is the actual problem you were trying to solve? how did your results answer it? what did we learn?