- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper attempts to mathematically explain what the deviations in error are between a model and its linearized clone for short timescales after initialization. Their analysis boils down to performing a Taylor series expansion of the models about their initialization and analyzing the dominant term that differs between the two, labeled as the “kernel velocity.” Their analysis heavily leans on mathematical theorems and propositions over empirical validation. The empirical checks show these theorems apply in the scope they were conceived in, and fail outside them.
Critical strengths
None
Critical weaknesses
- There is no structure to this paper which is accompanied by consistent jargon that doesn’t appear anywhere else in the kernel theory literature.
- There is little-to-no strong evidence to support the broader claims of the paper. To paraphrase the abstract, this paper studies how feature changes may alter predictions without improving them. The theory that is developed is applied empirically to a toy setting where the timescale shown (Fig. 1) is at with a difference in magnitude between a real model and its linearized counterpart of at most . How this relates to the features (not defined anywhere in the paper and thus is left vague), or their evolution is not discussed. There is no notable feature learning for models in a gradient flow regime at time .
Questions for authors
None