- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This manuscript explores the spectral behavior of the Neural Tangent Kernel in various regimes (finite sample, finite width, under evolution). Theoretical results are given for the finite sample and finite width case. The evolving case is more complicated and only an empirical study is provided.
Critical strengths
The discussion of the finite sample and finite width setting is reasonable (and expected behavior is recovered).
Critical weaknesses
This manuscript has two main weaknesses. While both are significant, the more productive path to improvement is substantially addressing the second.
First, while I find the theoretical results sensible they are also thoroughly unsurprising given the setting and, existing, relevant concentration results that power them. Unsurprising is not necessarily bad, but I also do no think that, on their own, these theoretical results are substantive enough to carry the manuscript. This is particularly true given the very limited stetting. I am not familiar enough with all the related literature to speak about novelty, but the technical pieces are certainly widespread. Moreover, I stand by my prior comment about substance even under the claim that these are fully novel results.
The second weakness is that the more interesting regime where the kernel is evolving is underdeveloped. It is completely reasonable for such an exploration to be largely empirical---mathematical study of that setting is challenging. Unfortunately, the manuscripts current study is not adequately developed around interesting narratives. In other words, the set of included experiments seem quite narrowly focused and don't really provided much outside of a narrow setting.
In fact, the lack of a strong central narrative is plainly evident in section 5. The prose is unfocused and unstructured. It makes it challenging to read it and come away with a clear understanding of what is being conveyed. A more focused message paired with appropriate experiments (which would depend on the message itself) could strengthen the manuscript.
Questions for authors
-
Is the sampling always uniform in section 3.2? More generally, perhaps I missed something, but its not clear why U_k has orthonormal columns for arbitrary finite samples. (All the discussion in that section is fine, I think it might just be that the definition of U_k is slightly more nuanced, particularly given the specific measure, and therefore normalization, used for the continuous case.)
-
Line 249 - 250: while pedantic, is in is a scalar.
-
How specific are some of these results to the target always being a finite Fourier mixture with, often, relatively small k?
Thank you for the detailed and constructive review. We agree that the concentration results are expected, and that the evolving-kernel study needs a clearer central message.
On your questions:
Sampling and Uk (Section 3.2). For i.i.d. uniform samples the sampled Fourier modes (scaled by 1/n) are only approximately orthonormal. The strength and alignment measurements require exact orthonormality, so on S1 we use n equally spaced points, where this holds exactly. We have clarified the definition of Uk accordingly.
Lines 249–250. Thank you. We have corrected this.
Dependence on the target. The theoretical results do not assume a Fourier-mixture target: the kernel and concentration results do not involve the target. The empirical evolving-kernel results are limited to finite Fourier mixtures with a relatively small k. Within that range, the low-frequency strengthening persists under different amplitudes, and when k=0,1,2 are absent from the target. We have not tested targets with substantial high-frequency content (k≥15).
For now, we have decided to withdraw the submission and will take your feedback into account in order to improve the manuscript.