- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces a new efficient way of computing empirical generalized neural tangent kernel (NTK) tensors. The generalized NTKs are given by where is the Jacobian of some state vector with respect to the parameters of the network. The state vector is a function of the parameters any may consist e.g. of activations of various layers of the network or different unrolling steps of an RNN. Traces of (products of) these tensors are estimated using a variant of Hutchinson’s trace estimator. Extensive empirical experiments show that the estimators converge quickly to the exact values and therefore offer a substantial speed-up which enables the computation of traces of generalized NTKs comprising of high-dimensional state vectors. Two applications are discussed, which both involve evaluating traces of the NTK throughout training and therefore require fast evaluations.
Critical strengths
- The proposed method seems to clearly outperform the naive baseline of computing the trace by instantiating the entire matrix by a large margin. This enables novel applications of NTK traces where fast evaluation is required, as demonstrated in the applications in Section 6.
Critical weaknesses
- The method is restricted to estimating operators which can be written in terms of the trace of (products of) the (generalized) NTK. If individual components of these tensors are required, the method is not applicable. The authors provide a list of examples of operators which can be computed with their method.
- The applications in Section 6 do illustrate that the proposed method can be used to compute NTK operators throughout training in a small, but not tiny setting. However, the results are not strong enough to show that enabling these computations also enables strong downstream applications in realistic settings. This is left open for future work.
Questions for authors
- In Figure 4, why is the effective rank in C harder to estimate than the norm and alignment in A and B?
- There is an oscillatory pattern in Figure 6 B, black line, which seems to vanish when the NTK regularization is used (red line). Do you know where this pattern comes from?
- In Section 6.2, why did you restrict to a single-layer MLP trained on Fashion-MNIST instead of e.g. a ResNet on CIFAR? Would your method scale to such a combination of network and data?
2 comments
Thank you for the updated manuscript and the rebuttal. I’m satisfied with the answers to my questions and will keep my positive score.
Response
Thank you for the positive and constructive review. We appreciate your recognition of the computational speedups and potential applications. We have addressed your questions below and added a new large-scale language-model experiment (Section 6.3, Figure 7) to further demonstrate the utility of these estimators. Changes are marked in blue in the revised PDF.
1. Effective-rank estimation (Figure 4C)
Effective rank is somewhat harder to estimate because it involves a ratio of two estimated quantities, tr(K)2/tr(K2), so errors in the numerator and denominator can compound. Nevertheless, Hutch++ achieves approximately 99% accuracy with 60 probes for all three statistics, at roughly 1% of the exact evaluation time.
2. Oscillations in teacher alignment (Figure 6B)
The oscillations in the cross-entropy-only trajectory may reflect fluctuations in the student's parameter sensitivities during training, but we have not isolated their precise origin. NTK regularization directly encourages alignment with the teacher's tangent geometry, which explains the higher and more stable alignment observed in the regularized trajectory. We do not claim that the regularizer eliminates the underlying source of the oscillations.
3. Scaling to larger architectures and applications
We chose Fashion-MNIST and MLPs as a controlled proof of principle, allowing repeated training across data sizes, regularization strengths, and random seeds. The estimators themselves are architecture-agnostic, requiring only JVPs/VJPs, and can therefore also be applied to CNNs such as ResNets, although we have not evaluated that particular setting.
To better demonstrate their utility at scale, we have added Section 6.3 and Figure 7, examining NTK effective rank across Pythia model sizes and throughout Pythia-410M pretraining. This involves a state space of 393,216 coordinates, estimated using only 21 Hutch++ probes. We observe substantial changes in effective rank during training, illustrating how the method enables large-scale, repeated measurements of NTK statistics beyond the trace. These experiments relate to progressive sharpening observed in prior work (which we cite), where the spectrum becomes more concentrated during learning. Interestingly, we find that the warmup period promotes expansion of the effective NTK rank, followed by concentration after that. We hope that the addition of these experiments help to provide a more large-scale experiment motivating the use of some of the statistics described.