acceptodds
Under review as a conference paper at ICLR 2027

Fast Generalized Neural Tangent Kernel Statistics via Trace Estimation

Abstract

The empirical state-space Neural Tangent Kernel (NTK) describes the local learning geometry of a finite-width neural network, but computing it explicitly quickly becomes impractical. We show that many useful NTK statistics that characterize, for example, the dimensionality of learned updates or how two models or learning rules relate, can instead be computed from matrix-free products using randomized trace estimation. Namely, we use Hutch++ to estimate the NTK trace, Frobenius norm, effective rank, and alignment. Furthermore, we show that the positive-semidefinite structure of the NTK yields one-sided estimators that require only forward- or reverse-mode automatic differentiation. We validate these estimators across MLPs, recurrent GRUs, and a natural-language Transformer with up to 410 million parameters. In the latter model, the state is a high-dimensional four-tensor. We demonstrate orders-of-magnitude speedups, with the fastest estimator depending on the ratio of parameter and state dimensions. Equipped with these estimators, we examine rich and lazy RNN training using hidden-state NTK alignment and use NTK alignment as a regularizer for data-scarce knowledge distillation. We find that this regularization can modestly improve generalization, especially in very data-scarce settings. Together, these results make state-space NTK diagnostics practical even at large scales.

Primary area:
learning theory
Keywords:
neural tangent kernel, recurrent neural networks, automatic differentiation, knowledge distillation, rich and lazy learning, language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_EA7XOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a new efficient way of computing empirical generalized neural tangent kernel (NTK) tensors. The generalized NTKs are given by JJ⊤JJ^\top where JJ is the Jacobian of some state vector with respect to the parameters of the network. The state vector is a function of the parameters any may consist e.g. of activations of various layers of the network or different unrolling steps of an RNN. Traces of (products of) these tensors are estimated using a variant of Hutchinson’s trace estimator. Extensive empirical experiments show that the estimators converge quickly to the exact values and therefore offer a substantial speed-up which enables the computation of traces of generalized NTKs comprising of high-dimensional state vectors. Two applications are discussed, which both involve evaluating traces of the NTK throughout training and therefore require fast evaluations.

Critical strengths
  1. The proposed method seems to clearly outperform the naive baseline of computing the trace by instantiating the entire matrix by a large margin. This enables novel applications of NTK traces where fast evaluation is required, as demonstrated in the applications in Section 6.
Critical weaknesses
  1. The method is restricted to estimating operators which can be written in terms of the trace of (products of) the (generalized) NTK. If individual components of these tensors are required, the method is not applicable. The authors provide a list of examples of operators which can be computed with their method.
  2. The applications in Section 6 do illustrate that the proposed method can be used to compute NTK operators throughout training in a small, but not tiny setting. However, the results are not strong enough to show that enabling these computations also enables strong downstream applications in realistic settings. This is left open for future work.
Questions for authors
  1. In Figure 4, why is the effective rank in C harder to estimate than the norm and alignment in A and B?
  2. There is an oscillatory pattern in Figure 6 B, black line, which seems to vanish when the NTK regularization is used (red line). Do you know where this pattern comes from?
  3. In Section 6.2, why did you restrict to a single-layer MLP trained on Fashion-MNIST instead of e.g. a ResNet on CIFAR? Would your method scale to such a combination of network and data?
2 comments

Thank you for the positive and constructive review. We appreciate your recognition of the computational speedups and potential applications. We have addressed your questions below and added a new large-scale language-model experiment (Section 6.3, Figure 7) to further demonstrate the utility of these estimators. Changes are marked in blue in the revised PDF.

1. Effective-rank estimation (Figure 4C)

Effective rank is somewhat harder to estimate because it involves a ratio of two estimated quantities, tr(K)2/tr(K2)\mathrm{tr}(K)^2/\mathrm{tr}(K^2), so errors in the numerator and denominator can compound. Nevertheless, Hutch++ achieves approximately 99% accuracy with 60 probes for all three statistics, at roughly 1% of the exact evaluation time.

2. Oscillations in teacher alignment (Figure 6B)

The oscillations in the cross-entropy-only trajectory may reflect fluctuations in the student's parameter sensitivities during training, but we have not isolated their precise origin. NTK regularization directly encourages alignment with the teacher's tangent geometry, which explains the higher and more stable alignment observed in the regularized trajectory. We do not claim that the regularizer eliminates the underlying source of the oscillations.

3. Scaling to larger architectures and applications

We chose Fashion-MNIST and MLPs as a controlled proof of principle, allowing repeated training across data sizes, regularization strengths, and random seeds. The estimators themselves are architecture-agnostic, requiring only JVPs/VJPs, and can therefore also be applied to CNNs such as ResNets, although we have not evaluated that particular setting.

To better demonstrate their utility at scale, we have added Section 6.3 and Figure 7, examining NTK effective rank across Pythia model sizes and throughout Pythia-410M pretraining. This involves a state space of 393,216 coordinates, estimated using only 21 Hutch++ probes. We observe substantial changes in effective rank during training, illustrating how the method enables large-scale, repeated measurements of NTK statistics beyond the trace. These experiments relate to progressive sharpening observed in prior work (which we cite), where the spectrum becomes more concentrated during learning. Interestingly, we find that the warmup period promotes expansion of the effective NTK rank, followed by concentration after that. We hope that the addition of these experiments help to provide a more large-scale experiment motivating the use of some of the statistics described.

Thank you for the updated manuscript and the rebuttal. I’m satisfied with the answers to my questions and will keep my positive score.

Reviewer_7j24OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at the empirical NTK defined on internal states (hidden units, timesteps, layers, tokens) and estimates its trace, Frobenius norm, alignment and effective rank without forming the matrix, using Hutch++ and one-sided Hutchinson estimators. The trace is benchmarked up to the hidden states of Pythia-410M for one 16-token prompt, and the other statistics are checked on an MNIST MLP. There are two applications: rich vs. lazy learning in RNNs, and teacher–student NTK alignment as a regularizer for distillation with few samples.

Critical strengths

The code is released and the setup is described in detail. Getting trace estimates for a 354M-parameter Transformer on one GPU is a useful practical result for people who already use these NTK diagnostics.

Critical weaknesses

My main concern is novelty. This is mostly an engineering paper. Computing NTK–vector products with vjp and jvp is standard autodiff, Hutch++ [5] and Frobenius norm estimators already exist (for example in Curvlinops [3]), and the one-sided estimators are the usual Hutchinson identities [1] (see also the Jacobian Frobenius-norm estimator in [2]). Algorithm 2 adds orthogonalized probes, but I could not find a direct comparison showing this helps over plain Hutchinson, and the Pythia runs use plain one-sided estimators anyway. Hazelden et al. already has a matrix-free operator interface with Hutch++ and trace-based effective rank, Frobenius norm and cosine similarity, so it is not clear what is new here beyond that. I also found two errors in Appendix B. In Proposition 1, the order of the Jacobians on the left side does not match the right side, which actually gives tr(J1J2⊤J3J4⊤)\mathrm{tr}(J_1 J_2^\top J_3 J_4^\top) rather than tr(J1J2⊤J4J3⊤)\mathrm{tr}(J_1 J_2^\top J_4 J_3^\top). Eq. (12) is missing the square in the numerator, so it does not match Eq. (3) and changes if JJ is rescaled. These look fixable and do not affect the reported experiments, since those use Hutch++ for the non-trace statistics.

My second concern is that it is not clear what these statistics are used for. The statistics used downstream are only computed for n≤6,000n \le 6{,}000, and the applications never compare exact and estimated computation, so the speed advantage is not shown where it matters. In the RNN experiment, the rich/lazy dependence on initialization rank, already shown in [6], shows up clearly in the representation alignment, while the NTK alignment is almost flat. I do not see what the NTK measurement adds beyond the initialization rank and the representation alignment. In the distillation experiment the output-logit kernel is only 160×160160 \times 160 and the gains are under 1 pp. The long-horizon runs in Fig. 6C show the gain going negative, only differences are reported (not the actual student accuracies), and there is no standard KD baseline [7].

References

[1] M. F. Hutchinson. A stochastic estimator of the trace of the influence matrix for Laplacian smoothing splines. Communications in Statistics – Simulation and Computation, 18(3):1059–1076, 1989.

[2] J. Hoffman, D. A. Roberts, S. Yaida. Robust learning with Jacobian regularization. arXiv:1908.02729, 2019.

[3] F. Dangel, R. Eschenhagen, W. Ormaniec, A. Fernandez, L. Tatzel, A. Kristiadi. Position: Curvature matrices should be democratized via linear operators. arXiv:2501.19183, 2025. (Curvlinops library: https://curvlinops.readthedocs.io)

[4] J. Hazelden, L. Driscoll, E. Shlizerman, E. Shea-Brown. The global empirical NTK: Self-referential bias and dimensionality of gradient descent learning. arXiv:2605.08746, 2026.

[5] R. A. Meyer, C. Musco, C. Musco, D. P. Woodruff. Hutch++: Optimal stochastic trace estimation. Symposium on Simplicity in Algorithms (SOSA), 2021.

[6] Y. H. Liu, A. Baratin, J. Cornford, S. Mihalas, E. Shea-Brown, G. Lajoie. How connectivity structure shapes rich and lazy learning in neural circuits. ICLR, 2024.

[7] G. Hinton, O. Vinyals, J. Dean. Distilling the knowledge in a neural network. arXiv:1503.02531, 2015.

Questions for authors

Is there any result where a statistic other than the trace is needed at a size where exact computation is not feasible? How do exact and estimated computation compare in the two applications? In Fig. 5A, representations barely change at high rank (about 0.95) while NTK alignment is low (about 0.15). Which of these predicts generalization or learning speed? For distillation, what are the absolute test accuracies of the CE-only and regularized students compared to the 90% teacher, and how does the method compare to standard KD with soft targets [7] on the same samples?

(Reference numbers refer to the list under Critical Weaknesses.)

1 comment

Thank you for the careful review, which helped us clarify the paper's scope and novelty. We corrected Appendix B, refined our claims, and added a large-scale language-model experiment related to progressive sharpening and exploring how the warm up during training affects this, addressing your question about non-trace NTK statistics.

Changes are marked in blue in the revised PDF.

  1. Novelty relative to standard estimators and earlier NTK work

We agree that these numerical ingredients are standard, and have made that explicit in the revised Introduction and Related Work. We do not claim to invent Hutch++ or the identity tr⁡(JJ⊤)=E∣J⊤v∣2=E∣Jp∣2\operatorname{tr}(JJ^\top)=\mathbb E|J^\top v|^2=\mathbb E|Jp|^2. Our contribution is the unified computational treatment and systematic runtime–accuracy evaluation of empirical state-space NTK statistics, comparing two-sided and single-AD-mode estimators across MLPs, GRUs, and pretrained Transformers, together with applications requiring repeated operator measurements. Appendix B additionally derives one-sided estimators for NTK products, including alignment and cross-Jacobian quantities.

We now acknowledge more precisely that earlier Hazelden et al. work already employed Hutch++ for trace-derived statistics within a different, dynamical-decomposition study; we distinguish the present paper's estimator comparisons, scaling measurements, and applications. We also clarify Curvlinops as complementary rather than implying that its general matrix-free ideas are unique to our work. The Pythia benchmark uses plain one-sided Hutchinson estimators, as specified in the appendix. We do not claim a dedicated controlled ablation proving that the orthogonalized variant of Algorithm 2 improves on plain Hutchinson in every setting.

  1. A non-trace statistic at genuinely large scale (new experiment)

Yes; this is now the focus of new Section 6.3 and Figure 7. We estimate the NTK effective rank for released Pythia models of different sizes and throughout pretraining for Pythia-410M, whose selected hidden-state space has 393,216 coordinates. We use 21 Hutch++ probes for the trace-derived rank measurement, rather than evaluating every state-basis direction. At initialization, effective rank generally grows with model size, while the trend is less clear after training. Over the 410M pretraining trajectory, effective rank first rises during learning-rate warmup and then falls by more than a factor of ten after warmup, accompanied by increasing trace. These are descriptive observations, not claims of a causal relationship to generalization or training stability. Crucially, this experiment uses a non-trace statistic in a setting where a direct state-basis approach would require 393,216 basis directions per operator pass.

  1. Utility and exact-versus-estimated costs in the applications

We agree that the distillation example is small and does not demonstrate large-scale savings. The RNN study instead involves 60 runs × 101 checkpoints = 6,060 measurements, totaling approximately 54,900 trace evaluations in 18.3 hours on a GTX Titan GPU. Its 6,000-dimensional state-space NTK uses 24 Hutch++ probes, versus 6,000 basis directions for direct trace evaluation. These are query counts, not a measured application-specific wall-clock speedup. The new Pythia experiment demonstrates the approach at much larger scale.

  1. What the RNN NTK adds beyond representation alignment

We agree that representation alignment more clearly captures the initialization-rank dependence in Figure 5A. However, representation alignment compares state values, whereas NTK alignment compares parameter sensitivities—the local geometry of gradient-driven updates. Similar representations therefore need not have similar tangent geometry. Figure 5A shows low NTK alignment even when representation alignment is high; Figures 5B–C show continued NTK drift and effective-rank changes at matched performance. We do not claim better prediction of accuracy or learning speed than representation alignment.

  1. Appendix corrections and distillation questions

We corrected the Jacobian ordering/transposes in Proposition 1 and the missing square in the effective-rank numerator. The reported results are unchanged: the main experiments use Hutch++ and the correct rank definition.

We also added absolute test accuracies (mean ± SEM) for CE-only and NTK-regularized students in both logit and hidden-state settings to Table 1. At the illustrative λ=0.7\lambda=0.7, logit-space accuracy gains are 0.53–0.77 percentage points; Figure 6A shows the full λ\lambda sweep. The hidden-space λ=0.5\lambda=0.5 is also illustrative, as is λ=0.4\lambda=0.4 for the alignment trajectory in Figure 6B. Students use labeled-example cross-entropy plus teacher-NTK alignment, not soft targets. Teacher accuracies are 90.05% (logit) and 90.44% (hidden). Gains are modest and can reverse with longer training (Figure 6C); we do not claim superiority to conventional distillation.

Reviewer_TREzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The manuscript develops matrix-free estimates of empirical state-space neural tangent kernel statistics, including trace, Frobenius norm, alignment, and participation-ratio effective rank. It combines Hutch++ and one-sided Jacobian-norm estimators, studies their runtime-error tradeoffs across MLPs, recurrent GRUs, and Pythia Transformers, and uses the resulting diagnostics to analyze recurrent learning and explore teacher-student regularization. The main contribution is a practical computational and empirical framework built from established numerical tools. The results demonstrate substantial savings relative to the specified exact state-basis reference. Several mathematical and experimental descriptions require correction, but the principal trace-estimation identities and reported runtime-accuracy evidence support a useful contribution.

Critical strengths
  1. Substantial demonstrated computational utility. The main acceleration claim is supported by measured fixed-configuration results as well as tolerance-dependent summaries. For Pythia-410M, Hutch++ at configured budget m = 96 reaches 0.58% median relative error in 8.03 seconds, compared with 8328.58 seconds for the stated exact state-basis reference. Their ratio is approximately 1037. This evidence remains useful independently of the separately reported 13499 times selected and interpolated speedup. The ratio is specific to the paper's baseline and timing setup.
  2. Broad and coherent validation. The experiments cover output-space MLP kernels, recurrent hidden-state kernels, and Transformer states. Exact references, repeated randomized estimates, and runtime-error curves provide a coherent validation of the practical approach. The smaller-model experiments also validate derived statistics rather than trace alone.
  3. A concrete analysis enabled by faster diagnostics. The recurrent study measures 60 training runs at 101 checkpoints each and compares geometry at matched task performance. Dense trajectories of alignment and effective rank provide a practical example of the scientific analysis enabled by the estimators, beyond a timing demonstration.
  4. Accessible implementation and AD tradeoffs. The operator formulation unifies several useful diagnostics without materializing the full kernel. Comparing forward-only, reverse-only, and two-sided estimators provides implementation guidance across different architectures. The contribution can be valuable through its synthesis and empirical demonstration even though its main numerical ingredients are established.
Critical weaknesses
  1. Speedup claims should distinguish measured configurations from retrospective summaries. Appendix D.4, lines 911-917, selects the fastest successful method using known exact errors, interpolates tolerance crossings, introduces an artificial starting point when the smallest budget succeeds, and excludes unsuccessful repeats. The 13499 times number should therefore be labeled as a retrospective benchmark summary. The fixed-configuration 8.03-second result already supports substantial utility; all-repeat success rates would clarify its reliability at a specified tolerance. The exact baseline also uses n complete NTK actions, while exact one-sided trace evaluation could use n VJPs or p JVPs. Its runtime has not been measured here, so the advantage over that stronger exact reference remains an open benchmarking question.
  2. Prior-work positioning should emphasize the actual contribution. The basic one-sided identities are standard Jacobian Frobenius-norm identities, and Hutch++ is an established algorithm. The paper should clearly state which additions concern estimator formulation, implementation, empirical characterization, and applications. Wang et al. (2023), Section 5.2, estimates NTK trace through random parameter perturbations without constructing the kernel; this is relevant precedent for the forward-only identity. Xiang et al. (2025) evaluates held-out task performance, so the broad statement that it did not examine generalization requires a narrower formulation. Hazelden et al. (2026), Appendix A.1.1, already discusses Hutch++ and several trace-derived statistics, while also crediting earlier work. Clarifying scope and chronology would make the present contribution easier to assess.
  3. Appendix B needs mathematical corrections. The reverse-mode identity in Proposition 1 interchanges J3 and J4 in the product on its left-hand side. Equation (12) omits the square on the expected squared-norm numerator of the participation ratio. The proof contains additional inconsistent transposes. These are genuine errors as written. However, the principal trace-estimation identities are correct, and the main Equation (3) and Appendix E use the correct squared-trace effective rank. The reported main experiments use Hutch++ for the derived statistics rather than the one-sided product estimator. Please correct the appendix and confirm that the implementation follows the intended formulas.
  4. The supporting distillation experiment needs a consistent specification. Section 6.2 describes equal-width 512-unit, one-hidden-layer teacher and student networks, whereas Appendix F gives widths 512 and 128 and uses two hidden layers plus a partial trace for the hidden-space experiment. Figure 6's caption refers to teacher-output supervision, while the objective and appendix specify classification cross-entropy plus NTK alignment. The architecture, targets, reduced operator, and selection of lambda and evaluation epochs should be reconciled. Figure 6C also shows the gain reversing over a longer horizon. These issues limit the distillation conclusion; they do not establish an error in the independent trace-estimation benchmarks or recurrent application.
Questions for authors
  1. For the already tested fixed method and probe budgets, what fraction of all repeats meets each target error? Please distinguish these measured results from the selected and interpolated headline speedups.
  2. Are the Proposition 1 and Equation (12) errors confined to the text? Please confirm the formulas used in the reported effective-rank computations.
  3. Which teacher/student architectures, supervision targets, and hidden-space partial-trace definition produced each Figure 6 panel and table row? How were lambda and reporting epochs selected?
  4. Can the authors state precisely the methodological and empirical additions relative to the cited earlier NTK trace and alignment work?
2 comments

Thank you for the detailed and constructive review. We appreciate your recognition of the computational utility of the framework. We have revised the manuscript to address the mathematical, reporting, and experimental-description issues, and have added a large-scale application to a pretrained language model.

All changes are made in the new pdf using blue text.

1. Speedup methodology and fixed-budget measurements

We agree that these are distinct comparisons and have revised the Introduction, Section 5.2, and figure captions accordingly. The reported 13,499× Pythia-410M speedup at 99% accuracy is a retrospective benchmark summary: it selects the fastest method meeting the tolerance in each repeat and interpolates between sampled probe budgets. This procedure excludes repeats that never meet the tolerance, and its interpolation uses an artificial starting point if the smallest sampled budget already succeeds. It should not be interpreted as the measured speedup of a prespecified configuration.

Independently of that summary, Hutch++ at fixed configured budget m=96m=96 achieved 0.58% median relative error in 8.03 s, versus 8,328.58 s for the exact state-basis evaluation: a measured 1,037× speedup against that reference. We now emphasize this fixed-configuration result. We also agree that an exact one-sided VJP/JVP baseline is a distinct comparison that we have not timed, and we do not claim this speedup against it.

2. Novelty and relationship to earlier methods

We agree that Hutch++, Hutchinson estimation, Jacobian-vector products, and the basic one-sided trace identities are established tools, and have clarified this in the Introduction and Related Work. Our goal is to make these methods practical for generalized empirical NTKs, where the state can be a large tensor of hidden activations across layers, time, and inputs. This was motivated partly by discussions with authors of related work (Xiang et al., Liu et al.), who noted computational limitations in their NTK analyses.

Our contributions are primarily computational and empirical: systematically benchmarking matrix-free estimation of NTK statistics (trace, norm, alignment, and effective rank), comparing Hutch++ with forward- and reverse-only estimators, extending one-sided estimation to NTK products, and demonstrating applications to recurrent learning, knowledge distillation, and large-scale language models. We have also clarified the distinctions from Hazelden et al., Curvlinops, and Xiang et al. in the revised manuscript.

3. Appendix B mathematical corrections

Thank you for catching these. We corrected the ordering and transpose inconsistencies in Proposition 1 and its proof, and restored the missing square in the participation-ratio numerator. Specifically, the product is tr⁡(J1J2⊤J3J4⊤)\operatorname{tr}(J_1J_2^\top J_3J_4^\top), while effective rank is tr⁡(K)2/tr⁡(K2)\operatorname{tr}(K)^2/\operatorname{tr}(K^2). These were manuscript errors, not changes to the reported experimental calculations: the main effective-rank experiments use the correct squared-trace definition and compute the derived statistics with Hutch++, not the appendix's one-sided product estimator.

4. Distillation specification and limitations

We have clarified the setup in Section 6.2: the teacher and student have widths 512 and 128, respectively, with one hidden layer for logit-space NTKs and two for hidden-state NTKs (where alignment uses a matrix-free partial trace over hidden units). Students use labeled-example cross-entropy plus alignment to the teacher's NTK, not teacher soft targets. The reported λ=0.7\lambda=0.7 (logit) and λ=0.5\lambda=0.5 (hidden) are illustrative settings, not required choices; Figure 6A shows results across a range of λ\lambda, while λ=0.4\lambda=0.4 in Figure 6B simply illustrates the alignment trajectory during training. Figure 6C shows that the modest test-accuracy gains do not necessarily persist over longer training. We do not claim superiority to conventional soft-target distillation.

5. New large-scale non-trace application

We have added Section 6.3 and Figure 7, using randomized trace estimates to study NTK effective rank, not just trace, across released Pythia model sizes and over the pretraining trajectory of Pythia-410M. The selected state of the 410M model has 393,216 coordinates. With 21 Hutch++ probes, we observe an initial increase in effective rank during learning-rate warmup, followed by a more than tenfold decrease in effective rank and an increase in trace after warmup. We describe this as an exploratory observation, not a causal explanation of warmup. It illustrates the kind of repeated, large-state non-trace diagnostic that motivated the method.

Thank you for the rebuttal that took in consideration all my points. I am convinced by the authors’ responses to my comments and am pleased to maintain my positive assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.