acceptodds
Withdrawn from ICLR 2027

Kernel Velocity: A Local Certificate for Feature Learning beyond the NTK

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The initial neural tangent kernel (NTK) does not reveal whether a network will keep that kernel fixed or learn task-dependent features. We isolate the missing local quantity: the task-effective kernel velocity (\dot K_0 r_0), where (K_t) is the empirical NTK and (r_t) is the training residual. For any smooth network trained by preconditioned gradient flow, we prove that the nonlinear predictor and its frozen-NTK trajectory (f_t^K) first separate as [ f_t-f_t^K =-t^22N\dot K_0r_0+O(t^3). ] We then construct a centered two-layer scaling family whose members have exactly the same initial predictor and NTK. Their kernel velocities obey the exact finite-width law (\dot K_0^(\alpha)=m^\alpha-1\cV_m); a nonzero population limit yields a vanishing lazy velocity at (\alpha=1/2) and order-one velocity at (\alpha=1). A rank-one linear model gives an exact adaptive-kernel closure and a monotone law for oriented teacher alignment. Experiments recover the predicted width exponents, verify second- and third-order trajectory expansions in two-layer and deep networks, and identify task-aligned nonlinear features. At a common training time, the adaptive scaling lowers test mean squared error from (0.294) for the matched frozen NTK to (0.160), while its feature spike has (0.93) teacher alignment. A stopping-time stress test separates this acceleration from a predictive advantage: on the three harder teachers, adaptation also beats the minimum test error on a dense exact-flow grid. Kernel velocity is a computable local certificate of departure from fixed-kernel training, not by itself a global guarantee of useful representation learning.

Primary area:
learning theory
Keywords:
kernel, NTK, theory

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_W66kOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper attempts to mathematically explain what the deviations in error are between a model and its linearized clone for short timescales after initialization. Their analysis boils down to performing a Taylor series expansion of the models about their initialization and analyzing the dominant term that differs between the two, labeled as the “kernel velocity.” Their analysis heavily leans on mathematical theorems and propositions over empirical validation. The empirical checks show these theorems apply in the scope they were conceived in, and fail outside them.

Critical strengths

None

Critical weaknesses
  1. There is no structure to this paper which is accompanied by consistent jargon that doesn’t appear anywhere else in the kernel theory literature.
  2. There is little-to-no strong evidence to support the broader claims of the paper. To paraphrase the abstract, this paper studies how feature changes may alter predictions without improving them. The theory that is developed is applied empirically to a toy setting where the timescale shown (Fig. 1) is at t=0.05t=0.05 with a difference in magnitude between a real model and its linearized counterpart of at most 4⋅10−64 \cdot 10^{-6}. How this relates to the features (not defined anywhere in the paper and thus is left vague), or their evolution is not discussed. There is no notable feature learning for models in a gradient flow regime at time t=0.05t=0.05.
Questions for authors

None

Reviewer_rgB2OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies feature changes during training through the derivative of the neural tangent kernel (NTK). The authors characterize the local difference between nonlinear training and frozen-kernel training. They derive several expressions for kernel velocity and prediction differences, analyze their scaling with network width, and conduct extensive experiments to investigate whether these quantities can serve as diagnostics for feature learning and predictive improvement.

Critical strengths

The distinction between feature changes and predictive improvement is an interesting question. The paper also includes extensive experiments and reports negative results.

Critical weaknesses
  1. The main theoretical contributions are unclear. The paper is very difficult to follow, and I find it hard to identify the main insights conveyed by the theoretical results. Proposition 1 gives an expression for the derivative of the NTK. Theorem 2 derives a second-order expansion of the difference between nonlinear and frozen-kernel predictions, and Proposition 3 provides a finite-time integral expression and a higher-order expansion. However, the paper does not sufficiently explain what we should learn from these formulas. In particular, how are these results related to feature learning?

  2. The experiments are difficult to follow and do not clearly support the proposed framework. The experimental section contains many different studies. However, the connections between these experiments and the main claims are not clearly explained. I do not understand how these experiments validate the usefulness of the proposed diagnostic. More importantly, Tables 2 and 7 show that the proposed framework has a negative result in practice.

Questions for authors

See weakness.

Reviewer_HoTrOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

In this work, the authors investigates how feature change during training affects the prediction of neural networks, through the derivative of the NTK applied to the training residual (called kernel velocity in the paper). They show that this quantity determines how fast nonlinear training departs from frozen-kernel training at early time, and build a matched two-layer network family that keeps the same initial predictor and kernel to isolate this effect. They further derive a signed held-out risk contrast that predicts whether this departure helps or hurts. Experiments show that the contrast predicts early risk ordering well, but becomes less accurate over longer training.

Critical strengths
  1. The idea of separating feature change from predictive benefit is clear and useful, and the matched construction removes the effect of different initializations.

  2. The experiments are careful, with paired seeds, validation-based selection and independent checks, and the authors also report the negative results.

Critical weaknesses

The theory mainly holds near initialization. Sign accuracy is 100% up to t=0.05 but drops to 47.9% at t=50, which the authors acknowledge.

Questions for authors

In Table 2, the learned rules have about 50% accuracy but AUC well below 0.5, and the text reports 67.6% for the context rule while the table shows 50.2%. Could the authors clarify these numbers?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.