acceptodds
Under review as a conference paper at ICLR 2027

On the Stability of the Neural Tangent Kernel in Classification

Abstract

In wide neural networks, the Neural Tangent Kernel (NTK) remains approximately constant during training, providing a powerful theoretical tool for studying training dynamics, generalization, and connections to kernel methods. However, this theory is largely restricted to regression losses. It was previously thought that training on a classification loss, or more generally losses involving nonlinear output transformations, breaks this property, leading to divergent logits and a breakdown of the linearization. In this paper, we extend NTK theory to classification by identifying conditions under which wide neural networks remain in the lazy training regime. We show that parameter-space regularization ensures a constant NTK during training for cross-entropy loss, while in the absence of regularization the regime is recovered when targets are non-degenerate, i.e. when all classes have strictly positive probability. Under these conditions, training is well-approximated by the linearized model, yielding an explicit characterization of the solution in terms of the NTK. We further analyze the distribution of trained predictors induced by random initialization and relate this notion of model uncertainty to Bayesian methods.

Primary area:
learning theory
Keywords:
NTK, Classification, Wide Neural Networks, Gradient Flow

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_eHVPOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies when the neural tangent kernel (NTK) approximation remains valid throughout classification training, including at infinite training time. The motivation is that ordinary cross-entropy training with one-hot labels can produce diverging logits and substantial parameter movement, making standard bounded-neighborhood NTK arguments insufficient.

The main result establishes a high-probability, uniform-in-time approximation between a wide neural network and its trained linearization, with error of order O((log⁡n)c/n)O((\log n)^c/\sqrt{n}). The result applies under either positive quadratic regularization around initialization or suitable curvature and gradient-growth conditions on the loss. For classification, full-support targets, such as label-smoothed targets, provide the relevant conditions in the unregularized setting. The analysis also addresses softmax shift invariance through centered logits.

The proof combines bounds on network Jacobians and Hessians, control of the total parameter displacement, and contraction arguments for the nonlinear and linearized training dynamics. The paper subsequently characterizes limiting kernel predictors through a nonlinear inverse map and develops a Gaussian approximation to their initialization-induced distribution, which it compares with a Laplace approximation to a Bayesian classification model.

Validation is primarily theoretical. Synthetic classification examples and an MNIST odd-versus-even experiment illustrate differences between ordinary cross-entropy training, label smoothing, and regularization around initialization. These experiments provide qualitative support, but do not systematically test the predicted width dependence or the Bayesian covariance comparison.

Critical strengths
  1. Uniform-in-time control addresses the central difficulty in extending NTK arguments to classification. The paper identifies concrete conditions under which classification training remains in a neighborhood where linearization is accurate, and establishes approximation guarantees extending to convergence. This is consequential because finite-time NTK results alone do not justify conclusions about fully trained classifiers, particularly when ordinary cross-entropy training produces diverging logits.

  2. The analysis connects loss geometry to concrete changes in training behavior. The distinction between one-hot targets, full-support targets, and regularization around initialization provides a useful explanation of when bounded training trajectories can be expected. The treatment of softmax shift invariance in the stability analysis also addresses a genuine mathematical obstacle. Together, these results make the sufficient conditions more informative than an abstract assumption that training remains close to initialization.

Critical weaknesses
  1. Two downstream claims require mathematical correction or qualification. The Bayesian comparison appears to compare the ensemble covariance of gg with the Laplace covariance of F=β gF=\sqrt{\beta}\,g, omitting the scale conversion required for a comparison of the same quantity. Consequently, the claimed universal covariance ordering does not follow for the stated Bayesian model. Separately, the unregularized inverse-map formula is not well-defined for standard softmax logits without a gauge restriction: full-support targets determine probabilities and centered logits, but not the common logit shift. These issues affect the predictor-distribution and uncertainty contributions, although they do not by themselves invalidate the core stability theorem.

  2. The empirical evidence is too limited to assess several advertised consequences. The experiments mainly show illustrative trajectories and selected diagonal NTK entries. They do not systematically measure network-versus-linearization prediction error, its dependence on width, changes in the full training Gram matrix, or the accuracy of the Gaussian covariance approximation. The experimental description also omits several details needed for reproduction, including network widths and complete optimization settings. This limits the evidence that the asymptotic results and distributional approximations describe the finite networks being studied.

Mathematical Consistency and Rigor:

1. The Bayesian covariance comparison mixes differently scaled variables

Section 4.1 defines F=τgF=\tau g and β=τ2\beta=\tau^2. Therefore,

Cov⁡(F)=β Cov⁡(g)\operatorname{Cov}(F)=\beta\,\operatorname{Cov}(g).

Let Θ=Θ(X,X)\Theta=\Theta(X,X) and H∗=∇2C(g∗)H^* = \nabla^2 C(g^*).

The Laplace approximation for the stated posterior, expressed in the variable gg, has training-point covariance

ΣLap,g=(H∗+βΘ−1)−1\Sigma_{\mathrm{Lap},g}=(H^*+\beta\Theta^{-1})^{-1}.

Writing A=ΘH∗+βIA=\Theta H^*+\beta I, this becomes

ΣLap,g=1β[Θ−ΘH∗A−1Θ]\Sigma_{\mathrm{Lap},g}=\frac{1}{\beta}\left[\Theta-\Theta H^*A^{-1}\Theta\right].

The corresponding expression in Equation (41) lacks the factor 1/β1/\beta and instead represents the covariance of FF. Equation (42) then compares this matrix with an ensemble covariance expressed in gg.

The printed matrix identity may be algebraically correct, but its interpretation as a comparison of uncertainty in the same predictions is not justified. After matching scales, the asserted ordering need not hold for arbitrary β\beta. This requires a corrected comparison or a clearly specified alternative Bayesian model.

2. The inverse-map formula does not directly cover unregularized standard softmax

For β=0\beta=0,

Φ(z)=Θ∇C(z)\Phi(z)=\Theta\nabla C(z).

Standard softmax cross-entropy is invariant under adding a common constant to every class logit at each training input. Thus distinct logit vectors can have the same gradient, making Φ\Phi non-injective even when Θ\Theta is invertible.

Consequently, the expression g∞(X)=Φ−1(0)g_\infty(X)=\Phi^{-1}(0) and the statement that the training logits are deterministic require qualification. Full-support targets can determine the limiting probabilities and centered logits, while a common shift can retain dependence on initialization.

The stability analysis addresses this symmetry through centered logits, but the subsequent inverse-map and distributional statements need an explicit compatible restriction, such as reference-class coordinates or a specified gauge.

3. Finite-dimensional Gaussian convergence does not establish the stated uniform bound

Lemma A.1 claims a high-probability bound on

sup⁡x∈Md∥f0(x)∥\sup_{x\in\mathcal{M}^d}\|f_0(x)\|

and attributes it directly to the Gaussian-process limit. Finite-dimensional convergence alone does not control a supremum over an uncountable input domain. Compactness of the domain does not remove the need for a tightness, equicontinuity, or direct supremum-bound argument.

This appears repairable, but the current justification is incomplete. If only bounded initialization outputs on the finite training set are needed for the trajectory argument, the lemma could be weakened accordingly, with the uniform Jacobian estimates used for test-point control justified separately.

Questions for authors
  1. Bayesian covariance scaling: In Section 4.1, the variables satisfy F=β gF=\sqrt{\beta}\,g. Is the covariance in Equation (41) intended to describe FF or gg? For the stated model, the training-point Laplace covariance of gg is (H∗+βΘ−1)−1(H^*+\beta\Theta^{-1})^{-1}. Please reconcile this with Equations (41)–(42) and clarify whether the claimed covariance ordering survives when both distributions are expressed in the same variable.

  2. Unregularized softmax inverse: How should Φ−1(0)\Phi^{-1}(0) in Lemmas 3.2–3.3 be interpreted for β=0\beta=0 and standard softmax logits? Since softmax is invariant to a common shift, Φ\Phi is not injective in this setting. Are these statements restricted to reference-class coordinates or a specified gauge? If so, how is the initialization-dependent shift represented in the claimed predictor distribution?

  3. Uniform initialization bound: Lemma A.1 appears to infer a uniform bound over the compact input domain from finite-dimensional convergence to a Gaussian process. What additional tightness or equicontinuity argument justifies this inference? Alternatively, can the proof use a bound only on the finite training set and establish the required uniform test-point bounds separately?

Reviewer_Yy2VOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper extends linearized training results for wide networks to more general losses, including cross-entropy. In particular, the main theorem states that whenever weight decay is present, under certain regularity conditions on the loss function, the trained model output stays close to that of the trained linearized model, uniformly in training time. When weight decay is not present, the same result holds under PL-condition on the loss function. The paper proceeds with deriving the exact expression for the fully trained linearized model. This model is random since the initial one is random. They relate predictions of the fully trained linearized model with a Maximal A-Posteiori (MAP) estimate in a certain probabilistic model. They also approximate the trained linearized model ensemble with a Gaussian process, and relate it to a Laplace approximate NTK-GP-posterior. The theory is briefly validated on a toy one-dimensional dataset and on MNIST.

Critical strengths
  1. The paper is easy to read and the results are clearly delivered.
  2. The results cover a certain gap in the NTK theory.
Critical weaknesses

The paper's results seem quite incremental to me. First, the authors claim "Our key contribution here is to, instead, treat the softmax as part of the function-space loss, which allows us to recover the NTK regime at the level of the pre-activation outputs." (Lines 51-53) does not seem particularly surprising, since treating softmax as part of the cross-entropy loss rather than the network architecture is a standard choice.

Second, although the main technical result, Theorem 3.1, appears to provide a new uniform-in-time approximation guarantee, its proof largely relies on existing, well-known techniques. In particular, the control of the network Jacobian and parameter Hessian follows [2], whose result is restated as Lemma 2.1. Moreover, the proof of lazy training and exponential convergence in the unregularized case (Appendix E.1) explicitly follows [1] and [3], relying on NTK conditioning and the PL condition. The regularized case similarly relies on standard stability arguments. Thus, the main technical contribution seems to be an adaptation and combination of existing techniques rather than a conceptually new mechanism. The observation that regularization around initialization, or label smoothing under appropriate conditions, can preserve the linearized regime is also rather intuitive.

I do not see anything conceptually new in Section 3.2 either: the characterization of the fully trained linearized model essentially amounts to writing the optimality conditions of a kernel optimization problem, without providing a closed-form solution for general losses. Similarly, the connection to MAP estimation in Section 4.1 follows from the standard interpretation of quadratic RKHS regularization as a Gaussian prior. Section 4.2 offers a potentially novel extension of previous ensemble-variance comparisons to non-quadratic losses, but this does not seem substantial enough to justify publication at ICLR.

[1] Oymak & Soltanolkotabi. Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path? ICML'2019.

[2] Liu, Zhu & Belkin. On the Linearity of Large Non-linear Models: When and Why the Tangent Kernel Is Constant. NeurIPS'2020.

[3] Liu, Zhu & Belkin. Loss Landscapes and Optimization in Over-parameterized Non-linear Systems and Neural Networks. 2022.

Questions for authors

None

Reviewer_bPjsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper extends lazy training and NTK linearisation to cross-entropy and general convex function-space losses. It does this by treating the softmax as part of the loss, so the analysis works at the level of the logits.

Critical strengths
  • Correct proofs and a genuinely useful infinite-horizon results, the theoretical contribution is strong and novel.
  • Uniform-in-time closeness to the linearisation for CE is a really strong contribution. Most prior classification NTK results only hold for finite training time.
Critical weaknesses
  • Width requirements are probably exponential in N and the papers does not discuss it.
  • The experimental evidence is not enough: There is no width sweep checking the (log n)^c/√n rate. There is no quantitative ensemble-versus-ODE comparison; Fig. 1 is visual only. There is no experiment for §4.2 (ensemble versus Laplace covariance).
Questions for authors

Can you give explicit dependence of the required width and of C on N, β, ε and K, or argue the exponential dependence is only depending in the proof?

In Fig. 3, what width is used? Does the smoothed-CE NTK drift shrink with width at the predicted rate?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.