- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies when the neural tangent kernel (NTK) approximation remains valid throughout classification training, including at infinite training time. The motivation is that ordinary cross-entropy training with one-hot labels can produce diverging logits and substantial parameter movement, making standard bounded-neighborhood NTK arguments insufficient.
The main result establishes a high-probability, uniform-in-time approximation between a wide neural network and its trained linearization, with error of order . The result applies under either positive quadratic regularization around initialization or suitable curvature and gradient-growth conditions on the loss. For classification, full-support targets, such as label-smoothed targets, provide the relevant conditions in the unregularized setting. The analysis also addresses softmax shift invariance through centered logits.
The proof combines bounds on network Jacobians and Hessians, control of the total parameter displacement, and contraction arguments for the nonlinear and linearized training dynamics. The paper subsequently characterizes limiting kernel predictors through a nonlinear inverse map and develops a Gaussian approximation to their initialization-induced distribution, which it compares with a Laplace approximation to a Bayesian classification model.
Validation is primarily theoretical. Synthetic classification examples and an MNIST odd-versus-even experiment illustrate differences between ordinary cross-entropy training, label smoothing, and regularization around initialization. These experiments provide qualitative support, but do not systematically test the predicted width dependence or the Bayesian covariance comparison.
Critical strengths
-
Uniform-in-time control addresses the central difficulty in extending NTK arguments to classification. The paper identifies concrete conditions under which classification training remains in a neighborhood where linearization is accurate, and establishes approximation guarantees extending to convergence. This is consequential because finite-time NTK results alone do not justify conclusions about fully trained classifiers, particularly when ordinary cross-entropy training produces diverging logits.
-
The analysis connects loss geometry to concrete changes in training behavior. The distinction between one-hot targets, full-support targets, and regularization around initialization provides a useful explanation of when bounded training trajectories can be expected. The treatment of softmax shift invariance in the stability analysis also addresses a genuine mathematical obstacle. Together, these results make the sufficient conditions more informative than an abstract assumption that training remains close to initialization.
Critical weaknesses
-
Two downstream claims require mathematical correction or qualification. The Bayesian comparison appears to compare the ensemble covariance of with the Laplace covariance of , omitting the scale conversion required for a comparison of the same quantity. Consequently, the claimed universal covariance ordering does not follow for the stated Bayesian model. Separately, the unregularized inverse-map formula is not well-defined for standard softmax logits without a gauge restriction: full-support targets determine probabilities and centered logits, but not the common logit shift. These issues affect the predictor-distribution and uncertainty contributions, although they do not by themselves invalidate the core stability theorem.
-
The empirical evidence is too limited to assess several advertised consequences. The experiments mainly show illustrative trajectories and selected diagonal NTK entries. They do not systematically measure network-versus-linearization prediction error, its dependence on width, changes in the full training Gram matrix, or the accuracy of the Gaussian covariance approximation. The experimental description also omits several details needed for reproduction, including network widths and complete optimization settings. This limits the evidence that the asymptotic results and distributional approximations describe the finite networks being studied.
Mathematical Consistency and Rigor:
1. The Bayesian covariance comparison mixes differently scaled variables
Section 4.1 defines and . Therefore,
.
Let and .
The Laplace approximation for the stated posterior, expressed in the variable , has training-point covariance
.
Writing , this becomes
.
The corresponding expression in Equation (41) lacks the factor and instead represents the covariance of . Equation (42) then compares this matrix with an ensemble covariance expressed in .
The printed matrix identity may be algebraically correct, but its interpretation as a comparison of uncertainty in the same predictions is not justified. After matching scales, the asserted ordering need not hold for arbitrary . This requires a corrected comparison or a clearly specified alternative Bayesian model.
2. The inverse-map formula does not directly cover unregularized standard softmax
For ,
.
Standard softmax cross-entropy is invariant under adding a common constant to every class logit at each training input. Thus distinct logit vectors can have the same gradient, making non-injective even when is invertible.
Consequently, the expression and the statement that the training logits are deterministic require qualification. Full-support targets can determine the limiting probabilities and centered logits, while a common shift can retain dependence on initialization.
The stability analysis addresses this symmetry through centered logits, but the subsequent inverse-map and distributional statements need an explicit compatible restriction, such as reference-class coordinates or a specified gauge.
3. Finite-dimensional Gaussian convergence does not establish the stated uniform bound
Lemma A.1 claims a high-probability bound on
and attributes it directly to the Gaussian-process limit. Finite-dimensional convergence alone does not control a supremum over an uncountable input domain. Compactness of the domain does not remove the need for a tightness, equicontinuity, or direct supremum-bound argument.
This appears repairable, but the current justification is incomplete. If only bounded initialization outputs on the finite training set are needed for the trajectory argument, the lemma could be weakened accordingly, with the uniform Jacobian estimates used for test-point control justified separately.
Questions for authors
-
Bayesian covariance scaling: In Section 4.1, the variables satisfy . Is the covariance in Equation (41) intended to describe or ? For the stated model, the training-point Laplace covariance of is . Please reconcile this with Equations (41)–(42) and clarify whether the claimed covariance ordering survives when both distributions are expressed in the same variable.
-
Unregularized softmax inverse: How should in Lemmas 3.2–3.3 be interpreted for and standard softmax logits? Since softmax is invariant to a common shift, is not injective in this setting. Are these statements restricted to reference-class coordinates or a specified gauge? If so, how is the initialization-dependent shift represented in the claimed predictor distribution?
-
Uniform initialization bound: Lemma A.1 appears to infer a uniform bound over the compact input domain from finite-dimensional convergence to a Gaussian process. What additional tightness or equicontinuity argument justifies this inference? Alternatively, can the proof use a bound only on the finite training set and establish the required uniform test-point bounds separately?