acceptodds
Under review as a conference paper at ICLR 2027

Scaling Law of Label-NTK Alignment and a Tighter Convergence Bound in the NTK Regime

Abstract

Neural tangent kernel (NTK) provides a central framework for understanding the training dynamics of wide neural networks. In this work, we empirically identify a scaling law that often holds on real-world datasets, governing the alignment of training labels and label residuals with NTK eigenspaces. Specifically, the squared eigen-components of labels and label residuals scale approximately as and , respectively, where is the corresponding NTK eigenvalue. We theoretically explain this phenomenon through the Lipschitz continuity of real-world label functions. Finally, leveraging this scaling law, we derive a tighter convergence bound for gradient descent in the NTK regime. Unlike existing bounds that depend on the smallest NTK eigenvalue, which is typically extremely small and leads to pessimistic convergence rates, our bound depends on the entire spectrum and closely matches empirical training curves.

Primary area:
optimization
Keywords:
Neural Tangent Kernel, optimization, generalization, NTK regime, scaling laws

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uvsDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how labels and prediction errors align with the NTK spectrum. The authors observe a similar pattern across several architectures and datasets, which weakens when the labels are randomized. They propose an explanation based on the smoothness of the label function. Under Condition 5.2, they derive upper and lower training-loss bounds for wide fully connected networks using the full NTK spectrum. These give a sharper account of loss decay than bounds based only on the smallest eigenvalue.

Critical strengths
  • The paper is well written and easy to follow.
  • The convergence analysis show why training can be fast despite small NTK eigenvalues. Under Condition 5.2, the full-spectrum upper and lower loss bounds give a sharper account of training speed than pessimistic guarantees based only on the smallest eigenvalue.
  • The experiment support a link between meaningful labels and NTK alignment across several networks and datasets.
Critical weaknesses
  • The experiments do not clearly show how well Condition 5.2 holds in practice. Although the condition allows deviations, the batch-averaged plots do not show whether the variation across eigendirections, including the outliers, stays within the allowed tolerance. This leaves the applicability of the convergence bound unclear.
  • The theoretical explanation claims more than the arguments. Equation (10) gives an upper bound, while Theorem 4.2 claims proportionality, which is not accurate and also does not follow from Lipschitz continuity alone.
  • Theorem 4.3 guarantees the relationship in at least one direction, but does not show more general rule of it across the bulk spectrum.
Questions for authors
  • How closely do the individual kernels used in Figures 3 and 10 satisfy Condition 5.2, including the outlier directions?
  • Equation (10) provides an upper bound. How does this imply the proportionality claimed in Theorem 4.2, given that identical labels can make the relevant component zero? Is an additional assumption needed, or should the conclusion be weakened?
1 comment

We thank the reviewer for the detailed and constructive comments, which substantially helped us improve the optimization theory during the rebuttal period. We address your concerns and questions below.

W1: "The experiments do not clearly show how well Condition 5.2 holds in practice. ... This leaves the applicability of the convergence bound unclear." & Q1: "How closely do the individual kernels used in Figures 3 and 10 satisfy Condition 5.2"

A: This is an excellent observation. We agree that Condition 5.2, which does not include averaging over batches, is inconsistent with the empirical observations reported in Section 3. We sincerely thank the reviewer for bringing this issue to our attention and apologize for this oversight in the development of our theoretical analysis.

In response, we have revised Condition 5.2, as well as redeveloped the optimization theory accordingly, to incorporate batch averaging and ensure consistency with the empirical law. Please see the revisions highlighted in blue in the updated submission PDF. Specifically, we now ask the two-sided relationship to hold for the batch-averaged quantities, while allowing these quantities of each batch to deviate from their respective average values. Despite these deviations and the additional terms they introduce into the convergence bounds, we can still establish similar convergence guarantees. In particular, the resulting bounds remain trace-based, which is a significant improvement over existing convergence rates that is controlled by the smallest NTK eigenvalue. Moreover, the additional term introduced by the deviations is typically much smaller than the dominant term tr[(I−ηK)2tK]tr[(I-\eta K)^{2t}K] (See Remark 5.6 in the revised PDF) and decays fast with time tt as well.

Regarding the eigenvalue outliers which seem not to obey the scaling law precisely, our analysis can still be readily extended. We make two observations: (1) these outlier eigenvalues (λ\lambda) are substantially larger than the eigenvalues in the remaining spectrum; and (2) their corresponding eigen-component squares (the vertical axis in the plots) are also larger than those of the lower-spectrum bulk, indicating that these outliers still make substantial contributions to the training loss. These contributions decay rapidly at a rate of (1−ηλ)2t(1-\eta\lambda)^{2t}, consequently, they do not fundamentally affect the overall convergence upper bound.

W2: "The theoretical explanation claims more than the arguments. Equation (10) gives an upper bound, while Theorem 4.2 claims proportionality, which is not accurate and also does not follow from Lipschitz continuity alone." & Q2: "... given that identical labels can make the relevant component zero? Is an additional assumption needed, or should the conclusion be weakened?"

A: Yes, we agree with the reviewer. Using a Lipschitz-based argument, we can establish only an upper bound theoretically, but not a two-sided bound. We apologize for this error. We have revised the theorems and corrected the corresponding statements (highlighted in blue in the revision) to reflect only the upper bound. We further note that the upper bound is attainable when the label difference ∣y1−y2∣|y_1-y_2| is of the same order as ∣∣x1−x2∣∣2||x_1-x_2||_2.

The theory in this paper aims to provide intuition and potential explanations for this phenomenon. The upper bound alone offers significant insights: it substantially suppresses the contributions of small NTK eigenvalues to the training dynamics, thereby potentially improving convergence, whereas a lower bound would not provide such benefits. Furthermore, the comparison with noisy labels in Section 2 together with the numerical verification in Figure 2, suggest that Lipschitz continuity could be one of the key factors explaining this empirical phenomenon.

W3: "Theorem 4.3 guarantees the relationship in at least one direction, but does not show more general rule of it across the bulk spectrum."

A: Theorem 4.3 aims to provide intuitions, showing the result of two-point analysis extends. An analysis of the whole bulk spectrum is challenging, and we consider it out of the scope of this work. The difficulty arises from the lack of mathematical tools that precisely recover the expressions of eigenvalues and eigenvectors of an NTK matrix. However, when nn is small, such as n=4n=4, this is doable, and we can exactly predict the power law. We included this example in the revision (see Appendix C).

Reviewer_bpZ1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how label and residual energy is distributed across the eigendirections of the neural tangent kernel (NTK), with the goal of explaining why training can progress faster than worst-case convergence bounds based on the smallest eigenvalue suggest. Across several neural network architectures and image datasets, the authors report approximate relationships of the form

\qquad |\langle v_i,r_0\rangle|^2\propto\lambda_i,$$ where $(r_0)$ is the residual at initialization. The authors propose Lipschitz continuity of the label function as a theoretical explanation for the label-alignment relationship. Under an additional assumption on the initial residual coefficients, they derive a trace-based characterization of squared-loss convergence for sufficiently wide networks in the NTK regime. They also obtain an iteration-complexity bound under power-law eigenvalue decay and compare the proposed predictions with empirical training curves.
Critical strengths

S1. The paper identifies a specific and potentially informative empirical regularity.

The distinction between the reported label and residual exponents, approximately (2) and (1), is interesting. It provides a concrete hypothesis about the distribution of target information across NTK eigendirections and could help explain why bounds based only on the smallest eigenvalue substantially overestimate the time needed to reduce the training loss.

S2. The empirical investigation covers several architectures and includes useful controls.

The experiments span MLPs, CNNs, ResNets, and selected parameter groups of a ViT across multiple image datasets. The random-label comparisons are informative for investigating whether the observed relationships depend on target structure. This breadth makes the empirical phenomenon worthy of further investigation, even though the precise scope of the scaling claim needs clarification.

S3. The conditional convergence characterization is a useful contribution.

Accounting for residual energy across the spectrum provides a more informative description of optimization than a worst-case bound based on (λmin⁡)(\lambda_{\min}). Under the stated residual-alignment assumption, the trace expression offers a simple characterization of the loss trajectory. The close agreement with several CNN training curves is encouraging, subject to clarification of how those predictions were computed.

Critical weaknesses

W1. The Lipschitz argument does not establish the claimed exponent.

Equation (10) establishes an (O(θ2))(O(\theta^2)) upper bound on the squared label projection. Together with the kernel expansion (λmin⁡=Θ(θ))(\lambda_{\min}=\Theta(\theta)), this gives ∣⟨vmin⁡,y⟩∣2=O(λmin⁡2).|\langle v_{\min},y\rangle|^2=O(\lambda_{\min}^2). However, Theorem 4.2 claims proportional scaling, and the accompanying discussion states that the theory precisely predicts the exponent (2). The stated Lipschitz assumption does not provide the matching lower bound needed for this conclusion.

For example, consider x1=(1,0),x2=(cos⁡θ,sin⁡θ),x_1=(1,0),\qquad x_2=(\cos\theta,\sin\theta), with the smooth, nonconstant, (1)-Lipschitz label function (y(x)=x(1))(y(x)=x^{(1)}). Under the paper’s two-point kernel construction,

=\frac{(1-\cos\theta)^2}{2} =\Theta(\theta^4) =\Theta(\lambda_{\min}^4).$$ Thus, the assumptions permit a different exponent. The scope of the extension also needs to be distinguished from the empirical claim: Theorem 4.3 concerns the existence of a mode associated with a closely spaced input pair. It does not establish a common relationship across the bulk spectrum. This affects a central theoretical contribution. Correcting the statement to an upper bound would still provide a meaningful explanation for small label projections along certain small-eigenvalue directions, but it would not explain why the observed exponent should be approximately (2). W2. **The averaged empirical trend does not establish the pointwise premise of the convergence result.** The alignment evidence is based on aggregation across batches and focuses on the bulk spectrum, with acknowledged exceptions among the largest eigenvalue outliers. In contrast, Condition 5.2 requires $$(c-\delta)\lambda_i \leq |\langle v_i,r_0\rangle|^2 \leq (c+\delta)\lambda_i \qquad\text{for every }i,$$ with $(\delta\ll c)$. This is a narrow, two-sided relationship for every eigenmode of a single initial kernel. An averaged bulk trend does not, by itself, establish such a condition. The reference slopes in the plots also do not quantify whether this uniform relationship is plausible. The convergence result remains meaningful as a conditional statement. My concern is whether the reported observations sufficiently support its applicability to the experiments. The label-regularity argument does not supply the required residual condition, and the authors appropriately acknowledge the absence of a theoretical explanation for the residual exponent. The presentation should therefore distinguish the empirical motivation for the assumption from evidence that it holds in the settings where the convergence result is applied. W3. **The construction of the convergence predictions and the multiclass quantities is insufficiently specified.** For Figures 3 and 10, the experimental description does not clearly distinguish between $$L_{\mathrm{trace}}(t) =\frac{c}{2}\sum_i\lambda_i(1-\eta\lambda_i)^{2t}$$ and $$L_{\mathrm{spectral}}(t) =\frac12\sum_i |\langle v_i,r_0\rangle|^2(1-\eta\lambda_i)^{2t}.$$ The first expression reflects the proposed residual-scaling relationship. The second is the frozen-kernel prediction using the measured initial residual coefficients and can be computed without assuming that relationship. Agreement with these two expressions supports different conclusions, so the exact quantity plotted matters for evaluating the convergence contribution. Rescaling a prediction using the initial loss is reasonable. The missing details concern the expression being rescaled, the constants used, and whether the plotted curve is a trace approximation or includes the error terms of the stated bound. The multiclass implementation also needs clarification. The theoretical formulation uses scalar outputs, whereas the experiments include multiclass classifiers. The target encoding, residual definition, treatment of output coordinates in the NTK, and normalization are not sufficiently specified to establish that the experimental quantities correspond to those analyzed theoretically. These appear to be reporting issues that could be resolved from the existing implementation.
Questions for authors
  1. What justifies proportional scaling rather than an upper bound?

Under the paper’s stated interpretation of (∼)(\sim), which existing assumption or proof step provides the lower bound needed to exclude the counterexample above? If the intended result is only an (O(λ2))(O(\lambda^2)) upper bound, please specify the corrected claim and explain how it changes the interpretation of the observed exponent and the extension to larger datasets.

This would determine whether the central concern is an omitted justification or a substantive limitation of the proposed explanation.

  1. How is Condition 5.2 intended to relate to the averaged empirical observations?

Is the pointwise condition an independent assumption motivated by the observations, or do the authors claim that the existing evidence establishes it? If its applicability was already assessed for the reported convergence runs, please identify the evidence or constants used and explain how spectral outliers were handled. If it was not assessed, please clarify the intended conditional interpretation. No new mode-wise measurements are requested.

This would help determine how much empirical support the paper provides for applying the convergence result.

  1. What exact expression is plotted in Figures 3 and 10?

Please specify whether the curves use (Ltrace),(Lspectral)(L_{\mathrm{trace}}), (L_{\mathrm{spectral}}), or another expression. Please also explain the normalization already used, how (c) was chosen, and whether the (δ)(\delta) and (ϵ)(\epsilon) terms are included. If the curves represent a central trace approximation rather than the full bound, please identify this explicitly.

This clarification would determine whether the experiments directly support the proposed scaling-based prediction or primarily validate frozen-NTK dynamics using measured residual coefficients.

  1. How are (y)(y), (r)(r), and (K)(K) defined in the multiclass experiments?

Please specify the target encoding, the outputs used to form the residual, and the dimensions and output-coordinate treatment of the kernel. In particular, is the implementation based on the full multioutput block NTK, a kernel for an individual output, or a scalarized or traced kernel? How do these choices and the loss normalization correspond to the scalar squared-loss dynamics analyzed in the paper?

2 comments

We thank the reviewer for the constructive comments, which substantially helped us improve the optimization theory during the rebuttal period.

W1: "The Lipschitz argument does not establish the claimed exponent..." & Q1: "What justifies proportional scaling rather than an upper bound?"

A: Yes, we agree with the reviewer. Using a Lipschitz-based argument, we can establish only an upper bound theoretically, but not a two-sided bound. We apologize for this error. We have revised the theorems and corrected the corresponding statements (highlighted in blue color in the revision) to reflect only the upper bound. We further note that the upper bound is attainable when the label difference ∣y1−y2∣|y_1-y_2| is of the same order as ∣∣x1−x2∣∣2||x_1-x_2||_2.

The theory in this paper aims to provide intuition and potential explanations for this phenomenon. The upper bound alone offers significant insights: it substantially suppresses the contributions of small NTK eigenvalues to the training dynamics, thereby potentially improving convergence, whereas a lower bound would not provide such benefits. Furthermore, the comparison with noisy labels in Section 2 together with the numerical verification in Figure 2, suggest that Lipschitz continuity could be one of the key factors explaining this empirical phenomenon.

W2: "The averaged empirical trend does not establish the pointwise premise of the convergence result." & Q2: "How is Condition 5.2 intended to relate to the averaged empirical observations?"

A: This is an excellent observation. We agree that Condition 5.2, which does not include averaging over batches, is inconsistent with the empirical observations reported in Section 3. We sincerely thank the reviewer for bringing this issue to our attention and apologize for this oversight in the development of our theoretical analysis.

In response, we have revised Condition 5.2, as well as redeveloped the optimization theory accordingly, to incorporate batch averaging and ensure consistency with the empirical law. Please see the revisions highlighted in blue in the updated submission PDF. Specifically, we now ask the two-sided relationship to hold for the batch-averaged quantities, while allowing these quantities of each batch to deviate from their respective average values. Despite these deviations and the additional terms they introduce into the convergence bounds, we can still establish similar convergence guarantees. In particular, the resulting bounds remain trace-based, which is a significant improvement over existing convergence rates that is controlled by the smallest NTK eigenvalue. Moreover, the additional term introduced by the deviations is typically much smaller than the dominant term tr[(I−ηK)2tK]tr[(I-\eta K)^{2t}K] (See Remark 5.6 in the revision) and decays fast with time tt as well.

Regarding the eigenvalue outliers which seem not to obey the scaling law precisely, our analysis can still be readily extended. We make two observations: (1) these outlier eigenvalues are substantially larger than the eigenvalues in the remaining spectrum; and (2) their corresponding eigen-component squares (the vertical axis in the plots) are also larger than those of the lower-spectrum bulk, indicating that these outliers still make substantial contributions to the training loss. These contributions decay rapidly at a rate of (1−ηλ)2t(1-\eta\lambda)^{2t}, consequently, they don't fundamentally affect the overall convergence bound.

W3: "The construction of the convergence predictions is insufficiently specified." & Q3: "What exact expression is plotted in Figures 3 and 10?"

A: In Figure 3 and 10, the curves are computed based on Ltrace(t)=ctr[K(1−ηK)2t]L_{trace}(t) = c tr[K(1-\eta K)^{2t}]. Given cc is unknown, we rescaled it with c=L(0)/tr[K]c=L(0)/tr[K] so that the curves start from L(0)L(0) (this does not change the fast convergence rate). In the curve, we didn't include the error terms ϵ/2\epsilon/2, which is often very small. Therefore, the experiments directly support the proposed scaling-based prediction.

W4: "The multiclass implementation also needs clarification." & Q4: "How are yy, rr and KK defined in the multiclass experiments?"

A: For the multiclass tasks, we use the one-hot encoding for the labels and prediction outputs, hence residual rr as well. We flatten them to vectors so that y,r∈RnCy,r\in\mathbb{R}^{nC}, where CC is the number of classes. The NTK is the full multioutput block matrix which is of dimension (nC)×(nC)(nC)\times (nC). The NTK eigenvectors are also of dimension nCnC, which matches that of yy and rr so that inner product can be computed for smaller scale MLP and CNN experiments.

We develop the theory for scalar-valued labels and outputs to remain consistent with existing optimization theories in the literature. The theoretical analysis can be extended to multiclass settings, provided that the underlying deep learning optimization theories can also be extended to such settings.

W1b: "Theorem 4.3 concerns the existence of a mode associated with a closely spaced input pair. It does not establish a common relationship across the bulk spectrum"

A: The main challenge in extending Theorem 4.3 to the bulk spectrum of the NTK lies in deriving explicit expressions for the eigenvalues and eigenvectors of a general (n \times n) NTK matrix, which is generally intractable with existing mathematical tools and beyond the scope of this work. To provide further evidence that our theory also applies to the bulk spectrum, we have included an illustrative example with (n=4) in the revised manuscript (see Appendix C), where the scaling law holds across the entire spectrum. As long as the label differences between close input pairs are large (for example ∣y1−y2∣>c∣∣x1−x2∣∣|y_1-y_2|>c||x_1-x_2||), the lower bound also typically holds.

Reviewer_F4eyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

For neural tangent kernel (NTK), the work presents an approximate power-law phenomenon that the squared labels projections and the squared residuals onto NTK eigenvectors scale as λi2\lambda_i^2 and λi.\lambda_i. This relationship is verified across several neural network architectures and real-world datasets. Then, it build an upper bound of the label projection to theoretically explain above results. Inspired by this, this paper derives a refined convergence bound for gradient descent that depends on the full NTK spectrum rather than solely on its smallest eigenvalue. Empirical results show this bound more closely matches training curves than conventional worst-case bounds.

Critical strengths
  1. The work provides abundant empirical results. The experiments include multiple datasets, architectures, random-label controls, and measurements at different training stages.

  2. This work is well-motivated. It starts from an empirical observation and use this to improve existing bounds.

Critical weaknesses
  1. The empirical evidence provided in this work is not strong enough to support the scaling law of NTK. The scope of eigenvalue is limited comparing with previous scaling law works makes the law less convincing. Moreover, there are some obvious deviations of the scaling law in the reported figures which requires more careful clarification.

  2. The theoretical results are incomplete to support the empirical power law. The power law needs the lower and upper bounds but theorem 4.3. only provides an upper bound.

Questions for authors

Theorem 4.2 establishes that $\langle v_{\min},y\rangle^2=O(\theta^2). $ Could you please clarify how the stronger scaling relationship follows from the current assumptions?

1 comment

We thank the reviewer for their evaluation and comments. We address your concerns and questions below.

W1a: "The empirical evidence ... is not strong enough to support the scaling law of NTK."

A: The experiments are run across various architectures (MLP, CNN, ResNet, Transformers) and datasets (MNIST, CIFAR-10, TinyImageNet etc). The scaling law of NTK consistently holds across various settings. We believe these empirical evidences are strong enough. If you have other suggestions regarding the empirical evidence, please specify. We are happy to provide more evidence, if time allows.

**W1b: " The scope of eigenvalue is limited comparing with previous scaling law works makes the law less convincing"

A: Honestly, we don't quite understand this assessment. We don't think we compare with "previous scaling law works" in this paper; we don't even notice a similar "previous scaling law works" exist. Could you please specify.

W1c: "there are some obvious deviations of the scaling law in the reported figures which requires more careful clarification"

A: From the figures, we can see that the relation holds tightly (very close to linear, except the few top outliers) in the log-log plot, indicating a close-to-perfect power law dependence. There is tiny deviation from a perfect theoretical line, which is quite common in empirical "laws" in deep learning (such as the scaling law that describes the performance vs. training resources, edge of stability, etc). If you refer to the deviation of the slope from α=1\alpha=1 or 22, we agree that the slopes do not perfectly align, but they align with lines with slightly different slops. The slight slop deviation does not falsify a power law dependence.

W2: "The theoretical results are incomplete to support the empirical power law. The power law needs the lower and upper bounds but theorem 4.3. only provides an upper bound."

A: The theory of this paper is set to provide intuitions and possible explanations of this phenomenon. A complete theory that fully explains and predicts this phenomenon is out of the scope of this submission (please note that many empirical phenomena, such as edge of stability, the performance-resource scaling law, haven't been fully theoretically understood). The difficulty arises from the lack of mathematical tools that precisely recover the expressions of the eigenvalues and eigenvectors of an NTK matrix. However, when nn is small, such as n=4n=4, this is doable, and we can exactly predict the upper bound of the power law on the whole spectrum.We included this example in the revision (see Appendix C). As long as the label differences between close input pairs are large (for example ∣y1−y2∣>c∣∣x1−x2∣∣|y_1-y_2|>c||x_1-x_2||), the lower bound also typically holds.

Q1: "Theorem 4.2 establishes that ⟨vmin,y⟩2=O(θ2)\langle v_{min},y\rangle^2 = O(\theta^2). Could you please clarify how the stronger scaling relationship follows from the current assumptions?"

A: We are not sure what "stronger scaling relationship" refers to. If it refers to ⟨vmin,y⟩2=O(θ2)\langle v_{min},y\rangle^2 = O(\theta^2), this simply follows Eq.(10). If it refers to Θ(θ2)\Theta(\theta^2), please see the revision: basically, from the Lipschitz continuity of target function, we can only have the upper bound ⟨vmin,y⟩2=O(θ2)\langle v_{min},y\rangle^2 = O(\theta^2), because Lipschitz continuity is an upper bound itself. The stronger result of Θ(θ2)\Theta(\theta^2) can be achieved if the labels differ enough.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.