- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies how labels and prediction errors align with the NTK spectrum. The authors observe a similar pattern across several architectures and datasets, which weakens when the labels are randomized. They propose an explanation based on the smoothness of the label function. Under Condition 5.2, they derive upper and lower training-loss bounds for wide fully connected networks using the full NTK spectrum. These give a sharper account of loss decay than bounds based only on the smallest eigenvalue.
Critical strengths
- The paper is well written and easy to follow.
- The convergence analysis show why training can be fast despite small NTK eigenvalues. Under Condition 5.2, the full-spectrum upper and lower loss bounds give a sharper account of training speed than pessimistic guarantees based only on the smallest eigenvalue.
- The experiment support a link between meaningful labels and NTK alignment across several networks and datasets.
Critical weaknesses
- The experiments do not clearly show how well Condition 5.2 holds in practice. Although the condition allows deviations, the batch-averaged plots do not show whether the variation across eigendirections, including the outliers, stays within the allowed tolerance. This leaves the applicability of the convergence bound unclear.
- The theoretical explanation claims more than the arguments. Equation (10) gives an upper bound, while Theorem 4.2 claims proportionality, which is not accurate and also does not follow from Lipschitz continuity alone.
- Theorem 4.3 guarantees the relationship in at least one direction, but does not show more general rule of it across the bulk spectrum.
Questions for authors
- How closely do the individual kernels used in Figures 3 and 10 satisfy Condition 5.2, including the outlier directions?
- Equation (10) provides an upper bound. How does this imply the proportionality claimed in Theorem 4.2, given that identical labels can make the relevant component zero? Is an additional assumption needed, or should the conclusion be weakened?
We thank the reviewer for the detailed and constructive comments, which substantially helped us improve the optimization theory during the rebuttal period. We address your concerns and questions below.
W1: "The experiments do not clearly show how well Condition 5.2 holds in practice. ... This leaves the applicability of the convergence bound unclear." & Q1: "How closely do the individual kernels used in Figures 3 and 10 satisfy Condition 5.2"
A: This is an excellent observation. We agree that Condition 5.2, which does not include averaging over batches, is inconsistent with the empirical observations reported in Section 3. We sincerely thank the reviewer for bringing this issue to our attention and apologize for this oversight in the development of our theoretical analysis.
In response, we have revised Condition 5.2, as well as redeveloped the optimization theory accordingly, to incorporate batch averaging and ensure consistency with the empirical law. Please see the revisions highlighted in blue in the updated submission PDF. Specifically, we now ask the two-sided relationship to hold for the batch-averaged quantities, while allowing these quantities of each batch to deviate from their respective average values. Despite these deviations and the additional terms they introduce into the convergence bounds, we can still establish similar convergence guarantees. In particular, the resulting bounds remain trace-based, which is a significant improvement over existing convergence rates that is controlled by the smallest NTK eigenvalue. Moreover, the additional term introduced by the deviations is typically much smaller than the dominant term tr[(I−ηK)2tK] (See Remark 5.6 in the revised PDF) and decays fast with time t as well.
Regarding the eigenvalue outliers which seem not to obey the scaling law precisely, our analysis can still be readily extended. We make two observations: (1) these outlier eigenvalues (λ) are substantially larger than the eigenvalues in the remaining spectrum; and (2) their corresponding eigen-component squares (the vertical axis in the plots) are also larger than those of the lower-spectrum bulk, indicating that these outliers still make substantial contributions to the training loss. These contributions decay rapidly at a rate of (1−ηλ)2t, consequently, they do not fundamentally affect the overall convergence upper bound.
W2: "The theoretical explanation claims more than the arguments. Equation (10) gives an upper bound, while Theorem 4.2 claims proportionality, which is not accurate and also does not follow from Lipschitz continuity alone." & Q2: "... given that identical labels can make the relevant component zero? Is an additional assumption needed, or should the conclusion be weakened?"
A: Yes, we agree with the reviewer. Using a Lipschitz-based argument, we can establish only an upper bound theoretically, but not a two-sided bound. We apologize for this error. We have revised the theorems and corrected the corresponding statements (highlighted in blue in the revision) to reflect only the upper bound. We further note that the upper bound is attainable when the label difference ∣y1−y2∣ is of the same order as ∣∣x1−x2∣∣2.
The theory in this paper aims to provide intuition and potential explanations for this phenomenon. The upper bound alone offers significant insights: it substantially suppresses the contributions of small NTK eigenvalues to the training dynamics, thereby potentially improving convergence, whereas a lower bound would not provide such benefits. Furthermore, the comparison with noisy labels in Section 2 together with the numerical verification in Figure 2, suggest that Lipschitz continuity could be one of the key factors explaining this empirical phenomenon.
W3: "Theorem 4.3 guarantees the relationship in at least one direction, but does not show more general rule of it across the bulk spectrum."
A: Theorem 4.3 aims to provide intuitions, showing the result of two-point analysis extends. An analysis of the whole bulk spectrum is challenging, and we consider it out of the scope of this work. The difficulty arises from the lack of mathematical tools that precisely recover the expressions of eigenvalues and eigenvectors of an NTK matrix. However, when n is small, such as n=4, this is doable, and we can exactly predict the power law. We included this example in the revision (see Appendix C).