acceptodds
Withdrawn from ICLR 2027

Spectral Bias in the Evolving Neural Tangent Kernel at Finite Width

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

This paper shows how spectral bias behaves in the finite sample size and finite network width setting, for regression on the unit circle using a shallow ReLU network. Finite samples and finite width are the setting of practical training, which theory mostly overlooks. Spectral bias is the tendency of gradient-based training to learn the low-frequency part of a target before its high-frequency part. In the infinite-width limit the Neural Tangent Kernel (NTK) under the uniform measure on the circle is a convolution, so its eigenfunctions are the Fourier modes, and low frequencies, corresponding to larger eigenvalues, are learned faster. In this specific case, the limiting kernel and its spectrum are known in closed form. For a network of finite width, the NTK is computed from its actual parameters, so it is random at initialization and changes during training. Using the limiting kernel as a reference, we can measure two things separately for a finite-width NTK: how close its eigenspaces are to the Fourier modes, through the cosine similarity between the two subspaces (eigenspace alignment), and how large its average eigenvalue is on each frequency (spectral strength). With the kernel held fixed at initialization, i.e. the lazy training regime, we prove that on the low frequencies its eigenvalues and eigenvectors stay close to the limiting ones, with a deviation of in the number of training points and in the width , and the measured deviations decay at these rates. We observe that at smaller widths the neural network with evolving weights reaches a lower training loss than the same network trained in the lazy training regime, and it is correlated with an increase in the spectral strength of the lowest frequencies while eigenspace alignment degrades at higher frequencies. The increase persists even for a target from which these low-frequency modes are absent, and we prove that the expected rate at which the finite-width NTK changes at initialization does not depend on the target, so we believe the strengthened directions are set by the network's input embedding and bias, architecture, and parameter distribution. A formal theory of the evolving-kernel regime remains open.

Primary area:
learning theory
Keywords:
spectral bias, neural tangent kernel, training dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_f5ZFOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This manuscript explores the spectral behavior of the Neural Tangent Kernel in various regimes (finite sample, finite width, under evolution). Theoretical results are given for the finite sample and finite width case. The evolving case is more complicated and only an empirical study is provided.

Critical strengths

The discussion of the finite sample and finite width setting is reasonable (and expected behavior is recovered).

Critical weaknesses

This manuscript has two main weaknesses. While both are significant, the more productive path to improvement is substantially addressing the second.

First, while I find the theoretical results sensible they are also thoroughly unsurprising given the setting and, existing, relevant concentration results that power them. Unsurprising is not necessarily bad, but I also do no think that, on their own, these theoretical results are substantive enough to carry the manuscript. This is particularly true given the very limited stetting. I am not familiar enough with all the related literature to speak about novelty, but the technical pieces are certainly widespread. Moreover, I stand by my prior comment about substance even under the claim that these are fully novel results.

The second weakness is that the more interesting regime where the kernel is evolving is underdeveloped. It is completely reasonable for such an exploration to be largely empirical---mathematical study of that setting is challenging. Unfortunately, the manuscripts current study is not adequately developed around interesting narratives. In other words, the set of included experiments seem quite narrowly focused and don't really provided much outside of a narrow setting.

In fact, the lack of a strong central narrative is plainly evident in section 5. The prose is unfocused and unstructured. It makes it challenging to read it and come away with a clear understanding of what is being conveyed. A more focused message paired with appropriate experiments (which would depend on the message itself) could strengthen the manuscript.

Questions for authors
  • Is the sampling always uniform in section 3.2? More generally, perhaps I missed something, but its not clear why U_k has orthonormal columns for arbitrary finite samples. (All the discussion in that section is fine, I think it might just be that the definition of U_k is slightly more nuanced, particularly given the specific measure, and therefore normalization, used for the continuous case.)

  • Line 249 - 250: while pedantic, Φ\Phi is in Rn×d,\mathbb{R}^{n\times d}, Φi,p\Phi_{i,p} is a scalar.

  • How specific are some of these results to the target always being a finite Fourier mixture with, often, relatively small k?

1 comment

Thank you for the detailed and constructive review. We agree that the concentration results are expected, and that the evolving-kernel study needs a clearer central message.

On your questions:

Sampling and UkU_k (Section 3.2). For i.i.d. uniform samples the sampled Fourier modes (scaled by 1/n1/\sqrt{n}) are only approximately orthonormal. The strength and alignment measurements require exact orthonormality, so on S1S^1 we use nn equally spaced points, where this holds exactly. We have clarified the definition of UkU_k accordingly.

Lines 249–250. Thank you. We have corrected this.

Dependence on the target. The theoretical results do not assume a Fourier-mixture target: the kernel and concentration results do not involve the target. The empirical evolving-kernel results are limited to finite Fourier mixtures with a relatively small kk. Within that range, the low-frequency strengthening persists under different amplitudes, and when k=0,1,2k = 0, 1, 2 are absent from the target. We have not tested targets with substantial high-frequency content (k≥15k \geq 15).

For now, we have decided to withdraw the submission and will take your feedback into account in order to improve the manuscript.

Reviewer_y1TwOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper explores the spectral bias of a two-layer (one-hidden-layer) network in finite samples and finite width. This is well understood for the uniform distribution and infinite width, where the NTK captures the residual dynamics, the eigenfunctions form a Fourier series, and the eigenvalues determine the convergence rate. While this was also demonstrated empirically for finite settings, this paper takes a further step by providing theoretical bounds in the lazy training regime. Furthermore, the paper provides empirical evidence about the evolving weights regime of better convergence at small widths, higher spectral strength of the lowest frequencies even when the target lacks them, and lower eigenspace alignment at higher frequencies.

Critical strengths

The setup is clear: finite data, finite size, and an evolving kernel is an interesting setup, and the paper provides new findings of higher spectral strength of the lowest frequencies even when the target lacks them. This is consequential to my overall assessment because it demonstrates its relevance to understanding spectral bias beyond the infinite-width limit.

Critical weaknesses
  • The theoretical contribution is small. The Fourier eigenvalues are known for a very similar setup (Basri et al., 2019, 2020. Note that Ronen et al. (2019) is a wrong citation, as Ronen is the author’s first name), and the trainable output weight is a minor and incremental contribution. The concentration bounds are loose.

  • The most interesting results about the spectral strength growing at k=0,1 are only empirical and unexplained, and the first-order theory provides an opposite-sign prediction.

Both are consequential to my overall assessment because they reduce the theoretical impact of the paper.

Questions for authors

Please address the weaknesses.

1 comment

Thank you for the review and for highlighting the paper's most relevant finding. We agree that the theoretical contribution is incremental given Basri et al., and that the strengthening still needs an explanation. On the first-order result: we intended to rule out the possibility that the initial velocity could account for the observed growth in the spectral strength. We think the effect is due to the kernel's accumulated higher-order motion. However, you are right that this leaves the main question open. Thank you also for pointing out the citation error; we have corrected it. For now, we have decided to withdraw the submission and will take your comments into account.

Reviewer_Ch65OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors prove some tail probability results regarding the spectrum of the NTK of a shallow neural network, for finite sample and finite width.

They also experimentally analyse how the spectrum changes during training, and find that the low-frequency terms, especially the first two constant and linear terms, show increasing strength, while higher-frequency terms show no change or a decrement, regardless of the target function.

The authors further find that this is not from the alignment of geometry; rather, it is due to the increment of eigenvalues.

Critical strengths

The result found in the empirical study, that only the spectral bias related to constant or linear terms was strengthened during training, is an interesting observation, and the authors’ claim that this is due to, possibly, the architecture already having first and second frequencies. It would be better if the authors could prove this concretely.

Critical weaknesses

The main weakness of the paper is its limited analysis. The paper only considers a two-layer shallow ReLU-NN on two-dimensional input, which is very narrow considering the general applicability of the NTK theory. Specifically, the spectral bias of NTK has been studied for general cases:

  • For deep ReLU-NN on n-dimensional input, eigenfunctions are spherical harmonics and eigenvalues follow d^{-k} decay rates. (https://arxiv.org/abs/2009.14397)
  • For deep NNs with arbitrary activation, eigenfunctions are spherical harmonics and eigenvalue decay rate is characterised by Hermite expansion of activations. (https://arxiv.org/abs/2211.07844)

One of the authors’ result is potentially misleading due to simplification, the unbiasedness property (14) would not be true when on begin considering deep networks, since neurons would be interacting.

Also, the objects that the authors study (G, H, …) are not justified enough. While it is true that they are related to the alignment of the spectrum, the authors do not provide an understanding of why these objects characterise the deviation. It would be helpful if the authors explained how these differences are used, for instance, regarding the spectral bias affecting the speed of learning each feature.

Questions for authors
  • Can you extend your result to at least the d-dimensional inputs? I believe extension to DNNs or general activations would be nearly impossible within the rebuttal period, but this might be a possible result that can increase its applicability.
  • Would you please give an intuitive description of the objects you study, and describe why they are meaningful, not just a proxy measure for deviation?
1 comment

Thank you for the careful review and the pointers to the literature on deeper networks and general activations. We agree that restricting the analysis to a shallow network on two-dimensional inputs limits the paper's scope. We also agree that the unbiasedness property does not carry over to deep networks, where neurons interact. Your question about the meaning of the objects we study, and how they relate to the speed at which each mode is learned, is well taken. We have decided to withdraw the submission and will take these points into account.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.