acceptodds
Under review as a conference paper at ICLR 2027

Learning between the peaks: sharp asymptotics for kernel ridge regression under power-law anisotropy

Abstract

We study Kernel Ridge Regression for polynomial inner-product kernels under anisotropic Gaussian data, where the eigenvalues of the input covariance follow a power-law decay with exponent . In the polynomial high-dimensional regime , we derive asymptotically sharp expressions for the kernel spectrum and the generalization error, revealing how anisotropy reshapes the learning curves. For weak anisotropy (), the problem remains effectively high-dimensional and retains some features of the isotropic case, while departing from it in others: the variance still peaks at integer sample complexities , though these peaks are progressively flattened as . Meanwhile, for targets strongly aligned with the data's principal directions, the bias drops at fractional sample complexities . This early transition effectively decouples the bias from the interpolation peaks. For strong anisotropy (), the effective dimension is constant. The variance stops depending on sample size, plateauing under ridgeless interpolation or vanishing at an explicit rate under fixed ridge penalty. The bias undergoes a sharp transition governed by the target's decay rate: below a threshold, learning is abrupt rather than gradual; above it, the bias decays as a power law, recovering classical source and capacity rates. We finally specialize these results to single-index targets, showing how the alignment of the index with the data's principal directions determines the effect of anisotropy on learning. Together, our results clarify how the input geometry shapes the kernel features and fundamentally impacts its generalization properties.

Primary area:
learning theory
Keywords:
Kernel Ridge Regression, Power-law Anisotropy, Kernel Spectrum, High-Dimensional Asymptotics

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_mpNyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies the generalization error of kernel regression for anisotropic Gaussian inputs and polynomial kernels. They derive asymptotic bounds for the bias and variance, as well as a tighter characterization of the kernel eigenvalues than in past works. The results highlight how anisotropy shifts the dependence of the generalization error on the ratio between the samples and the dimension.

Critical strengths
  • Overall the paper is fairly well written, and the text is easy to follow (albeit the technical details are somewhat heavy to parse).
  • I think the underlying question is reasonably well motivated within the KRR literature.
  • The figures are great and really help to digest the results.
Critical weaknesses
  • My main issue with this paper is that it is unclear whether it contributes novel techniques or insights. The results and techniques seem to be entirely based on [1, 2, 3]. There are no proof sketches or discussions of theoretical insights that led to these results. It is unclear to me what this paper contributes beyond a standalone result.
  • Several of the theorems are very difficult to interpret, with bounds depending implicitly on somewhat complicated integrals. I think that more concrete and explicit corollaries in canonical cases would improve the readability.
  • The single index chapter currently reads somewhat weakly. If you can state a definite and interesting result for single-index models, then in my opinion, just state it.

[1] - Kernel ridge regression under power-law data: spectrum and generalization, Wortsman and Loureiro, 2025 [2] - Spectrum of inner-product kernel matrices in the polynomial regime and multiple descent phenomenon in kernel ridge regression, Misiakiewicz, 2022 [3] - A non-asymptotic theory of kernel ridge regression: deterministic equivalents, test error, and gcv estimator, Misiakiewicz and Saeed, 2024

Questions for authors

Can you please address my first weakness in particular, raised above (about the scope of the technical contributions)?

2 comments
AuthorsOpenReview ↗

Response to Reviewer mpNy

We thank the reviewer for the helpful feedback which will help in improving the manuscript. Below we present our answer to each concern.

  1. Technical contributions

Our work builds on the technical results of [1–3]. However, these works do not address our central question -- "Can we sharply characterize how data anisotropy and target alignment jointly shape KRR learning curves?" -- or the resulting phenomenology. This question is part of a broader effort to understand neural network performance on real data with this covariance structure, starting with the simpler lazy regime (see [4,5] and Levi and Oz (2025), suggested by Reviewer CYcu).

Establishing these results requires three steps beyond the tools developed in [1–3]:

  • Asymptotically exact spectrum. Proposition 1 sharpens the two-sided spectral bounds of [1] to an asymptotic equality under weak anisotropy. The proof controls the transformation between monomial and Hermite features, showing that it approaches the identity in operator norm. This refinement enables exact limiting learning curves rather than bounds up to unspecified constants.

  • Validity of deterministic equivalents. Appendix B verifies the spectral and concentration conditions needed to apply [3] to anisotropic Gaussian polynomial features, without an explicit Mercer eigenbasis. The spectral concentration part for this type of spectrum is not part of the literature, and the conditions on the eigenfunctions that are needed to apply the results from [3] are not directly available as we don't have an explicit Mercer Basis. The latter is a difficult problem, which we solve by proving that Proposition 1 allows us to transfer the conditions from a simpler basis (Hermite Polynomials) to the (unkown) Kernel Basis.

  • Evaluation of the learning curves. The deterministic equivalents still involve an implicit self-consistency equation and sums over a multi-indexed spectrum. We identify the scaling of the effective regularization and use the Irwin–Hall representation in Lemma D.1 to reduce the resulting multidimensional integrals to one-dimensional expressions. These steps yield Theorems 1–4.

The resulting mechanism is explicit: anisotropy spreads the eigenvalues within each polynomial degree. An aligned target can therefore be learned through the leading eigenmodes before the bulk of that degree contributes to the variance peak. For weak anisotropy, this produces bias transitions at n≍dm(1−α)n \asymp d^{m(1-\alpha)}, while variance peaks remain at integer powers of dd.

We will add a proof overview to the main text that explains these steps and distinguishes the tools inherited from [1–3] from our contributions.

  1. Interpretability of the results

We will supplement the integral formulas with explicit corollaries for canonical cases and state their implications before the technical expressions. For example, when α=1/4\alpha=1/4 and the target decay satisfies ω>1/2\omega>1/2, the degree-one target component becomes learnable at n≍d3/4n \asymp d^{3/4}, before the first variance peak at n≍dn \asymp d. More generally, degree-mm components become learnable at n≍d3m/4n \asymp d^{3m/4}. These statements make the sample-complexity implications directly accessible without evaluating the integrals.

  1. Single-index models

We will state Eq. (19) and its learning consequences as an explicit proposition conditional on the Hermite ansatz. The main implication is that, for an index with vj∝j−γ/2v_j \propto j^{-\gamma/2}, the effective target decay is ω=(α+γ)/2\omega=(\alpha+\gamma)/2. Under weak anisotropy, the fractional learning thresholds therefore occur when α+γ>1\alpha+\gamma>1. This gives a direct geometric criterion for when a single-index target is learned between the variance peaks.

References:

[1] Wortsman and Loureiro. "Kernel ridge regression under power-law data: spectrum and generalization." 2025.

[2] Misiakiewicz. "Spectrum of inner-product kernel matrices in the polynomial regime and multiple descent phenomenon in kernel ridge regression." 2022.

[3] Misiakiewicz and Saeed. "A non-asymptotic theory of kernel ridge regression: deterministic equivalents, test error, and gcv estimator." 2024.

Reviewer_mpNyOpenReview ↗

Response to Authors

I thank the authors for their response. I will raise my score in good faith that the authors will use the extra page in the camera-ready version to add a proof overview and explicit corollaries for canonical cases.

Reviewer_CYcuOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper provides a theoretical expression for the excess risk of a kernel ridge regression learner on anisotropic Gaussian data in terms of the hyperparameters of a kernel alongside the dataset’s statistics (namely the dataset dimension, sample size, powerlaw exponents). The excess risk is shown to be composed of two understandable parts, a bias and variance term, which are studied independently.

These variance and bias equations are derived for both weak and strong anisotropy of the data, yielding different quantitative behavior. In the weakly anisotropic regime, the variance consistently peaks when the number of samples nn are approximately integer powers of the dataset dimension dd. The bias may drop at or before these peaks, depending on the powerlaw exponent for the target function when indexed by the kernel’s eigensystem. In the strong anisotropic regime, both the bias and variance become smoother functions, decaying gradually as the sample size is increased. Finally, the single-index model is studied and characterized.

Critical strengths

Using only the kernel’s fixed hyperparameters and the dataset statistics itself is the absolute highlight of this paper. Previous literature on kernel regression has slightly overindexed on the kernel’s eigensystem for explaining the excess risk, and studying how properties of the dataset itself clarifies critical learning details that were obfuscated by the eigensystem.

Critical weaknesses

Although this paper is highly theoretical in nature, having no connection to anything experimental made it difficult to understand the extent to which these results can be applied. For instance, Levi and Oz find that most natural datasets have a scaling exponent of above 11, making the study of the weakly anisotropic regime not substantially motivated. I would consider raising my score if (a) the authors show agreement between experiment and theory, and (b) if the authors would discuss under what natural circumstances we would find a scaling exponent below 11 in.

Additionally, I found the paper lacking clarity at times, both in terms of notation and scope. More concretely, Section 3 decomposes the risk in terms of complicated bias and variance expressions, yet the relation to scaling laws is only discussed at the end of each subsection. Having a more comprehensive plan of attack earlier in the paper would aid in guiding the reader. Similarly, the mathematical notation is at times convoluted: λ\lambda is both the ridge parameter and the kernel eigenvalues, β\beta is a multi-index that’s not standard in the literature and that later gets decomposed, ν⋆\nu_\star nor ζ⋆\zeta_\star are intuitive as to their meaning. Finally, α=1/2\alpha = 1/2 and ω=1/2\omega = 1/2 appear to be some type of critical points, although they weren’t discussed in much detail compared to α=1\alpha=1.

Levi, Noam, and Oz, Yarom. “The Underlying Scaling Laws and Universal Statistical Structure of Complex Datasets.” International Conference on Machine Learning. 2025.

Questions for authors

See the section above. Additionally:

  1. Is there any work (theoretical or empirical) detailing the scale of the Lipschitzs constants appearing in 198-199, Eq. 17? Are these closer to order 11 or order 10610^6?
  2. Are there any conditions required on the kernel’s series coefficients hmh_m? In a related work of Karkada et al., they require a “coefficient decay” wherein the ℓ\ell-th coefficient is significantly larger than that of the ℓ+1\ell+1-th for their analysis to hold, and I’m wondering if a similar requirement is needed here.

Karkada, Dhruva, et al. “Predicting Kernel Regression Learning Curves from only Raw Data Statistics.” International Conference on Learning Representations. 2026.

1 comment
AuthorsOpenReview ↗

Response to Reviewer CYcu

We appreciate the reviewer's feedback, which will help us better motivate our main ideas. Below, we address the main points raised.

  1. Weakly anisotropic regime: We thank the reviewer for bringing Levi and Oz (2025) to our attention. Their work provides further motivation for our setting, which we added in the updated manuscript (Appendix D, Empirical and Theoretical Motivation for the Anisotropic Regimes).

    Our motivation for studying α<1\alpha < 1 is twofold. First, a large literature studies the generalization properties of KRR and related models, such as random features, under isotropic Gaussian data (see [1-7]). These results have played a central role in our mathematical understanding of neural networks, including the benefits of feature learning over lazy learning [1,5-7], double descent, and benign overfitting [1-4]. We view our work as a fundamental step toward incorporating data geometry into this discussion. From this perspective, it is natural to study how anisotropy continuously reshapes generalization as we move away from α=0\alpha=0. Second, α<1\alpha < 1 can arise in embeddings and feature maps of data. For instance, in the framework of Levi and Oz, covariance eigenvalues scaling as λk∼k−α\lambda_k \sim k^{-\alpha} with α<1\alpha < 1 emerge when correlations between features ii and jj decay as a power law with distance, Cij∼∣i−j∣α−1C_{ij} \sim |i-j|^{\alpha-1} (their Eqs. 3–4), as well as when raw datasets with α>1\alpha > 1 are corrupted by noise. Studies of deep representation geometry (e.g., Agrawal et al., NeurIPS 2022) also show that DNNs learn representations whose feature covariance spectra follow a power law with exponents below one, corresponding to dense encoding. To illustrate this, we extracted GPT-2 token embeddings from several text datasets and computed their empirical feature covariance matrices. We confirmed that their eigenspectra follow a power-law decay with α<1\alpha < 1; see Fig. 5 in Appendix D of the updated manuscript.

  2. Notation and clarity: We appreciate the suggestions regarding exposition and notation. In the final manuscript, we will add an upfront roadmap to guide the reader toward the main findings. We will also improve the mathematical notation (e.g changing the kernel's eigenvalues symbol for other greek letter) following your feedback and clarify the intuition behind the critical points, as we did for α=1\alpha = 1 (for example noting that the limits ω→1/2+\omega \to 1/2^+ and ω→1/2−\omega \to 1/2^- yield compatible results).

  3. On the Lipschitz constant: The scale of the constants is kernel dependent. However, for a normalized kernel (that is, a kernel whose trace is Θ(1)\Theta(1)), these constants should also remain bounded. In our particular case, the constants we find grow with the degree of the polynomial, so they can be very large (polynomial in the kernels degree). On the other hand, by Proposition 1 we intuitively expect this constants to be close to the case where the kernel is an Hermite polynomial, so they should be ≈1\approx 1.

  4. On the scaling of hmh_m: As long as the max polynomial degree of the kernel, DD, remains finite (even if arbitrarily large), we don't need to impose any specific assumption on the hmh_m. However, care must be taken when considering the limit D→∞D \to \infty. In that case, we require hm∼1m!h_m \sim \frac{1}{m!} to ensure that expressions such as the effective regularization in Lemma 3.1, λeff=λ+∑m>κhm\lambda_{\text{eff}} = \lambda + \sum_{m > \kappa} h_m, properly converge. We also note that given the nature of the problem, where hh represents a 1D1D polynomial, making assumptions on the scale of this coefficients depending on the dimension would not allow to isolate the effect of dimensionality in this setting as we did.

[1] B. Ghorbani, S. Mei, T. Misiakiewicz, and A. Montanari. When do neural networks outperform kernel methods? Advances in Neural Information Processing Systems, 2020.

[2] B. Ghorbani, S. Mei, T. Misiakiewicz, and A. Montanari. Linearized two-layers neural networks in high dimension. The Annals of Statistics, 2021

[3] S. Mei, T. Misiakiewicz, and A. Montanari. Generalization error of random feature and kernel methods: Hypercontractivity and kernel matrix concentration. Applied and Computational Harmonic Analysis, 2022.

[4] T. Misiakiewicz. Spectrum of inner-product kernel matrices in the polynomial regime and multiple descent phenomenon in kernel ridge regression. arXiv preprint

[5] A. Damian, J. Lee, and M. Soltanolkotabi. "Neural networks can learn representations with gradient descent." PMLR, 2022.

[6] J. Ba, M. A. Erdogdu, T. Suzuki, Z. Wang, D. Wu, and G. Yang. "High-dimensional asymptotics of feature learning: How one gradient step improves the representation." Advances in Neural Information Processing Systems 35 (2022):

[7] Y. Dandi, F. Krzakala, B. Loureiro, L. Pesce, and L. Stephan. "How two-layer neural networks learn, one (giant) step at a time." Journal of Machine Learning Research 2024

Reviewer_iZ3fOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work analyzes kernel ridge regression with polynomial kernels and Gaussian inputs whose covariance eigenvalues exhibit power-law decay. It characterizes the kernel spectrum and asymptotic bias and variance under weak and strong anisotropy. Of particular interest, sufficiently aligned targets can exhibit bias transitions at fractional sample-complexity exponents, separated from the variance peaks. A single-index case study relates the spectral analysis to target geometry.

Critical strengths

Well written. The joint treatment of covariance decay, target alignment, and regularization makes the results informative beyond an isolated asymptotic calculation.

Critical weaknesses

The conclusions rely on Gaussian inputs, polynomial kernels, and structured target spectra (See question below).

Questions for authors

Do you believe that any of the three above assumtpions are strictly necessary or can they be realxed (maybe more intuition for me atleast).

1 comment
AuthorsOpenReview ↗

Response to Reviewer iZ3f

We thank the reviewer for the opportunity to explain our setting and its motivation. Below we provide the corresponding answers.

  1. Gaussian covariates: Our work aims to understand how data structure, specifically the power-law covariance observed in real data, impacts the generalization properties of kernel ridge regression. Unlike the long list of works that assume power-laws on the kernel spectrum (a.k.a. source and capacity conditions [1]), we explicitly relate the geometry of the data to generalization.

    Despite the apparent simplicity of the setting, this is a mathematically challenging task, as relating the geometry of the inputs to the features require diagonalizing the kernel operator. With the exception of a few distributions (all of them isotropic), this is a mathematically intractable problem. To our knowledge, all of the results in the literature either consider these distributions [2-4] or rely on universality results to reduce the problem to them.

    Given this challenge, we believe that breaking isotropy in the Gaussian (or asymptotically equivalent distributions) case is a natural starting point.

  2. Polynomial kernels: While we state our main assumptions for polynomial kernels of degree DD, this is primarily a technical convenience. By taking DD arbitrarily large (D→∞D \to \infty in Assumption 1.1), our framework approximates any smooth inner-product kernel admitting a Taylor expansion k(x,x′)=h(⟨x,x′⟩)=∑m=0∞hm⟨x,x′⟩mk(\mathbf{x}, \mathbf{x}') = h(\langle \mathbf{x}, \mathbf{x}' \rangle) = \sum_{m=0}^\infty h_m \langle \mathbf{x}, \mathbf{x}' \rangle^m.

  3. Target spectral decay: The spectral and variance results do not require the target decay assumption in Eq. (14). This assumption is used to evaluate the bias explicitly and determine how target alignment changes the learning thresholds. It plays a role analogous to a source condition [1], while specifying alignment within each polynomial degree.

    The underlying mechanism is more general: target energy concentrated in leading kernel eigenmodes can be learned before the bulk modes generate a variance peak. Equation (14) provides a tractable family in which this separation can be quantified sharply.

References:

[1] A. Caponnetto and E. De Vito. Optimal rates for the regularized least-squares algorithm. Founda- tions of Computational Mathematics, 7(3):331–368, 2007

[2] Misiakiewicz (2022), Spectrum of inner-product kernel matrices in the polynomial regime and multiple descent phenomenon in kernel ridge regression. (Works on the hypersphere Sd−1(d)\mathbb{S}^{d-1}(\sqrt{d}) and hypercube Qd\mathcal{Q}^d, both asymptotically equivalent to our isotropic Gaussian setting as d→∞d\to\infty)

[3] Mei & Montanari (2022), The Generalization Error of Random Features Regression: Precise Asymptotics and the Double Descent Curve.

[4] Bordelon, Canatar, & Pehlevan (2020), Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.