- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies the generalization error of kernel regression for anisotropic Gaussian inputs and polynomial kernels. They derive asymptotic bounds for the bias and variance, as well as a tighter characterization of the kernel eigenvalues than in past works. The results highlight how anisotropy shifts the dependence of the generalization error on the ratio between the samples and the dimension.
Critical strengths
- Overall the paper is fairly well written, and the text is easy to follow (albeit the technical details are somewhat heavy to parse).
- I think the underlying question is reasonably well motivated within the KRR literature.
- The figures are great and really help to digest the results.
Critical weaknesses
- My main issue with this paper is that it is unclear whether it contributes novel techniques or insights. The results and techniques seem to be entirely based on [1, 2, 3]. There are no proof sketches or discussions of theoretical insights that led to these results. It is unclear to me what this paper contributes beyond a standalone result.
- Several of the theorems are very difficult to interpret, with bounds depending implicitly on somewhat complicated integrals. I think that more concrete and explicit corollaries in canonical cases would improve the readability.
- The single index chapter currently reads somewhat weakly. If you can state a definite and interesting result for single-index models, then in my opinion, just state it.
[1] - Kernel ridge regression under power-law data: spectrum and generalization, Wortsman and Loureiro, 2025 [2] - Spectrum of inner-product kernel matrices in the polynomial regime and multiple descent phenomenon in kernel ridge regression, Misiakiewicz, 2022 [3] - A non-asymptotic theory of kernel ridge regression: deterministic equivalents, test error, and gcv estimator, Misiakiewicz and Saeed, 2024
Questions for authors
Can you please address my first weakness in particular, raised above (about the scope of the technical contributions)?
2 comments
Response to Authors
I thank the authors for their response. I will raise my score in good faith that the authors will use the extra page in the camera-ready version to add a proof overview and explicit corollaries for canonical cases.
Response to Reviewer mpNy
We thank the reviewer for the helpful feedback which will help in improving the manuscript. Below we present our answer to each concern.
Our work builds on the technical results of [1–3]. However, these works do not address our central question -- "Can we sharply characterize how data anisotropy and target alignment jointly shape KRR learning curves?" -- or the resulting phenomenology. This question is part of a broader effort to understand neural network performance on real data with this covariance structure, starting with the simpler lazy regime (see [4,5] and Levi and Oz (2025), suggested by Reviewer CYcu).
Establishing these results requires three steps beyond the tools developed in [1–3]:
Asymptotically exact spectrum. Proposition 1 sharpens the two-sided spectral bounds of [1] to an asymptotic equality under weak anisotropy. The proof controls the transformation between monomial and Hermite features, showing that it approaches the identity in operator norm. This refinement enables exact limiting learning curves rather than bounds up to unspecified constants.
Validity of deterministic equivalents. Appendix B verifies the spectral and concentration conditions needed to apply [3] to anisotropic Gaussian polynomial features, without an explicit Mercer eigenbasis. The spectral concentration part for this type of spectrum is not part of the literature, and the conditions on the eigenfunctions that are needed to apply the results from [3] are not directly available as we don't have an explicit Mercer Basis. The latter is a difficult problem, which we solve by proving that Proposition 1 allows us to transfer the conditions from a simpler basis (Hermite Polynomials) to the (unkown) Kernel Basis.
Evaluation of the learning curves. The deterministic equivalents still involve an implicit self-consistency equation and sums over a multi-indexed spectrum. We identify the scaling of the effective regularization and use the Irwin–Hall representation in Lemma D.1 to reduce the resulting multidimensional integrals to one-dimensional expressions. These steps yield Theorems 1–4.
The resulting mechanism is explicit: anisotropy spreads the eigenvalues within each polynomial degree. An aligned target can therefore be learned through the leading eigenmodes before the bulk of that degree contributes to the variance peak. For weak anisotropy, this produces bias transitions at n≍dm(1−α), while variance peaks remain at integer powers of d.
We will add a proof overview to the main text that explains these steps and distinguishes the tools inherited from [1–3] from our contributions.
We will supplement the integral formulas with explicit corollaries for canonical cases and state their implications before the technical expressions. For example, when α=1/4 and the target decay satisfies ω>1/2, the degree-one target component becomes learnable at n≍d3/4, before the first variance peak at n≍d. More generally, degree-m components become learnable at n≍d3m/4. These statements make the sample-complexity implications directly accessible without evaluating the integrals.
We will state Eq. (19) and its learning consequences as an explicit proposition conditional on the Hermite ansatz. The main implication is that, for an index with vj∝j−γ/2, the effective target decay is ω=(α+γ)/2. Under weak anisotropy, the fractional learning thresholds therefore occur when α+γ>1. This gives a direct geometric criterion for when a single-index target is learned between the variance peaks.
References:
[1] Wortsman and Loureiro. "Kernel ridge regression under power-law data: spectrum and generalization." 2025.
[2] Misiakiewicz. "Spectrum of inner-product kernel matrices in the polynomial regime and multiple descent phenomenon in kernel ridge regression." 2022.
[3] Misiakiewicz and Saeed. "A non-asymptotic theory of kernel ridge regression: deterministic equivalents, test error, and gcv estimator." 2024.