acceptodds
Under review as a conference paper at ICLR 2027

Power-Law Data Spectra as Implicit Regularization in Learning

Abstract

Neural networks often exhibit substantially different learning performance on real-world and synthetic data, suggesting that structural properties of real data may play an important role in learning. Large-scale datasets used in modern neural-network applications, including images and language, commonly exhibit approximately power-law covariance spectra. Such spectral structure may therefore contribute to the learning properties observed on real data. However, how this ubiquitous power-law structure affects learning remains unclear. Here, we show that power-law spectral structure can act as an implicit regularizer of learning. To characterize this effect quantitatively, we analyze double-descent learning curves in ridge regression while systematically varying the power-law exponent. We establish an explicit relation between the power-law exponent and the ridge penalty that reproduces the leading behavior of the double-descent peak in the weak-ridge regime. This relation shows that increasing the power-law exponent suppresses noise amplification in a manner analogous to increasing explicit regularization. Furthermore, we find that the spectral structure affects the dependence of prediction error on sample size beyond the interpolation regime, reshaping the overall learning curve. Establishing the power-law structure of real-world data as an intrinsic source of implicit regularization that shapes learning performance, our findings advance the theoretical understanding of the role of data structure in learning.

Primary area:
learning theory
Keywords:
double descent, implicit regularization, power-law spectra, ridge regression

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DXXROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how power-law covariance spectra govern double descent in ridge regression, proposing that faster spectral decay produces an implicit regularization effect analogous to increasing explicit ridge penalties. Focus is placed on the weak-ridge regime near interpolation, where the noise-risk curve depends primarily on a single scaled spectral sum; this yields an explicit mapping between changes in spectral decay and an equivalent scalar ridge penalty. The authors decompose this behavior into a scale effect that dominates near interpolation and an irreducible shape effect away from the peak, while establishing an asymptotic threshold that dictates when the interior noise peak disappears entirely. Synthetic Gaussian experiments and covariance-rescaled MNIST representations confirm that the predicted equivalent ridge adjustments reproduce nearly all of the variance suppression observed near interpolation, though the theoretical framework characterizes label-noise risk rather than total generalization error or nonlinear architectures.

Critical strengths

Following are the strengths:

  • Theorem 1 establishes an explicit, quantitative mapping between arbitrary positive covariance spectra and scalar ridge penalties, moving beyond qualitative arguments about data anisotropy suppressing variance peaks to provide an interpretable, uniform leading-order approximation near the interpolation boundary.
  • The decomposition into scale and shape effects offers valuable theoretical insight, explaining precisely why isotropic ridge regularization closely matches spectral sharpening near the interpolation peak while fundamentally failing to capture the full trajectory away from it.
  • The empirical validation is carefully executed, pairing synthetic Gaussian baselines with non-Gaussian MNIST representations and utilizing paired sampling to demonstrate convincing 97–99% empirical agreement with the theoretically predicted ridge shifts near interpolation.
Critical weaknesses

Following are the weaknesses:

  • The core theoretical guarantees are derived for deterministic equivalents and continuum limits rather than exact finite-dimensional risk, leaving a theoretical gap in the critical weak-ridge regime where small eigenvalues and vanishing ridge penalties challenge standard deterministic equivalent convergence guarantees.
  • The primary claims characterize only label-noise amplification rather than total generalization error, which obscures the bias-variance trade-off; as shown in Appendix E, spectral sharpening can inflate bias and increase net prediction error under uniform signal distributions even as noise risk monotonically declines.
  • The novelty relative to established spectral learning theory and benign overfitting literature remains somewhat incremental, particularly regarding closely related analyses like Rizzi et al. (2026), making it unclear whether the results represent fundamentally new phenomena or reparameterized asymptotic variants of existing spectral risk formulas.
  • The experimental scope is narrow, relying exclusively on synthetic Gaussian inputs and artificially rescaled MNIST representations at d=300d=300 rather than evaluating naturally emergent power-law spectra across diverse modern neural architectures, tasks, or modalities.
Questions for authors

Following are the questions for the authors:

  • Can the framework be extended to incorporate target-alignment spectra and bound the accompanying bias term, yielding a closed-form condition for when increasing α\alpha strictly improves total generalization error?
  • How robust is the scalar equivalence to spectral non-idealities, such as truncated tails or isolated outlier eigenvalues, and can higher-order spectral moments bound the resulting approximation error?
  • Does the predicted equivalence hold for representation spaces extracted across intermediate layers of real pre-trained networks, where spectral shifts are intrinsically coupled to feature rotation and higher-order dependencies?
  • What concrete error bounds or convergence guarantees justify the deterministic equivalent in the singular regime where λ→0\lambda \to 0, n≈dn \approx d, and μmin⁡\mu_{\min} approaches zero?
  • Does the predicted equivalence hold for representation spaces extracted across intermediate layers of real pre-trained networks, where spectral shifts are intrinsically coupled to feature rotation and higher-order dependencies?
5 comments
AuthorsOpenReview ↗

Response to Reviewer DXXR (1/2)

We thank the reviewer for the careful reading and for the constructive comments. We respond to each point below. Added or changed text in the revised PDF is shown in red.

W1. The core theoretical guarantees are derived for deterministic equivalents and continuum limits rather than exact finite-dimensional risk, leaving a theoretical gap in the critical weak-ridge regime where small eigenvalues and vanishing ridge penalties challenge standard deterministic equivalent convergence guarantees.

We agree that this gap remains, and the Limitations paragraph states it. No available bound controls the error of the deterministic equivalent near interpolation at weak ridge, so we measured this error instead (Appendix D.4). For Gaussian inputs at n=dn = d, it is at most 0.5% at the ridge penalty of our real-data experiments (λS1=90\lambda S_1 = 90) and grows to 11 to 12% at λS1=0.3\lambda S_1 = 0.3. The comparison at the center of the paper is much less affected. The ratio between the exponent change and the ridge change agrees with the deterministic equivalent within 0.1%, also where the deterministic equivalent is off by more than 10%. Regarding the continuum limit, Theorem 2 is a statement about this limit, and Section 5 explains how the criterion applies at finite dimension and tests it on MNIST inputs with d=300d = 300 (Figure 2).

W2. The primary claims characterize only label-noise amplification rather than total generalization error, which obscures the bias-variance trade-off; as shown in Appendix E, spectral sharpening can inflate bias and increase net prediction error under uniform signal distributions even as noise risk monotonically declines.

Thank you for this observation. You are right that the submitted version characterized only the label-noise part, and that Appendix E already showed how the bias can grow. We turned that example into a result. Theorem 1 now gives the prediction error near the peak as the sum of the noise risk and the bias term. At n=dn = d these are (d/2) σ2/λS1(d/2)\,\sigma^2/\sqrt{\lambda S_1} and (d/2) ωλS1(d/2)\,\omega\sqrt{\lambda S_1}, where ω\omega is a coefficient determined by the teacher, so at fixed ω\omega a larger λS1\lambda S_1 lowers the first and raises the second. Proposition E.1 gives the condition under which their sum decreases with the exponent, and Proposition C.3 extends Theorem 2 to the prediction error.

W3. The novelty relative to established spectral learning theory and benign overfitting literature remains somewhat incremental, particularly regarding closely related analyses like Rizzi et al. (2026), making it unclear whether the results represent fundamentally new phenomena or reparameterized asymptotic variants of existing spectral risk formulas.

We agree that the relation to prior work needed to be stated more precisely, and the revision does so. For 0≤α<10 \le \alpha < 1 our continuum equation is indeed a reparameterization of the state equations of Rizzi et al. (2026), as Section 2 says. Two results, however, are not reparameterizations. For α>1\alpha > 1, Theorem 2 gives the threshold (α−1)r=1(\alpha-1)r = 1 at which the peak disappears. Rizzi et al. treat the ridgeless case and a ridge penalty of order one, which are the two ends r=0r = 0 and r→∞r \to \infty, and the threshold does not follow from either. Theorem 1 is an expansion at finite dimension for an arbitrary spectrum, which gives the ridge penalty equivalent to a change of the exponent. Relative to the benign-overfitting literature, which bounds the risk of a single problem through effective ranks, our results convert a change of the spectrum into a change of the ridge penalty at the interpolation peak. Section 2 now states this difference.

W4. The experimental scope is narrow, relying exclusively on synthetic Gaussian inputs and artificially rescaled MNIST representations at d=300d = 300 rather than evaluating naturally emergent power-law spectra across diverse modern neural architectures, tasks, or modalities.

We agree that the scope of the submitted experiments was narrow. The revision adds the actual labels of MNIST and CIFAR-10 (Appendix D.3) and 14 representations of a ResNet-18 and a ResNet-50, whose spectra arise from the networks and are not rescaled (Appendix H, please see our reply to Q3). Other architectures, tasks and modalities are still not covered, as stated in the Limitations paragraph

AuthorsOpenReview ↗

Response to Reviewer DXXR (2/2)

Q1. Can the framework be extended to incorporate target-alignment spectra and bound the accompanying bias term, yielding a closed-form condition for when increasing α\alpha strictly improves total generalization error?

Thank you for this question. Yes, this is possible at interpolation and at weak ridge, and the revision now includes such a condition. Theorem 1 now holds for arbitrary signal powers qkq_k, and it gives the bias term at n=dn = d as (d/2) ωλS1(d/2)\,\omega\sqrt{\lambda S_1} up to a relative error O(κ0/μmin⁡)O(\kappa_0/\mu_{\min}), for any alignment of the target. The resulting condition is ρ (4b−3a)<a\rho\,(4b-3a) < a (Proposition E.1), where ρ\rho is the ratio of the bias term to the noise risk and aa and bb are means of log⁡k\log k with weights 1/μk1/\mu_k and qk/μk2q_k/\mu_k^2. For a power-law alignment qk∝k−βq_k \propto k^{-\beta} with β<2α+1\beta < 2\alpha+1, the threshold on ρ\rho is 1+4(β−α)(α+1)(2α−β+1)log⁡d1 + \frac{4(\beta-\alpha)}{(\alpha+1)(2\alpha-\beta+1)\log d} for large dd, up to smaller terms (Corollary E.2). We should note that the condition is stated for the leading value of the prediction error and is local in the exponent.

Q2. How robust is the scalar equivalence to spectral non-idealities, such as truncated tails or isolated outlier eigenvalues, and can higher-order spectral moments bound the resulting approximation error?

We examined both kinds of non-idealities in Appendix I. The error of Theorem 1 is controlled by κ0/μmin⁡\kappa_0/\mu_{\min}, so the harmful non-ideality is a very small eigenvalue, whether isolated or in a tail of very small positive eigenvalues. When the smallest eigenvalue of a power law is reduced by a factor of 1000, the leading curve is off by more than a factor of two, yet the exponent change and the ridge change still differ by only 8.3%, because the outlier dominates S1S_1 of both spectra. An exactly truncated tail is covered by the theorem with the rank in place of dd.

On your second question, the answer is yes at the interpolation point. Proposition B.2 bounds the error of the leading value at n=dn = d by 4e24e_2 whenever e2≤1/4e_2 \le 1/4, with e2=λS1 S2/S12e_2 = \sqrt{\lambda S_1}\,S_2/S_1^2, where S2S_2 is the sum of the inverse squared eigenvalues. This bound does not involve μmin⁡\mu_{\min}.

Q3. Does the predicted equivalence hold for representation spaces extracted across intermediate layers of real pre-trained networks, where spectral shifts are intrinsically coupled to feature rotation and higher-order dependencies?

We agree that this is an important test, and we carried it out on 14 representations (Appendix H). Across layers, the noise peaks differ by a factor of five at a common ridge penalty and lie within 17% of the leading value of Theorem 1 at equal λS1\lambda S_1, although the layers differ in their eigenvectors and higher-order statistics. Within each layer, the exponent change and the equivalent ridge change give noise risks at n=dn = d that differ by about 1% (Δα=0.25\Delta\alpha = 0.25) and by 3 to 4% (Δα=0.5\Delta\alpha = 0.5). We should be clear about what these experiments do not show. They do not show that the change of a representation produced by training is equivalent to a ridge change, because the rotation and the higher-order dependencies then change together with the spectrum, as you point out. The Limitations paragraph states this.

Q4. What concrete error bounds or convergence guarantees justify the deterministic equivalent in the singular regime where λ→0\lambda \to 0, n≈dn \approx d, and μmin⁡\mu_{\min} approaches zero?

To our knowledge there is no such bound or guarantee, and we do not provide one. We would like to clarify, however, which regime our results use. Theorem 1 does not rely on the singular limit. It is an expansion in the ratio κ0/μmin⁡\kappa_0/\mu_{\min}, where κ0=λ/S1\kappa_0 = \sqrt{\lambda/S_1}, and for a power law this ratio is about (α+1)λS1/d(\alpha+1)\sqrt{\lambda S_1}/d. It applies at a fixed positive ridge penalty, where the peak has a finite height. At λS1=90\lambda S_1 = 90, the value of our real-data experiments, this ratio is 0.09 for μk=k−2\mu_k = k^{-2} with d=300d = 300, and the measured error of the deterministic equivalent is small, as reported in our reply to W1. In the continuum limit of Theorem 2 with α>1\alpha > 1, λ\lambda and μmin⁡\mu_{\min} tend to zero, but λS1\lambda S_1 grows as d2d^2. The limit λ→0\lambda \to 0 at a fixed spectrum is the case in which the deterministic equivalent is least reliable (Appendix D.4), and the Limitations paragraph states that our theorems concern the deterministic equivalent.

Q5. Does the predicted equivalence hold for representation spaces extracted across intermediate layers of real pre-trained networks, where spectral shifts are intrinsically coupled to feature rotation and higher-order dependencies?

This question repeats Q3. Please see our reply to Q3 above.

AuthorsOpenReview ↗

Follow-up on our response

Thank you again for the time and effort you put into reviewing our paper. Your comments helped us improve it considerably.

As the discussion period is coming to a close, we would like to ask whether our response and the revised paper address your concerns. If any point remains unclear, or if there is anything further we can do, please let us know. We would be glad to provide additional clarification.

Reviewer_rXz2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how power law data spectra affects the double-descent in ridge regression.

There are two main contributions. First, near the interpolation threshold with small ridge penalty , the noise risk depends on a composed factor λS1\lambda S_1, and thus changing the exponent α\alpha can be approximated by the change in λ\lambda. The authors also discuss the scale effect and shape effect.

Second, the authors establish a criterion for the existence of an interior maximum: under the scaling r=λdα−1r=\lambda d^{\alpha-1}, the criterion for α>1\alpha>1 is (α−1)r<1(\alpha-1)r<1. The authors then conduct numerical experiments on finite-dimensional Gaussian experiments and experiments on spectrally modified MNIST representations to support their conclusions.

Critical strengths

The organization of this paper is clear, so the two main strengths are easy to be identified:

  1. The equivalence condition built in Theorem 1 is precise and interesting. This is a useful connection between (power law) data spectra and ridge regularization.

  2. The sharp boundary separating limiting noise curves with an interior maximum from those without one (Theorem 2) is particularly interpretable, and this precise boundary is novel to my best of knowledge.

Critical weaknesses

I think the main weaknesses fall into the novelty and significance, as shown below.

  1. For the novelty, the suppression of variance peaks by power law spectra has already been studied by Wortsman et al. (2025), and Rizzi et al. (2026), which also builds on established deterministic equivalents and classical connections between feature rescaling and ridge regularization. Thus, the contributions, while being useful quantitative refinements, are not a fundamentally new explanation of the phenomenon.

  2. For the significance, the theoretical results do not sufficiently establish the broader claim that power law spectra provide beneficial implicit regularization of learning. I think this point is particularly important as the paper's motivation comes from connecting power law data spectra to the learning properties of real-world data. The experiments do not directly validate this broader connection.

Reference:

Wortsman et al. Kernel ridge regression under power-law data: spectrum and generalization. 2025.

Rizzi et al. Learning between the peaks: Sharp asymptotics for kernel ridge regression under power-law anisotropy. 2026.

Questions for authors
  1. Could the authors more explicitly identify which results cannot be obtained from existing deterministic-equivalent analyses?

  2. Under what natural conditions does increasing the power law exponent lead to improved total prediction error? Can the authors demonstrate that these conditions hold in realistic tasks?

5 comments
AuthorsOpenReview ↗

Response to Reviewer rXz2 (1/2)

We thank the reviewer for the careful reading and for the constructive comments. We respond to each point below. Added or changed text in the revised PDF is shown in red.

W1. For the novelty, the suppression of variance peaks by power law spectra has already been studied by Wortsman et al. (2025), and Rizzi et al. (2026), which also builds on established deterministic equivalents and classical connections between feature rescaling and ridge regularization. Thus, the contributions, while being useful quantitative refinements, are not a fundamentally new explanation of the phenomenon.

Thank you for pointing us to these works. We agree that there is an overlap, and we do not claim the suppression of the peak itself as new. For 0≤α<10 \le \alpha < 1 our continuum equation coincides with the state equations of Rizzi et al. (2026), who show that the variance peak is damped as α\alpha grows, and Appendix F now compares the results in detail. We would like to point out three results that, to our knowledge, are not contained in these works.

  1. For α>1\alpha > 1, the peak exists if and only if (α−1)r<1(\alpha-1)r < 1 (Theorem 2). Rizzi et al. treat only the two ends r=0r = 0 and r→∞r \to \infty of our scaling, and the threshold lies between them. Exponents measured in data are mostly above one (1.48 for MNIST, 0.88 to 2.15 for the network representations).
  2. The classical identity turns a rescaling of the features into a ridge penalty with a different weight on each direction. Theorem 1 shows that near the peak this weighted penalty acts on the noise risk like a single ridge penalty, and it gives the value of that penalty.
  3. The effect of the exponent splits into a part that a rescaling of the ridge penalty reproduces and a part that no ridge penalty reproduces (Eq. (11)).

Regarding Wortsman & Loureiro (2025), as far as we can tell their result concerns 0≤α<10 \le \alpha < 1 at sample sizes away from the peaks, and we could not find a statement on the suppression of variance peaks there. We may have missed it, and we would be grateful if you could point us to the result you have in mind.

W2. For the significance, the theoretical results do not sufficiently establish the broader claim that power law spectra provide beneficial implicit regularization of learning. I think this point is particularly important as the paper's motivation comes from connecting power law data spectra to the learning properties of real-world data. The experiments do not directly validate this broader connection.

We agree with this assessment. The submitted version showed that the spectrum suppresses the label-noise part of the error, but not that this is beneficial, and the abstract claimed more than the results showed. The revision addresses the benefit directly, in theory and in experiments. On the theory side, Theorem 1 now covers the prediction error with a teacher, and Proposition E.1 gives the condition under which a larger exponent lowers it at interpolation (please see our reply to Q2). On the experimental side, with the actual labels of MNIST and CIFAR-10 the test error near interpolation falls as the exponent grows (for MNIST from 6.03 to 3.35 and 1.96 at n=dn = d), by nearly the same amount as under the equivalent ridge change (Appendix D.3). When the ridge penalty is tuned on a validation set, the gain almost disappears (below 0.55%). The spectrum is therefore beneficial in the way ridge regularization is, for a model that is under-regularized near interpolation. We revised the abstract accordingly. It now says that the spectrum can act as an implicit regularizer in ridge regression, and that the resulting suppression of noise amplification lowers the prediction error near the peak when label noise dominates the signal.

AuthorsOpenReview ↗

Response to Reviewer rXz2 (2/2)

Q1. Could the authors more explicitly identify which results cannot be obtained from existing deterministic-equivalent analyses?

Thank you for the opportunity to clarify this. All our results are derived from the known deterministic equivalent, and none requires a new one. What we did not find stated in existing analyses are the following consequences.

  • The closed-form curve of the prediction error on the whole peak region, with an error bound, and the ridge penalty λeq=λ S1(α+Δα)/S1(α)\lambda_{\rm eq} = \lambda\,S_1(\alpha+\Delta\alpha)/S_1(\alpha) that reproduces a change of the exponent (Theorem 1). Known before were the role of S1S_1 at n=dn = d (Dobriban & Wager, 2018; Kamisoyama et al., 2026) and the peak height for equal eigenvalues (Canatar et al., 2021).
  • The threshold (α−1)r<1(\alpha-1)r < 1 for the existence of the peak for α>1\alpha > 1 (Theorem 2).
  • For α>1\alpha > 1, no ridge change reproduces an exponent change over the whole noise curve in the continuum limit (Corollary C.1).
  • The condition under which a larger exponent lowers the prediction error at interpolation (Proposition E.1).

The paragraph after Theorem 1 now makes this separation explicit.

Q2. Under what natural conditions does increasing the power law exponent lead to improved total prediction error? Can the authors demonstrate that these conditions hold in realistic tasks?

Thank you for this question, which is closely related to your second concern above. The revision gives the condition in Proposition E.1. At interpolation and weak ridge, a larger exponent lowers the prediction error, to leading order, if and only if ρ (4b−3a)<a\rho\,(4b-3a) < a. Here ρ\rho is the ratio of the bias term to the noise risk, and aa and bb are means of log⁡k\log k with weights 1/μk1/\mu_k and qk/μk2q_k/\mu_k^2. When the signal is aligned with the spectrum, the condition is ρ<1\rho < 1, that is, the label noise contributes more to the error than the bias.

For the second part of your question, we evaluated the condition for MNIST and CIFAR-10 with actual labels. There ρ\rho is 1.0×10−41.0\times10^{-4} and 3.0×10−53.0\times10^{-5}, far below the threshold of about one, and the measured test error at n=dn = d decreases with the exponent as predicted (Appendix D.3). We note that the condition is specific to the neighborhood of the peak. At n=0.4dn = 0.4d the test error increases with the exponent.

AuthorsOpenReview ↗

Follow-up on our response

Thank you again for the time and effort you put into reviewing our paper. Your comments helped us improve it considerably.

As the discussion period is coming to a close, we would like to ask whether our response and the revised paper address your concerns. If any point remains unclear, or if there is anything further we can do, please let us know. We would be glad to provide additional clarification.

Reviewer_3fCDOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper studies how the covariance structure of input data affects double descent in ridge regression, where the test error rises to a peak as the number of samples approaches the number of parameters. This peak is caused by label noise, which the model is forced to fit almost exactly near this point. Real data, such as images and text, often have covariance eigenvalues that decay as a power law, and the authors ask how the decay exponent affects the height and shape of this peak. The analysis focuses on the part of the error caused by label noise and is carried out within a standard approximation of this error for a linear model with Gaussian inputs. The authors show that, near the peak and for a small ridge penalty, the effect of the spectrum on this error is equivalent to that of the ridge penalty, as the error depends on both only through a single combined quantity. As a result, making the spectrum decay faster has the same effect as increasing the ridge penalty by an explicit factor, which the authors interpret as implicit regularization. In the limit of large dimension, they further show that this equivalence breaks down away from the peak, and derive an explicit condition under which a fast enough decay removes the peak entirely. Empirically, the authors report that increasing the ridge penalty by the predicted factor closely reproduces the effect of a faster-decaying spectrum at the peak, both on Gaussian inputs and on MNIST data rescaled so that its spectrum decays faster.

Critical strengths
  • Understanding how the structure of input data affects generalization is an important problem in modern machine learning. Power-law spectra are widely observed in real data, which makes this setting a natural one to study.
  • The main results are explicit and relatively simple, and they make precise how a faster-decaying spectrum lowers the error caused by label noise near the peak. The derivations also appear correct.
Critical weaknesses

My main concern is the paper’s presentation, which in my opinion makes its contributions and their significance difficult to follow and evaluate.

  • The paper lacks a clear structure and a proper setup. The central question only becomes clear in Section 4, and there is no preliminaries section. The approximation on which all results rely is introduced briefly in Section 3, with little explanation, which makes the paper hard to follow for readers unfamiliar with this line of work.
  • The framing of the results goes beyond what is shown. The abstract presents the power-law spectrum as an implicit regularizer that shapes learning performance, but the results concern only the part of the error caused by label noise, and a lower error caused by label noise does not necessarily mean a lower test error. In addition, implicit regularization in modern machine learning usually refers to the bias induced by the optimizer, so the term may confuse readers here.

An additional concern is the limited scope of the experiments.

  • The experiments do not involve any prediction task on real labels. They verify the theory’s prediction for the error caused by label noise, but do not test whether a faster-decaying spectrum leads to better generalization. An experiment fitting the model to the real labels of the modified MNIST data, and measuring how the test error changes as the spectrum decays faster, would greatly strengthen the paper’s claims.
  • The data used in the experiments do not reflect the setting the paper motivates. The spectrum of the MNIST data is modified by hand rather than arising from learned representations, even though the introduction cites learned neural representations as motivation. Experiments with features from a trained network would help show whether the results apply to the spectra that arise in practice.
  • While restricting the analysis to linear models is standard in this literature, the experiments use only a single real-world dataset. Additional datasets would help show whether the results hold more broadly.
Questions for authors
  • The paper presents Theorem 1 as extending known results on the error caused by label noise near the peak to general spectra. Could the authors clarify why this extension is significant, and which of the results they consider the main new contribution?
  • What are the practical implications of the results? Can they inform the design of representations in practice?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 3fCD (1/2)

We thank the reviewer for the careful reading and for the constructive comments. We respond to each point below. Added or changed text in the revised PDF is shown in red.

W1. The paper lacks a clear structure and a proper setup. The central question only becomes clear in Section 4, and there is no preliminaries section. The approximation on which all results rely is introduced briefly in Section 3, with little explanation, which makes the paper hard to follow for readers unfamiliar with this line of work.

Thank you for pointing this out. We agree that the submitted version made the question and the setup hard to find, and we reorganized the presentation in three ways. First, the central question is now stated in the Introduction, before the results. Second, Section 3 is retitled "Problem setup and deterministic equivalent" and serves as the preliminaries. Third, Section 3 now explains the approximation on which the results rely. The deterministic equivalent replaces the average of the prediction error over the random inputs by an explicit function of the eigenvalues of the input covariance, and we state there that all our theorems are statements about it.

W2. The framing of the results goes beyond what is shown. The abstract presents the power-law spectrum as an implicit regularizer that shapes learning performance, but the results concern only the part of the error caused by label noise, and a lower error caused by label noise does not necessarily mean a lower test error. In addition, implicit regularization in modern machine learning usually refers to the bias induced by the optimizer, so the term may confuse readers here.

We agree with this point, and it led to the main change of the revision. As you note, a lower error caused by label noise does not by itself mean a lower test error. We therefore extended Theorem 1 to the prediction error with a teacher, and the result supports your point. On the noise part the spectrum acts like the ridge penalty, but on the bias part it has an additional effect that depends on the teacher. Proposition E.1 gives the condition under which the prediction error at interpolation decreases, which for a signal aligned with the spectrum is that the noise part exceeds the bias part. We also revised the abstract. It now says that the suppression of noise amplification lowers the prediction error near the peak when label noise dominates the signal, and it no longer says that the spectrum shapes learning performance.

Regarding the term implicit regularization, we agree that it is most often used for the bias induced by the optimizer. To avoid confusion, the Introduction now states explicitly that we use it for a regularizing effect of the data spectrum and not for optimization dynamics. This usage follows Kobak et al. (2020) and Jacot et al. (2020), where low-variance directions or a finite number of random features act as an implicit ridge penalty.

W3. The experiments do not involve any prediction task on real labels. They verify the theory’s prediction for the error caused by label noise, but do not test whether a faster-decaying spectrum leads to better generalization. An experiment fitting the model to the real labels of the modified MNIST data, and measuring how the test error changes as the spectrum decays faster, would greatly strengthen the paper’s claims.

Thank you for this suggestion. We carried out the experiment you describe on the modified MNIST data, and also on CIFAR-10 (Appendix D.3). Near interpolation, a faster-decaying spectrum does lower the test error. For MNIST at n=dn = d it falls from 6.03 to 3.35 and 1.96 for Δα=0.25\Delta\alpha = 0.25 and 0.50.5, and the ridge change predicted by Theorem 1 gives the same test error within 1 to 3%. We also report where the improvement ends. At n=0.4dn = 0.4d the test error rises, and when the ridge penalty is tuned on a validation set the gain almost disappears (below 0.55%). A faster decay thus improves generalization in the way additional ridge regularization does, that is, when the model is under-regularized near interpolation.

AuthorsOpenReview ↗

Response to Reviewer 3fCD (2/2)

W4. The data used in the experiments do not reflect the setting the paper motivates. The spectrum of the MNIST data is modified by hand rather than arising from learned representations, even though the introduction cites learned neural representations as motivation. Experiments with features from a trained network would help show whether the results apply to the spectra that arise in practice.

We agree that representations from a trained network are closer to the setting that motivates the paper, and we added them (Appendix H). We use 14 representations taken from a ResNet-18, at initialization and after training on CIFAR-10, and from a ResNet-50 trained on ImageNet. Their spectra are not modified. At the same ridge penalty their noise peaks differ by a factor of five, and once the ridge penalties are chosen with equal λS1\lambda S_1 the peaks lie within 17% of the leading value of Theorem 1. The reduction to λS1\lambda S_1 therefore also holds for the spectra that arise in these networks.

W5. While restricting the analysis to linear models is standard in this literature, the experiments use only a single real-world dataset. Additional datasets would help show whether the results hold more broadly.

In addition to MNIST, the revision includes CIFAR-10 with actual labels and the network representations mentioned above, and the results agree with those on MNIST. We acknowledge that the experiments remain limited to image data, and the Limitations paragraph states this.

Q1. The paper presents Theorem 1 as extending known results on the error caused by label noise near the peak to general spectra. Could the authors clarify why this extension is significant, and which of the results they consider the main new contribution?

Thank you for the question. It showed us that the submitted version did not separate clearly what was known from what is new. It was known that the sum S1S_1 of the inverse eigenvalues appears in ridge asymptotics at interpolation (Dobriban & Wager, 2018) and that the effective ridge at n=dn = d is λ/S1\sqrt{\lambda/S_1} to leading order (Kamisoyama et al., 2026). Theorem 1 adds the closed-form curve on the whole peak region for an arbitrary spectrum. Its significance is that it turns the analogy between the spectrum and the ridge penalty into a conversion rule. It gives the ridge penalty that reproduces a given change of the exponent, λeq=λ S1(α+Δα)/S1(α)\lambda_{\rm eq} = \lambda\,S_1(\alpha+\Delta\alpha)/S_1(\alpha), and it identifies the part of the change that no ridge penalty reproduces (the shape effect). The paragraph after Theorem 1 now makes this separation.

We consider two results the main new contributions. The first is the conversion between the exponent and the ridge penalty near the peak (Theorem 1). The second is the threshold (α−1)r<1(\alpha-1)r < 1 for the existence of an interior peak of the noise curve in the continuum limit for exponents above one (Theorem 2). To our knowledge the threshold has no counterpart in prior work. The closest work, Rizzi et al. (2026), treats only the two extreme cases r=0r = 0 and r→∞r \to \infty.

Q2. What are the practical implications of the results? Can they inform the design of representations in practice?

We appreciate this question. The submitted version did not spell out the practical implications, and we added a paragraph on them to Appendix H. For a linear readout trained near interpolation, the regularization that a representation provides is summarized by one number, S1S_1, since the noise peak has height σ2d/(2λS1)\sigma^2 d/(2\sqrt{\lambda S_1}) to leading order. Two consequences follow for the design of representations. First, downweighting the low-variance directions of a representation and raising the ridge penalty are interchangeable at the rate given by Theorem 1. Conversely, whitening removes this regularization and has to be compensated by a larger ridge penalty. Whitening a spectrum k−2k^{-2} in dimension 300 raises the noise peak by a factor of about 13. Second, comparing representations at equal λS1\lambda S_1 removes the difference in noise amplification that is due to the spectrum alone. We should add that these implications concern readouts with a weak or untuned ridge penalty, because the gain from a steeper spectrum is small once the penalty is tuned.

I thank the authors for their detailed response. The revised presentation of the paper, along with the additional experiments and improved theoretical results mostly address my concerns. Accordingly, I raised my score.

We would like to thank the reviewer for reading our response and the revised paper, and for raising the score. We appreciate the time and effort put into reviewing this work.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of the revision and correction of numerical values

We sincerely thank all reviewers for their time and for their careful and constructive comments. The comments helped us to clarify what the paper shows and to extend it. We summarize the main changes of the revision below and respond to each reviewer in detail in separate comments. In the revised PDF, added or changed text is shown in red.

Summary of the main changes

  1. From the label-noise part to the prediction error (Reviewers 3fCD, rXz2, DXXR). All three reviewers pointed out that the results of the submitted version concerned only the part of the error caused by label noise. We extended Theorem 1 to the prediction error with a teacher (Section 4, Appendix B) and Theorem 2 to the prediction error in the continuum limit (Proposition C.3). We also derived the condition under which a larger exponent lowers the prediction error at interpolation (Proposition E.1), and we revised the abstract so that its claim matches these results.
  2. Experiments with actual labels and learned representations (Reviewers 3fCD, rXz2, DXXR). We added experiments with the actual labels of MNIST and CIFAR-10, including the case in which the ridge penalty is tuned on a validation set (Appendix D.3), and with 14 representations taken from a ResNet-18 and a ResNet-50 (Appendix H).
  3. Relation to prior work (Reviewers 3fCD, rXz2, DXXR). The paragraph after Theorem 1 now separates what was known from what is new, and Appendix F compares our results with Rizzi et al. (2026) and Wortsman & Loureiro (2025).
  4. Accuracy of the deterministic equivalent and robustness (Reviewer DXXR). We measured the error of the deterministic equivalent at weak ridge (Appendix D.4) and examined spectra that deviate from a power law, together with a bound in terms of the second moment of the inverse spectrum (Appendix I, Proposition B.2).
  5. Presentation (Reviewer 3fCD). The central question is now stated in the Introduction, Section 3 serves as the preliminaries and explains the deterministic equivalent, and the Introduction states in which sense we use the term implicit regularization.

Correction of numerical values

While preparing the revision, we found an error in numbers reported in the submitted version. We apologize for this error. The quantity concerned is the fraction of the exponent change that the ridge change reproduces at n=dn = d (Section 6). In our computation, the size of the shape effect relative to the exponent change was subtracted from 100% where it should have been added, so the deviation from 100% was reported with the wrong sign.

Submitted versionCorrected
Gaussian inputs97.9%, 96.9%102.1%, 103.1%
MNIST98.5 to 99.0%101.0 to 101.5%
Summary in the Introduction97 to 99%101 to 103%

The corrected values are above 100%, which means that the ridge change lowers the noise risk slightly more than the exponent change. The definition of the quantity, the plotted curves and the conclusions of the paper are unchanged. Reviewer DXXR quoted the previous values in the summary of strengths, so we wanted to point out the correction explicitly.

We thank the reviewers again, and we would be glad to answer any further questions.

AuthorsOpenReview ↗

Follow-up on Author-Reviewer Discussion

We thank the reviewers and the Area Chair for their time and consideration.

We are pleased that reviewer 3fCD found that the revision mostly addressed their concerns and raised their score. Reviewer rXz2 said that our response addressed their concrete questions. They recognized that the equivalence condition gives new, precise results but saw them mainly as refinements of the established relationship between spectral structure and regularization. They would not oppose acceptance but remained reluctant to raise their score.

Reviewer DXXR kept their score and raised three remaining concerns. We have since added two results and replied to all three points. (1) On finite-dimensional guarantees at weak ridge, Proposition J.1 now bounds the error of the deterministic equivalent of the noise risk near interpolation for Gaussian inputs. (2) On representations that evolve during training, Appendix H now tests the equivalence for the noise risk on 100 representations taken over the training of a ResNet-18. (3) On the bias-variance results beyond local approximations, our reply explains which results hold for finite exponent changes and outside the peak region (Corollary B.1, Lemma C.2, Proposition C.3).

We would be grateful if the Area Chair and the reviewers could take the new results into account when assessing these concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.