- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how power-law covariance spectra govern double descent in ridge regression, proposing that faster spectral decay produces an implicit regularization effect analogous to increasing explicit ridge penalties. Focus is placed on the weak-ridge regime near interpolation, where the noise-risk curve depends primarily on a single scaled spectral sum; this yields an explicit mapping between changes in spectral decay and an equivalent scalar ridge penalty. The authors decompose this behavior into a scale effect that dominates near interpolation and an irreducible shape effect away from the peak, while establishing an asymptotic threshold that dictates when the interior noise peak disappears entirely. Synthetic Gaussian experiments and covariance-rescaled MNIST representations confirm that the predicted equivalent ridge adjustments reproduce nearly all of the variance suppression observed near interpolation, though the theoretical framework characterizes label-noise risk rather than total generalization error or nonlinear architectures.
Critical strengths
Following are the strengths:
- Theorem 1 establishes an explicit, quantitative mapping between arbitrary positive covariance spectra and scalar ridge penalties, moving beyond qualitative arguments about data anisotropy suppressing variance peaks to provide an interpretable, uniform leading-order approximation near the interpolation boundary.
- The decomposition into scale and shape effects offers valuable theoretical insight, explaining precisely why isotropic ridge regularization closely matches spectral sharpening near the interpolation peak while fundamentally failing to capture the full trajectory away from it.
- The empirical validation is carefully executed, pairing synthetic Gaussian baselines with non-Gaussian MNIST representations and utilizing paired sampling to demonstrate convincing 97–99% empirical agreement with the theoretically predicted ridge shifts near interpolation.
Critical weaknesses
Following are the weaknesses:
- The core theoretical guarantees are derived for deterministic equivalents and continuum limits rather than exact finite-dimensional risk, leaving a theoretical gap in the critical weak-ridge regime where small eigenvalues and vanishing ridge penalties challenge standard deterministic equivalent convergence guarantees.
- The primary claims characterize only label-noise amplification rather than total generalization error, which obscures the bias-variance trade-off; as shown in Appendix E, spectral sharpening can inflate bias and increase net prediction error under uniform signal distributions even as noise risk monotonically declines.
- The novelty relative to established spectral learning theory and benign overfitting literature remains somewhat incremental, particularly regarding closely related analyses like Rizzi et al. (2026), making it unclear whether the results represent fundamentally new phenomena or reparameterized asymptotic variants of existing spectral risk formulas.
- The experimental scope is narrow, relying exclusively on synthetic Gaussian inputs and artificially rescaled MNIST representations at rather than evaluating naturally emergent power-law spectra across diverse modern neural architectures, tasks, or modalities.
Questions for authors
Following are the questions for the authors:
- Can the framework be extended to incorporate target-alignment spectra and bound the accompanying bias term, yielding a closed-form condition for when increasing strictly improves total generalization error?
- How robust is the scalar equivalence to spectral non-idealities, such as truncated tails or isolated outlier eigenvalues, and can higher-order spectral moments bound the resulting approximation error?
- Does the predicted equivalence hold for representation spaces extracted across intermediate layers of real pre-trained networks, where spectral shifts are intrinsically coupled to feature rotation and higher-order dependencies?
- What concrete error bounds or convergence guarantees justify the deterministic equivalent in the singular regime where , , and approaches zero?
- Does the predicted equivalence hold for representation spaces extracted across intermediate layers of real pre-trained networks, where spectral shifts are intrinsically coupled to feature rotation and higher-order dependencies?
5 comments
Response to Reviewer DXXR (2/2)
Q1. Can the framework be extended to incorporate target-alignment spectra and bound the accompanying bias term, yielding a closed-form condition for when increasing strictly improves total generalization error?
Thank you for this question. Yes, this is possible at interpolation and at weak ridge, and the revision now includes such a condition. Theorem 1 now holds for arbitrary signal powers , and it gives the bias term at as up to a relative error , for any alignment of the target. The resulting condition is (Proposition E.1), where is the ratio of the bias term to the noise risk and and are means of with weights and . For a power-law alignment with , the threshold on is for large , up to smaller terms (Corollary E.2). We should note that the condition is stated for the leading value of the prediction error and is local in the exponent.
Q2. How robust is the scalar equivalence to spectral non-idealities, such as truncated tails or isolated outlier eigenvalues, and can higher-order spectral moments bound the resulting approximation error?
We examined both kinds of non-idealities in Appendix I. The error of Theorem 1 is controlled by , so the harmful non-ideality is a very small eigenvalue, whether isolated or in a tail of very small positive eigenvalues. When the smallest eigenvalue of a power law is reduced by a factor of 1000, the leading curve is off by more than a factor of two, yet the exponent change and the ridge change still differ by only 8.3%, because the outlier dominates of both spectra. An exactly truncated tail is covered by the theorem with the rank in place of .
On your second question, the answer is yes at the interpolation point. Proposition B.2 bounds the error of the leading value at by whenever , with , where is the sum of the inverse squared eigenvalues. This bound does not involve .
Q3. Does the predicted equivalence hold for representation spaces extracted across intermediate layers of real pre-trained networks, where spectral shifts are intrinsically coupled to feature rotation and higher-order dependencies?
We agree that this is an important test, and we carried it out on 14 representations (Appendix H). Across layers, the noise peaks differ by a factor of five at a common ridge penalty and lie within 17% of the leading value of Theorem 1 at equal , although the layers differ in their eigenvectors and higher-order statistics. Within each layer, the exponent change and the equivalent ridge change give noise risks at that differ by about 1% () and by 3 to 4% (). We should be clear about what these experiments do not show. They do not show that the change of a representation produced by training is equivalent to a ridge change, because the rotation and the higher-order dependencies then change together with the spectrum, as you point out. The Limitations paragraph states this.
Q4. What concrete error bounds or convergence guarantees justify the deterministic equivalent in the singular regime where , , and approaches zero?
To our knowledge there is no such bound or guarantee, and we do not provide one. We would like to clarify, however, which regime our results use. Theorem 1 does not rely on the singular limit. It is an expansion in the ratio , where , and for a power law this ratio is about . It applies at a fixed positive ridge penalty, where the peak has a finite height. At , the value of our real-data experiments, this ratio is 0.09 for with , and the measured error of the deterministic equivalent is small, as reported in our reply to W1. In the continuum limit of Theorem 2 with , and tend to zero, but grows as . The limit at a fixed spectrum is the case in which the deterministic equivalent is least reliable (Appendix D.4), and the Limitations paragraph states that our theorems concern the deterministic equivalent.
Q5. Does the predicted equivalence hold for representation spaces extracted across intermediate layers of real pre-trained networks, where spectral shifts are intrinsically coupled to feature rotation and higher-order dependencies?
This question repeats Q3. Please see our reply to Q3 above.
Follow-up on our response
Thank you again for the time and effort you put into reviewing our paper. Your comments helped us improve it considerably.
As the discussion period is coming to a close, we would like to ask whether our response and the revised paper address your concerns. If any point remains unclear, or if there is anything further we can do, please let us know. We would be glad to provide additional clarification.
Response to Reviewer DXXR (1/2)
We thank the reviewer for the careful reading and for the constructive comments. We respond to each point below. Added or changed text in the revised PDF is shown in red.
We agree that this gap remains, and the Limitations paragraph states it. No available bound controls the error of the deterministic equivalent near interpolation at weak ridge, so we measured this error instead (Appendix D.4). For Gaussian inputs at n=d, it is at most 0.5% at the ridge penalty of our real-data experiments (λS1=90) and grows to 11 to 12% at λS1=0.3. The comparison at the center of the paper is much less affected. The ratio between the exponent change and the ridge change agrees with the deterministic equivalent within 0.1%, also where the deterministic equivalent is off by more than 10%. Regarding the continuum limit, Theorem 2 is a statement about this limit, and Section 5 explains how the criterion applies at finite dimension and tests it on MNIST inputs with d=300 (Figure 2).
Thank you for this observation. You are right that the submitted version characterized only the label-noise part, and that Appendix E already showed how the bias can grow. We turned that example into a result. Theorem 1 now gives the prediction error near the peak as the sum of the noise risk and the bias term. At n=d these are (d/2)σ2/λS1 and (d/2)ωλS1, where ω is a coefficient determined by the teacher, so at fixed ω a larger λS1 lowers the first and raises the second. Proposition E.1 gives the condition under which their sum decreases with the exponent, and Proposition C.3 extends Theorem 2 to the prediction error.
We agree that the relation to prior work needed to be stated more precisely, and the revision does so. For 0≤α<1 our continuum equation is indeed a reparameterization of the state equations of Rizzi et al. (2026), as Section 2 says. Two results, however, are not reparameterizations. For α>1, Theorem 2 gives the threshold (α−1)r=1 at which the peak disappears. Rizzi et al. treat the ridgeless case and a ridge penalty of order one, which are the two ends r=0 and r→∞, and the threshold does not follow from either. Theorem 1 is an expansion at finite dimension for an arbitrary spectrum, which gives the ridge penalty equivalent to a change of the exponent. Relative to the benign-overfitting literature, which bounds the risk of a single problem through effective ranks, our results convert a change of the spectrum into a change of the ridge penalty at the interpolation peak. Section 2 now states this difference.
We agree that the scope of the submitted experiments was narrow. The revision adds the actual labels of MNIST and CIFAR-10 (Appendix D.3) and 14 representations of a ResNet-18 and a ResNet-50, whose spectra arise from the networks and are not rescaled (Appendix H, please see our reply to Q3). Other architectures, tasks and modalities are still not covered, as stated in the Limitations paragraph