acceptodds
Under review as a conference paper at ICLR 2027

Double Descent and Malign Overfitting in Diffusion Models

Abstract

Conventional wisdom in deep learning holds that overparameterization—having more parameters than training samples —is benign: larger models generalize better and, even without regularization, interpolating models generalize well, the test error following a double-descent curve. One might expect the same benign overfitting for diffusion models, whose training reduces to regression, i.e. to minimizing a quadratic denoising score-matching loss. Yet the opposite is observed: overfitting here is catastrophic, driving the model into a memorization regime. We resolve this paradox by combining experiments on U-Nets trained on CelebA with a random-features model for which we derive closed-form learning curves. We show that with a fixed number of noise realizations per training sample, an interpolation peak does occur, but at rather than at as in standard regression. The rise of the test loss, however, sets in much earlier, at , independently of . This overfitting is malign because, although the implicit regularization of training is fully at work, it drives the model toward the empirical score, which memorizes the training set, rather than toward the true score. A bias–variance decomposition pinpoints the mechanism: the bias of the score estimator starts to grow at ; past the peak the variance decays, as in regression, whereas the bias keeps growing and both saturate at a large value. Since diffusion models are trained with , the peak is pushed to very large model sizes, and therefore models sit on the rising branch that precedes it, where malign overfitting is already in play. Nevertheless, overparameterization remains beneficial when paired with regularization: in the random-features theory and in U-Net experiments, optimally regularized large models—via a ridge penalty or early stopping, respectively—outperform unregularized models of any size.

Primary area:
generative models
Keywords:
Diffusion Models, Double Descent, Benign Overfitting

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mouZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This manuscript compares overfitting in diffusion models with overfitting in supervised learning, operating on the premise that benign overfitting and the corresponding double descent in the test loss could be a universal property of the regression loss. The authors ask whether and how diffusion models exhibit benign overfitting, since they are trained on a regression objective. They study this problem using heavily overparametrized random-features models which they are able to treat analytically. The answer they give is both double descent and benign overfitting indeed occur, but the latter applies only to a quantity they call L_{test}^{emp}, which uses the training data, but averaged over independent noise observations, opposed to a training loss with a set of m independent noise observations per training example.

Critical strengths
  • The authors deliver the answer to the question they ask: double descent occurs in diffusion models, benign overfitting also, but only if one measures the correct quantity
  • The authors show that the interpolation peak in diffusion models appears when the number of parameters scales with nm, the number of data times the number of noise realizations per training point
Critical weaknesses

I would like to preface this by saying that I think that the authors have established non-trivial mathematical results, evidenced by the strengths section, but the manuscript does not present itself to me as a lucid and coherent combination of these results. My main criticisms are:

  • I found it difficult to appreciate the merits of the derivation of the theoretical results. For example, I don't understand how the theory motivates the measurement of L_{test}^{emp}, and why it leads to a double descent curve. The theoretical quantities of interest appear to be determined self-consistently, and the manuscript to me does not yet connect these equations to the results in an illuminating manner - intuitively, it would have been nice to place more emphasis on explaining what the order parameters of the system are, what they measure, and how they determine the loss curves? Moreover, the derivation appears to make the assumption that m and the training dynamics can be decoupled. However, diffusion models are trained using one noise realization per training step, such that m, the number of noise realizations per training example, is increasing as a function of training time. Here, as I understand, m is fixed, but training is considered to be converged. Can it be shown that early stopping is equivalent to choosing m or could there be training dynamics effects not captured by the picture here?
  • The connection to practice: One empirical observation on the more realistic experiments with U-nets do not appear to support the generality of the phenomena: there appears to be no peak in L_{test}^{emp}, in the middle of Figure 3, as I would have expected given the theoretical predictions. Is such a peak in L_{test}^{emp} not expected here? The double descent in the FID curve for early-stopped using the test loss as a performance measure suggests, as the authors also acknowledge, that there are other phenomena in diffusion model training affecting performance that are not adequately captured here. Additionally, the authors do promote early stopping as a way to mitigate overfitting, but otherwise, I found it difficult to decipher what the high-level practical takeaways of the manuscript are. For example, one could easily increase manipulate m, by doubling the number of noise samples per training point per batch, thereby shifting the interpolation peak. Would the authors expect this to have any practical merit?
Questions for authors

Most of my questions have already been asked in the weaknesses section. Here are the remaining ones:

In the introduction, the authors make the relation to Marion and Wu's observation regarding the peak in FID as a function of training time, in a manner that made me expect that their manuscript contains an explanation of this phenomenon. In the appendix, they clarify that Marion and Wu's observation is an independent effect. Perhaps the phrasing in the related work section could make this scope more explicit?

What kind of distributions can the random features model learn? If it is effectively linear, does that mean it is comparable to a linear score, that learns Gaussian distributions? Or is the linearization valid only with respect to performance measures such as test loss?

3 comments
AuthorsOpenReview ↗

Response to Reviewer mouZ

Answer to Weaknesses

We first address the two critical weaknesses raised. Please, see also the "general response to reviewers".

W1. We thank the reviewer for this detailed comment.

Empirical test loss. The motivation to introduce Ltestemp\mathcal{L}^{\mathrm{emp}}_{\mathrm{test}} is that it is the natural loss one optimizes when training a diffusion model. Moreover, given that data is formed by samples noised mm times, it is natural to disentangle the role of taking fresh samples and fresh noises, from just taking fresh noises. Up to a constant, this loss measures the distance to its minimizer, the empirical score (Appendix C.3). The main result of our work is that it is this loss that exhibits double descent and benign overfitting instead of what we call the test loss.

Fixed mm and training dynamics. Fixing the noises and considering the converged minimizer is an idealization (Sect. 1), standard in theoretical analyses (Cui et al., 2024; George et al., 2026) and matched by our U-Net experiments, which use a fixed pool of mm noises per image. In this framework, mm sets the total number nmnm of training points, hence the interpolation threshold. As stated by the referee, early stopping the training dynamics limits the number of fresh noises seen in practice, hence effectively fixes the number mm. As we have stressed in the introduction, we decided to work at fixed mm noises to make the connection with the double descent literature. The regime relevant for realistic applications corresponds to m≫1m\gg1, in which fixed noises and fresh noises become essentially equivalent.

Order parameters. The "order parameters" qq and rr, which solve a set of algebraic equations, are normalized traces of resolvents of random matrices (Stieltjes transforms). They are auxiliary quantities commonly introduced to compute the asymptotic losses of random-feature neural networks (see, e.g., D'Ascoli et al., 2020; George et al., 2026).

W2. As is visible in the RFNN case, the prominence of the peak in Ltestemp\mathcal{L}^{\mathrm{emp}}_{\mathrm{test}} depends on mm and also tt. We agree that the U-Net curves we display do not exhibit a clear peak in this loss. However, we added in Appendix B.6 the evolution of the gap between the train and empirical test losses revealing a peak, followed by a decrease.

Our practical message is not to manipulate mm to move the interpolation peak. Increasing mm can improve early-stopped performance, with diminishing returns in our experiments, but we establish no computational advantage from using a finite, fixed noise pool. Finite mm serves to explain the mechanism behind the transition from benign to malign overfitting, rather than to prescribe a new training procedure. Note, however, that the goal of our paper is mainly foundational as explained in "general response to reviewers".

Answer to Questions

We now answer the questions asked.

Q1. We have clarified our statement concerning the relation with Marion and Wu's work in the introduction. We believe the two double descents are caused by different phenomena. Their double descent occurs during learning, observed on the statistics of the generated distribution, and likely rooted in the transition of the model's functional representation from the generalizing to the memorizing score. Ours is static, appearing in the test loss, and stems from overfitting the wrong target.

Q2. The RFNN is nonlinear in the input: sA(x)=Aσ(Wx/d)/ps_A(x)=A\sigma(Wx/\sqrt{d})/\sqrt{p}. It is therefore not restricted to learning linear scores or Gaussian distributions, although its approximation capacity depends on the chosen features and on its width. The Gaussian equivalence principle applies to the asymptotic quantities entering our learning curves, not to a replacement of the learned score by a linear function. This is also compatible with the possibility of memorization. Indeed, George et al. (2026) demonstrate that memorization can happen in these models. We would be of course happy to provide additional detail if needed.

I thank the authors for their clarifications, which in part address my concerns. With these clarifications and also the additional discussion between the authors and the other reviewers, I was able to split my remaining concerns into two parts.

These are: A) highlighting and clarifying the difference to the supervised setting, which leads to the definitions of the three losses, especially LtestempL_{\text{test}}^{\text{emp}} B) Based on this, demonstrating how the double descent follows from the self-consistent equations for the order parameters.

Regarding point A), I was able to convince myself why LtestempL^{\text{emp}}_{test} is a natural loss to use using an ajacent calculation. It is close to what the authors also treat in their paper, but helped me understand their contribution better, so I decided to report it here in case others also find it useful.

During training, one could consider that LtrainL^{train} is computed from N=nm samples drawn from an underlying distribution pempp_{\text{emp}}, such that the training examples are

Y∼pemp=1n∑νN(e−txν,ΔtId)Y \sim p_{\text{emp}}= \frac{1}{n} \sum_{\nu} \mathcal{N} \left( e^{-t} x^{\nu}, \Delta_t \text{Id} \right)

For such samples, we know that 1ΔtE[ξ∣Y]=−∇ln⁡pemp(Y)=−semp∗(Y)\frac{1}{\sqrt{\Delta_t}} \mathbb{E} \left[ \xi|Y \right] = -\nabla \ln p_{\text{emp}}(Y) =- s^{*}_{\text{emp}}(Y)

In this sense, we can view each training sample as a noised version of a "true target" plus a zero-mean remaining noise:

ξ=−Δt∇ln⁡pemp(Y)+ϵ\xi= - \sqrt{\Delta_t} \nabla \ln p_{\text{emp}}(Y)+ \epsilon

The classical test loss then becomes the average over infinitely many such samples drawn from the same distribution, which here, is crucially, pempp_{\text{emp}}. This naturally leads to the empirical test loss considered by the authors, Lemp, testL^{\text{emp, test}}, which would be the "classical equivalent" of the test loss in the regression setting, which has the double descent. Therefore, using Lemp, testL^{\text{emp, test}}, the problem reduces to the known double descent. The problem, as the authors also show, is that this is a biased estimator of the test loss drawn over the true distribution. This is then demonstrated by the bias variance decomposition treated in the paper.

The remaining point is now B) clarifying how the self-consistent equations result in the different behaviours the Lemp, testL^{\text{emp, test}}, for example through asymptotic analysis or a simplified case. Another avenue that could be taken is to say that using Lemp, testL^{\text{emp, test}} reduces the problem to established results in supervised settings (unless there are technical points of which I am unaware which prevent this).

Overall, however, I now see more clearly where the difference to the standard regression setting lies and I have thus increased my score.

We thank the reviewer for increasing their score and for the discussion, which helped us improve the presentation of our results.

Reviewer_zXwgOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies double descent in diffusion models trained on nn samples, each with mm fixed noise realizations. Using a random-features theory at finite mm and U-Nets on CelebA, it shows the interpolation peak shifts to p∼nmp\sim nm, while the test loss already starts rising at p∼np\sim n, driven by a growing bias toward the empirical score. It also shows that optimally regularized large models outperform unregularized ones.

Critical strengths

The paper is well written and easy to follow. It analyzes the finite-mm regime, extending prior random-features results that only covered m=1m=1 and m=∞m=\infty. It also supports the theory with U-Net experiments on CelebA that reproduce the predicted trends.

Critical weaknesses

First, the contribution is incremental. Malign overfitting, attraction to the empirical score, and the benefit of early stopping are already established in the cited works. What is new is the finite-mm extension.

Second, by the paper’s own scalings, practical training is effectively in the m=∞m=\infty regime. Finite mm matters only near p∼nmdp\sim nmd, while for p≪nmdp\ll nmd the test loss matches the m=∞m=\infty curve. With the paper’s CIFAR-10 numbers (m≈2⋅103m\approx2\cdot10^3), nmd≈3⋅1011nmd\approx3\cdot10^{11}, far beyond realistic model sizes. The finite-mm phenomena (the nmnm peak and the benign-to-malign crossover) therefore do not arise in practice, and the practically relevant picture reduces to the m=∞m=\infty case already covered by prior work.

Questions for authors

A minor issue: the variance in Eq. (4) is missing the average over D,Θ\mathcal D,\Theta.

8 comments
AuthorsOpenReview ↗

Response to Reviewer zXwg

Answer to Weaknesses

Please, see first the discussion in "general response to all reviewers" which already addresses these points. We will reiterate our arguments below.

W1. We agree that memorization and the benefits of early stopping are already established findings, and we do not claim these observations as new. Our main result is to identify the mechanism responsible for them in a way that connects with prior work on double descent and overfitting in standard regression.

The finite-mm analysis is essential to this explanation. It's a tool to obtain the result above, not a result per se (which would indeed be incremental if it was the only purpose of the paper). Our asymptotic characterization separates two distinct scales: the onset of increasing test error at p∼np\sim n and the interpolation peak at p∼nmp\sim nm. Increasing mm moves the interpolation peak without moving the onset of overfitting. This connects the double-descent curve at finite mm to the peak-free behavior at m→∞m\to\infty observed by previous works, while showing that malign overfitting persists as the peak recedes to infinity (in Figs. 3 and 5).

Our regularization results address a more specific question than whether early-stopping helps: they show that suitably regularized large models can outperform fully trained smaller models despite this malign-overfitting behavior (Fig. 6).

We therefore view the finite-mm extension as the analytical means to establish this broader picture, rather than as an extension of previously known learning curves. We have sharpened the presentation to distinguish these contributions from the established results.

W2. We agree that, in practice, large-scale training likely operates far below the interpolation peak, where the m=∞m=\infty description is appropriate. Our contribution is to explain theoretically how this regime connects to double descent: the peak moves with nmnm, whereas the onset of malign overfitting remains at around p∼np\sim n in the RFNN. The bias–variance analysis further explains why pushing the peak to infinity does not restore the benign nature of the overfitting. We therefore do not claim a new practically accessible interpolation threshold, but a characterization of the transition linking regression-like double descent to the previously studied diffusion regime. We've made this goal clearer in the paper.

Answer to Questions

We now answer the question asked.

Q1. The average over D,Θ\mathcal{D},\Theta is already included through the outer angular brackets in Eq. (4), with this notation defined immediately below the equation. We have made these brackets more visually explicit to avoid confusion.

Thank you for your reply. If I understand correctly, the main contribution of this work, if not the only one, is the finite-mm analysis. However, I am still unclear to what extent this distinction necessitates new techniques or constitutes a genuinely new framework. As noted in my earlier assessment and in the comments from the other two reviewers, the bound itself may not be sufficiently informative to explain phenomena beyond the reach of existing approaches in the literature. Could you please elaborate on the novelty and significance of the contribution from this perspective?

Sure, here it is (we hope to have understood correctly what you meant by bound).

The m=∞m=\infty analysis alone cannot explain why overfitting in diffusion models is malign, or how it relates to existing theories of overfitting. The m=∞m=\infty learning curves could fit two very different scenarios:

(1) Diffusion models behave like modern overparameterized models in supervised learning, but the second descent, where overfitting becomes benign, is out of reach because m≫1m \gg 1 (cf ours estimate of pp vs nmnm in practice).

(2) Overfitting is malign, classical statistical learning theory holds for diffusion models: overfitting is driven by a growing variance, while the bias decreases.

We show that neither scenario holds, and telling them apart requires finite mm. At finite mm, we show that the rise of the test loss, and memorization, are driven by the bias. This mechanism is absent from classical regression, where overfitting is driven by the variance. It is also absent from modern overparameterized supervised learning, where overfitting is benign because the variance decreases beyond the interpolation peak (and the rise before the peak is due to variance and not to biais). Finite mm is also what shows that making models larger, even up to and beyond the interpolation peak, does not restore generalization (Fig.1 and 2).

Without finite mm, one cannot locate practical diffusion models relative to the interpolation threshold. One therefore cannot see that they are underparameterized with respect to the number of noisy training points nmnm, which is the opposite of the modern supervised picture in which models are overparametrized with respect to the training points. Nor can one see how they still differ from the classical setting: although they are underparameterized, the test loss rises because of the bias (that we compute for the first time for diffusion models).

In summary, our contribution is to show that diffusion models are a new paradigm for overfitting:

Classical statistical learning theory: overfitting is malign and driven by the variance when models are overparametrized.

Modern supervised learning: models work beyond the interpolation peak (p>np > n) and are overparameterized with respect to the number of data. Overfitting is benign because the variance decreases past the peak.

Diffusion models: models work before the interpolation peak, in the window n<p<nmn < p < nm. They are overparameterized with respect to the number of training samples but underparameterized with respect to the number of noisy training points. Overfitting is malign, and it is driven by the growth of the bias for p>np > n. This window has no counterpart in supervised learning.

Reviewer_xTvmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper analyzes the different phases of diffusion models with score function parameterized as random feature model. They establish generalization, interpolation, and memorization phase analytically, and provide experimental results on synthetic data and real data.
Moreover, they provide the scaling for the transitions between these regimes in terms of model parameters.

Critical strengths

S1. The authors analyze an important problem of studying over-parameterized regimes for diffusion models, which has a direct impact in practice.

Critical weaknesses

W1. Theorem 2.1 in the current presentation lacks clarity and a connection to the rest of paper. Many quantities are defined in Appendix but lack explanation in main draft by what they are supposed to mean.

W2. The curves in figure 2 do establish the regimes as illustrated in figure 1. However, despite having the analytical expressions for all the quantities of interest the authors did not provide convergence rates for generalization and memorization which is standard in the literature.

Questions for authors

Q1. Could the authors state the analytical expressions of Ltrainn,mL_{\mathrm{train}}^{n,m}, LtestL_{\mathrm{test}} and LtestempL_{\mathrm{test}}^{\mathrm{emp}} and provide growth/decay of these functions over different phases ?

Q2. Can the authors provide what is the optimal scaling for λ\lambda ?

Q3. In the figure 3 (right) does the peak for U-Net occur at theoretical prediction?

Q4. Minor corrections:

  1. The notation is confusing. The authors use superscript as indexing in line 100 for x\mathbf{x}, and in equation (4) they define bias term as B2\mathcal{B}^2, is the 22 here supposed to be index or exponent? If it is exponent, should the authors just define B\mathcal{B}?

  2. In Equation (4), ⟨\langle and ⟩\rangle are in the wrong order.

  3. The fonts in the figures needs to be increased for better readability.

Update: The authors have adequately addressed all of my questions and included the necessary changes in the draft. Therefore, I change my evaluation to weak accept.

4 comments
AuthorsOpenReview ↗

Response to Reviewer xTvm

We thank the reviewer for their careful reading of the manuscript and for comments that will greatly improve our submission. We answer each of the points raised below.

Answer to Weaknesses

We first address the two critical weaknesses raised.

W1. We agree with the reviewer that Theorem 2.1, as written in the first version of the manuscript, is compact and could be improved. In the revised manuscript, we propose a new version with more details. Does the reviewer find this new version clearer? We hope that this is the case, but if not, please give us suggestions and we will implement them.

W2. Regarding the convergence rates and the growth/decay of the losses: the revised Theorem 2.1 now gives the losses explicitly, and these closed forms yield the following behaviors:

  • Generalization rates (Fig. 3): for p<np<n, Ltest\mathcal{L}_{\mathrm{test}} decreases and depends on ψp\psi_p only; for n<p<nmn<p<nm, it increases and depends on ψp/ψn\psi_p/\psi_n only; the height of the peak at p=nmp=nm and the plateau beyond it are independent of nn. The generalization gap also depends on ψp/ψn\psi_p/\psi_n (Fig. 18).
  • Overparameterized limit ψp→∞\psi_p\to\infty at fixed λ\lambda: Ltrain\mathcal{L}_{\mathrm{train}} vanishes as 1/ψp1/\psi_p, while Ltest\mathcal{L}_{\mathrm{test}} and Ltestemp\mathcal{L}^{\mathrm{emp}}_{\mathrm{test}} converge to a finite plateau (Proposition C.3).
  • Sample complexity at ψp=∞\psi_p=\infty (Fig. 16, right): for m=1m=1, both the bias and the variance decay as power laws in ψn\psi_n; for m>1m>1, the variance still decays as a power law, but the bias saturates at a value that remains O(1)O(1) in ψn\psi_n.

Did the reviewer have other rates in mind?

Answer to Questions

We now answer the questions asked.

Q1. We have written explicitly the formulas for the three losses in the theorem of the revised version. The growth/decay rates of the different losses can be found in our answer to the second weakness the reviewer raised.

Q2. In the appendix (Fig. 15), we provide the optimal scaling of the ridge regularization strength with ψp=p/d\psi_p=p/d, and find that it is λ=O(ψp)\lambda=O(\psi_p). Is the reviewer also interested in its scaling with other parameters, such as tt and ψn\psi_n?

Q3. In Fig. 3 (right), the U-Net peak occurs approximately at p=nmdp=nmd, consistent with the theoretical prediction. Additional experiments in Appendix B.1 varying nn and mm further support this scaling. We do not, however, expect this peak location to hold in general. The theory is derived for a simplified model, and the agreement observed here is an empirical finding for the architecture and training setup studied. In previous studies on double descent for supervised learning, it was found that the scaling in practical applications can be qualitatively similar but quantitatively different from the one obtained by Random Features models.

Q4-1. The superscript 22 in B2\mathcal{B}^2 denotes an exponent: it corresponds to the squared bias, which enters the bias–variance decomposition directly. We have clarified this in the text.

Q4-2. Thank you for flagging this. The brackets in Eq. (4) are in the correct order. We have enlarged the outer brackets to remove all ambiguity.

Q4-3. We agree with the reviewer, and we will increase the font size in the figures of the camera-ready version of the manuscript.

Thank you for the response and the revision. I have some follow-ups:

W1. Theorem 2.1 now states the fixed-point equations for (q,r)(q, r) and the maps f1,f2f_1, f_2. However, it still does not show how these determine the losses in Eqs. (9)–(11). Specifically:

  1. Eqs. (10) and (11) require ∂zT1\partial_z T_1, ∂ζT1\partial_\zeta T_1, and ∂ϵT1\partial_\epsilon T_1. The theorem only provides f1f_1 and f2f_2 at ϵ=0\epsilon = 0, and the ϵ≠0\epsilon \neq 0 system is still deferred to Appendix C.8.3 (Eq. 155). As a result, LtestL_{\text{test}} and LtestempL^{\text{emp}}_{\text{test}} cannot be evaluated from the main text.
  2. The main text does not explain how qq and rr behave in each regime. The descent, rise, and interpolation peak of LtestL_{\text{test}} are therefore shown only by the figures, not by the theorem.
  3. The concentration of (q,r)(q, r) as d→∞d \to \infty is stated without a rate, so the theorem gives no control on problem parameters.

I would ask the authors to add a short corollary, or a paragraph after the theorem, that states the limiting values of Eqs. (9)–(11) in each regime and explains how they follow from (q,r)(q, r).

Q1. The response gives asymptotic limits (d→∞d \to \infty, then ψp→∞\psi_p \to \infty or m→∞m \to \infty) but not the convergence or growth rates I asked for in the generalization, memorization, and interpolation phases. To answer the authors' question: yes, I had in mind the leading-order behaviour of LtrainL_{\text{train}}, LtestL_{\text{test}}, and LtestempL^{\text{emp}}_{\text{test}} within each phase.

I would ask the authors to:

  • (a) state the leading-order rates in each phase, including the rate at which LtestL_{\text{test}} and LtestempL^{\text{emp}}_{\text{test}} approach their plateaus;
  • (b) report the explicit power-law exponents;
  • (c) include these results in the main text.

We thank the reviewer for their follow-up questions.

  • W1.1 We have uploaded a new version of the manuscript in which Theorem 2.1 gives the explicit expressions of f1f_1, f2f_2 and the self-consistent equations at all ϵ\epsilon. All three losses can now be evaluated from the main text.

  • W1.2 qq and rr are auxiliary variables in our computation and have no direct interpretation: they are Stieltjes transforms of random matrices, as defined in Eq. (13). Their behavior is not particularly enlightening, except for the interpolation peak. The peak at ψp=mψn\psi_p=m\psi_n can be obtained analytically from the fact that qq is the normalized trace of the inverse of the p×pp\times p matrix FFT/(nm)+λIp\mathbf{F}\mathbf{F}^T/(nm)+\lambda\mathbf{I}_p, where FFT\mathbf{F}\mathbf{F}^T has rank nmnm. For p>nmp>nm, it has p−nmp-nm zero eigenvalues, so λq→1−mψn/ψp\lambda q\to1-m\psi_n/\psi_p as λ→0\lambda\to0 and the training loss vanishes. At p=nmp=nm, its smallest nonzero eigenvalue vanishes, which produces the peak. The behavior of the three losses in each regime was obtained by analyzing the numerical solutions of the asymptotic equations. We have also added a paragraph after the theorem summarizing this behavior (see the end of this answer).

  • W1.3 We have added finite-size simulations (new Fig. 15 in Appendix C.13.1) showing that qq, rr, Ltrain\mathcal{L}_{\mathrm{train}} and Ltest\mathcal{L}_{\mathrm{test}} all converge to the predictions of Theorem 2.1 at rate 1/d1/d in the generalization, rise and memorization regimes. This is consistent with random matrix theory. qq and rr are normalized traces of resolvents, whose fluctuations are O(1/d)O(1/d) because the density of states satisfies a large-deviation principle with speed d2d^2. The losses are likewise normalized traces of products of random matrices: they average over the dd output dimensions and the nm=O(d)nm=O(d) noisy training points, i.e. over O(d2)O(d^2) scalar terms. We have stated this 1/d1/d rate in Theorem 2.1 of the new version and given the argument in Appendix C.13.1. A rigorous proof could be obtained by generalizing to our setting the results of Mei & Montanari (2022) for random-features ridge regression, and the non-asymptotic, dimension-free deterministic equivalents of Defilippis, Loureiro & Misiakiewicz (2024).

  • Q1 (a) Approach to the plateau. Beyond the interpolation peak, with all other parameters fixed, the losses approach their ψp=∞\psi_p=\infty values (Proposition C.3) as O(1/ψp)O(1/\psi_p). We have added this result to the main text.

We have added the following short summary paragraph after Theorem 2.1:

  • Generalization (ψp≪ψn\psi_p\ll\psi_n): the three losses depend only on ψp\psi_p and behave as e−2t+O(1/ψp)e^{-2t}+O(1/\psi_p).
  • Rise (ψn≲ψp≪mψn\psi_n\lesssim\psi_p\ll m\psi_n): Ltest\mathcal{L}_\text{test} increases and depends only on ψp/ψn\psi_p/\psi_n.
  • Memorization (ψp≫mψn\psi_p\gg m\psi_n): the losses approach their plateau at a rate 1/ψp1/\psi_p. The asymptotic value of the empirical test loss goes to 0 as O(1/m)O(1/m).

Thank you for revising the draft. The authors have adequately addressed all of comments, and therefore I am willing to increase my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.