- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This manuscript compares overfitting in diffusion models with overfitting in supervised learning, operating on the premise that benign overfitting and the corresponding double descent in the test loss could be a universal property of the regression loss. The authors ask whether and how diffusion models exhibit benign overfitting, since they are trained on a regression objective. They study this problem using heavily overparametrized random-features models which they are able to treat analytically. The answer they give is both double descent and benign overfitting indeed occur, but the latter applies only to a quantity they call L_{test}^{emp}, which uses the training data, but averaged over independent noise observations, opposed to a training loss with a set of m independent noise observations per training example.
Critical strengths
- The authors deliver the answer to the question they ask: double descent occurs in diffusion models, benign overfitting also, but only if one measures the correct quantity
- The authors show that the interpolation peak in diffusion models appears when the number of parameters scales with nm, the number of data times the number of noise realizations per training point
Critical weaknesses
I would like to preface this by saying that I think that the authors have established non-trivial mathematical results, evidenced by the strengths section, but the manuscript does not present itself to me as a lucid and coherent combination of these results. My main criticisms are:
- I found it difficult to appreciate the merits of the derivation of the theoretical results. For example, I don't understand how the theory motivates the measurement of L_{test}^{emp}, and why it leads to a double descent curve. The theoretical quantities of interest appear to be determined self-consistently, and the manuscript to me does not yet connect these equations to the results in an illuminating manner - intuitively, it would have been nice to place more emphasis on explaining what the order parameters of the system are, what they measure, and how they determine the loss curves? Moreover, the derivation appears to make the assumption that m and the training dynamics can be decoupled. However, diffusion models are trained using one noise realization per training step, such that m, the number of noise realizations per training example, is increasing as a function of training time. Here, as I understand, m is fixed, but training is considered to be converged. Can it be shown that early stopping is equivalent to choosing m or could there be training dynamics effects not captured by the picture here?
- The connection to practice: One empirical observation on the more realistic experiments with U-nets do not appear to support the generality of the phenomena: there appears to be no peak in L_{test}^{emp}, in the middle of Figure 3, as I would have expected given the theoretical predictions. Is such a peak in L_{test}^{emp} not expected here? The double descent in the FID curve for early-stopped using the test loss as a performance measure suggests, as the authors also acknowledge, that there are other phenomena in diffusion model training affecting performance that are not adequately captured here. Additionally, the authors do promote early stopping as a way to mitigate overfitting, but otherwise, I found it difficult to decipher what the high-level practical takeaways of the manuscript are. For example, one could easily increase manipulate m, by doubling the number of noise samples per training point per batch, thereby shifting the interpolation peak. Would the authors expect this to have any practical merit?
Questions for authors
Most of my questions have already been asked in the weaknesses section. Here are the remaining ones:
In the introduction, the authors make the relation to Marion and Wu's observation regarding the peak in FID as a function of training time, in a manner that made me expect that their manuscript contains an explanation of this phenomenon. In the appendix, they clarify that Marion and Wu's observation is an independent effect. Perhaps the phrasing in the related work section could make this scope more explicit?
What kind of distributions can the random features model learn? If it is effectively linear, does that mean it is comparable to a linear score, that learns Gaussian distributions? Or is the linearization valid only with respect to performance measures such as test loss?
3 comments
I thank the authors for their clarifications, which in part address my concerns. With these clarifications and also the additional discussion between the authors and the other reviewers, I was able to split my remaining concerns into two parts.
These are: A) highlighting and clarifying the difference to the supervised setting, which leads to the definitions of the three losses, especially B) Based on this, demonstrating how the double descent follows from the self-consistent equations for the order parameters.
Regarding point A), I was able to convince myself why is a natural loss to use using an ajacent calculation. It is close to what the authors also treat in their paper, but helped me understand their contribution better, so I decided to report it here in case others also find it useful.
During training, one could consider that is computed from N=nm samples drawn from an underlying distribution , such that the training examples are
For such samples, we know that
In this sense, we can view each training sample as a noised version of a "true target" plus a zero-mean remaining noise:
The classical test loss then becomes the average over infinitely many such samples drawn from the same distribution, which here, is crucially, . This naturally leads to the empirical test loss considered by the authors, , which would be the "classical equivalent" of the test loss in the regression setting, which has the double descent. Therefore, using , the problem reduces to the known double descent. The problem, as the authors also show, is that this is a biased estimator of the test loss drawn over the true distribution. This is then demonstrated by the bias variance decomposition treated in the paper.
The remaining point is now B) clarifying how the self-consistent equations result in the different behaviours the , for example through asymptotic analysis or a simplified case. Another avenue that could be taken is to say that using reduces the problem to established results in supervised settings (unless there are technical points of which I am unaware which prevent this).
Overall, however, I now see more clearly where the difference to the standard regression setting lies and I have thus increased my score.
We thank the reviewer for increasing their score and for the discussion, which helped us improve the presentation of our results.
Response to Reviewer mouZ
Answer to Weaknesses
We first address the two critical weaknesses raised. Please, see also the "general response to reviewers".
W1. We thank the reviewer for this detailed comment.
Empirical test loss. The motivation to introduce Ltestemp is that it is the natural loss one optimizes when training a diffusion model. Moreover, given that data is formed by samples noised m times, it is natural to disentangle the role of taking fresh samples and fresh noises, from just taking fresh noises. Up to a constant, this loss measures the distance to its minimizer, the empirical score (Appendix C.3). The main result of our work is that it is this loss that exhibits double descent and benign overfitting instead of what we call the test loss.
Fixed m and training dynamics. Fixing the noises and considering the converged minimizer is an idealization (Sect. 1), standard in theoretical analyses (Cui et al., 2024; George et al., 2026) and matched by our U-Net experiments, which use a fixed pool of m noises per image. In this framework, m sets the total number nm of training points, hence the interpolation threshold. As stated by the referee, early stopping the training dynamics limits the number of fresh noises seen in practice, hence effectively fixes the number m. As we have stressed in the introduction, we decided to work at fixed m noises to make the connection with the double descent literature. The regime relevant for realistic applications corresponds to m≫1, in which fixed noises and fresh noises become essentially equivalent.
Order parameters. The "order parameters" q and r, which solve a set of algebraic equations, are normalized traces of resolvents of random matrices (Stieltjes transforms). They are auxiliary quantities commonly introduced to compute the asymptotic losses of random-feature neural networks (see, e.g., D'Ascoli et al., 2020; George et al., 2026).
W2. As is visible in the RFNN case, the prominence of the peak in Ltestemp depends on m and also t. We agree that the U-Net curves we display do not exhibit a clear peak in this loss. However, we added in Appendix B.6 the evolution of the gap between the train and empirical test losses revealing a peak, followed by a decrease.
Our practical message is not to manipulate m to move the interpolation peak. Increasing m can improve early-stopped performance, with diminishing returns in our experiments, but we establish no computational advantage from using a finite, fixed noise pool. Finite m serves to explain the mechanism behind the transition from benign to malign overfitting, rather than to prescribe a new training procedure. Note, however, that the goal of our paper is mainly foundational as explained in "general response to reviewers".
Answer to Questions
We now answer the questions asked.
Q1. We have clarified our statement concerning the relation with Marion and Wu's work in the introduction. We believe the two double descents are caused by different phenomena. Their double descent occurs during learning, observed on the statistics of the generated distribution, and likely rooted in the transition of the model's functional representation from the generalizing to the memorizing score. Ours is static, appearing in the test loss, and stems from overfitting the wrong target.
Q2. The RFNN is nonlinear in the input: sA(x)=Aσ(Wx/d)/p. It is therefore not restricted to learning linear scores or Gaussian distributions, although its approximation capacity depends on the chosen features and on its width. The Gaussian equivalence principle applies to the asymptotic quantities entering our learning curves, not to a replacement of the learned score by a linear function. This is also compatible with the possibility of memorization. Indeed, George et al. (2026) demonstrate that memorization can happen in these models. We would be of course happy to provide additional detail if needed.