- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper examines whether benign overfitting occurs in diffusion models. Its main result gives a sample-size lower bound for a data-dependent score estimator to achieve both small empirical ESM loss and good population performance, including a condition expressed through the KL divergence of the generated distribution. The bound depends on intrinsic dimension, noise scale, and time weighting. The paper then analyzes linear random feature networks to compare score matching with regression and studies how time smoothness and early stopping prevent overfitting. Random feature and CIFAR-10 experiments support parts of this picture.
Critical strengths
-
The general lower bound goes beyond a particular network model. Theorem 3.2 allows the score estimator to depend on the training data and does not restrict its architecture. Its dependence on intrinsic dimension and noise scale makes the limitation more informative than the observation that the exact empirical score reproduces training data.
-
The linear analysis provides useful explanations of overfitting and regularization. The comparison with regression identifies the role of the empirical covariance null space. The analysis of time integration and early stopping gives explicit results that help explain how generalization can be maintained before the model approaches the empirical solution.
Critical weaknesses
-
The absence of benign overfitting and the absence of double descent are different claims. Theorem 3.2 concerns whether empirical fitting and population accuracy can be achieved simultaneously; it does not rule out a second decrease in test error as model width grows. Section 4 studies linear random feature networks, whose function class no longer expands once (p\geq d), so the resulting curves do not establish the same behavior for general nonlinear networks. George et al. [1], already cited in the submission, explicitly report double descent for DSM with finitely many noise samples. This does not contradict Theorem 3.2, but requires a clearer distinction between the settings. Figure 1 also needs clarification: Appendix C.1 lists (n=20{,}000), (N_g=10), and a maximum width of (10{,}000), which does not reach the stated number of training samples. It is therefore unclear what interpolation threshold is marked in the figure.
-
Theorem 3.2 assumes small empirical ESM loss, which measures approximation to the empirical score. The reported training curves mainly use DSM loss, differing from ESM by a data-dependent constant. The paper should establish whether the experimental models satisfy the required ESM tolerance. In addition, the numerical discussion in Appendix B.1 substitutes an estimated intrinsic dimension for the lower Rényi dimension and applies a bound that holds for sufficiently small noise. The paper acknowledges that the dimension notions may differ, but does not verify either this correspondence or the required noise scale. Consequently, the numerical sample-size estimate is suggestive rather than a demonstrated lower bound for the CIFAR-10 setting.
[1] Denoising Score Matching with Random Features: Insights on Diffusion Models from Precise Learning Curves.
Questions for authors
- Can the Figure 1 hyperparameters and the marked interpolation threshold be reconciled? How does the curve change with the number of noise samples per training example?
- Can the empirical ESM loss be evaluated, or related quantitatively to the reported DSM loss, to check the premise of Theorem 3.2?
- Section 5.1 studies time-independent networks. What conclusions can be drawn for networks that retain time conditioning?
4 comments
Question 1
Can the Figure 1 hyperparameters and the marked interpolation threshold be reconciled
As explained in the answer to the first weakness above, there was indeed a typo in the hyperparameters listed in the appendix. The sample size used in Figure 1 (right) is , which is consistent with the marked interpolation threshold on the figure.
How does the curve change with the number of noise samples per training example?
As remarked in our answer to weakness 1, we do not believe the case where a finite number of Gaussians is used in the train loss is particularly relevant. We believe that this setting shares much more in common with traditional regression than how diffusion models are trained in practice. However, we agree with the reviewer that this setting deserves consideration. We have added plots in Section C where we vary (with ). With such a low dimension, a peak is barely visible. However, a peak is expected at the effective dataset size for higher dimensions, as suggested by [George et al]. It is clear that there are some distinctions between this finite Gaussian diffusion model setting and traditional regression (namely that the effective training data is no longer i.i.d.) and so we agree that this is an interesting direction of research that we will suggest as future work in the updated manuscript.
Question 2
Can the empirical ESM loss be evaluated, or related quantitatively to the reported DSM loss, to check the premise of Theorem 3.2?
We agree with the reviewer that there would be a clearer connection to the theory if the U-Net experiments reported ESM loss instead of DSM loss. This is a minor change as we can estimate the DSM/ESM gap directly from data and subtract it from the existing plots. We will provide this update during the rebuttal period and we thank the reviewer for the suggestion.
Question 3
Section 5.1 studies time-independent networks. What conclusions can be drawn for networks that retain time conditioning?
This is a good question. To the best of our knowledge, our analysis in Section 5.1 is the first to explicitly take into account the time integral in the score matching losses. A time-independent network is an extreme example of a network that is very smooth in time. As such, this result is informative on the impact of time smoothness. Obtaining similar results for a random feature analysis of time-dependent networks with time-integrated losses is beyond the scope of our work and is a fascinating direction for future work. Intuitively, we believe that similar conclusions should hold for networks that have a smooth dependence on time. In fact, a realistic time dependence should interpolate between the two settings discussed in Figure 3 (right): between perfectly smooth and perfectly non-smooth networks.
Thank you for the detailed response and clarifications. I appreciate the correction to Figure 1 and the additional experiments. I also agree that the distinction between benign overfitting and double descent should be made clearer.
I find the discussion of time smoothness and regularization interesting. It is also related to existing explanations of diffusion generalization, such as Bonnaire et al. [1], which studies implicit dynamical regularization and early stopping, and Zhou et al. [2], which investigates how smoothing empirical score weights can reduce single-sample dominance and memorization. Although these works consider different mechanisms, I think they offer complementary perspectives on why regularization can improve generalization in diffusion models.
My main remaining concern is the connection between Theorem 3.2 and practical diffusion models. While I appreciate the generality of the theoretical result, it remains unclear whether the empirical ESM losses and noise scales in the U-Net experiments satisfy the conditions required by the theorem. The proposed additional ESM measurements would help clarify this connection.
Overall, I find the theoretical contribution valuable and appreciate the authors' efforts to address my concerns. I will maintain my original score of 3.
References
[1] Bonnaire, T., Urfin, R., Biroli, G., and Mézard, M. (2025). Why Diffusion Models Don't Memorize: The Role of Implicit Dynamical Regularization in Training. Advances in Neural Information Processing Systems (NeurIPS 2025). https://arxiv.org/abs/2505.17638
[2] Zhou, X., Zhang, J., and Wright, S. J. (2026). Smoothing the Score Function to Enhance Generalization in Diffusion Models. (CVPR 2026). https://arxiv.org/abs/2601.19285
We thank the reviewer for their detailed response to our rebuttal and for maintaining the positive evaluation of our work.
Thank you for taking an interest in the time smoothness discussion. We appreciate the reference [Zhou et al.], whose unconditioned score analysis is related to our Section 5.1, and also analyze other implicit regularization mechanisms, such as temperature smoothing. We have now added this reference alongside the already existing discussion in the related works section. We agree that [Bonnaire et al] is an important reference (especially regarding early stopping) and a paragraph in the related works section is dedicated to the comparison with this work.
We understand the concerns of the reviewer regarding the applicability of Theorem 3.2 to practical settings and the U-Net experiments. To that end, we make two remarks:
- Regarding whether the base assumptions of the theorem hold, we refer to the numerical section paragraph in Section B, showing that, at least, the condition to make the lower bound exponential in is realistic in practice and holds for the case of CIFAR-10 according to best estimates of the effective dimension.
- We have now included a discussion in section C.3 that estimates the ESM/DSM gap, demonstrating that for the setup in the U-Net experiment it is negligible. It is not known to the authors whether the population ESM/DSM gap is possible to estimate but we compute the empirical gap over the test set as an (unjustified) estimate.
We thank the reviewer for their time and we hope that our answer helped clarifying the remaining concerns.
We thank the reviewer for their interesting remarks and the positive evaluation of our work. We answer your questions below.
Weakness 1
The absence of benign overfitting and the absence of double descent are different claims ...
This is a good remark. What Section 3 shows is that benign overfitting is impossible for diffusion models. This means, at least, that there cannot be a second descent all the way down to zero. The reviewer is correct that [George et al] point out that, when a finite number of Gaussian samples is used, a peak might be observed. This is not the setting that we are considering in our theory, where we use the true expectation over the noise. In practice, during training, Gaussian samples are sampled at every iterations, so we consider that the true expectation over noise is what is being targeted. In a sense, having a finite number of Gaussian samples returns the learning problem closer to a traditional regression problem and so we would expect benign overfitting and double descent to follow.
In Figure 1 (right) the sample size is n=30, the data dimension is d=3, and we sample Ng=10 Gaussian noises. In that case the number of gaussian noises is large enough compared to the data dimension for this setup to approximate the behavior of the (true) expected loss, which is the goal of our experiment. When the true expectation is used, no double descent is observed in this experiment.
We thank the reviewer for this interesting question and leave further investigation for future works.
Figure 1 also needs clarification: ...
We thank the reviewer for spotting this typo in Section C. The sample size used in this experiment is n=30 an not n=20,000. Therefore, the range of model widths cover the interpolation threshold and the vertical line on the diagram is correct.
In the revised version, we have now corrected the hyperparameters for this figure and have also included additional plots in the appendix in Section C, where we vary some of the hyperparameters used for this experiment.
Weakness 2
In addition, the numerical discussion in Appendix B.1 substitutes an estimated intrinsic dimension for the lower Rényi dimension ...
This is a good remark. There are two different reasons for the asymptoticity in the noise scale:
First, we need 2αϵ−1σϵ(log(64)+d)<1 for the lower bound to be exponential in d′. This first condition on σϵ only depends on the ambient dimension, not the intrinsic dimension itself, and we have precise estimate of the required values for ϵ (see the numerical evaluation).
Secondly, the last step of the proof of Proposition B.5 uses the limit defining the lower Renyi dimension. This asymptoticity in ϵ depends on how fast this limit is converging. This can only be made more explicit under additional geometric assumptions, such as Ahlfors regularity of the data distribution. We discuss this in detail in the response to reviewer bfd6, Question 2. In the revised version, we have added a remark in Section B.
Moreover, we agree that it is unclear how the lower Renyi dimension can be estimated from a finite dataset. However, under the assumption that the data is supported on a manifold, the lower Renyi dimension coincides the manifold dimension. The dimension estimators we report in our discussion also target the same intrinsic/manifold dimension, hence why we discuss them interchangeably. That being said, we agree that our estimation of the lower bound is only suggestive and is only meant to provide intuition about its order of magnitude. In the revised version, we have added remarks to discuss these facts more precisely in Section B.