- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies why diffusion models learn distributional structure before memorizing training samples. For a dataset with (m) fixed noise realizations per sample, it analyzes the NTK Gram matrix and identifies generalization and memorization bulks with different learning timescales. The theoretical contributions include eigenvector characterizations, a bias–variance decomposition, and an extension to polynomial sample sizes. Experiments use CNTKs on CelebA and finite-width U-Nets to study spectral truncation and regularization.
Critical strengths
-
The analysis goes beyond identifying two groups of eigenvalues. The eigenvector characterization explains how the generalization and memorization bulks relate to data directions and noise realizations. The bias–variance decomposition and polynomial scaling results add useful detail to this picture.
-
The CNTK truncation experiments provide a controlled way to examine how retaining different parts of the spectrum affects generation. The finite-width experiments also make a useful attempt to investigate the relevance of the analysis beyond the lazy regime.
Critical weaknesses
-
The contribution relative to prior work needs clearer positioning, and the mathematical comparison requires clarification. Bonnaire et al. [1] already connect generalization and memorization timescales to separated spectral bulks. Appendix C.2 acknowledges the shared nonzero spectrum of the corresponding Hessian and Gram matrix, so the main additions are the finite-(m) analysis, eigenvectors, bias–variance decomposition, and polynomial extension. More importantly, Appendix D.12 appears to lose the correlation shared across noise copies. Under the Gaussian covariance in Proposition D.3, one can write (\Omega^\alpha=\sqrt{\kappa_t}Z+\sqrt{1-\kappa_t}E^\alpha). Averaging over copies then leaves ((1-\kappa_t)I_p+\kappa_t ZZ^\top/n), which is generally not spectrally equivalent to (I_p) when (p/n) stays nonzero. Eq. (199), and consequently Eq. (201), therefore need justification. This matters for the claimed connection between the fixed-(m) results and fresh-noise training.
-
The experiments do not yet fully support the claimed generality of the mechanism. Appendix E.1 evaluates memorization only on trajectories starting from training noises, while FID is evaluated using fresh noises. Returning training noises to training images is informative, but does not establish the copying rate during generation from independent noise. The finite-width results also measure a traced eNTK whose connection to the full vector-valued training dynamics needs clarification. Finally, the regularization experiment lacks comparisons with ordinary (L_2) penalties or alternative spectral cutoffs, so it does not isolate the benefit of selecting the penalty from the proposed spectral boundary.
[1] Why Diffusion Models Don't Memorize: The Role of Implicit Dynamical Regularization in Training
Questions for authors
- How should Eq. (199) be interpreted given the shared component in Proposition D.3? What order of limits justifies the connection to the noise-averaged setting?
- Can the authors report memorization fractions from independent fresh noises, including for the regularized U-Net?
- How well does the traced eNTK represent the relevant spectrum of the full vector-valued NTK in the finite-width U-Net?
- Does Theorem 2.2 require Gaussian data? Its statement assumes isotropic covariance, whereas Appendix D.9 uses Gaussianity and the corresponding population score.
4 comments
Answer to Questions
We now answer the questions asked. Note that Q1, Q2 & Q3 were answered in the weaknesses above.
Q4 (Condition on Gaussian data). Theorem 2.2, and hence its corollary Result 2.1, does require Gaussian data, and we have added this assumption to both statements. The spectral results (Theorems 2.1 and 2.3) only assume sub-Gaussian data. Gaussianity is used through the linear form of the exact score, with . For non-Gaussian isotropic data, and assuming that the moments of the predictor are universal, the bias acquires an extra term , which does not decay with .
Thank you for the detailed response and clarifications.
I appreciate the additional analysis and experiments, particularly the clarification of the connection to Bonnaire et al. and the new fresh-noise memorization results. However, my main concerns remain.
First, while the paper provides more detailed spectral characterizations, including eigenvectors, a bias–variance decomposition, and polynomial scaling results, the central mechanism and separation of generalization and memorization timescales were already established in Bonnaire et al. The rebuttal does not sufficiently clarify what fundamentally new insights or predictions arise from these extensions.
Second, the new fresh-noise experiments partially address my concern, but they are limited to small datasets in the CNTK setting. More importantly, the authors acknowledge that no fresh-noise memorization was detected in their finite-width U-Net experiments, which limits the evidence connecting the proposed mechanism to memorization during standard generation.
Finally, I appreciate the correction to Appendix D.12, but the practical advantage of the proposed spectral regularization remains unclear without comparisons to standard regularization or early stopping.
Overall, I recognize the additional mathematical contributions and the authors' efforts to address the concerns. Nevertheless, I remain unconvinced that the paper establishes sufficiently distinct novelty and empirical support beyond prior work. I therefore maintain my original score.
We thank the reviewer for their prompt reply. We address the remaining concerns below.
New insights relative to Bonnaire et al. The main new insight comes from the polynomial scaling analysis. It shows that the model successively learns the best polynomial approximations of the score. Once it has learned the best approximation available at a given sample size, it starts to memorize. In Bonnaire et al., it was an open question whether the generalization and memorization timescales could still overlap. Here we show that they are strictly separated, and that the generalization phase itself splits into several successive timescales.
Practical advantage of the spectral regularization. The L2 experiments were not meant to propose a new regularization for practitioners. Their aim is to validate our spectral picture: choosing the penalty so as to suppress the memorization bulk suppresses memorization. We therefore do not claim a competitive hyperparameter-selection procedure.
We hope this addresses the reviewer's main concerns, and we thank them again for their careful and constructive feedback.
Response to Reviewer HDV7
We thank the reviewer for this very careful review, and in particular for spotting some issues in the appendix.
Answer to Weaknesses
We first address the two critical weaknesses raised.
W1
Positioning with respect to the literature. Our goal is to identify the general mechanism responsible for memorization in diffusion models. Bonnaire et al. analyzed a random-features model; here we go further and develop a general theory based on the NTK limit of realistic networks, and then test it in numerical experiments. In the NTK setting we study the eigenvectors of the Gram matrix, the bias–variance decomposition, and the polynomial regime. All results hold at finite m, and the case m→∞ is recovered as a special case. Each of these analyses bring new insights. The eigenvector decomposition identifies the mechanism of memorization and explains why it is tied to the bulk of small eigenvalues. The bias–variance decomposition further characterizes this secondary bulk. Finally, the polynomial regime shows how these results extend to a more general setting, in which the learned score approximates the true score increasingly well. These results extend the analysis of Bonnaire et al. in both scope and depth.
Link between random features and the NTK. The reviewer is right. With Ωα=κtZ+1−κtEα, averaging over copies leaves μIIp+μBZZ⊤/n, and replacing ZZ⊤/n by Ip is only valid if p/n→0. The same issue affects the cross term, Eq. (198). The correct m=∞ Hessian is U=μIIp+μ1ΔtdWW⊤+n1∑νhνhν⊤,hν=μ1′e−tdWxν+μ∗′κtZν. This does not affect the results of the main text. Regarding the order of limits, Bonnaire et al. work at finite width p≍n, whereas our kernel corresponds to p/n→∞. In that limit, the non-zero spectrum of the corrected U (divided by p) coincides with that of our Gram matrix at large m (divided by nm), so the two descriptions agree. We have rewritten the appendix accordingly.
W2
Initialization from training noise vs. fresh noise. We thank the reviewer for pointing out this distinction. We agree that reconstruction from training-noise initializations does not, by itself, establish the copying rate during standard generation from independent noise. We have added Appendix E.4 and Fig. 14 to distinguish fmem from fmemfresh and report additional evaluations using independent Gaussian initializations. We report that fresh-noise memorization still occurs for the CNTK when r is large enough on CelebA with m=1024 and for n from 2 to 16, with a linear scaling of the onset of memorization. For the U-Net experiments, we detect no fresh-noise memorization in either case. These runs use only m=8 fixed noise realizations per training image. Fitting this finite-noise objective need not recover the exact empirical score that leads to fresh-start memorization. Consequently, reconstruction from training noises can coexist with little or no copying from independent noises.
Traced eNTK vs. full Gram matrix. We agree that the output-traced eNTK and the full, matrix-valued eNTK are distinct objects and need not share the same spectral structure. However, our U-Net experiments directly establish a two-bulk structure for the traced eNTK. Together with the suppression of training-noise memorization when the regularization is chosen using the corresponding spectral scale, this provides evidence consistent with the existence of a memorization bulk in the full eNTK. We now clarify this scope in Sect. 3.2 of the revised paper. We will also include a more thorough empirical investigation of the comparison between the full and traced eNTK spectra in the camera-ready version of the manuscript.
L2 baselines. We thank the reviewer for raising this point. The penalty itself is an ordinary L2 penalty with a coefficient that depends on the training time, calibrated using the measured spectra. Our objective was to test the proposed spectral interpretation rather than establish a hyperparameter selection procedure. The observed suppression of training-noise memorization is consistent with this interpretation, but does not establish that the choice is unique nor optimal. Furthermore, minimizing the test loss need not minimize the dependence on training samples, as shown by Garnier-Brun et al., 2026. This motivates evaluating memorization or correlation to the training set alongside the losses.