acceptodds
Under review as a conference paper at ICLR 2027

First Learn, Then Memorize: The Spectral Bias of Diffusion Models

Abstract

Diffusion models trained on a finite dataset first learn to generate novel, high-quality samples and only much later collapse onto their training set. We identify the mechanism behind this separation of timescales and the object that probes it. The training dynamics of the score function are governed—exactly, and at any width—by the Gram matrix of the Neural Tangent Kernel (NTK) evaluated on the noisy training data, so the timescales of generalization and of memorization must be encoded in its spectrum. We show that they are, and that the structure responsible has no analogue in standard kernel settings. The use of multiple noise realizations per sample ( noised copies at a fixed noise level) in the score-matching loss is what restructures the Gram matrix spectrum into two distinct parts. The first, of large eigenvalues, carries the global features of the target distribution and is present already for . The second, which the repeated noising creates, consists of the smallest eigenvalues and is supported on eigenvectors aligned with the sample-specific noise directions; it sets a memorization timescale parametrically larger in the training set size . We establish this picture on two fronts. Analytically, we solve the spectrum in the lazy high-dimensional limit for both linear () and polynomial () sample complexities, and prove through a bias–variance decomposition that the first bulk minimizes the approximation error while the second drives the error associated with memorization. Empirically, we show the same two-bulk structure in Convolutional NTKs on CelebA and in finite-width U-Nets trained well beyond the lazy regime, and we make the link causal: truncating the Gram matrix at rank tunes the generalization–memorization transition, and an penalty targeting the second bulk suppresses memorization in feature-learning U-Nets.

Primary area:
generative models
Keywords:
Diffusion Models, lazy regime, memorization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_HDV7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies why diffusion models learn distributional structure before memorizing training samples. For a dataset with (m) fixed noise realizations per sample, it analyzes the NTK Gram matrix and identifies generalization and memorization bulks with different learning timescales. The theoretical contributions include eigenvector characterizations, a bias–variance decomposition, and an extension to polynomial sample sizes. Experiments use CNTKs on CelebA and finite-width U-Nets to study spectral truncation and regularization.

Critical strengths
  1. The analysis goes beyond identifying two groups of eigenvalues. The eigenvector characterization explains how the generalization and memorization bulks relate to data directions and noise realizations. The bias–variance decomposition and polynomial scaling results add useful detail to this picture.

  2. The CNTK truncation experiments provide a controlled way to examine how retaining different parts of the spectrum affects generation. The finite-width experiments also make a useful attempt to investigate the relevance of the analysis beyond the lazy regime.

Critical weaknesses
  1. The contribution relative to prior work needs clearer positioning, and the mathematical comparison requires clarification. Bonnaire et al. [1] already connect generalization and memorization timescales to separated spectral bulks. Appendix C.2 acknowledges the shared nonzero spectrum of the corresponding Hessian and Gram matrix, so the main additions are the finite-(m) analysis, eigenvectors, bias–variance decomposition, and polynomial extension. More importantly, Appendix D.12 appears to lose the correlation shared across noise copies. Under the Gaussian covariance in Proposition D.3, one can write (\Omega^\alpha=\sqrt{\kappa_t}Z+\sqrt{1-\kappa_t}E^\alpha). Averaging over copies then leaves ((1-\kappa_t)I_p+\kappa_t ZZ^\top/n), which is generally not spectrally equivalent to (I_p) when (p/n) stays nonzero. Eq. (199), and consequently Eq. (201), therefore need justification. This matters for the claimed connection between the fixed-(m) results and fresh-noise training.

  2. The experiments do not yet fully support the claimed generality of the mechanism. Appendix E.1 evaluates memorization only on trajectories starting from training noises, while FID is evaluated using fresh noises. Returning training noises to training images is informative, but does not establish the copying rate during generation from independent noise. The finite-width results also measure a traced eNTK whose connection to the full vector-valued training dynamics needs clarification. Finally, the regularization experiment lacks comparisons with ordinary (L_2) penalties or alternative spectral cutoffs, so it does not isolate the benefit of selecting the penalty from the proposed spectral boundary.

[1] Why Diffusion Models Don't Memorize: The Role of Implicit Dynamical Regularization in Training

Questions for authors
  1. How should Eq. (199) be interpreted given the shared component in Proposition D.3? What order of limits justifies the connection to the noise-averaged setting?
  2. Can the authors report memorization fractions from independent fresh noises, including for the regularized U-Net?
  3. How well does the traced eNTK represent the relevant spectrum of the full vector-valued NTK in the finite-width U-Net?
  4. Does Theorem 2.2 require Gaussian data? Its statement assumes isotropic covariance, whereas Appendix D.9 uses Gaussianity and the corresponding population score.
4 comments
AuthorsOpenReview ↗

Response to Reviewer HDV7

We thank the reviewer for this very careful review, and in particular for spotting some issues in the appendix.

Answer to Weaknesses

We first address the two critical weaknesses raised.

W1

  • Positioning with respect to the literature. Our goal is to identify the general mechanism responsible for memorization in diffusion models. Bonnaire et al. analyzed a random-features model; here we go further and develop a general theory based on the NTK limit of realistic networks, and then test it in numerical experiments. In the NTK setting we study the eigenvectors of the Gram matrix, the bias–variance decomposition, and the polynomial regime. All results hold at finite mm, and the case m→∞m\to\infty is recovered as a special case. Each of these analyses bring new insights. The eigenvector decomposition identifies the mechanism of memorization and explains why it is tied to the bulk of small eigenvalues. The bias–variance decomposition further characterizes this secondary bulk. Finally, the polynomial regime shows how these results extend to a more general setting, in which the learned score approximates the true score increasingly well. These results extend the analysis of Bonnaire et al. in both scope and depth.

  • Link between random features and the NTK. The reviewer is right. With Ωα=κtZ+1−κtEα\Omega^\alpha=\sqrt{\kappa_t}Z+\sqrt{1-\kappa_t}E^\alpha, averaging over copies leaves μIIp+μBZZ⊤/n\mu_I I_p+\mu_B ZZ^\top/n, and replacing ZZ⊤/nZZ^\top/n by IpI_p is only valid if p/n→0p/n\to0. The same issue affects the cross term, Eq. (198). The correct m=∞m=\infty Hessian is U=μIIp+μ1ΔtWW⊤d+1n∑νhνhν⊤,hν=μ1′e−tWxνd+μ∗′κt Zν.U=\mu_I I_p+\mu_1\Delta_t\frac{WW^\top}{d}+\frac1n\sum_\nu h^\nu h^{\nu\top},\qquad h^\nu=\mu_1' e^{-t}\frac{Wx^\nu}{\sqrt d}+\mu_*'\sqrt{\kappa_t}\,Z^\nu. This does not affect the results of the main text. Regarding the order of limits, Bonnaire et al. work at finite width p≍np\asymp n, whereas our kernel corresponds to p/n→∞p/n\to\infty. In that limit, the non-zero spectrum of the corrected UU (divided by pp) coincides with that of our Gram matrix at large mm (divided by nmnm), so the two descriptions agree. We have rewritten the appendix accordingly.

W2

  • Initialization from training noise vs. fresh noise. We thank the reviewer for pointing out this distinction. We agree that reconstruction from training-noise initializations does not, by itself, establish the copying rate during standard generation from independent noise. We have added Appendix E.4 and Fig. 14 to distinguish fmemf_{\mathrm{mem}} from fmemfreshf_{\mathrm{mem}}^{\mathrm{fresh}} and report additional evaluations using independent Gaussian initializations. We report that fresh-noise memorization still occurs for the CNTK when rr is large enough on CelebA with m=1024m=1024 and for nn from 2 to 16, with a linear scaling of the onset of memorization. For the U-Net experiments, we detect no fresh-noise memorization in either case. These runs use only m=8m=8 fixed noise realizations per training image. Fitting this finite-noise objective need not recover the exact empirical score that leads to fresh-start memorization. Consequently, reconstruction from training noises can coexist with little or no copying from independent noises.

  • Traced eNTK vs. full Gram matrix. We agree that the output-traced eNTK and the full, matrix-valued eNTK are distinct objects and need not share the same spectral structure. However, our U-Net experiments directly establish a two-bulk structure for the traced eNTK. Together with the suppression of training-noise memorization when the regularization is chosen using the corresponding spectral scale, this provides evidence consistent with the existence of a memorization bulk in the full eNTK. We now clarify this scope in Sect. 3.2 of the revised paper. We will also include a more thorough empirical investigation of the comparison between the full and traced eNTK spectra in the camera-ready version of the manuscript.

  • L2L_2 baselines. We thank the reviewer for raising this point. The penalty itself is an ordinary L2L_2 penalty with a coefficient that depends on the training time, calibrated using the measured spectra. Our objective was to test the proposed spectral interpretation rather than establish a hyperparameter selection procedure. The observed suppression of training-noise memorization is consistent with this interpretation, but does not establish that the choice is unique nor optimal. Furthermore, minimizing the test loss need not minimize the dependence on training samples, as shown by Garnier-Brun et al., 2026. This motivates evaluating memorization or correlation to the training set alongside the losses.

Answer to Questions

We now answer the questions asked. Note that Q1, Q2 & Q3 were answered in the weaknesses above.

Q4 (Condition on Gaussian data). Theorem 2.2, and hence its corollary Result 2.1, does require Gaussian data, and we have added this assumption to both statements. The spectral results (Theorems 2.1 and 2.3) only assume sub-Gaussian data. Gaussianity is used through the linear form of the exact score, sexact(y)=−y/Γts_{\rm exact}(y)=-y/\Gamma_t with Γt=Tr Σt/d\Gamma_t=\mathrm{Tr}\,\Sigma_t/d. For non-Gaussian isotropic data, and assuming that the moments of the predictor are universal, the bias acquires an extra term 1dE∥sexact(y)+y/Γt∥2=1dE∥sexact∥2−1/Γt≥0\frac1d\mathbb{E}\|s_{\rm exact}(y)+y/\Gamma_t\|^2=\frac1d\mathbb{E}\|s_{\rm exact}\|^2-1/\Gamma_t\ge0, which does not decay with ψn\psi_n.

Thank you for the detailed response and clarifications.

I appreciate the additional analysis and experiments, particularly the clarification of the connection to Bonnaire et al. and the new fresh-noise memorization results. However, my main concerns remain.

First, while the paper provides more detailed spectral characterizations, including eigenvectors, a bias–variance decomposition, and polynomial scaling results, the central mechanism and separation of generalization and memorization timescales were already established in Bonnaire et al. The rebuttal does not sufficiently clarify what fundamentally new insights or predictions arise from these extensions.

Second, the new fresh-noise experiments partially address my concern, but they are limited to small datasets in the CNTK setting. More importantly, the authors acknowledge that no fresh-noise memorization was detected in their finite-width U-Net experiments, which limits the evidence connecting the proposed mechanism to memorization during standard generation.

Finally, I appreciate the correction to Appendix D.12, but the practical advantage of the proposed spectral regularization remains unclear without comparisons to standard L2L_2 regularization or early stopping.

Overall, I recognize the additional mathematical contributions and the authors' efforts to address the concerns. Nevertheless, I remain unconvinced that the paper establishes sufficiently distinct novelty and empirical support beyond prior work. I therefore maintain my original score.

We thank the reviewer for their prompt reply. We address the remaining concerns below.

New insights relative to Bonnaire et al. The main new insight comes from the polynomial scaling analysis. It shows that the model successively learns the best polynomial approximations of the score. Once it has learned the best approximation available at a given sample size, it starts to memorize. In Bonnaire et al., it was an open question whether the generalization and memorization timescales could still overlap. Here we show that they are strictly separated, and that the generalization phase itself splits into several successive timescales.

Practical advantage of the spectral regularization. The L2 experiments were not meant to propose a new regularization for practitioners. Their aim is to validate our spectral picture: choosing the penalty so as to suppress the memorization bulk suppresses memorization. We therefore do not claim a competitive hyperparameter-selection procedure.

We hope this addresses the reviewer's main concerns, and we thank them again for their careful and constructive feedback.

Reviewer_sfXfOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the separation between generalization and memorization in diffusion models trained on finite datasets. The authors analyze the Neural Tangent Kernel Gram matrix evaluated on noisy training examples and show that repeated noising of each training sample creates a two-bulk spectral structure. The large-eigenvalue generalization bulk, which is already present with one noise realization per sample, captures global features of the target distribution and is learned on a relatively fast timescale. The small-eigenvalue memorization bulk is induced by repeated noisy copies of the same sample, contains directions aligned with sample-specific noise, and is learned on a timescale that grows with the dataset size. The paper derives this structure analytically in both linear and polynomial sample-complexity regimes and connects the two spectral components to a bias–variance decomposition. Experiments with convolutional Neural Tangent Kernels on CelebA and finite-width U-Nets support the predicted spectral hierarchy beyond the lazy regime. Spectral truncation controls the transition between generalization and memorization, while targeted L2 regularization of the memorization bulk suppresses memorization without substantially degrading test performance.

Critical strengths
  1. The paper provides a clear mechanism linking repeated noising, the two-bulk spectrum of the Neural Tangent Kernel Gram matrix, and the separation between generalization and memorization timescales. This offers a useful theoretical explanation for an important empirical property of diffusion models.
  2. The analysis covers both linear and polynomial sample-complexity regimes, while experiments with convolutional Neural Tangent Kernels, spectral truncation, and finite-width U-Nets test whether the proposed mechanism persists beyond the idealized lazy setting. The targeted regularization experiment also provides evidence that the memorization bulk is causally related to memorization.
Critical weaknesses
  1. The theoretical assumptions do not fully match standard diffusion training. The main analysis assumes a lazy regime, a fixed diffusion time, and a finite set of noise realizations that are sampled once and then held fixed. In practical diffusion training, noise is typically resampled online at every optimization step, and models often operate in a feature-learning regime across a range of noise levels. The paper provides empirical evidence beyond these assumptions, but not a rigorous justification that the same spectral mechanism must hold in the fully realistic setting.
  2. The empirical scope and quantitative agreement with theory are limited. The experiments focus primarily on grayscale, downsampled CelebA, and the paper acknowledges a mismatch between the theoretically predicted and empirically observed number of generalization eigenvalues. It remains unclear how well the proposed spectral picture transfers to larger-scale diffusion models, other datasets, or architectures, and how robust the memorization metric is to the choice of generation protocol and nearest-neighbor criterion.
Questions for authors
  1. How does the proposed two-bulk mechanism change when noise is resampled online at every optimization step, as in standard diffusion training, rather than fixed for a finite set of realizations? Can the authors provide either a theoretical connection to the (m\to\infty) setting or empirical evidence that the same spectral separation persists under online noise resampling?
  2. What explains the discrepancy between the theoretically predicted number of generalization eigenvalues and the number observed on CelebA? Can the authors quantify how image anisotropy, intrinsic dimension, and the covariance spectrum modify the predicted bulk locations, eigenvalue multiplicities, and generalization–memorization transition?
  3. How robust is the proposed spectral mechanism across datasets, dimensionalities, resolutions, and architectures? It would be nice to have experiments on datasets with substantially different structure, such as CIFAR-10, ImageNet subsets, and non-image or synthetic data with controlled intrinsic dimension. It would also be useful to vary the ambient dimension and image resolution, for example from (32\times32) to (64\times64) or higher, and to compare convolutional networks, U-Nets, Vision Transformers, and other architectures with different inductive biases. For each setting, the authors should report the Gram-matrix spectrum, the locations and sizes of the two proposed bulks, the scaling of the memorization threshold with dataset size, and the resulting FID, test loss, and memorization rate. Such experiments would clarify whether the two-bulk picture is a general property of diffusion training or is specific to the CelebA and convolutional settings studied here.
  4. How robust are the measured memorization and generalization transitions to the evaluation protocol? In particular, do the conclusions remain unchanged under different trajectory initializations, nearest-neighbor thresholds, sample-generation procedures, noise schedules, and definitions of memorization?
3 comments
AuthorsOpenReview ↗

Response to Reviewer sfXf

We are grateful to the reviewer for their interest in our work and for their detailed questions.

Answer to Weaknesses

We first address the two critical weaknesses raised.

W1 (Simplifying assumptions). We agree that our theoretical analysis relies on simplifying assumptions (the lazy regime, a fixed diffusion time tt, and a finite number mm of noise realizations). Some of them have been done to develop a better analysis. In particular, working at a fixed diffusion time allows us to isolate and study the new phenomenon, tied to the spectrum of the Gram matrix, that we associate with the generalization and memorization regimes. As discussed in the paper, this effect is pronounced at small times (low noise) and absent at large times. Treating all times at once would therefore entangle distinct regimes and make the analysis less transparent. Similarly, studying a finite number of noise realizations allows us to show the difference between the standard "regression" case (m=1m=1) and the denoising score-matching loss (large mm). Note that our results, obtained for finite mm, allow us to obtain the infinite noise realization limit (m→∞m\rightarrow \infty) as a particular case. Considering the lazy regime is instead a simplifying assumption indeed. We do agree with the reviewer that a theoretical analysis able to also describe the feature regime would be very valuable. We respectfully note that this is a very exciting, but at the moment very challenging, direction.

Given the limitations above, it was indeed important for us to test the predictions of the theory. Namely, that the separation of timescales and its link with the spectrum of the Gram matrix hold not only for the lazy CNTK on natural images but also for a U-Net trained in the feature-learning regime, which suggests that the mechanism is general.

W2 (Limited quantitative agreement with theory). Despite quantitative discrepancies between theory and experiment, the central qualitative picture remains consistent: the Gram-matrix spectrum exhibits two bulks, with evidence linking one bulk to generalization and the lower bulk to memorization. The observed number of eigenvalues in the generalization bulk differs from the theoretical prediction and we believe this discrepancy may reflect differences between the assumptions of our theoretical setting and the statistics of natural images.

Our experiments support the proposed mechanism in controlled settings, without establishing universality across large-scale diffusion models. The CelebA results cover both CNTK regression in the lazy regime and a finite-width U-Net in the feature-learning regime. We nevertheless agree that their scope remains limited in dataset diversity, resolution, and architecture. Broader validation is an important direction for future work, with the computational cost of spectral analysis posing a practical challenge at larger scales. We therefore interpret the current experiments as evidence for the qualitative spectral mechanism.

To address dependence on the generation protocol, we added Appendix E.4 and Fig. 14, reporting fresh-noise memorization for CNTK regression on CelebA with m=1024m=1024 and nn varying from 2 to 16. These results show that copying also occurs from independent Gaussian initializations. They complement the training-noise evaluations, while robustness to alternative nearest-neighbor distances and thresholds remains to be investigated.

Answer to Questions

We now answer the reviewer's questions.

Q1 (Finite mm vs. m=∞m=\infty). The case m=∞m=\infty, which is closer to practice, is a special case of our analysis. In this limit, online resampling and fixed noise are equivalent, since both minimize the loss averaged over the noise. In this limit our results continue to hold. We find that the eigenvalues grow with mm, but the associated timescales τλ∼nm/(dλ)\tau_\lambda\sim nm/(d\lambda) have a finite limit as m→∞m\to\infty (taking n,d→∞n,d\to\infty first): τgen=1/(μ1λt)\tau_{\rm gen}=1/(\mu_1\lambda_t) for every mm, and τmem→ψnλt/(μBΔt)\tau_{\rm mem}\to\psi_n\lambda_t/(\mu_B\Delta_t). The ratio τmem/τgen\tau_{\rm mem}/\tau_{\rm gen} saturates at ψnμ1λt2/(μBΔt)\psi_n\mu_1\lambda_t^2/(\mu_B\Delta_t), which is proportional to ψn\psi_n.

Q2 (Number of generalization eigenvalues). There is indeed a discrepancy between our theory for sub-Gaussian data (Sect. 2) and the experiments regarding the number of eigenvalues in the first bulk (dd vs. nn). We emphasize, however, that the main claim of the paper concerns the scaling of each bulk of the Gram spectrum with nn, and this prediction does hold for natural images (Fig. 3). Our theory, like the universality results for kernel methods it builds on, assumes a weakly anisotropic covariance, λmax⁡(Σ)=O(1)\lambda_{\max}(\Sigma)=O(1) with Tr Σ/d\mathrm{Tr}\,\Sigma/d finite. Natural images do not satisfy this condition since they have a power-law spectrum. Wortsman and Loureiro have shown that, for strongly anisotropic data, the bulks associated with different groups of eigenfunctions can merge. We leave a theory of diffusion models in the lazy regime for strongly anisotropic data to future work.

Q3 (Additional experiments). We agree that such experiments would strengthen the paper. Unfortunately, our computational resources are limited, and we cannot run them during the rebuttal period. We also believe that the current experiments are sufficient to support the main theoretical findings. Additional extensive empirical studies could be the topic of further works.

Q4 (Robustness of fmemf_{\rm mem}). The 1/31/3 ratio is the criterion used by Yoon et al. and Bonnaire et al. It has been shown in the latter paper that the exact value of the threshold does change the exact moment the first datum is memorized but not the scaling of the timescales with nn. As for the noise schedule, App. B shows that the OU and linear interpolants are reparametrizations of each other, so we expect our theory to apply to other schedules as well.

Reviewer_sfXfOpenReview ↗

response to rebuttal

Thank you for the detailed explanations and the additional fresh-noise experiments. The clarification of the m→∞m\to\infty limit helps connect the finite-noise analysis to the noise-averaged objective. The new evaluations using independent Gaussian initializations also provide valuable evidence beyond reconstruction from training noises. Validation across different datasets, resolutions, and architectures is still limited. Furthermore, fresh-noise memorization has only been demonstrated in small-scale CNTK experiments, while the U-Net experiments have not demonstrated this phenomenon. Robustness to alternative memorization metrics and thresholds also remains insufficiently evaluated. I maintain my rating.

Reviewer_SDsDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why diffusion models generalize before they memorize. By studying the learning dynamics under a NTK framework, the authors show that the top eigenvectors of the NTK gram matrix is learned at a faster speed compared to the non-leading eigenvectors. Since the top ones represent the dominant mode of the population data, while the non-leading ones correspond to the sample-specific noise fluctuations, diffusion models generalize before memorize. The authors further provide empirical justification for the thoery.

Critical strengths
  1. This paper provides a solid nontrivial analysis on the spectrum evolution of the NTK Gram matrix and associate it to the generalization--memorization in diffusion models. The analysis provides an interesting explanation for why generalization happens before memorization.

  2. The paper performs extensive experiments including both NTK regime and the real world network case. The results align well with the theory.

Critical weaknesses
  1. Gap between theoretical assumptions and practical settings.

The theoretical results rely on the lazy-training regime, a fixed diffusion timestep, and a finite number of fixed noise realizations per training sample. These assumptions deviate substantially from practical diffusion training, raising concerns about whether the conclusions generalize to realistic settings. Although the authors provide experiments with finite-width U-Nets, these experiments primarily demonstrate the persistence of the two-bulk spectral structure, rather than directly validating the predicted separation between generalization and memorization timescales.

  1. Unclear connection to the conventional notion of memorization and generalization in diffusion models.

In the existing literature, memorization is often associated with learning the score of the empirical data distribution, whereas generalization involves approximating the population score. However, the NTK model considered in this paper has restricted representational capacity, and it is unclear whether it can accurately approximate either the empirical score or the population score. Consequently, the extent to which the spectral separation identified in the paper explains the conventional generalization–memorization transition remains unclear. A more explicit connection between the proposed spectral analysis and the approximation of these two score functions would strengthen the theoretical interpretation.

Questions for authors

The theoretical analysis primarily focuses on the learning dynamics of the NTK eigenspectrum. However, the connection between these spectral dynamics and the evolution of the learned generative distribution remains unclear. In particular, can the authors explain how learning the generalization and memorization eigenmodes relates to approximating the population score and the empirical score, respectively?

3 comments
AuthorsOpenReview ↗

Response to Reviewer SDsD

We thank the reviewer for their careful reading and for the positive assessment of our analysis. We address each point below.

Answer to Weaknesses

We first address the two critical weaknesses raised.

W1. In their first weakness, the reviewer raises two points: (i) the simplifying assumptions and (ii) the link between the two bulks of the spectrum and the timescales for generalization and memorization in the U-Net experiments.

  • Simplifying assumptions. We agree that our theoretical analysis relies on simplifying assumptions (the lazy regime, a fixed diffusion time tt, and a finite number mm of noise realizations). Some of them have been done to develop a better analysis. In particular, working at a fixed diffusion time allows us to isolate and study the new phenomenon, tied to the spectrum of the Gram matrix, that we associate with the generalization and memorization regimes. As discussed in the paper, this effect is pronounced at small times (low noise) and absent at large times. Treating all times at once would therefore entangle distinct regimes and make the analysis less transparent. Similarly, studying a finite number of noise realizations allows us to show the difference between the standard "regression" case (m=1m=1) and the denoising score-matching loss (large mm). Note that our results, obtained for finite mm, allow us to obtain the infinite noise realization limit (m→∞m\rightarrow \infty) as a particular case. Considering the lazy regime is instead a simplifying assumption indeed. We do agree with the reviewer that a theoretical analysis able to also describe the feature regime would be very valuable. We respectfully note that this is a very exciting, but at the moment very challenging, direction.

    Given the limitations above, it was indeed important for us to test the predictions of the theory. Namely, that the separation of timescales and its link with the spectrum of the Gram matrix hold not only for the lazy CNTK on natural images but also for a U-Net trained in the feature-learning regime, which suggests that the mechanism is general.

  • U-Net experiments. Regarding the U-Net experiments, Bonnaire et al. have already shown that U-Nets exhibit a separation between the generalization and memorization timescales. Our goal is not to re-establish this separation but to explain its origin. We trace it to the spectrum of the NTK Gram matrix, and specifically to an observable feature, the "two bulks," which is generic and can be probed in realistic settings. Two experiments test this link directly. In the lazy regime, truncating the CNTK Gram matrix at rank rr amounts to stopping training at τ∼nm/(dλr)\tau\sim nm/(d\lambda_r) (App. E.2); the memorization threshold then scales as r∝nr\propto n, while the generalization threshold does not depend on nn (Fig. 4), i.e. τmem/τgen∝n\tau_{\rm mem}/\tau_{\rm gen}\propto n. In the feature-learning U-Net, an L2L_2 penalty that damps only the modes below the gap keeps fmemf_{\rm mem} at zero without degrading the test loss (Sect. 3.2 and Fig. 13).

W2 (Score learned by the NTK). At the end of training (γ→0+\gamma\to0^+), the kernel ridge estimator reaches zero training loss with mm frozen noises. As m→∞m\to\infty, the learned function approaches the empirical score, which memorizes (Biroli et al., 2024). If the training loss at fixed mm is small, then the m→∞m\to\infty training loss, which is, up to a constant, the squared L2L^2 distance to the empirical score, is bounded by this same quantity plus a term of order 1/nm1/\sqrt{nm}. Hence, as mm grows, the learned score approaches the empirical score, and at the end of training the model memorizes for mm large enough.

In our analysis, we consider the high-dimensional setting, where different regimes arise depending on how the number of samples nn scales with the dimension dd. In each regime, the score learned by the NTK approximates both the empirical and the population score, and the approximation improves as nn scales with a higher power of dd. (Taking the high-dimensional limit discards contributions to the loss that vanish as d→∞d\to\infty. These contributions are nevertheless needed (1) to reach exactly zero training loss, and hence full memorization, and (2) to learn the generalizing score.)

In the linear regime n≍dn\asymp d, the score learned by the NTK is linear on test points (App. D.9). Since a linear score generates Gaussian data, in this regime the model generates a Gaussian approximation of the target distribution, which approaches the Gaussian with the data covariance when training is stopped at the optimal time and ψn≫1\psi_n\gg1 (Result 2.1). In the polynomial regimes, the kernel can learn non-linear functions and can therefore generate distributions with higher-order correlations. We have added a paragraph on this in the revised version of the paper.

Answer to Questions

Q1. We believe that the reviewer's question was answered when we addressed their second weakness. Please, let us know if a more detailed explanation is needed.

Thank you for your rebuttal. Unfortunately, the technical content of this work is beyond my current expertise, and I do not feel sufficiently qualified to assess the mathematical arguments rigorously. Therefore, I will maintain my original score, but I kindly ask the Area Chair (AC) to assign less weight to my evaluation when making the final decision.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General comment

We thank the three reviewers for their careful and constructive reports. We are glad that they found that the paper provides "a clear mechanism linking repeated noising, the two-bulk spectrum of the Neural Tangent Kernel Gram matrix, and the separation between generalization and memorization timescales", which "offers a useful theoretical explanation for an important empirical property of diffusion models" (sfXf); that it gives "a solid nontrivial analysis" with experimental results that "align well with the theory" (SDsD).

Scope of the paper. We would like first to clarify the scope of the paper. Our aim is to identify the mechanism behind the separation of timescales between generalization and memorization, and the object that reveals it: the spectrum of the Gram matrix of the Neural Tangent Kernel (NTK), which governs the training dynamics. We show that it's the multiple mm noise realizations which creates a second bulk of small eigenvalues, with no analogue in standard kernel settings. This sets a memorization timescale growing with nn. In this setting we study the eigenvectors of the Gram matrix, the bias–variance decomposition, and the polynomial regime, each of which brings new insights. We then test the mechanism identified by this theoretical analysis, which relies on minimal assumptions, beyond its range of validity: the CNTK Gram matrix on CelebA and finite-width U-Nets display the same behavior, supporting that it is a general phenomenon.

We have uploaded a revised version of the paper, in which all changes are highlighted in teal. The main ones are:

  • Correction of App. D.12 (thanks to Reviewer HDV7). The comparison with the random-features Hessian of Bonnaire et al. wrongly dropped the component shared across the noise copies of each sample. We now present the corrected m=∞m=\infty Hessian and state the order of limits: at infinite width, p/n→∞p/n\to\infty, its spectrum coincides with that of our Gram matrix at large mm. The results of the main text are unaffected.
  • Learned Score. A new paragraph explains that, in the linear regime, the learned score is linear and the model generates a Gaussian approximation of the data, while the polynomial regime progressively improve the approximation. A new appendix relates the training loss at fixed mm to the m→∞m\to\infty (denoising score-matching) loss, whose minimizer is the empirical score.
  • Gaussian assumption in Theorem 2.2. We now state that Theorem 2.2 and Result 2.1 assume Gaussian data (the results Theorems 2.1 and 2.3 only require sub-Gaussian data).
  • Memorization metric. In the main text, fmemf_{\rm mem} is measured on backward trajectories started from the training noises. The new Appendix E.4 and Fig. 14 also reports fmemfreshf_{\rm mem}^{\rm fresh}, measured from independent Gaussian noises with the same nearest-neighbor criterion. For the CNTK on CelebA with m=1024m=1024, fresh-noise memorization appears once the rank rr is large enough, and its onset scales linearly with nn, as in the main text.
  • Traced vs. full NTK. We clarify in Sect. 3.2 that the U-Net experiment uses the traced eNTK Gram matrix. Although its spectrum need not coincide with that of the full eNTK, the results remain consistent with the proposed spectral separation and its connection to memorization.

Finally, we agree with the reviewers that experiments on other datasets and architectures, as well as a theory of the feature-learning regime and of strongly anisotropic data, are important next steps. These are challenging problems that go beyond the scope of the present paper and would require substantial new work; we plan to address them in future studies.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.