- Rating
- 3(+1)
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies memorization and generalization in Transformer-based diffusion models under finite-sample training. It provides a unified theoretical analysis of gradient-descent convergence, generalization gap, and sample-specific memorization, and further proposes the empirical-prototype attraction mechanism to explain how the denoiser transitions from fitting individual training samples to learning population-level statistics as the dataset grows. The theoretical framework is relatively complete, but the novelty is somewhat incremental compared with recent diffusion memorization theory, and the analysis relies on simplified assumptions such as Gaussian-mixture data and a one-layer single-head Transformer. Overall, the work is theoretically interesting, although my understanding of this theoretical framework is relatively limited.
Critical strengths
The paper provides a unified finite-sample analysis that connects gradient-descent convergence, generalization, and sample-specific memorization for Transformer-based diffusion models, and further gives an explicit memorization-rate characterization together with the empirical-prototype attraction mechanism. This makes the theoretical picture relatively complete rather than analyzing only one isolated aspect.
Critical weaknesses
-
Several recent works as this work mentioned have already studied the transition between memorization and generalization, finite-sample effects, or diffusion training dynamics. The main novelty here is largely extending these analyses to a Transformer-DDPM setting and providing a more unified theoretical characterization, rather than introducing a fundamentally new phenomenon.
-
The analysis relies on a one-layer, single-head Transformer and Gaussian-mixture data with the number of learnable prototypes matched to the number of modes. It remains unclear how well the proposed EPA mechanism and sample-complexity conclusions extend to realistic multi-layer DiTs and natural image distributions.
Questions for authors
-
How well do the theoretical conclusions extend beyond the one-layer, single-head Transformer setting to deeper and more realistic DiT architectures?
-
How sensitive is the proposed memorization–generalization transition to the Gaussian-mixture assumption? Would a similar phenomenon still appear on more complex natural data distributions?
-
Compared with recent works on diffusion memorization and generalization, what is the most important new insight provided by the proposed empirical-prototype attraction mechanism?
4 comments
Weakness 2 & Question 1: It remains unclear how well the proposed EPA mechanism and sample-complexity conclusions extend to realistic multi-layer DiTs and natural image distributions. How well do the theoretical conclusions extend beyond the one-layer, single-head Transformer setting to deeper and more realistic DiT architectures?
A2: This is a great question. First, we would like to clarify that analyzing multi-layer Transformers, or even multi-layer neural networks in general, is extremely challenging, while the one-layer case represents a stepping stone towards the multi-layer case. To the best of our knowledge, no existing works analyze the convergence of multi-layer nonlinear Transformers in DDPM training, due to the highly non-convex nature of the problem. Up till now, one-layer nonlinear Transformers are still among the most complicated models that can currently be analyzed theoretically in the family of Transformer models. In addition, we believe that our current analysis of a one-layer Transformer is useful for future extensions to multi-layer settings. A potential direction is to build upon our analysis of how a one-layer model learns a Gaussian mixture distribution, and extend it to characterize how multi-layer Transformers learn hierarchical mixture distributions that can represent features with multiple scales and levels. Therefore, our current analysis of a one-layer nonlinear Transformer is a crucial first step.
Second, we verify that our proposed memorization mechanism also holds on CIFAR-10 using a 12-layer DiT model. Specifically, we linearly fit the noisy inputs and denoised outputs to estimate the coefficient and the bias term in Proposition 2. We further apply K-means clustering to the data features to identify different modes. By splitting the data into training and test sets, we can approximately compute the empirical mode means and the true mode means. The left panel of Figure 2(B) shows the fitted as a function of under different sample sizes. We observe that when is large, the empirically fitted curve aligns well with the theoretical prediction. The right panel of Figure 2(B) reports the difference between the fitting errors of the estimated bias term with respect to the true mode mean and the empirical mode mean. Denote the estimated bias term as . The right panel of Figure 2(B) essentially computes . The results show that the fitted bias term is closer to the empirical mean. Together, the results in Figure 2(B) demonstrate that the EPA mechanism characterized in Proposition 2 also holds for real-world data distributions and DiT models.
Regarding sample complexity, the left panel of Figure 2(A) shows that once the sample size exceeds a certain threshold, the memorization rate drops rapidly from 1. This threshold plays a similar role to the sample complexity in Corollary 2 that separates memorization and generalization. This indicates that, under real-world data and deep DiT models, the memorization rate follows the same theoretical trend characterized in Corollary 2.
Question 2: How sensitive is the proposed memorization–generalization transition to the Gaussian-mixture assumption? Would a similar phenomenon still appear on more complex natural data distributions?
A3: We thank the reviewer for this question. Gaussian mixture models have been widely used in prior works [Li et al., 2025; Wang et al., 2026; Li et al., 2026] on memorization and generalization in diffusion models, but these works do not provide a quantitative characterization of the memorization rate. Our Theorem 2 provides, to the best of our knowledge, the first quantitative characterization of the memorization rate. We do not claim that the same closed-form threshold must hold for arbitrary natural-data distributions. Our experiments, however, suggest that the qualitative phenomenon of memorization to generalization persists well beyond the theoretical GMM setting.
Specifically, we train a DiT model on CIFAR-10 images, whose distribution is not a Gaussian mixture. We still observe the same qualitative phenomenon, a transition from memorization to generalization. Figure 2(A) shows that once the number of training data exceeds a certain threshold, the memorization rate rapidly drops from 1, indicating that the model no longer perfectly reconstructs the training samples. At the same time, as the number of training data increases, the reconstruction error on the training set gradually increases, while the error on the testing set decreases correspondingly, and the two become nearly identical around that threshold. Figure 5 in the Appendix further reflects this phenomenon by plotting the curve of the memorization rate against .
These results suggest that although the exact quantitative transition derived in our theory depends on the Gaussian-mixture assumptions, a similar phenomenon also appears in standard diffusion models trained on complex natural-image data. This phenomenon can be summarized as: when the sample size is small, the model reconstructs training samples with a high memorization rate, whereas with a large sample size, it achieves a low memorization rate and a smaller reconstruction error on unseen samples. We have clarified this part in the revised manuscript.
Li et al., Neurips 2025. Understanding Representation Dynamics of Diffusion Models via Low-Dimensional Modeling
Wang et al., JMLR 2026. Breaking the Curse of Dimensionality: Diffusion Models Efficiently Learn Low-Dimensional Distributions
Li et al., ICML 2026. Transformers Learn the Optimal DDPM Denoiser for Multi-Token GMMs.
Thanks for the rebuttal, I am not familiar with the theory of diffusion models, so I keep the low confidence score and suggest other reviewers to discuss how the rebuttal has solved their concerns.
We thank the reviewer for the time and effort in the evaluation. We address the concerns as follows.
Weakness 1 & Question 3: Several recent works as this work mentioned have already studied the transition between memorization and generalization, finite-sample effects, or diffusion training dynamics. The main novelty here is largely extending these analyses to a Transformer-DDPM setting and providing a more unified theoretical characterization, rather than introducing a fundamentally new phenomenon. Compared with recent works on diffusion memorization and generalization, what is the most important new insight provided by the proposed empirical-prototype attraction mechanism?
A1: We thank the reviewer for raising this important point. We agree that the existence of a memorization-generalization transition itself has been observed and analyzed in several recent works, and we have revised the manuscript to avoid presenting this phenomenon alone as the main novelty. Our main contribution is to connect this transition to the finite-sample optimization dynamics of a nonlinear Transformer DDPM and to identify the specific finite-sample statistics learned by gradient descent.
First, our analysis addresses finite-sample gradient-based training of a Transformer denoiser. This distinction is significant because Transformer backbones have become an important scalable architecture for modern diffusion models, while their softmax attention and coupled parameterization lead to a nontrivial nonconvex training problem. Existing analyses [Han et al., 2024; Wang & Pehlevan, 2025; Bonnaire et al., 2025] of diffusion training dynamics mainly rely on simplified architectures. Only [Li et al., 2026] provides a convergence analysis for nonlinear Transformer DDPM training, but considers a population objective and therefore does not characterize finite-sample memorization or generalization. Our work closes this gap by proving convergence under a finite training set and then deriving the resulting generalization gap and sample-specific memorization behavior.
Second, the optimization analysis allows us to characterize what denoiser is actually learned by training. Many existing works [Li et al., 2024; Buchanan et al., 2025; Wang et al., 2026] use Tweedie’s identity only to establish the existence of a minimizer for the denoising objective, but they do not show that the identified minimizer corresponds to the parameters reached by gradient-based training. Our mechanistic understanding is built upon the optimization analysis that characterizes the converged model. The experiments on real-world data in Figure 2(B) further validate the accuracy of the interpolation mechanism described in Proposition 2.
Third, the optimal denoiser characterized in our work more explicitly reflects the effects of practical training and finite samples. Eqn. (11) in [Buchanan et al. 2025] gives a result similar to our Proposition 2. However, since [Buchanan et al., 2025] does not analyze the training dynamics, it cannot further simplify the softmax term. The μi in their Eqn. (11) is interpreted as a learned model parameter, whereas our Proposition 2 further shows that it is close to the empirical mode mean. Moreover, when the number of modes equals to the capacity of the model, [Buchanan et al., 2025] provides an example of a generalizing denoiser. Our work further shows that as the sample size increases, the bias term of the denoiser changes from an individual sample to the empirical mode mean and eventually to the population mode mean. This transition also corresponds to a phase transition from memorizing individual samples, to memorizing finite-sample statistics, and finally to generalization. This is the main new insight of our EPA mechanism.
Han et al., ICLR 2024. Neural network-based score estimation in diffusion models: Optimization and generalization.
Li et al., Neurips 2024. Understanding Generalizability of Diffusion Models Requires Rethinking the Hidden Gaussian Structure
Buchanan et al., Neurips 2025. On the Edge of Memorization in Diffusion Models.
Wang & Pehlevan, Neurips 2025. An analytical theory of spectral bias in the learning dynamics of diffusion models.
Bonnaire et al., Neurips 2025. Why diffusion models don’t memorize: The role of implicit dynamical regularization in training.
Wang et al., JMLR 2026. Breaking the Curse of Dimensionality: Diffusion Models Efficiently Learn Low-Dimensional Distributions