acceptodds
Under review as a conference paper at ICLR 2027

Finite-Sample Training, Generalization, and Memorization in Transformer DDPMs

Abstract

Diffusion models can either generate novel samples or reproduce data from their training samples, exhibiting different generalization and memorization behaviors. However, the theoretical mechanisms underlying these behaviors remain poorly understood. Specifically, it is unclear how finite-sample training shapes the learned denoiser and when it leads to memorization. To the best of our knowledge, this paper provides the first finite-sample convergence analysis of Transformer-based diffusion models trained by gradient descent and a theoretical characterization of the resulting sample-specific memorization performance. We consider learning data from a Gaussian mixture distribution under the Denoising Diffusion Probabilistic Model (DDPM) objective with a one-layer single-head Transformer model. We show that gradient descent learns the empirical mode means and within-mode variance, driving the training loss close to the empirical oracle denoising risk, while the generalization gap diminishes with the sample size. We further derive an explicit Gaussian-CDF characterization of a reconstruction-based sample-specific memorization rate, which yields a sample complexity threshold separating sample-specific memorization from generalization. A more fundamental investigation uncovers an empirical-prototype attraction mechanism underlying memorization in diffusion models, i.e., the trained denoiser interpolates between the scaled noisy input and a learned empirical mode prototype, which transitions from an individual training sample to the population mean as the number of samples increases. Numerical experiments validate these theoretical findings.

Primary area:
learning theory
Keywords:
Diffusion models, Transformers, memorization, generalization, mechanism understanding, optimization

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_5vXwOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies memorization and generalization in Transformer-based diffusion models under finite-sample training. It provides a unified theoretical analysis of gradient-descent convergence, generalization gap, and sample-specific memorization, and further proposes the empirical-prototype attraction mechanism to explain how the denoiser transitions from fitting individual training samples to learning population-level statistics as the dataset grows. The theoretical framework is relatively complete, but the novelty is somewhat incremental compared with recent diffusion memorization theory, and the analysis relies on simplified assumptions such as Gaussian-mixture data and a one-layer single-head Transformer. Overall, the work is theoretically interesting, although my understanding of this theoretical framework is relatively limited.

Critical strengths

The paper provides a unified finite-sample analysis that connects gradient-descent convergence, generalization, and sample-specific memorization for Transformer-based diffusion models, and further gives an explicit memorization-rate characterization together with the empirical-prototype attraction mechanism. This makes the theoretical picture relatively complete rather than analyzing only one isolated aspect.

Critical weaknesses
  1. Several recent works as this work mentioned have already studied the transition between memorization and generalization, finite-sample effects, or diffusion training dynamics. The main novelty here is largely extending these analyses to a Transformer-DDPM setting and providing a more unified theoretical characterization, rather than introducing a fundamentally new phenomenon.

  2. The analysis relies on a one-layer, single-head Transformer and Gaussian-mixture data with the number of learnable prototypes matched to the number of modes. It remains unclear how well the proposed EPA mechanism and sample-complexity conclusions extend to realistic multi-layer DiTs and natural image distributions.

Questions for authors
  1. How well do the theoretical conclusions extend beyond the one-layer, single-head Transformer setting to deeper and more realistic DiT architectures?

  2. How sensitive is the proposed memorization–generalization transition to the Gaussian-mixture assumption? Would a similar phenomenon still appear on more complex natural data distributions?

  3. Compared with recent works on diffusion memorization and generalization, what is the most important new insight provided by the proposed empirical-prototype attraction mechanism?

4 comments

We thank the reviewer for the time and effort in the evaluation. We address the concerns as follows.

Weakness 1 & Question 3: Several recent works as this work mentioned have already studied the transition between memorization and generalization, finite-sample effects, or diffusion training dynamics. The main novelty here is largely extending these analyses to a Transformer-DDPM setting and providing a more unified theoretical characterization, rather than introducing a fundamentally new phenomenon. Compared with recent works on diffusion memorization and generalization, what is the most important new insight provided by the proposed empirical-prototype attraction mechanism?

A1: We thank the reviewer for raising this important point. We agree that the existence of a memorization-generalization transition itself has been observed and analyzed in several recent works, and we have revised the manuscript to avoid presenting this phenomenon alone as the main novelty. Our main contribution is to connect this transition to the finite-sample optimization dynamics of a nonlinear Transformer DDPM and to identify the specific finite-sample statistics learned by gradient descent.

First, our analysis addresses finite-sample gradient-based training of a Transformer denoiser. This distinction is significant because Transformer backbones have become an important scalable architecture for modern diffusion models, while their softmax attention and coupled parameterization lead to a nontrivial nonconvex training problem. Existing analyses [Han et al., 2024; Wang & Pehlevan, 2025; Bonnaire et al., 2025] of diffusion training dynamics mainly rely on simplified architectures. Only [Li et al., 2026] provides a convergence analysis for nonlinear Transformer DDPM training, but considers a population objective and therefore does not characterize finite-sample memorization or generalization. Our work closes this gap by proving convergence under a finite training set and then deriving the resulting generalization gap and sample-specific memorization behavior.

Second, the optimization analysis allows us to characterize what denoiser is actually learned by training. Many existing works [Li et al., 2024; Buchanan et al., 2025; Wang et al., 2026] use Tweedie’s identity only to establish the existence of a minimizer for the denoising objective, but they do not show that the identified minimizer corresponds to the parameters reached by gradient-based training. Our mechanistic understanding is built upon the optimization analysis that characterizes the converged model. The experiments on real-world data in Figure 2(B) further validate the accuracy of the interpolation mechanism described in Proposition 2.

Third, the optimal denoiser characterized in our work more explicitly reflects the effects of practical training and finite samples. Eqn. (11) in [Buchanan et al. 2025] gives a result similar to our Proposition 2. However, since [Buchanan et al., 2025] does not analyze the training dynamics, it cannot further simplify the softmax term. The μi\mu^i in their Eqn. (11) is interpreted as a learned model parameter, whereas our Proposition 2 further shows that it is close to the empirical mode mean. Moreover, when the number of modes equals to the capacity of the model, [Buchanan et al., 2025] provides an example of a generalizing denoiser. Our work further shows that as the sample size increases, the bias term of the denoiser changes from an individual sample to the empirical mode mean and eventually to the population mode mean. This transition also corresponds to a phase transition from memorizing individual samples, to memorizing finite-sample statistics, and finally to generalization. This is the main new insight of our EPA mechanism.

Han et al., ICLR 2024. Neural network-based score estimation in diffusion models: Optimization and generalization.

Li et al., Neurips 2024. Understanding Generalizability of Diffusion Models Requires Rethinking the Hidden Gaussian Structure

Buchanan et al., Neurips 2025. On the Edge of Memorization in Diffusion Models.

Wang & Pehlevan, Neurips 2025. An analytical theory of spectral bias in the learning dynamics of diffusion models.

Bonnaire et al., Neurips 2025. Why diffusion models don’t memorize: The role of implicit dynamical regularization in training.

Wang et al., JMLR 2026. Breaking the Curse of Dimensionality: Diffusion Models Efficiently Learn Low-Dimensional Distributions

Weakness 2 & Question 1: It remains unclear how well the proposed EPA mechanism and sample-complexity conclusions extend to realistic multi-layer DiTs and natural image distributions. How well do the theoretical conclusions extend beyond the one-layer, single-head Transformer setting to deeper and more realistic DiT architectures?

A2: This is a great question. First, we would like to clarify that analyzing multi-layer Transformers, or even multi-layer neural networks in general, is extremely challenging, while the one-layer case represents a stepping stone towards the multi-layer case. To the best of our knowledge, no existing works analyze the convergence of multi-layer nonlinear Transformers in DDPM training, due to the highly non-convex nature of the problem. Up till now, one-layer nonlinear Transformers are still among the most complicated models that can currently be analyzed theoretically in the family of Transformer models. In addition, we believe that our current analysis of a one-layer Transformer is useful for future extensions to multi-layer settings. A potential direction is to build upon our analysis of how a one-layer model learns a Gaussian mixture distribution, and extend it to characterize how multi-layer Transformers learn hierarchical mixture distributions that can represent features with multiple scales and levels. Therefore, our current analysis of a one-layer nonlinear Transformer is a crucial first step.

Second, we verify that our proposed memorization mechanism also holds on CIFAR-10 using a 12-layer DiT model. Specifically, we linearly fit the noisy inputs and denoised outputs to estimate the coefficient vtNv_t^N and the bias term μˉm\bar{\mu}_m in Proposition 2. We further apply K-means clustering to the data features to identify different modes. By splitting the data into training and test sets, we can approximately compute the empirical mode means and the true mode means. The left panel of Figure 2(B) shows the fitted vtNv_t^N as a function of tt under different sample sizes. We observe that when NN is large, the empirically fitted vtNv_t^N curve aligns well with the theoretical prediction. The right panel of Figure 2(B) reports the difference between the fitting errors of the estimated bias term with respect to the true mode mean and the empirical mode mean. Denote the estimated bias term as z^m\hat{z}_m. The right panel of Figure 2(B) essentially computes ∑m=1M∣∣z^m−μm∣∣2/(dM)−∑m=1M∣∣z^m−μˉm∣∣2/(dM)\sum_{m=1}^M ||\hat{z}_m-\mu_m||^2/(dM)-\sum_{m=1}^M ||\hat{z}_m-\bar{\mu}_m||^2/(dM). The results show that the fitted bias term is closer to the empirical mean. Together, the results in Figure 2(B) demonstrate that the EPA mechanism characterized in Proposition 2 also holds for real-world data distributions and DiT models.

Regarding sample complexity, the left panel of Figure 2(A) shows that once the sample size NN exceeds a certain threshold, the memorization rate drops rapidly from 1. This threshold plays a similar role to the sample complexity in Corollary 2 that separates memorization and generalization. This indicates that, under real-world data and deep DiT models, the memorization rate follows the same theoretical trend characterized in Corollary 2.

Question 2: How sensitive is the proposed memorization–generalization transition to the Gaussian-mixture assumption? Would a similar phenomenon still appear on more complex natural data distributions?

A3: We thank the reviewer for this question. Gaussian mixture models have been widely used in prior works [Li et al., 2025; Wang et al., 2026; Li et al., 2026] on memorization and generalization in diffusion models, but these works do not provide a quantitative characterization of the memorization rate. Our Theorem 2 provides, to the best of our knowledge, the first quantitative characterization of the memorization rate. We do not claim that the same closed-form threshold must hold for arbitrary natural-data distributions. Our experiments, however, suggest that the qualitative phenomenon of memorization to generalization persists well beyond the theoretical GMM setting.

Specifically, we train a DiT model on CIFAR-10 images, whose distribution is not a Gaussian mixture. We still observe the same qualitative phenomenon, a transition from memorization to generalization. Figure 2(A) shows that once the number of training data exceeds a certain threshold, the memorization rate rapidly drops from 1, indicating that the model no longer perfectly reconstructs the training samples. At the same time, as the number of training data increases, the reconstruction error on the training set gradually increases, while the error on the testing set decreases correspondingly, and the two become nearly identical around that threshold. Figure 5 in the Appendix further reflects this phenomenon by plotting the curve of the memorization rate against β\beta.

These results suggest that although the exact quantitative transition derived in our theory depends on the Gaussian-mixture assumptions, a similar phenomenon also appears in standard diffusion models trained on complex natural-image data. This phenomenon can be summarized as: when the sample size is small, the model reconstructs training samples with a high memorization rate, whereas with a large sample size, it achieves a low memorization rate and a smaller reconstruction error on unseen samples. We have clarified this part in the revised manuscript.

Li et al., Neurips 2025. Understanding Representation Dynamics of Diffusion Models via Low-Dimensional Modeling

Wang et al., JMLR 2026. Breaking the Curse of Dimensionality: Diffusion Models Efficiently Learn Low-Dimensional Distributions

Li et al., ICML 2026. Transformers Learn the Optimal DDPM Denoiser for Multi-Token GMMs.

Thanks for the rebuttal, I am not familiar with the theory of diffusion models, so I keep the low confidence score and suggest other reviewers to discuss how the rebuttal has solved their concerns.

Reviewer_wYJ6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper addresses the problem of memorization in Transformer-based DDPMs. It provides the first finite-sample convergence analysis of Transformer-based DMs. The authors show that the model learns the empirical mode means and within-mode variance, and that the generalization gap diminishes with the sample size, agreeing with existing large empirical evidence.

Critical strengths
  • The paper provides the first theoretical analysis of memorization for Transformer-based DMs and a useful mechanism that provides explanations about the memorization phenomenon (EPA)
  • The simple experiments validate the theoretical findings
Critical weaknesses
  • Although the second experiment can be considered as a "real-data experiment", in practice, the scenario considered is still a mixture of Gaussians
Questions for authors
  • Can you run an experiment where the DiT is trained on raw CIFAR-10 images?
  • What do you expect to happen if the number of prototypes does not match the number of underlying modes? Can you run an empirical experiment on such a scenario?
1 comment

We thank the reviewer for the time and effort in the evaluation. We address the concerns as follows.

Weakness & Question 1: Although the second experiment can be considered as a "real-data experiment", in practice, the scenario considered is still a mixture of Gaussians. Can you run an experiment where the DiT is trained on raw CIFAR-10 images?

A1: We thank the reviewer for this suggestion. We have added a new experiment in which the DiT is trained directly on raw CIFAR-10 images, without using DINOv2 features and K-means clustering to fit a Gaussian-mixture model. Specifically, we construct class-balanced subsets of CIFAR-10 with N∈10,80,640,5120,40960N\in\\{10,80,640,5120,40960\\} for training. We also keep a fixed held-out set of 2,0002,000 images for testing.

The new results IMG_rate and IMG_risk show a clear memorization-to-generalization transition. In the small-data regime, e.g., at N=10N=10, the memorization rate on training images is 11, whereas the corresponding rate for testing data is 00. The empirical reconstruction risk for training data is also nearly zero, while the testing risk is much larger. As the number of training samples increases, the memorization rate on training data decreases, while the rate on testing data increases. Likewise, the reconstruction risk on training data increases, while the risk on testing data decreases. In the large-data regime, training and unseen samples become nearly indistinguishable under both the memorization-rate and reconstruction-risk metrics. This result is aligned with our CIFAR-10 results in Figure 2 after using K-means clustering.

These results indicate that the qualitative memorization-to-generalization phenomenon does not only hold for the Gaussian mixture data or data fitted to the mixture model. In contrast, it also appears when a standard DiT is trained directly on raw natural images. We have added these results to the revised manuscript.

Question 2: What do you expect to happen if the number of prototypes does not match the number of underlying modes? Can you run an empirical experiment on such a scenario?

A2: We thank the reviewer for this suggestion. We added a controlled experiment in which the data distribution contains M=10M=10 true Gaussian modes, while the Transformer denoiser uses K∈{5,10,15}K\in\{5,10,15\} learnable prototypes. All other settings are kept identical across the three values of KK.

The results can be found in IMG_varyingK. The number of training samples is chosen as 120120 in (A) and (B). When K=5<MK=5<M, the number of prototypes is insufficient to learn all the modes. The result shows that the training loss remains substantially higher than other cases and does not approach the risk attained by K=10K=10. Meanwhile, the K=5K=5 model exhibits essentially no sample-specific memorization across the tested sample sizes. Its memorization curve as a function of the tolerance is also shifted toward larger tolerances, indicating poorer reconstruction of individual training samples. In contrast, when K=10=MK=10=M, the model can learn the underlying mode structure given the choice of other parameters. When K=15>MK=15>M, the training loss is slightly lower than in the K=10K=10 case, while the memorization rate is slightly higher under the same tolerance or sample size.

These results show that when the number of prototypes is smaller than the number of modes, the model lacks sufficient capacity to learn all modes, resulting in larger reconstruction errors. When the number of prototypes exceeds the number of modes, the training loss becomes smaller, and the model exhibits stronger memorization.

Reviewer_wWHUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies finite-sample training, generalization, and reconstruction-based memorization for a diffusion denoiser trained on a mixture of spherical Gaussians. The model combines a residual connection with Gaussian-kernel attention over learnable prototypes and a learnable scalar coefficient at each diffusion timestep. Under local initialization and high-dimensional assumptions, the paper claims that gradient descent learns the empirical component means and timestep-dependent coefficients determined by the pooled within-component variance, achieves a training risk close to an empirical oracle denoising risk, and exhibits a decreasing generalization gap. It further proposes a Gaussian-CDF characterization of a sample-specific reconstruction metric and a critical sample-size scale of N=Θ(Md)N=\Theta(M\sqrt d). The interpretation is an empirical-prototype attraction mechanism. Synthetic GMM experiments and CIFAR-10 experiments with a DiT investigate these predictions.

Critical strengths
  1. An interesting attempt to connect optimization, finite-data statistics, and memorization. The paper aims to explain not only what an optimal denoiser looks like, but also which statistics a trained model retains and how those statistics affect individual training samples’ reconstruction probabilities. A rigorous connection between these aspects would be informative and consequential.

  2. An interpretable setting with complementary empirical investigations. The GMM setting makes the proposed mechanism explicit. The experiments compare local and K-means initialization, examine training versus held-out reconstruction, and include sensitivity analyses for the proposed memorization metric. These are useful investigations.

Critical weaknesses
  1. Several important steps in the theoretical proofs need further justification. My main concern is Appendix C.3, particularly Eqs. (39), (42), and (44)–(45). The proof centers the squared residual of an individual noisy sample at dρ^N2d\hat\rho_N^2, where ρ^N2\hat\rho_N^2 is the pooled empirical within-component variance. However, even for clean samples, the expected squared residual depends on the component occupancy NmN_m, whereas the pooled variance averages over all components. Moreover, xtix_t^i includes additional forward noise. It is unclear how the stated assumptions justify this concentration bound for individual noisy samples, particularly when component occupancies are unequal. Moreover, Eqs. (34)–(35) appear to average over forward noise while conditioning on xtix_t^i, whereas Definition 1 fixes x0ix_0^i. Eq. (44) also provides only a sufficient condition, but Eq. (45) states a probability equality without justifying the reverse implication or bounding the approximation error. These issues raise concerns about the Gaussian-CDF characterization in Theorem 2. I also have concerns about the optimization analysis. Eq. (3) uses finite forward-noise samples, but Eq. (95) appears to omit their empirical noise contribution, making the claimed convergence to the empirical means unclear unless the objective averages over noise. Additionally, the correct-component attention argument around Eq. (67) appears to require positive λt\lambda_t, although its initialization is a zero-mean Gaussian. These steps need further clarification.

  2. The novelty and quantitative experimental validation could be better established. The related-work discussion does not sufficiently distinguish the contribution from closely related studies. In particular, [4] already studies gradient-based learning of GMM parameters under the DDPM objective, while [5] analyzes Gaussian denoisers characterized by empirical means and covariances. A clearer comparison would help identify the novel contributions in convergence analysis, learned variance, and memorization characterization. The broadly titled theoretical related-work paragraph could also be more comprehensive. For experimental validation, the synthetic experiments fix d=128d=128 and M=10M=10, while the claimed memorization threshold scales as N=Θ(Md)N=\Theta(M\sqrt d). Varying only NN does not directly verify this scaling. Experiments varying MM and dd, together with a quantitative comparison between Eq. (15) and the measured memorization curves, would provide stronger support for the theoretical claims.

[1] Kazusato Oko, Shunta Akiyama, and Taiji Suzuki. Diffusion Models are Minimax Optimal Distribution Estimators. ICML, 2023.

[2] Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru R. Zhang. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. ICLR, 2023.

[3] Minshuo Chen, Kaixuan Huang, Tuo Zhao, and Mengdi Wang. Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Data. ICML, 2023.

[4] Kulin Shah, Sitan Chen, and Adam Klivans. Learning Mixtures of Gaussians Using the DDPM Objective. NeurIPS, 2023.

[5] Xiang Li, Yixiang Dai, and Qing Qu. Understanding Generalizability of Diffusion Models Requires Rethinking the Hidden Gaussian Structure. NeurIPS, 2024.

Questions for authors
  1. Clarification of the memorization proof. How are Eqs. (39), (42), and the Gaussian approximation in Eq. (45) justified when the empirical component occupancies are unequal? Does Theorem 2 require balanced sampling? Please also clarify whether Eqs. (34)–(35) condition on the clean sample and training set or on the noisy observation, and justify the probability equality following the sufficient condition in Eq. (44). Please also explain how Eqs. (38) and (39) are derived.

  2. Clarification of the optimization setup. Equation (3) defines the empirical DDPM loss using sampled forward noises, while Eq. (4) analyzes gradient descent on this objective. However, the subsequent convergence analysis appears to rely on averaging out certain noise-dependent gradient terms. It would be helpful to clarify whether the theoretical analysis considers the noise-averaged empirical objective or stochastic gradient descent with freshly sampled noises, and how the corresponding gradient fluctuations are controlled. How is correct-component attention established from the stated initialization, which allows negative λt(0)\lambda_t^{(0)}?

  3. Compatibility of the theorem assumptions and asymptotic interpretations. What joint scaling of N,d,M,ε,δN,d,M,\varepsilon,\delta is intended? For fixed d,M,δd,M,\delta, Theorem 1 requires Nlog⁡(N/δ)≲dM2N\log(N/\delta)\lesssim dM^2, rather than allowing arbitrary growth in NN. Moreover, substituting the choice ε=Θ(d−1/2)\varepsilon=\Theta(d^{-1/2}) discussed in lines 317–319 into the same theorem requires 1≳Mlog⁡(N/δ)1\gtrsim M\log(N/\delta). How is the claimed connection between the corollaries established under compatible assumptions?

  4. Positioning and validation of the current claims. What are the specific technical advances over the GMM optimization results of Shah et al. [4] and the empirical Gaussian-denoiser perspective of Li et al. [5]? Which results directly validate the quantitative CDF approximation and the MdM\sqrt d scaling, rather than only the qualitative decrease of memorization with sample size?

Minor Issue on Presentation and Notation

  1. Data-distribution notation. Please explicitly define π=(π1,…,πM)\pi=(\pi_1,\ldots,\pi_M), πm=Pr⁡(Y=m)\pi_m=\Pr(Y=m), and the probability simplex ΔM−1\Delta^{M-1}. Although the intended meaning is inferable from the categorical distribution, the notation should be self-contained. Likewise, introduce the latent label YiY_i associated with each observation x0ix_0^i before using it to define NmN_m in Section 3. Appendix C.3 subsequently reuses YiY_i for a reconstruction loss in Eq. (32); a different symbol should be used there.

  2. Learning-model motivation and organization. The "Learning model" paragraph should first state that f(Θ;xt,t)f(\Theta;x_t,t) predicts the clean observation x0x_0, rather than the noise or score. The forward noising process should be defined before invoking its posterior mean to motivate Eq. (2). The paragraph does provide a posterior-mean motivation, but its current ordering leaves the prediction target and the relationship between the probabilistic model and the learnable architecture insufficiently explicit.

  3. Explanation of Theorem 1's conditions. The "Data dimension" paragraph refers to "the first lower bound" and "the second lower bound" in an order different from their appearance in the theorem. In addition, it writes the accuracy-related term as ε−2log⁡(N/δ)\varepsilon^{-2}\log(N/\delta), whereas the theorem contains Mε−2log⁡(N/δ)M\varepsilon^{-2}\log(N/\delta).

  4. Gaussian-CDF notation. Please define Ψ\Psi and Ψ−1\Psi^{-1} before their first use in Theorem 2. The subsequent paragraph suggests that Ψ\Psi is the standard normal CDF, but the theorem should not require the reader to infer this afterward.

  5. Posterior-mean derivation and typographical errors. In Appendix D.1, Eq. (59), Tweedie's formula requires ∇xlog⁡pt(x)\nabla_x\log p_t(x), not ∇xpt(x)\nabla_xp_t(x). The following expansion appears to use the correct log-density gradient, so this may be a typo rather than an error in the final architecture. Please also correct E[x0∣xtx]E[x_0\mid x_t x] to E[x0∣xt=x]E[x_0\mid x_t=x], zcz_c to zmz_m, and the inconsistent summation index CC versus MM. The phrases describing "a noisy data xtx_t obtained from xtx_t" in Definition 2 and the clean/noisy notation in Proposition 2 should likewise be corrected.

[4] Kulin Shah, Sitan Chen, and Adam Klivans. Learning Mixtures of Gaussians Using the DDPM Objective. NeurIPS, 2023.

[5] Xiang Li, Yixiang Dai, and Qing Qu. Understanding Generalizability of Diffusion Models Requires Rethinking the Hidden Gaussian Structure. NeurIPS, 2024.

5 comments

We thank the reviewer for the time and effort in the evaluation. We address the concerns as follows.

Weakness 1 & Question 1: Clarification of the memorization proof. How are Eqs. (39), (42), and the Gaussian approximation in Eq. (45) justified when the empirical component occupancies are unequal? Does Theorem 2 require balanced sampling? Please also clarify whether Eqs. (34)–(35) condition on the clean sample and training set or on the noisy observation, and justify the probability equality following the sufficient condition in Eq. (44). Please also explain how Eqs. (38) and (39) are derived.

A1: We thank the reviewer for identifying these issues. We have carefully revised this part of the proof.

Unequal component occupancies and Eqns. (38), (39), and (42). After reviewing the paper, we agree that the variance of within-component residuals depends on NmN_m. To make the explicit characterization of memorization rate in Theorem 2 rigorous, we have added the condition of balanced component occupancies, i.e., Nm=N/MN_m=N/M for all m∈[M]m\in[M] in Theorem 2. Note that this condition is only required for Theorem 2 and does not affect the optimization and generalization results in Theorem 1.

We then explain how explanation Eqns. (38) and (39) are derived. We have revised them to be the (40) and (41) in the updated version, respectively. We have first changed the xti−zm(S)x_t^i-z_m^{(S)} from the old version into x0i−μˉmx_0^i-\bar\mu_m. The discrepancy zm(S)−μˉmz_m^{(S)}-\bar{\mu}_m is instead included in the error term e~it\tilde{e}_i^t in Eqn. (33). Note that for any training sample from mode mm, we have $x_0^i-\bar{\mu}_m \sim\mathcal{N}(0,\rho^2(1-M/N)I_d$. Eqn. (38) (Eqn. (40) in the revision) uses Gaussian concentration to control the coordinate-wise fluctuations of x0i−μˉmx_0^i-\bar{\mu}_m. Eqn. (39) (Eqn. (41) in the revision) comes from the gap between ∥x0i−μˉm∥2\|x_0^i-\bar\mu_m\|^2 and dρ2(1−M/N)d\rho^2(1-M/N) by Laurent-Massart inequality and the gap between dρ2(1−M/N)d\rho^2(1-M/N) and ρ^N2\hat{\rho}_N^2 by the relationship between ρ2\rho^2 and ρ^N2\hat{\rho}_N^2.

Conditioning in Eqns. (34) and (35). We agree that the conditioning in the original presentation needs revision. Consistently with Definition 1, the clean training sample x0ix_0^i and the realized training set R\mathcal R are fixed, and the inner probability is only over a forward noise ϵt\epsilon_t. We therefore redefine mY(x0i;R)=Eϵt[Yi∣x0i,R]m_Y(x_0^i;\mathcal{R})=\mathbb{E}_{\epsilon_t}[Y_i\mid x_0^i,\mathcal{R}], and sY2(x0i;R)=Varϵt(Yi∣x0i,R)s_Y^2(x_0^i;\mathcal{R})=\text{Var}_{\epsilon_t}(Y_i\mid x_0^i,\mathcal{R}). Together with the bound on e~it\tilde{e}_i^t, this yields the revised conditional mean and variance in Eqns. (34)–(35), including the term from the optimization error.

Derivation of Eqns. (44) and (45). We agree that the previous proof only computes a sufficient condition and cannot lead to the probability equality. In the revision, after the first Berry-Esseen approximation of g(Θ(S);β,x0i,t)g(\Theta^{(S)};\beta,x_0^i,t), we explicitly derive both a sufficient and a necessary condition for $g(\Theta^{(S)};\beta,x_0^i,t)\geq 1-\gamma $. Specifically, for b=Cd−1/2b=Cd^{-1/2} for some constant C>0C>0, the two conditions become the two ends of {Hi≥−Ψ−1(γ−b)}⊆{g(Θ(S);β,x0i,t)≥1−γ}⊆{Hi≥−Ψ−1(γ+b)}\{H_i\geq-\Psi^{-1}(\gamma-b)\}\subseteq\{g(\Theta^{(S)};\beta,x_0^i,t)\geq1-\gamma\} \subseteq \{H_i\geq-\Psi^{-1}(\gamma+b)\}, where HiH_i represents the standardized margin between β\beta and the distribution center of reconstruction-error for the ii-th training sample. Since that the Gaussian CDF and its inverse function is globally Lipschitz continuous, the probabilities associated with the two bounding events differ by only O(d−1/2)O(d^{-1/2}) after applying the Berry-Esseen approximation. Thus, this yields the probability equality in the revised Eqn. (45) (Eqn. (51) of the updated version), up to the stated O(d−1/2+ϵ)O(d^{-1/2}+\epsilon) approximation error.

Weakness 1 & Question 2: It would be helpful to clarify whether the theoretical analysis considers the noise-averaged empirical objective or stochastic gradient descent with freshly sampled noise. How is correct-component attention established from the stated initialization, which allows negative λt(0)\lambda_t^{(0)}?

A2: We thank the reviewer for pointing out these two issues. We clarify and revise them separately.

Forward noise averaging in the optimization analysis. We agree with the reviewer’s underlying concern. Our intended theoretical analysis considers finite-sample training with respect to the NN clean training examples, while averaging over the Gaussian corruption noise. More precisely, we have revised Eqn. (3) as $ L_{N,T}(\Theta) := \frac{1}{2NT}\sum_{t=1}^T\sum_{n=1}^N \mathbb{E}_{\epsilon_t}||f(\Theta;\sqrt{\bar{\alpha}_t}x_0^n+\sqrt{1-\bar{\alpha}_t}\epsilon_t^n, t)-x_0^n||^2.$ This formulation is consistent with prior works [Buchanan et al., 2025; Ye et al., 2026; Zhang et al., 2026]. We have also made revisions across the paper. Since this update can reduce some error terms when using the concentration bound, our current theoretical results still hold.

The initialization of λt\lambda_t. We also agree that our initial Eqn. (67) indeed requires λt(0)>0\lambda_t^{(0)}>0. We have corrected this in the manuscript. For simplicity, we set λt(0)=1\lambda_t^{(0)}=1. A practical justification of this initialization is the learnable temperature parameter that is initialized as positive. Specifically, as stated in Remark 1, after expanding the input to the softmax function, the attention logits contain λtαˉtzm⊤x\lambda_t\sqrt{\bar{\alpha}_t}z_m^\top x, which corresponds to scaled dot-product attention. This means that our λt\lambda_t plays the role of the inverse temperature in dot-product attention. In [Radford et al., 2021], the learnable temperature is initialized to 0.07>0, which is consistent with our initialization.

Buchanan et al., Neurips 2025. On the Edge of Memorization in Diffusion Models.

Ye et al., ICLR 2026. Provable separations between memorization and generalization in diffusion models.

Zhang et al., ICLR 2026. Generalization of Diffusion Models Arises with a Balanced Representation Space

Radford et al., 2021. Learning Transferable Visual Models From Natural Language Supervision.

Weakness 2 & Question 4 Part I: What are the specific technical advances over the GMM optimization results of Shah et al. [4] and the empirical Gaussian-denoiser perspective of Li et al. [5]?

A3: Great question! We summarize the differences and technical advances between our work and [4,5] as follows.

Comparison with [4]. First, our work characterizes the convergence of the empirical DDPM objective under finite training data, whereas [4] studies the population DDPM loss. This requires us to derive finite-sample error bounds for the gradients and characterize their impact on convergence. We show that the empirical DDPM loss converges to the empirical oracle denoising risk and characterize the required sample size, number of iterations, dimension, and learning rate. In contrast, [4] does not study convergence in the finite-sample regime. Second, our work jointly optimizes the prototypes zmm=1M\\{z_m\\}_{m=1}^M and coefficients λtt=1T\\{\lambda_t\\}_{t=1}^T, whereas [4] only learns the prototypes. We consider the more general setting where the Gaussian mixture variance is unknown and must also be learned through λtt=1T\\{\lambda_t\\}_{t=1}^T. This joint optimization makes the problem more challenging. We leverage prototype initialization to simplify the loss landscape and characterize the local convergence of each parameter. Third, our work analyzes both generalization and memorization, which are not considered in [4]. We characterize the memorization rate and derive the critical sample complexity separating the memorization and generalization regimes, where a key step is analyzing the distribution of the reconstruction error. Since [4] does not consider finite-sample training, it does not address memorization or generalization.

Comparison with [5]. First, we provide a convergence analysis of the DDPM loss under finite training data, whereas [5] only characterizes the optimal denoiser without analyzing convergence. Second, we consider a nonlinear Transformer with softmax attention, while [5] studies a linear denoiser. Our nonlinear model leads to a substantially more challenging nonconvex optimization problem. Third, our work provides, for the first time, a theoretical characterization of the sample-specific memorization rate and derives the sample complexity separating memorization and generalization, neither of which is theoretically characterized in [5].

We agree with the reviewer that these comparisons help highlight our novel contributions. We have incorporated these revisions into the updated manuscript.

Weakness 2 & Question 4 Part II: Which results directly validate the quantitative CDF approximation and the MdM\sqrt{d} scaling, rather than only the qualitative decrease of memorization with sample size? Experiments varying MM and dd, together with a quantitative comparison between Eq. (15) and the measured memorization curves, would provide stronger support for the theoretical claims.

A4: We thank the reviewer for this suggestion. We agree that the original experiments primarily varied NN with fixed dd and MM. We have added new synthetic experiments that directly vary dd and MM, and compare the empirical memorization rates with the theoretical prediction from Theorem 2, without fitting any parameters.

Varying dimension dd. In IMG_d, we vary dd from 128 to 1024 while fixing the other parameters. The measured memorization rates closely follow the theoretical prediction, with an RMSE of 0.0093.

Varying the number of modes MM. In IMG_M, we vary MM from 5 to 60 while fixing the other parameters. The empirical results again closely match Theorem 2 with an RMSE of 0.0148.

Joint scaling with N/(Mρ2d)N/(M\rho^2\sqrt d). We further vary NN, MM, and dd jointly and plot the memorization rate against the ratio N/(Mρ2d)N/(M\rho^2\sqrt d) as in Corollary 2. The results IMG_joint across different settings exhibit the predicted transition, and the theoretical predictions remain close to the empirical memorization rates, with an RMSE of 0.01820.0182 over 48 joint settings.

These results directly validate the predicted dependence on dd and MM, as well as the sample-complexity scale N=Θ(Mρ2d)N=\Theta(M\rho^2\sqrt d). We have included these quantitative comparisons in the revised manuscript.

Question 3: What joint scaling of NN, dd, MM, ϵ\epsilon, δ\delta is intended? For fixed dd, MM, δ\delta, Theorem 1 requires Nlog⁡(N/δ)≲dM2N\log(N/\delta)\lesssim dM^2, rather than allowing arbitrary growth in NN. Moreover, substituting the choice ϵ=Θ(d−1/2)\epsilon=\Theta(d^{-1/2}) discussed in lines 317–319 into the same theorem requires 1≳Mlog⁡(Nδ)1\gtrsim M\log(N\delta). How is the claimed connection between the corollaries established under compatible assumptions?

A5: We thank the reviewer for pointing out this issue. We agree that the joint asymptotic regime was not stated sufficiently clearly in the original manuscript.

First, we would like to clarify that our results are intended in a joint high-dimensional regime rather than an N→∞N\to\infty limit with fixed dd. Since we have added the expectation over the forward noise to the training objective in the revised Eqn. (3), the error terms in the gradient of λt\lambda_t can be bounded to be tighter, such that the condition (ii) of Theorem 1 is updated as d≳NM−2log⁡(N/δ)d\gtrsim NM^{-2}\log(N/\delta), or equivalently, Nlog⁡(N/δ)≲dM2N\log(N/\delta)\lesssim dM^2. Thus, N,d,M,δN,d,M,\delta are allowed to scale jointly subject to this condition. In particular, the regime relevant to Corollary 2, N=Θ(Md)N=\Theta(M\sqrt d) is compatible with Theorem 1 whenever log⁡(Md/δ)=O(Md)\log(M\sqrt{d}/\delta)=O(M\sqrt d), which includes, for example, fixed M,δM,\delta as d→∞d\to\infty.

Second, the issue of 1≳Mlog⁡(Nδ)1\gtrsim M\log(N\delta) with ϵ=Θ(d−1/2)\epsilon=\Theta(d^{-1/2}) comes from a loose dimension requirement in the original proof. After revision, Theorem 1 has been updated, and the current dimension condition does not rely on ϵ\epsilon. Thus, the choice of ϵ\epsilon no longer imposes an additional polynomial dimension requirement.

Minor Issue on Presentation and Notation

A6}: We thank the reviewer for pointing out these issues. We have made revisions accordingly.

Data-distribution notation: We have updated the illustration as "π=(π1,…,πM)\boldsymbol{\pi}=(\pi_1,\ldots,\pi_M) denotes the mixture weights, with πm=Pr⁡(Y=m)=1/M\pi_m=\Pr(Y=m)=1/M for any m∈[M]m\in[M] for simplicity in this work." We have also commented that the latent variable YiY^i is actually the mode-membership variable of x0ix_0^i introduced in Eqn. 1.

Learning-model motivation and organization: We have added the explanation that the learning model aims to predict the clean sample x0x_0. We have also reorganized the presentation of Section 2. In the updated version, we start by introducing the forward noising process of DDPM so that we can include the noise schedule when proposing the learning model.

Explanation of Theorem 1's conditions} We have removed Mϵ−2log⁡(N/δ)M\epsilon^{-2}\log (N/\delta) in the updated condition for data dimension condition. This issue is therefore addressed.

Gaussian-CDF notation: In the revision, we have added the definition of Ψ(⋅)\Psi(\cdot) and its inverse function Ψ−1(⋅)\Psi^{-1}(\cdot) in red words before Theorem 2.

Posterior-mean derivation and typographical errors: Thank you for the careful reading. We have revised them accordingly.

Reviewer_hVGSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates finite-sample training, generalization, and memorization in Transformer-based diffusion models. Using a Gaussian mixture model and a one-layer, single-head Transformer, the authors analyze gradient-descent convergence toward empirical denoising statistics. They derive a Gaussian-CDF characterization of sample-specific reconstruction memorization and identify a sample-complexity threshold. The proposed empirical-prototype attraction mechanism explains how learned prototypes transition from individual training samples toward population means as the dataset grows. The theoretical findings are supported by synthetic experiments and experiments with DiT on CIFAR-10.

Critical strengths

The paper's idea is interesting. it addresses an important question: how finite-sample training relates to memorization and generalization in diffusion models. Particularly, connecting gradient-descent convergence with learned empirical statistics and reconstruction behavior provides an interesting theoretical perspective. In addition, the paper goes beyond qualitative observations by deriving a sample-specific memorization rate and a sample-complexity threshold. The authors also provide experiments on both synthetic data and CIFAR-10, including held-out reconstruction comparisons.

Critical weaknesses

I have several concerns as follows: The theoretical analysis relies on a highly structured Gaussian mixture model with orthogonal mode means, shared isotropic variance, and locally initialized prototypes. Furthermore, the Transformer architecture is explicitly motivated by the Gaussian posterior mean, and the interpolation underlying EPA is closely related to Bayesian denoising. While these assumptions are reasonable for theoretical analysis, it remains unclear which aspects of the proposed mechanism are genuinely new and how broadly the findings apply to general diffusion models. Additionally, the paper derives a Gaussian-CDF memorization rate and a sample-complexity threshold. However, i believe that the synthetic experiments mainly examine changes in training-set size while keeping the dimension and number of modes fixed. Therefore, the predicted scaling is not directly validated.

Questions for authors

Accordingly, i have following questions:

  • Since the model architecture is derived from the Gaussian posterior mean, what is the key new insight provided by EPA beyond classical Bayesian denoising? Is the main contribution the finite-sample convergence analysis rather than the interpolation mechanism itself?
  • How essential are orthogonal Gaussian means, shared isotropic variance, and prototype initialization near empirical means? Are there experiments showing whether the proposed behavior persists when these assumptions are relaxed?
  • Could the authors provide experiments varying both the data dimension (d) and the number of modes (M)? A quantitative comparison between the predicted Gaussian-CDF and empirical memorization curves would also strengthen the results.
3 comments

We thank the reviewer for the time and effort in the evaluation. We address the concerns as follows.

Weakness & Question 1: Furthermore, the Transformer architecture is explicitly motivated by the Gaussian posterior mean, and the interpolation underlying EPA is closely related to Bayesian denoising. It remains unclear which aspects of the proposed mechanism are new and how broadly the findings apply to general diffusion models. What is the key new insight provided by EPA beyond classical Bayesian denoising? Is the main contribution the finite-sample convergence analysis rather than the interpolation mechanism itself?

A1: We thank the reviewer for raising this question. We agree that it is important to distinguish the classical Bayesian denoising from the EPA mechanism introduced in our work.

Relation between EPA and classical Bayesian denoising. Bayesian denoising refers to that, under squared loss as the denoising objective, is the posterior mean is computed as the Bayes-optimal denoiser. For an isotropic Gaussian component, this posterior mean indeed interpolates between the noisy observation and the population component mean. For a Gaussian mixture, the posterior additionally involves the component posterior probabilities. Therefore, we agree that the interpolation underlying EPA is closely related to Bayesian denoising and do not view the interpolation form itself as the novelty of EPA.

The key distinction is that, in our learning problem, the distribution parameters are unknown and the model is trained from a finite dataset by gradient descent. We prove that the Transformer model can be trained to learn data-dependent finite-sample statistics, including the prototype parameter that converges to the empirical mode mean μˉm\bar{\mu}_m, while the denoising coefficient is determined by the empirical within-mode variance. In the updated version of Proposition 2, we characterize the learned prototype parameters under different sample complexities, thereby providing a theoretical explanation for the distinct generation mechanisms associated with memorization and generalization.

Moreover, we would like to clarify our contributions as follows. First, we provide a finite-sample convergence analysis for gradient-based training of a nonlinear softmax-attention Transformer under the DDPM objective. We show that the training loss converges towards the empirical oracle denoising risk. This differs from prior diffusion model analysis with Transformers that characterizes population-level optimal denoising but does not capture data-dependent memorization [Li et al., 2026]. Second, we derive a quantitative sample-specific memorization law, i.e., the memorization rate is approximated by an explicit Gaussian CDF, which can characterize a critical sample complexity separating sample-specific memorization from generalization. This goes beyond only identifying an optimal posterior denoiser [Buchanan et al., 2025; Li et al., 2025; Wang et al., 2026; Zhang et al., 2026; Ye et al., ICLR 2026] without showing how finite-data training produces such behavior. Third, the above analysis leads to the EPA mechanism for generation. We emphasize that the contribution is not merely the existence of an interpolation formula. Rather, we prove that training a Transformer with unknown distribution parameters produces this interpolation based on the prototypes learned by the empirical DDPM objective, and we characterize how the prototypes themselves change with sample size. This connects the learned representation of the Transformer model directly to different phases of generalization and memorization, which differs from prior work that either assumes the distribution parameters are known [Buchanan et al., 2025] or does not analyze the training dynamics of Transformer models [Li et al., 2024; Zhang et al., 2026].

We have made the corresponding revisions to the manuscript. We provide a more detailed discussion of the related work in Table 1 and have revised Section 1.1 to better highlight our contributions. We have also revised Proposition 2 and the related discussion to emphasize the quantitative impact of sample complexity on the generation mechanisms in different phases.

Li et al., ICML 2026. Transformers Learn the Optimal DDPM Denoiser for Multi-Token GMMs.

Li et al., Neurips 2024. Understanding Generalizability of Diffusion Models Requires Rethinking the Hidden Gaussian Structure

Buchanan et al., Neurips 2025. On the Edge of Memorization in Diffusion Models

Li et al., Neurips 2025. Understanding Representation Dynamics of Diffusion Models via Low-Dimensional Modeling

Zhang et al., ICLR 2026. Generalization of Diffusion Models Arises with a Balanced Representation Space

Ye et al., ICLR 2026. Provable separations between memorization and generalization in diffusion models.

Wang et al., JMLR 2026. Breaking the Curse of Dimensionality: Diffusion Models Efficiently Learn Low-Dimensional Distributions

Weakness & Question 2: The theoretical analysis relies on a highly structured Gaussian mixture model with orthogonal mode means, shared isotropic variance, and locally initialized prototypes. How essential are orthogonal Gaussian means, shared isotropic variance, and prototype initialization near empirical means? Are there experiments showing whether the proposed behavior persists when these assumptions are relaxed?

A2: This is a great question. We first emphasize that analyzing the convergence and generalization of Transformer models trained on the DDPM objective with finite data is highly challenging. Making assumptions on the data can therefore be viewed as a stepping stone toward studying more general settings. To the best of our knowledge, due to the highly nonconvex nature of this problem, no existing work has established convergence results for Transformer models under more general data assumptions. The setting considered in our work is among the most complex models that are currently theoretically tractable. Orthogonal Gaussian means and shared isotropic variance are also adopted in [Li et al., 2026] for diffusion model analysis, while a similar initialization assumption is also used in other analyses of nonconvex neural networks.

Second, we have also conducted experiments on cases where the assumptions are relaxed as suggested. For each setting, we examine (a) the training loss relative to the Bayes denoising risk, (b) the sample-specific memorization rate, and (c) the error between the learned prototypes and the matched empirical mode means.

Orthogonality. We construct correlated mode means with normalized pairwise inner product μj⊤μm/d=c∈[0,0.6]\mu_j^\top\mu_m/d=c\in[0,0.6], j≠mj\neq m. As shown in IMG_orth, the training loss still converges close to the corresponding Bayes denoising risk; the memorization-rate curves remain nearly unchanged, and the learned prototypes continue to converge close to the empirical mode means. The difference is that increasing cc leads to a modest increase in denoising risk and prototype error. This is because less-separated modes are more difficult to distinguish during training.

Variance. We next replace the shared isotropic covariance by mode-dependent anisotropic variances, while keeping the average within-mode variance fixed. We set Σm=ρ2diag(1+hsm,1,⋯ ,1+hsm,d)\Sigma_m=\rho^2\text{diag}(1+hs_{m,1},\cdots,1+hs_{m,d}) for m∈[M]m\in[M], where sm,j∼Unif1,−1s_{m,j}\sim\text{Unif}\\{1,-1\\}, and h∈[0,0.75]h\in[0,0.75] is the anisotropy strength. IMG_var shows that the curves of the learned prototypes and the memorization rate are very close to each other for varying hh. The Bayes denoising risk decreases as anisotropy increases. This is because, when the average within-mode variance is fixed, the reduction in uncertainty along certain directions induced by anisotropy becomes the dominant factor, thereby making the learning problem easier.

Initialization. We vary the initialization distance from the empirical mode means from r0=0.2r_0=0.2 to r0=1.2r_0=1.2 beyond the requirement in Assumption 1. As shown in IMG_initialization, Although the starting points of the curves differ, all of curves converge to essentially the same training loss, memorization curve, and empirical prototypes. However, when the prototypes are initialized completely randomly, training fails to recover the empirical prototypes and attains a higher loss. This indicates that initialization alignment is necessary, while Assumption 1 provides a sufficient condition that enables a tractable analysis of the training dynamics.

These experiments show that the central qualitative phenomena remain unchanged, while also revealing several new observations. Therefore, our current assumptions and analysis is a crucial first step. We have incorporated the above results and discussion into our revision.

Li et al., ICML 2026. Transformers Learn the Optimal DDPM Denoiser for Multi-Token GMMs.

Weakness & Question 3: Additionally, the paper derives a Gaussian-CDF memorization rate and a sample-complexity threshold. However, i believe that the synthetic experiments mainly examine changes in training-set size while keeping the dimension and number of modes fixed. Therefore, the predicted scaling is not directly validated. Could the authors provide experiments varying both the data dimension (d) and the number of modes (M)? A quantitative comparison between the predicted Gaussian-CDF and empirical memorization curves would also strengthen the results.

A3: We thank the reviewer for this suggestion. We agree that the original experiments primarily varied NN with fixed dd and MM. We have added new synthetic experiments that directly vary dd and MM, and compare the empirical memorization rates with the theoretical prediction from Theorem 2, without fitting any parameters.

Varying dimension dd. In IMG_d, we vary dd from 128 to 1024 while fixing the other parameters. The measured memorization rates closely follow the theoretical prediction, with an RMSE of 0.0093.

Varying the number of modes MM. In IMG_M, we vary MM from 5 to 60 while fixing the other parameters. The empirical results again closely match Theorem 2 with an RMSE of 0.0148.

Joint scaling with N/(Mρ2d)N/(M\rho^2\sqrt d). We further vary NN, MM, and dd jointly and plot the memorization rate against the ratio N/(Mρ2d)N/(M\rho^2\sqrt d) as in Corollary 2. The results IMG_joint across different settings exhibit the predicted transition, and the theoretical predictions remain close to the empirical memorization rates, with an RMSE of 0.01820.0182 over 48 joint settings.

These results directly validate the predicted dependence on dd and MM, as well as the sample-complexity scale N=Θ(Mρ2d)N=\Theta(M\rho^2\sqrt d). We have included these quantitative comparisons in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.