- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This work studies momorization and generalization in diffusion models. The authors introduce the noise dependent generalization gap, a metric that subtracts the training from validation reconstruction error. The authors then consider a 2D, fully analytic toy model, and qualitatively reproduce the phenomena of image diffusion models. The authors then continue to study "training affinity", how close the denoiser outputs are to training elements in the dataset. Finally, the authors consider what happens in class conditional diffusion models.
Critical strengths
- I appreciate the effort to dive deep into the "generalization mystery" of diffusion models, which has been discussed by ample papers, but no satisfying solution has been found yet.
- I like that the authors use a combination of empirical results, as well as an analytic model to build intuition
Critical weaknesses
- Overall I was hoping for stronger conclusions, or perhaps suggestions on how to improve diffusion models based on existing results. It would be great if the authors could address this in the rebuttal.
- The amount of content in the paper seems limited for a full conference submission.
Questions for authors
- Why do the plots go negative (Fig 2a/b, Fig 4b)? It implies that in the high and sometimes low noise regime, the validation loss is lower than the training loss, why? I would expect them to converge on the same value, because at the endpoints the predictions are trivial (either dataset mean, or closest training sample)
- Can the authors define "traj" samples, how are they different from the train and val ones? Do the authors mean these are reverse process samples (traj) vs forward process samples (train/val)?
- At what sigma is figure 6 taken, and why is the train affinity of the val and train sets constants with training images seen? I would expect it to change since it the metric includes the predictor y_theta for train/val too.
Reply to reviewer Z7fY
We thank the reviewer for the positive assessment and the helpful questions.
W1: Stronger conclusions and implications. We agree that concrete improvements regarding generalization would have been great. The goal of our paper was to shed light on the problem of memorization/generalization in diffusion models from what we see as an underexplored angle: Structural memorization. As you mentioned, this is a notoriously difficult problem. We see the value of our contributions
W2: Amount of content. The revision adds null and positive controls, a calibrated memorization test across five EDM2 sizes and CIFAR-10, and a train/val decomposition of the toy model.
Q1: Why does G go negative? The endpoint values are not model effects: You can see in Fig.13 and Fig. 15 that these tails are identical across all checkpoints and model sizes (e.g., ImageNet-64: −0.010 at σ=80). At high σ, the prediction is the (class) mean, so the error measures how spread out each image set is around that mean. At low σ, a real model passes the input through, so the error is again determined by the images, not the model. Because the two splits differ slightly (the ImageNet validation set is known to be shifted), G converges to small split-dependent constants of either sign rather than to zero. For ImageNet-512, this happens to be negative at both tails, and for ImageNet-64 it's negative at high σ and positive at low σ. The model-dependent signal is the growth of G over training at intermediate σ.
With the toy model, the symmetries in the data and how they interact with the noise level can lead to a negative gap. Both tails eventually converge to 0, as the splits are i.i.d. here.
Q2: Definition of "traj". Yes. Trajectory queries are the intermediate states x(t) of the reverse process, i.e., the states the EDM sampler visits while denoising pure noise into a sample, fed to the denoiser at their own σ(t). Training and validation queries are real images, forward-noised to the same σ. We will make this more explicit in Sec. 4.
Q3: σ in Fig. 6, and why are the train/val curves flat? σ=0. We will clarify that in the revised PDF. There, the model is not called: trajectory queries are the final samples, and training and validation queries are clean images. Their affinity depends only on the images and reference sets, so it is constant across checkpoints by construction. This observation, together with Reviewer kxpp's, led us to the calibrated test described in our reply to kxpp (will be posted soon, if it is not yet).