- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The authors identify that the noise prediction by the diffusion model has norm that does not correspond to the norm of the injected noise during training. Based on this, the authors propose a regulariser that encourages the average squared norm of predicted noise to match that of the injected noise. The authors demonstrate that precision, density, and coverage improve using their regularisation.
Critical strengths
- The authors motivate a straightforward regularisation technique using observations regarding the predicted noise's scale.
- The authors provide evaluation on various metrics. Additionally, the authors use a variety of models for both pixel- and latent-space generation.
Critical weaknesses
- Questionable method for evaluating performance. PRDC is computed in pixel space (lines 864-865), rather than in feature space. Raw pixel distances aren't reliable measures of perceptual/semantic similarity. [1] discusses this issue. This is particularly consequential since the paper's main improvements are in precision and density.
- Poor FID, with limited improvement over the baseline. EnDiff worsens FID in 11/12 settings in table 2, eg Improved Diffusion (CIFAR-10) worsens from 4.072 to 10.060. While higher fidelity can come at the expense of diversity, the results provide little evidence of improved distributional similarity. The paper also lacks other feature-space evaluation metrics, such as FDr proposed in [11].
- The theoretical motivation for EnDiff is not sufficiently justified. The authors argue that optimal noise prediction has trace energy that is lower than 1, resulting in overly conservative reverse updates (lines 171-260). However, lower predicted-noise energy does not imply an incorrect denoising field. For example, if the data distribution is standard Gaussian, the optimal noise predictor has trace energy below 1 while representing the exact score, so the regulariser biases the predictor away from the exact score. The authors have not established why this should improve generation.
- Authors omit comparisons with simple training-free alternatives. The authors don't compare against, for example, scaling the noise prediction during sampling as a first-step diagnostic. This is a particularly important baseline, since their method modifies the magnitude of the noise prediction. [3] proposes training-free Epsilon Scaling (though its motivation and scaling direction differ). What benefit does training-time regularisation provide over simply tuning the noise scale during sampling?
- Authors provide a weak treatment of related work and limited comparisons with other existing methods. Sec 2 primarily discusses standard diffusion architectures and broad improvements to diffusion models. More directly relevant methods are missing, including [4], autoguidance [6]. Min-SNR [5] is mentioned but is not compared with EnDiff empirically. The paper does not adequately position EnDiff relative to existing fidelity-oriented methods.
- Limited experimental scope. Experiments are restricted to synthetic datasets and CIFAR-10/100, CelebA and CelebA-HQ, using DDPM, Improved Diffusion and LDM. There are no larger scale ImageNet experiments or modern transformer backbones such as DiT/SiT/JiT [8] [9] [10]. This limits the relevance of the results to modern image generation frameworks and further limits the significance of this work.
- Weak empirical evidence for the proposed mechanism for how EnDiff improves performance. Fig 3 shows a relatively small change in the predicted noise trace energy alongside higher density. This correlation does not establish that the original reverse updates were incorrect or overly conservative. A more useful experiment would compare the learned denoising field with an analytically computed optimal field (Appendix B.3 of EDM [2] derives the ideal denoiser in closed form).
- The Introduction (lines 30-40) emphasises the importance of high-fidelity synthetic data for downstream model training. However, the paper does not evaluate whether data produced using EnDiff improves downstream model performance. Therefore, this motivation is not supported experimentally.
- Limited theoretical novelty. The main result (Lemma 1) follows from a standard conditional-expectation identity. In particular, the fact that the optimal predictor is the conditional mean is well known (see [7] [2]). The paper provides no further theoretical results explaining why its regulariser should improve generation.
- The authors use high coverage to argue that their method doesn't cause substantial loss of support (lines 420-424). However, recall decreases substantially in several settings, eg Improved Diffusion on CIFAR-10 decreases from 0.558 to 0.416 (tab 2). The increase in coverage alone does not establish that the method preserves diversity.
The issues with this paper do not appear to be resolvable during rebuttal. First and foremost, I think the theoretical motivation for the proposed regularisation isn't sufficiently justified. Additionally, the theoretical and empirical contributions of this paper are limited. Therefore, I don't think this paper should be accepted.
Questions for authors
See weaknesses.
[1] Reliable Fidelity and Diversity Metrics for Generative Models, https://arxiv.org/abs/2002.09797
[2] Elucidating the Design Space of Diffusion-Based Generative Models, https://arxiv.org/abs/2206.00364
[3] Elucidating the Exposure Bias in Diffusion Models, https://arxiv.org/abs/2308.15321
[4] Perception Prioritized Training of Diffusion Models, https://arxiv.org/abs/2204.00227
[5] Efficient Diffusion Training via Min-SNR Weighting Strategy, https://arxiv.org/abs/2303.09556
[6] Guiding a Diffusion Model with a Bad Version of Itself, https://arxiv.org/abs/2406.02507
[7] Noise2Noise: Learning Image Restoration without Clean Data, https://arxiv.org/abs/1803.04189
[8] Scalable Diffusion Models with Transformers, https://arxiv.org/abs/2212.09748
[9] SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers, https://arxiv.org/abs/2401.08740
[10] Back to Basics: Let Denoising Generative Models Denoise, https://arxiv.org/abs/2511.13720
[11] Representation Fréchet Loss for Visual Generation, https://arxiv.org/abs/2604.28190