- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors observe that, in diffusion models, as the noise level increases, the geometry of the model's prediction distribution exhibits structural distortion. The authors attribute this phenomenon to the Neural Tangent Kernel (NTK).To address this issue, the authors propose Elucidated Representation Diffusion (ERD), a plug-and-play approach that dynamically adjusts optimization effort. The method is shown to be effective on UNet- and Transformer-based diffusion models.
Critical strengths
- Substantial mathematical analysis. The authors provide extensive mathematical derivations. The paper analyzes the training process of diffusion models from the perspectives of Bayes Risk and NTK Dynamics, offering a relatively comprehensive theoretical discussion.
- Plug-and-play framework. The proposed ERD method is implemented by adjusting the loss weights across different noise levels. It requires no additional pretrained models or modifications to the network architecture, making it practical to apply.
- Effectiveness on DiT models. The results in Tables 1 and 3 show that ERD improves generation quality across DiT models of different scales. The results in Figure 6 also indicate improvements in training efficiency to some extent.
Critical weaknesses
- Unfair Comparison. In Table 2, the authors train their model based on the approach in Efficient Diffusion Training via Min-SNR Weighting Strategy (referred to as the baseline below), but compare it only with methods such as DDIM from 2020, rather than with the baseline itself. This seems unreasonable. The baseline already achieves an FID of 1.6, whereas this paper reports an FID of 1.53.
- Relatively weak experimental baselines and a lack of more representative comparisons. The paper mainly validates ERD on DiT (2023), U-ViT (2023), and UNet-based frameworks, without evaluating its applicability to more recent generative model frameworks.
- Unclear interpretation of Representation Degradation in Figure 1. I find the interpretation of Representation Degradation in Figure 1 somewhat confusing. For different inputs (), the output distributions of a diffusion model are naturally expected to differ.
Questions for authors
- I have some concerns regarding Figure 1. Please refer to Weakness 3.
- In Figure 2, for ()-prediction and ()-prediction, the gap does not reach its maximum at the highest noise level, but instead peaks at intermediate noise levels. This seems somewhat inconsistent with the authors' claim. Could the authors explain this observation?
2 comments
Response to Question & Weakness 3
Question 1 & Weakness 3 (Reviewer's Intuition: Output Variation is Natural )
Since Weakness 3 and Question 1 concern the same issue, we address them jointly in our response. The reviewer argues that:
We generally agree with the reviewer's intuition. However,
-
Premature geometric distortion. In Fig. 1, we only show the results at the endpoints, without illustrating the gradual evolution across uniformly spaced noise levels. In fact, the distortion does not progressively increase with noise. Instead, it emerges rapidly and nearly saturates at a noise level of 0.2, where the input is not yet heavily corrupted. This early saturation cannot be fully explained by the intuition that prediction becomes progressively harder as noise increases.
-
Representation degradation vs. ordinary output variation. We cite Gao et al. (2019) [1] immediately after the term "representation degradation" in the Introduction. Gao et al. showed that learned word embeddings can collapse into a narrow cone, becoming highly correlated and losing representational expressiveness. Thus, degeneration refers to the loss of geometric diversity and structure, rather than simply differences between outputs. In our setting, Fig. 1 illustrates analogous geometric distortions in diffusion model predictions.
Why can output geometry reflect internal representations? Modern diffusion architectures, such as DiT, use a linear projection in their final layer to map hidden representations to prediction targets. Consistently, our formulation in Sec. 2.4 adopts the representation–readout factorization: Therefore, the observed output geometry can partially reflect the geometric properties of the underlying hidden representations, rather than merely capturing arbitrary variations in predictions.
Question 2 (Contradiction in Loss Gap)
We do not claim that the optimal prediction loss must be highest at the highest noise level; it depends on the prediction target. Moreover, the loss trends in Fig. 2 (Fig. 3 in the latest version) do not contradict the noise-dependent geometric distortions observed in Fig. 1.
1. The Bayes-optimal prediction and its corresponding Bayes risk depend on the prediction target. Fig. 2 (Fig. 3 in the latest version) illustrates the gap between empirical MSE and Bayes risk for different prediction targets. Thus, neither the optimal prediction loss nor the excess risk is expected to peak at the highest noise level for all targets. For clarity, under squared-error loss:
| Concept | Definition | Interpretation |
|---|---|---|
| Irreducible Bayes Risk | The unavoidable prediction error | |
| Excess Risk | The additional error beyond the Bayes risk | |
| Local Loss Density | The total prediction loss density |
Here, irreducible error describes the unavoidable error floor, whereas excess risk measures the remaining gap to Bayes optimality.
2. Excess risk does not directly reflect geometric structure. Consider a counterexample: let , where , with equal probability, and is small. The predictor incurs excess risk of only , yet reduces the output covariance rank from two to one. Conversely, with can incur large MSE while preserving relative geometry under uniform scaling (see Fig. 1b and Fig. 3 -prediction).
Reference
[1] Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie-Yan Liu. Representation degeneration problem in training natural language generation models. arXiv preprint arXiv:1907.12009, 2019. 1
Response to Weakness 1&2
Weakness 1 (Unfair Comparison)
We respectfully clarify that our paper already includes extensive comparisons with Min-SNR under controlled experimental settings. Specifically, we compare with Min-SNR on DiT (Fig. 5), U-ViT-H/2 (Tab. 4), and three DiT configurations plus U-ViT-H/2 at 100K iterations (Appendix Tab. 10). As shown below, Ours consistently achieves lower FID across all four configurations.
Regarding the reported FID of 1.60 on UNet, the Min-SNR paper does not provide sufficient implementation details for a faithful reproduction. For ViT, we found that Min-SNR obtains its reported FID of 2.14 on CelebA 64×64 by switching to patch size 2 instead of patch size 4, without explicitly reporting this change; this substantially increases computational cost while barely affecting the parameter count.
To further ensure a fair comparison, we delved into our previous experimental records and found a controlled ViT experiment using the same learning rate (0.001), patch size (4), and otherwise identical settings to those of Min-SNR. The results are as follows:
Under this controlled setting, Ours consistently outperforms Min-SNR, reducing FID from 2.56 to 2.14 at 500K iterations, with particularly substantial gains during early training.
Weakness 2 (Lack of Representative Comparisons)
We disagree with this criticism. Our experimental evaluation is comprehensive in two complementary aspects: backbone coverage and baseline comparisons.
First, regarding backbone coverage, we evaluate our method on 3 widely used diffusion backbones: DiT, U-ViT, and UNet, covering different architectures and scales. This is broader than many closely related works, which typically evaluate their methods on only 1 or 2 backbones:
Note that SiT does not introduce a new backbone; it retains the DiT architecture and instead adopts a more general interpolant framework, i.e., flow matching formulations.
In contrast, our evaluation covers 3 distinct and widely used diffusion backbones.
Second, regarding baseline comparisons, beyond backbone diversity, we also compare against representative training methods. Specifically, Tab. 4 includes Min-SNR and SiT, while Tab. 5 includes REPA, SD-DiT, and DiffiT, providing a broad comparison with existing approaches.