acceptodds
Desk rejected at ICLR 2027

Elucidating Representation Degradation Problem in Diffusion Model Training

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Diffusion models have achieved remarkable success, yet their training remains inefficient due to a severe optimization bottleneck, which we term Representation Degradation. As noise levels increase, the outputs of the trained model exhibit progressive structural distortion, which can destabilize training and impair generation quality. Our analysis suggests that this instability is driven by mismatched target recoverability, which is associated with Neural Tangent Kernel (NTK) spectral weakening and effective low-rank behavior. To address this, we propose Elucidated Representation Diffusion (ERD), a plug-and-play framework that dynamically reallocates optimization effort according to effective recoverability. By stabilizing representation learning without external supervision, ERD accelerates convergence and achieves strong empirical performance across diffusion backbones.

Primary area:
generative models
Keywords:
Diffusion Models, Representation Learning, Generative Modeling, Training Dynamics, Optimization, Training Acceleration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_c3x2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors observe that, in diffusion models, as the noise level increases, the geometry of the model's prediction distribution exhibits structural distortion. The authors attribute this phenomenon to the Neural Tangent Kernel (NTK).To address this issue, the authors propose Elucidated Representation Diffusion (ERD), a plug-and-play approach that dynamically adjusts optimization effort. The method is shown to be effective on UNet- and Transformer-based diffusion models.

Critical strengths
  1. Substantial mathematical analysis. The authors provide extensive mathematical derivations. The paper analyzes the training process of diffusion models from the perspectives of Bayes Risk and NTK Dynamics, offering a relatively comprehensive theoretical discussion.
  2. Plug-and-play framework. The proposed ERD method is implemented by adjusting the loss weights across different noise levels. It requires no additional pretrained models or modifications to the network architecture, making it practical to apply.
  3. Effectiveness on DiT models. The results in Tables 1 and 3 show that ERD improves generation quality across DiT models of different scales. The results in Figure 6 also indicate improvements in training efficiency to some extent.
Critical weaknesses
  1. Unfair Comparison. In Table 2, the authors train their model based on the approach in Efficient Diffusion Training via Min-SNR Weighting Strategy (referred to as the baseline below), but compare it only with methods such as DDIM from 2020, rather than with the baseline itself. This seems unreasonable. The baseline already achieves an FID of 1.6, whereas this paper reports an FID of 1.53.
  2. Relatively weak experimental baselines and a lack of more representative comparisons. The paper mainly validates ERD on DiT (2023), U-ViT (2023), and UNet-based frameworks, without evaluating its applicability to more recent generative model frameworks.
  3. Unclear interpretation of Representation Degradation in Figure 1. I find the interpretation of Representation Degradation in Figure 1 somewhat confusing. For different inputs (xtx_t), the output distributions of a diffusion model are naturally expected to differ.
Questions for authors
  1. I have some concerns regarding Figure 1. Please refer to Weakness 3.
  2. In Figure 2, for (vθv_\theta)-prediction and (uθu_\theta)-prediction, the gap does not reach its maximum at the highest noise level, but instead peaks at intermediate noise levels. This seems somewhat inconsistent with the authors' claim. Could the authors explain this observation?
2 comments
AuthorsOpenReview ↗

Response to Weakness 1&2

Weakness 1 (Unfair Comparison)

We respectfully clarify that our paper already includes extensive comparisons with Min-SNR under controlled experimental settings. Specifically, we compare with Min-SNR on DiT (Fig. 5), U-ViT-H/2 (Tab. 4), and three DiT configurations plus U-ViT-H/2 at 100K iterations (Appendix Tab. 10). As shown below, Ours consistently achieves lower FID across all four configurations.

ModelBaselineSiTMin-SNROurs
DiT-B/275.862.262.5760.0
DiT-L/248.042.543.4439.9
DiT-XL/244.538.737.8235.9
U-ViT-H/233.1--29.521.6

Regarding the reported FID of 1.60 on UNet, the Min-SNR paper does not provide sufficient implementation details for a faithful reproduction. For ViT, we found that Min-SNR obtains its reported FID of 2.14 on CelebA 64×64 by switching to patch size 2 instead of patch size 4, without explicitly reporting this change; this substantially increases computational cost while barely affecting the parameter count.

To further ensure a fair comparison, we delved into our previous experimental records and found a controlled ViT experiment using the same learning rate (0.001), patch size (4), and otherwise identical settings to those of Min-SNR. The results are as follows:

Steps100K200K300K400K500K
Min-SNR11.093.473.052.722.56
Ours4.572.612.352.202.14

Under this controlled setting, Ours consistently outperforms Min-SNR, reducing FID from 2.56 to 2.14 at 500K iterations, with particularly substantial gains during early training.

Weakness 2 (Lack of Representative Comparisons)

We disagree with this criticism. Our experimental evaluation is comprehensive in two complementary aspects: backbone coverage and baseline comparisons.

First, regarding backbone coverage, we evaluate our method on 3 widely used diffusion backbones: DiT, U-ViT, and UNet, covering different architectures and scales. This is broader than many closely related works, which typically evaluate their methods on only 1 or 2 backbones:

  • 1 backbone: P2 (ADM/UNet), EDM (DDPM++/NCSN++), SiT (DiT), SD-DiT (DiT), DiffiT (Transformer–UNet hybrid), and REPA (DiT/SiT).
  • 2 backbones: Min-SNR (UNet and ViT), VDM++ (UNet and U-ViT), ScaleLong (UNet and U-ViT), and SpeeD (UNet and DiT).

Note that SiT does not introduce a new backbone; it retains the DiT architecture and instead adopts a more general interpolant framework, i.e., flow matching formulations.

In contrast, our evaluation covers 3 distinct and widely used diffusion backbones.

Second, regarding baseline comparisons, beyond backbone diversity, we also compare against representative training methods. Specifically, Tab. 4 includes Min-SNR and SiT, while Tab. 5 includes REPA, SD-DiT, and DiffiT, providing a broad comparison with existing approaches.

AuthorsOpenReview ↗

Response to Question & Weakness 3

Question 1 & Weakness 3 (Reviewer's Intuition: Output Variation is Natural )

Since Weakness 3 and Question 1 concern the same issue, we address them jointly in our response. The reviewer argues that: noise↑  ⟹  input information↓  ⟹  prediction difficulty↑  ⟹  output structure naturall changes\text{noise} \uparrow \implies \text{input information} \downarrow \implies \text{prediction difficulty} \uparrow \implies \text{output structure naturall changes}

We generally agree with the reviewer's intuition. However,

  1. Premature geometric distortion. In Fig. 1, we only show the results at the endpoints, without illustrating the gradual evolution across uniformly spaced noise levels. In fact, the distortion does not progressively increase with noise. Instead, it emerges rapidly and nearly saturates at a noise level of 0.2, where the input is not yet heavily corrupted. This early saturation cannot be fully explained by the intuition that prediction becomes progressively harder as noise increases.

  2. Representation degradation vs. ordinary output variation. We cite Gao et al. (2019) [1] immediately after the term "representation degradation" in the Introduction. Gao et al. showed that learned word embeddings can collapse into a narrow cone, becoming highly correlated and losing representational expressiveness. Thus, degeneration refers to the loss of geometric diversity and structure, rather than simply differences between outputs. In our setting, Fig. 1 illustrates analogous geometric distortions in diffusion model predictions.

Why can output geometry reflect internal representations? Modern diffusion architectures, such as DiT, use a linear projection in their final layer to map hidden representations to prediction targets. Consistently, our formulation in Sec. 2.4 adopts the representation–readout factorization: fθ(x,λ)=Wθhθ(x,λ).f_\theta(x,\lambda)=W_\theta h_\theta(x,\lambda). Therefore, the observed output geometry can partially reflect the geometric properties of the underlying hidden representations, rather than merely capturing arbitrary variations in predictions.

Question 2 (Contradiction in Loss Gap)

We do not claim that the optimal prediction loss must be highest at the highest noise level; it depends on the prediction target. Moreover, the loss trends in Fig. 2 (Fig. 3 in the latest version) do not contradict the noise-dependent geometric distortions observed in Fig. 1.

1. The Bayes-optimal prediction and its corresponding Bayes risk depend on the prediction target. Fig. 2 (Fig. 3 in the latest version) illustrates the gap between empirical MSE and Bayes risk for different prediction targets. Thus, neither the optimal prediction loss nor the excess risk is expected to peak at the highest noise level for all targets. For clarity, under squared-error loss:

ConceptDefinitionInterpretation
Irreducible Bayes Riskℓλ⋆\ell_\lambda^\starThe unavoidable prediction error
Excess Riskℓˉτ,m(λ)\bar{\ell}_{\tau,m}(\lambda)The additional error beyond the Bayes risk
Local Loss Densityℓτ,m(λ)=ℓλ⋆+ℓˉτ,m(λ)\ell_{\tau,m}(\lambda)=\ell_\lambda^\star+\bar{\ell}_{\tau,m}(\lambda)The total prediction loss density

Here, irreducible error describes the unavoidable error floor, whereas excess risk measures the remaining gap to Bayes optimality.

2. Excess risk does not directly reflect geometric structure. Consider a counterexample: let X=Y=(Z,δS)X=Y=(Z,\delta S), where Z∼N(0,1)Z\sim\mathcal{N}(0,1), S=±1S=\pm1 with equal probability, and δ>0\delta>0 is small. The predictor f(X)=(Z,0)f(X)=(Z,0) incurs excess risk of only δ2\delta^2, yet reduces the output covariance rank from two to one. Conversely, f(X)=cYf(X)=cY with 0<c<10<c<1 can incur large MSE while preserving relative geometry under uniform scaling (see Fig. 1b and Fig. 3 ϵθ\epsilon_\theta-prediction).

Reference

[1] Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie-Yan Liu. Representation degeneration problem in training natural language generation models. arXiv preprint arXiv:1907.12009, 2019. 1

Reviewer_uVV5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the "Representation Degradation" problem: trained diffusion model outputs become progressively distorted and collapse as input noise increases. It analyzes training as continuous-time, noise-conditioned regression in the NTK regime and proposes and tests a fix.

I have to admit I didn't have time to examine the theory part in detail, so my assessment may be misguided.

Critical strengths
  1. The phenomenon is well documented, and the paper uses the 2D GMM study as a useful validation.
  2. The fix, ERD, is elegant. It is architecture-agnostic, free, and easy to adopt.
  3. Unified rule across prediction parameterizations is a clean contribution.
  4. The theory presentation is well organized. Although I didn't have time to go through it in detail, I appreciated the effort.
Critical weaknesses
  1. The causal chain from phenomenon to method is not closed. The paper does not really show that ERD training reduces these measured degradations or the spectral weakening.
  2. The experiments do not reach SOTA. The paper itself says it is "narrowing the gap".
Questions for authors
  1. Does ERD actually reduce the measured high-noise degradation? A direct before/after of the phenomenon would be useful to demonstrate this
  2. Can you extend the system-level comparison to even longer epochs to test whether ERD's advantage grows with scale?
6 comments

Weakness 1 & Question 1 (Causal Chain Closure)

The revised Sec. 4.5 provides direct empirical evidence that our method mitigates representation degradation, evaluating representation quality from three perspectives:

  1. Linear probing (Fig. 6a). We retain the linear-probing evaluation and further connect it to our revised theoretical analysis. Our method substantially improves representation quality over SiT despite fewer training epochs, narrowing the gap toward REPA without external representation supervision.
ModelTrainingt=0.0t=0.0t=0.25t=0.25t=0.5t=0.5
SiT-XL/21400 epochs52.359.142.6
REPA800 epochs72.469.548.7
Ours800 epochs68.265.945.4
  1. Effective rank (Fig. 6b). Under matched backbone and training settings, our method preserves higher feature diversity, particularly in deeper layers across noise levels.
  2. NTK eigenspectra (Fig. 7). Our method maintains a stronger lower spectral envelope across training stages, especially for smaller eigenvalues.

Weakness 2 & Question 2 (Performance and Training Scale)

Our primary contribution is identifying and analyzing representation degradation as an intrinsic bottleneck in diffusion training, rather than achieving SOTA through another weighting strategy. We show that mitigating this degradation improves both representation quality and generation quality without external representation guidance. Thus, our goal is to address this underlying limitation of diffusion training.

Regarding the training scale, our system-level comparison has already reached 800 epochs (4M iterations), requiring approximately one month of training on an A100 server. Given the substantial computational cost, we do not extend the comparison to longer training schedules in this revision. We also demonstrate the complementarity of our method with existing representation learning approaches, as shown in Appendix Table 12.

Method100K200K300K400K
Baseline75.8057.5048.2043.40
Ours60.0043.6036.8032.30
REPA55.6542.6837.0834.47
REPA + Ours52.5836.6231.2028.87

Finally, there is still room for further improving our method. Layerwise allocation is a promising direction, as prior studies show that layerwise representation properties provide useful insights into representation-learning performance [1, 2]. We adopt a simple allocation design in this work to maintain consistency with our theoretical analysis, which considers the NTK globally rather than performing a layerwise decomposition.

References

[1] Lucrezia Valeriani, Diego Doimo, Francesca Cuturello, Alessandro Laio, Alessio Ansuini, and Alberto Cazzaniga. The geometry of hidden representations of large transformer models. NeurIPS, 2023.

[2] Daniel Bolya, Po-Yao Huang, Peize Sun, et al. Perception Encoder: The best visual embeddings are not at the output of the network. NeurIPS, 2025.

AuthorsOpenReview ↗

Follow-up on Rebuttal and Further Discussion

We sincerely appreciate your time and thoughtful feedback.

We understand that reviewing takes considerable time and effort, and we are grateful for your attention to our work. As we are now halfway through the rebuttal discussion period, we would welcome any further thoughts or questions whenever your schedule permits.

We would also like to share our recent discussion with Reviewer c3x2, particularly "Further Perspectives on Representation Learning", where we provide additional perspectives on representation degradation in diffusion models, its connection to information bottlenecks, and its relationship with REPA.

We hope these perspectives may be helpful in clarifying our contributions and addressing any remaining concerns. We would be happy to discuss further and sincerely appreciate your consideration.

Thanks for the responses.

On point 1 (causal): I apologize for not being clearer earlier. What I meant was as follows. The paper first examined representation degradation with several measures, as illustrated in Figures 1-4, which is nice. But the paper didn't directly compare those measurements with the ERD framework. So the empirical performance improvement seen in ERD is not directly connected to the earlier analyses.

Reviewer_Tr8fOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies inefficiency in diffusion-model training through what the authors call “representation degradation”: structural distortion of model outputs and hidden features across noise levels. Its analysis decomposes prediction loss into irreducible conditional uncertainty and reducible estimation error, then uses joint NTK dynamics to describe interactions between training updates at different noise levels. Under additional surrogate and contraction assumptions, it derives excess-risk bounds involving corrective dynamics and conditional-residual forcing. Motivated by this analysis, the authors propose Elucidated Representation Diffusion (ERD), a target-dependent loss-weighting scheme based on an effective-amplitude proxy. Experiments on synthetic data, ImageNet, and CelebA investigate the proposed mechanism and demonstrate improved generation quality within the evaluated training budgets.

Critical strengths
  • The paper addresses an important training-efficiency problem and provides a quite comprehensive theory involving target uncertainty, noise-level allocation, and shared parameter interactions.
  • It's quite interesting to see the theoretical parts are considered with all noise levels altogether and the authors point out training for each noise level is affected by other noise levels.
Critical weaknesses
  • Theorem 3.4 relies on the assumption in Equation (5). Could the authors check whether this condition holds during training? Also, it would be nice to see some empirical verification of the bound with observed excess-risk trajectories.
  • I find the narrative of the paper not so coherent. Although I appreciate the authors' analysis on linking output reconstruction with representation change, I feel the connection between Proposition 3.7 and other analyses is quite weak or not justified enough. Moreover, I don't quite understand why the proposed method is called "Elucidated Representation Diffusion", it's a loss reweighting strategy. Where does "representation" come into play?
Questions for authors
  • Can the authors clarify the core message they want to convey in the paper? I found some of the results quite interesting but slightly scattered.
  • In principle, can a noise-level sampling strategy achieve similar behavior to ERD loss reweighting? EDM (Karras et al) uses a loss sampling to focus on intermediate noise levels, where they find loss reduction is more possible. Could the authors clarify this relationship and provide a small controlled ablation (if time and resources permit)? For example, in the existing DiT/SiT setup, retain the baseline loss weights and sample noise levels in proportion to the baseline sampling density multiplied by ERD’s weight? Or in the EDM framework, retain EDM’s noise-level sampling and network preconditioning while comparing its original loss weighting with an appropriately parameterized ERD weighting?
6 comments
AuthorsOpenReview ↗

Response to Weakness

Weakness 1 (Assumption Validity and Empirical Bound Verification)

We are sincerely grateful to the reviewer for raising this important point.

After careful consideration, we realized that the assumption required by the pathwise loss analysis, without taking expectation, does not accurately reflect actual stochastic training dynamics. Following the reviewer's suggestion, we directly conducted an empirical check of this condition during training and confirmed that it is not consistently satisfied. We therefore reformulated the analysis in expectation.

Specifically, the revised analysis explicitly accounts for the covariance between the evolving estimation error and the sampled irreducible residual, rather than assuming that this correlation vanishes within each sampling interval.

Under revised assumptions that better reflect practical stochastic training (using a local linear approximation, allowing the model to represent the optimal predictor, and requiring only standard independence, stability, and finite-variance conditions), we establish an expected excess-risk bound. The bound shows that the expected excess risk initially decays at a geometric rate O(ρj)O(\rho^j), where 0<ρ<10<\rho<1, and eventually converges to a nonzero error floor of O(Δ)O(\Delta). In the long-time limit, we further show that

E[LˉjΔ+s,m]=Vξ,m4Δ+O(Δ2).\mathbb{E}[\bar{L}_{j\Delta+s,m}]=\frac{V_{\xi,m}}{4}\Delta+O(\Delta^2).

Weakness 2 (Theoretical Coherence and Method Motivation)

(a) Connection between Proposition 3.7 and other analyses.

We agree that the previous presentation did not make this connection sufficiently clear. In the revised version, we have moved the original Proposition 3.7 to an earlier position and reformulated it as Lemma 2.6, serving as a preliminary result for the subsequent analysis.

The intuition is straightforward: for a model with the representation–readout factorization f=Whf=Wh, changes in the output can arise from changes in both the readout and hidden representations. The NTK captures the output sensitivity to parameter changes. The contribution from the readout parameters alone is proportional to the hidden-feature Gram matrix, while the remaining parameter contributions are positive semidefinite. Consequently, the full NTK spectrum provides an upper bound on the hidden-feature spectrum. This establishes the connection between output-level dynamics and hidden representations, which is then used to derive the representation fluctuations and their impact on linear-probe risk in the subsequent analysis.

(b) Motivation for the method name.

To clarify the method's objective, we have renamed ERD as Representation Preservation Diffusion Models (RPDM). Our goal is to preserve learned representations by mitigating representation degradation, not to introduce an additional representation loss (noise-level sampling, loss weighting, and noise scheduling are equivalent, as we will explain in detail in our subsequent response to Q2). Thus, our method is fundamentally about reallocating training effort to preserve representations, rather than prescribing a particular weighting strategy. This is also the key message of our conclusion: "We show that mitigating degradation improves representations and generation quality without external representation guidance."

AuthorsOpenReview ↗

Response to Question

Question 1 (Clarify the Core Message)

Our core message is that diffusion models lose representation quality because one network must fit every noise level at once. Depending on the prediction target, the corresponding Bayes-optimal predictor can have an irreducible residual that no predictor can eliminate. Because all noise levels share the same parameters, these residuals keep perturbing the parameters throughout training. In turn, these perturbations reach the hidden representations and persist longest along weak NTK modes, which correct errors slowly (Theorem 2.7), and raise linear-probe risk (Corollary 2.8). How training allocates effort across noise levels determines how large these perturbations remain at equilibrium (Theorem 3.3). RPDM uses this result to reallocate effort, and it improves both representation and generation quality without external representation supervision.

Question 2 (Relationship Between Noise-Level Sampling and Loss Reweighting)

(a) Theoretical Relationship and Equivalence

The reviewer asks whether noise-level sampling can achieve an effect similar to ERD loss reweighting and how the two formulations are related.

This equivalence was established in the previous version and further clarified in Remark 3.2 of the revised manuscript. The effective allocation M(λ)=ρ(λ)w(λ)∣λ˙t(λ)∣M(\lambda)=\rho(\lambda)w(\lambda)|\dot{\lambda}_t(\lambda)| jointly captures noise-level sampling, loss weighting, and the noise schedule. Under a linear log-SNR schedule, loss reweighting is equivalent, at the population-objective level, to nonuniform sampling. Given uniform sampling density ρ0\rho_0 and weight w(λ)w(\lambda), the equivalent sampling density is ρ1(λ)=ρ0(λ)w(λ)Eρ0[w],\rho_1(\lambda)=\frac{\rho_0(\lambda)w(\lambda)}{\mathbb{E}_{\rho_0}[w]}, with a constant loss weight Eρ0[w]\mathbb{E}_{\rho_0}[w]. The two formulations yield identical population objectives but may differ in stochastic gradient covariances.

Connection to EDM. We derive two equivalent VP formulations for vv-prediction (see Mathematical Appendix):

Case 1: EDM data scalingCase 2: Original data
Dataz0=x0/σdataz_0=x_0/\sigma_{\mathrm{data}}x0x_0
Forward processzt=az0+bϵz_t=az_0+b\epsilonxt=αx0+βϵx_t=\alpha x_0+\beta\epsilon
Prediction targetvt=aϵ−bz0v_t=a\epsilon-bz_0vt=αϵ−βx0v_t=\alpha\epsilon-\beta x_0
Equivalent loss weight11σdata2+σ2σdata2(1+σ2)\frac{\sigma_{\mathrm{data}}^2+\sigma^2}{\sigma_{\mathrm{data}}^2(1+\sigma^2)}
Equivalent objectiveUnweighted −v-v predictionWeighted vv prediction

Here, a=σdata/σdata2+σ2a=\sigma_{\mathrm{data}}/\sqrt{\sigma_{\mathrm{data}}^2+\sigma^2}, b=σ/σdata2+σ2b=\sigma/\sqrt{\sigma_{\mathrm{data}}^2+\sigma^2}, α=1/1+σ2\alpha=1/\sqrt{1+\sigma^2}, and β=σ/1+σ2\beta=\sigma/\sqrt{1+\sigma^2}.

Thus, under the corresponding predictor reparameterization, EDM is equivalent to either unweighted −v-v prediction with EDM data scaling or weighted vv prediction on the original data.

(b) Controlled Ablation

The reviewer suggests a controlled ablation in either the DiT/SiT or EDM setting:

  • DiT/SiT: Keep baseline loss weights and adjust sampling according to the ERD weight.
  • EDM: Keep EDM sampling and preconditioning, and compare original and ERD loss weights.

We first clarify that the reviewer's characterization of EDM's sampling strategy is inaccurate under VP-normalized time. With their default settings, EDM's and SD3's sampling densities peak at t≈0.2t\approx0.2 and t=0.5t=0.5, respectively, under t=σ/(σ+σdata)t=\sigma/(\sigma+\sigma_{\mathrm{data}}). Thus, EDM's sampling density peaks at a lower normalized noise level than SD3's [1, 2]. Our vv-prediction weighting, wv∝αtσtw_v\propto\alpha_t\sigma_t, induces a similar sampling emphasis to SD3, also focusing on intermediate noise levels.

Moreover, SpeeD [3] has already explored combining asymmetric timestep sampling with change-aware loss weighting, demonstrating that the two strategies can be jointly optimized. Given this prior work, we do not conduct an additional ablation on this combination. Instead, we highlight a more important finding: our method offers benefits beyond training allocation. Appendix Tab. 12 demonstrates its complementarity with existing representation learning methods:

Method100K200K300K400K
Baseline75.8057.5048.2043.40
Ours60.0043.6036.8032.30
REPA55.6542.6837.0834.47
REPA + Ours52.5836.6231.2028.87

References

[1] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. NeurIPS, 2022.

[2] Patrick Esser, Sumith Kulal, Andreas Blattmann, et al. Scaling rectified flow transformers for high-resolution image synthesis. ICML, 2024.

[3] Kai Wang, Yukun Zhou, Mingjia Shi, Zhihang Yuan, Yuzhang Shang, Xiaojiang Peng, Hanwang Zhang, and Yang You. A closer look at time steps is worthy of triple speed-up for diffusion model training. arXiv preprint arXiv:2405.17403, 2024.

AuthorsOpenReview ↗

Mathematical Appendix for Question

Equivalence of EDM Denoising Loss

Let σ=σ(t)>0\sigma=\sigma(t)>0. The following equivalences hold at each noise level, regardless of the sampling distribution. EDM uses xσ=x0+σϵx_\sigma=x_0+\sigma\epsilon, where ϵ∼N(0,I)\epsilon\sim\mathcal N(0,I), and

Dθ=cskipxσ+coutFθ(cinxσ,cnoise),D_\theta=c_{\mathrm{skip}}x_\sigma+c_{\mathrm{out}}F_\theta(c_{\mathrm{in}}x_\sigma,c_{\mathrm{noise}}),

where cskip=σdata2/(σdata2+σ2)c_{\mathrm{skip}}=\sigma_{\mathrm{data}}^2/(\sigma_{\mathrm{data}}^2+\sigma^2), cout=σdataσ/σdata2+σ2c_{\mathrm{out}}=\sigma_{\mathrm{data}}\sigma/\sqrt{\sigma_{\mathrm{data}}^2+\sigma^2}, cin=1/σdata2+σ2c_{\mathrm{in}}=1/\sqrt{\sigma_{\mathrm{data}}^2+\sigma^2}, and cnoise=log⁡σ/4c_{\mathrm{noise}}=\log\sigma/4. Its loss weight is λ(σ)=(σdata2+σ2)/(σdata2σ2)=cout−2\lambda(\sigma)=(\sigma_{\mathrm{data}}^2+\sigma^2)/(\sigma_{\mathrm{data}}^2\sigma^2)=c_{\mathrm{out}}^{-2}.

Case 1: vv prediction with EDM data scaling.

Let z0=x0/σdataz_0=x_0/\sigma_{\mathrm{data}} and define

v_t=\frac{\sigma_{\mathrm{data}}\epsilon-\sigma z_0}{\sqrt{\sigma_{\mathrm{data}}^2+\sigma^2}}.$$ Since $c_{\mathrm{in}}x_\sigma=z_t$ and $D_\theta-x_0=c_{\mathrm{out}}(F_\theta+v_t)$, we obtain $$\mathcal L_{\mathrm{EDM}} =\mathbb E_{t,x_0,\epsilon}[\lambda(\sigma)\|D_\theta-x_0\|^2] =\mathbb E_{t,z_0,\epsilon}[\|F_\theta(z_t,c_{\mathrm{noise}})+v_t\|^2].$$ Thus, EDM is equivalent to unweighted $-v$ prediction with EDM consistent data scaling. **Case 2: $v$ prediction on the original data.** Without EDM's data scaling, define $$x_t=\frac{x_0+\sigma\epsilon}{\sqrt{1+\sigma^2}},\qquad v_t=\frac{\epsilon-\sigma x_0}{\sqrt{1+\sigma^2}}.$$ Here $c_{\mathrm{in}}x_\sigma=c_{\mathrm{in}}\sqrt{1+\sigma^2}\,x_t$. Define the equivalent predictor as $$v_\theta(x_t,t)=\frac{x_t-\sqrt{1+\sigma^2}D_\theta}{\sigma}.$$ Since $D_\theta-x_0=-\sigma(v_\theta-v_t)/\sqrt{1+\sigma^2}$, the loss becomes $$\begin{aligned} \mathcal L_{\mathrm{EDM}} &=\mathbb E_{t,x_0,\epsilon}[\lambda(\sigma)\|D_\theta-x_0\|^2]\\\\ &=\mathbb E_{t,x_0,\epsilon}\left[\lambda(\sigma)\frac{\sigma^2}{1+\sigma^2}\|v_\theta-v_t\|^2\right]\\\\ &=\mathbb E_{t,x_0,\epsilon}\left[ \frac{\sigma_{\mathrm{data}}^2+\sigma^2}{\sigma_{\mathrm{data}}^2(1+\sigma^2)} \|v_\theta-v_t\|^2\right]. \end{aligned}$$ Thus, EDM is equivalent to weighted $v$ prediction on the original data, with $$w_v(\sigma)=\frac{\sigma_{\mathrm{data}}^2+\sigma^2}{\sigma_{\mathrm{data}}^2(1+\sigma^2)}.$$

Comments on the paper

3 comments

We thank the reviewers for their feedback. We had already studied the causal link between representation degradation and our method and completed the supporting experiments before this review cycle. However, the tight revision schedule and 9 page limit prevented us from including these results in the original main text.

We have now added Section 4.5 under the 10 page rebuttal limit. Fig. 6 and Fig. 7 present Linear Probe Accuracy, Effective Feature Rank, and NTK Eigenspectrum, providing direct evidence that ERD mitigates representation degradation. We have also revised the manuscript to improve its clarity and organization.

Most other concerns are addressed by existing analyses and experiments in the appendices, which we reference in our responses below.

AuthorsOpenReview ↗

Further Perspectives on Representation Learning

We sincerely thank the reviewer for the thoughtful follow-up and for reconsidering our work.

Regarding the reviewer's intuition about Fig. 1, we had already been raised by fellow researchers before we began writing this paper. Therefore, we fully acknowledge that the reviewer's question is reasonable and well-motivated. Given the paper's page limit, it is natural that some questions may remain unanswered in the main text. Thus, the rebuttal process provides a valuable opportunity to clarify such concerns through constructive discussion.

We also agree that this is a worthwhile question for future evaluation. In our response to Reviewer uVV5, we emphasized that:

Our primary contribution is identifying and analyzing representation degradation as an intrinsic bottleneck in diffusion training, rather than achieving SOTA through a weighting.

We also mentioned that:

Layerwise allocation is a promising direction, as prior studies show that layerwise representation properties provide useful insights into representation-learning performance.

However, since this direction requires additional computational resources and involves considerable engineering efforts, we have not pursued it further at this stage.

Nevertheless, we have been continuously exploring representation learning in diffusion models.

Here, we recall the history of representation learning and diffusion models. Early work on denoising autoencoders (DAEs, Bengio et al.) studied representation learning through a Gaussian corruption and denoising process of the form xt=x0+ϵ,x_t=x_0+\epsilon, which later led to Langevin sampling (an early form of diffusion models).

Diffusion models use xt=αtx0+σtϵ.x_t=\alpha_t x_0+\sigma_t\epsilon. Thus, diffusion training is also a special form of DAE. Our ongoing work further reveals the following decomposition:

\mathcal{L} &\triangleq \mathbb{E}_{(x_0,x_t)\sim q}\left[\mathrm{KL}\left(q(x_{t-1}\mid x_t,x_0)\,\|\,p_\theta(x_{t-1}\mid x_t)\right)\right]\\\\ &= \mathbb{E}_{q(x_0,x_t,x_{t-1})}\left[\log\frac{q(x_{t-1}\mid x_t,x_0)}{p_\theta(x_{t-1}\mid x_t)}\right]\\\\ &= \mathbb{E}_{q}\left[\log\left(\frac{q(x_{t-1}\mid x_t,x_0)}{q(x_{t-1}\mid x_t)}\cdot\frac{q(x_{t-1}\mid x_t)}{p_\theta(x_{t-1}\mid x_t)}\right)\right]\\\\ &= \mathbb{E}_{q}\left[\log\frac{q(x_{t-1}\mid x_t,x_0)}{q(x_{t-1}\mid x_t)}\right] + \mathbb{E}_{q}\left[\log\frac{q(x_{t-1}\mid x_t)}{p_\theta(x_{t-1}\mid x_t)}\right]\\\\ &= I_q(x_0;\,x_{t-1}\mid x_t) + \mathbb{E}_{x_t\sim q}\left[\mathrm{KL}\left(q(x_{t-1}\mid x_t)\,\|\,p_\theta(x_{t-1}\mid x_t)\right)\right]. \end{aligned}$$ Here, $q$ denotes the forward process, $p_\theta$ the learned reverse process, and $I_q$ the conditional mutual information under $q$. Thus, the diffusion forward process acts as a progressive information bottleneck, gradually erasing information about the data. Naturally, REPA can be viewed as a form of externally guided representation distillation, with the objective $$\mathcal{L}_{\mathrm{REPA}}(\phi;\psi)=\int_0^T \lambda(t)\,\mathbb{E}_{x_0\sim q_0}\mathbb{E}_{x_t\sim q(\cdot\mid x_0)}\left[\mathrm{KL}\left(q_\psi(z\mid x_0)\|\,r_\phi(z\mid x_t,t)\right)\right]\mathrm{d}t.$$ Here, the teacher encoder $q_\psi(z\mid x_0)$ provides clean semantic targets, while the student encoder $r_\phi(z\mid x_t,t)$ learns to recover the corresponding representations from noisy inputs. In short, REPA addresses this problem through representation distillation. Returning to this submission, we treat diffusion models as denoising encoders and use weighting to mitigate representation degradation, improving representation learning and generation quality. We will continue exploring simpler ways to fundamentally address this problem. At the same time, we understand that the current rating of Weak Acceptance may indicate that some concerns remain and prevent a stronger assessment. Therefore, if the reviewer has time and is willing to discuss further, we would be very happy to address any remaining questions or concerns. Please feel free to continue the discussion. Finally, thank you again for your constructive feedback and encouraging words. Best wishes!
AuthorsOpenReview ↗

Summary of Rebuttal

We sincerely thank all reviewers for their constructive feedback. As the rebuttal period draws to a close, we would like to summarize the major concerns and our responses below. We welcome any further feedback or questions from the reviewers.


Reviewer Tr8f

Concern 1: Validity of theoretical assumptions

See: "Weakness 1 (Assumption Validity and Empirical Bound Verification)"

  • Following the reviewer's suggestion, we empirically examined the original pathwise assumption and revised our analysis to better reflect stochastic training dynamics.
  • We established a new expected excess-risk bound that explicitly accounts for error–residual covariance, demonstrating geometric convergence toward a nonzero stationary error floor.

Concern 2: Connections between theoretical results and representation learning

See: "Weakness 2 (Theoretical Coherence and Method Motivation)" and "Question 1 (Clarify the Core Message)"

  • We reorganized the theoretical analysis to establish a clearer connection between NTK dynamics, representation fluctuations, and linear-probe risk, moving the original Prop. 3.7 earlier as Lemma 2.6.
  • We renamed ERD to Representation Preservation Diffusion Models (RPDM) to better reflect the motivation of preserving learned representations.
  • We clarified the core theoretical mechanism: shared-parameter optimization introduces stochastic perturbations that affect hidden representations, persist longer along weak NTK modes, and increase linear-probe risk. Training allocation controls the magnitude of these fluctuations at equilibrium, motivating our representation-preserving method.

Concern 3: Relationship between sampling and loss weighting

See: "Question 2 (Relationship Between Noise-Level Sampling and Loss Reweighting)"

  • We clarified the equivalence between noise-level sampling and loss weighting at the population-objective level, while noting differences in stochastic gradient covariance.
  • We further established connections with EDM and existing timestep-sampling strategies, and highlighted our method's complementarity with REPA.

Reviewer uVV5

Concern 1: Whether our method actually mitigates representation degradation

See: "Weakness 1 & Question 1 (Causal Chain Closure)" We added empirical evidence in Sec. 4.5 to directly demonstrate that our method mitigates representation degradation:

  • Linear probing: Our method improves representation quality without relying on external representation supervision.
  • Effective rank: Our method better preserves hidden-feature diversity across noise levels, particularly in deeper layers.
  • NTK eigenspectra: We added a new spectral comparison showing that our method maintains a stronger lower spectral envelope, especially for smaller eigenvalues.

Concern 2: SOTA performance and longer training schedules

See: "Weakness 2 & Question 2 (Performance and Training Scale)"

  • We clarified our primary contribution: identifying and mitigating representation degradation as an intrinsic optimization bottleneck in diffusion training.
  • We demonstrated the effectiveness of our method at scale, with system-level experiments extending to 800 epochs (4M iterations).
  • We established the complementarity of our method with REPA, where combining both consistently achieves better performance than either method alone.

Reviewer c3x2

Concern 1: Fairness and breadth of experimental comparisons

See: "Weakness 1 (Unfair Comparison)" and "Weakness 2 (Lack of Representative Comparisons)"

  • We clarified the reproducibility issues in Min-SNR's reported results, including missing UNet implementation details and an unreported patch-size change in ViT. We further demonstrated consistent improvements over Min-SNR under matched ViT training settings.
  • We highlighted that our evaluation covers three diffusion backbones (DiT, U-ViT, and UNet), whereas related works typically consider only one or two.

Concern 2: Interpretation of representation degradation

See: "Question 1 & Weakness 3 (Reviewer's Intuition: Output Variation is Natural)"

  • We showed that geometric distortion nearly saturates at a noise level of 0.2, rather than increasing progressively with noise, challenging the explanation based solely on prediction difficulty.
  • We clarified that representation degradation concerns the loss of geometric diversity and structure, rather than natural output variation, and explained its connection to hidden representations through linear readout.

Concern 3: Relationship between prediction loss and geometric distortion

See: "Question 2 (Contradiction in Loss Gap)"

  • We clarified that excess risk depends on the prediction target and need not peak at the highest noise level.
  • We provided counterexamples demonstrating that excess risk and geometric distortion are not directly correlated: small excess risk can accompany rank collapse, while large prediction errors can preserve geometric structure.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.