acceptodds
Under review as a conference paper at ICLR 2027

Representation-Augmented Diffusions

Abstract

Diffusion models achieve state-of-the-art sample quality but lack an inherent mechanism to learn semantic representations from data. We propose Representation-Augmented Diffusions (RAD), a new class of diffusion models that natively embeds compact representations within the denoising process. Training maximizes a variational lower bound on the data likelihood, in which a recent class of models based on representation alignment emerge as special cases. Interestingly, our framework reveals a design choice overlooked by prior work: feeding the learned representation back into the denoiser. This simple idea leads to an order of magnitude gains in training convergence speed, data and sampling efficiency over prevalent representation alignment framework while enabling semantic discriminative representations. Furthermore, RAD extends naturally to other modalities like text-to-image synthesis, demonstrating that strong representations can be learned without compromising sample quality. Overall, RAD emerges as a strong alternative for efficiently training diffusion models.

Primary area:
generative models
Keywords:
Diffusion Models, Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_9fxLOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Representation-Augmented Diffusions (RAD). At each denoising step, the model predicts a semantic representation and feeds it back into the subsequent denoiser; the representation is used during both training and inference. The paper formulates this process as a sequential conditional latent-variable model and derives a variational objective with a conditional denoising term and a representation-policy KL term.

Critical strengths
  1. A simple, clear core mechanism with a large early-FID gain. Feeding the predicted representation back into the denoiser improves FID from 22.9 to 10.3, and the method can be combined with REG.
  2. Potentially useful trends in early training and low-step sampling. RAD reaches lower FID more quickly across model scales and training budgets, and its 10-step Euler result outperforms the reported 100-step REPA result. The exact efficiency multiplier requires stronger controls, but the trend is worth further investigation.
Critical weaknesses
  1. The main efficiency and generalization claims are not established under sufficiently matched conditions. RAD is trained for at most 400K steps and is not compared with REPA/SiT at mutual convergence. At 400K steps, RAD-B/L/XL recall is 0.49/0.49/0.50, whereas the corresponding REPA recall is approximately 0.65. The current evidence therefore shows faster attainment of a particular FID at a high-precision, low-coverage operating point. The GenEval improvement of XL/2 text-to-image generation from 0.258 to 0.266 is marginal. Evaluating gains at such low absolute performance levels does not provide compelling evidence for strong generalization.
  2. The theory strictly derives the structure of Eq. (27), not main-text Eq. (9) or the final algorithm. Eq. (9) replaces the ELBO denoising weight (w_t) with 1. The practical objective further replaces an unbounded low-noise SNR weight with a saturated weight and adds a mixed flow objective and per-sample stop-gradient adaptive reweighting. Consequently, “training maximizes a variational lower bound” does not apply to the best-performing experimental model.
Questions for authors

Can the authors provide RAD learning curves to at least 1M/4M steps and compare FID, precision, and recall after both methods approach convergence?

6 comments
AuthorsOpenReview ↗

Author Response

We thank the reviewer for the detailed feedback. We first list the relevant changes in the revision (highlighted in blue) and then answer each weakness and question.

Relevant changelog:

  1. [W1, Q1] Added RAD-XL/2 results at 1M training iterations (200 epochs) to Tables 1, 7 and 9, and REPA-XL/2 at 1M to Table 1.
  2. [W1] Revised the text-to-image claims in the abstract, the introduction, Sec. 3 (now: RAD improves FID without sacrificing GenEval at both scales), and the caption of Table 3.
  3. [W2] Revised the claim "Training maximizes a variational lower bound" in the abstract, the introduction, and Sec. 2.2.
  4. [W2] Added the step-by-step derivation of the flow-policy objective to App. A.9, and showed that, up to the reweighting of time steps, it is a bound on the log-likelihood of the flow-policy model ("Combining the Gaussian and flow policies", Eqs. 43–44).

[W1, Q1] Efficiency and generalization claims.

As requested, we trained RAD-XL/2 for 1M iterations on class-conditional ImageNet and compared it with REPA at several training budgets in Table R1 below (also Tables 1 and 7 in the paper; full learning curves in Table 9). Apart from FID, REPA reports no metrics at 1M iterations (and we could not train a 1M REPA baseline ourselves due to compute limitations), so we also compare against REPA at 4M iterations. We make the following observations:

ModelIter.FID ↓sFID ↓IS ↑Prec. ↑Rec. ↑
REPA-XL/2400K7.95.06122.60.700.65
RAD-XL/2400K3.75.66217.80.850.50
REPA-XL/21M6.4––––
RAD-XL/21M3.45.12239.60.840.51
REPA-XL/24M5.95.73157.80.700.69

Table R1: REPA and RAD at different training budgets. ImageNet 256×256, FID-50k, Euler–Maruyama SDE with 250 steps, no CFG. REPA numbers from its paper (Tables 3 and 8). RAD leads in FID, IS and precision in every available comparison, whereas REPA has higher recall.

  • RAD keeps improving in sample quality with more training. RAD at 400K iterations already outperforms REPA at 4M iterations and keeps improving with longer training: at 1M iterations its FID drops to 3.4. REPA, in contrast, changes by only 0.5 FID from 1M to 4M iterations, so it is close to convergence. Therefore, the gap is not an artifact of comparing both methods early in training. We could not train RAD to 4M iterations within the rebuttal period.
  • Precision and recall barely change from 400K to 1M/4M iterations. Recall improves by 0.04 for REPA from 400K to 4M and by 0.01 for RAD from 400K to 1M iterations. Similar trends hold for precision. Longer training thus does not change either method's operating point, so the 400K comparisons in the paper are representative of the comparison at convergence.
  • On lower recall. We have stated the lower recall as a limitation of our work since the first version of the paper (Sec. 4). We hypothesize that the drop in recall (with very high precision) observed in RAD is due to a peaked policy induced by the inject block in RAD. However, a more detailed examination of this hypothesis remains a promising direction for future work.
  • Text-to-image synthesis. The text-to-image experiment tests whether RAD transfers without changes to a different conditioning setup (caption tokens instead of a class label) and is not meant to compete with state-of-the-art text-to-image models. Within this setup, compared to REPA, RAD improves FID at both scales (by 22% at B/2 and 9% at XL/2) without losing prompt adherence: GenEval also rises at B/2 (by 20% relative) and is on par at XL/2. We have softened the claim accordingly in the abstract, the introduction, Sec. 3, and the caption of Table 3.
AuthorsOpenReview ↗

Author Response (Contd.)

[W2] The bound vs. the trained objective.

We agree, and we have revised "Training maximizes a variational lower bound" in the abstract, the introduction and Sec. 2.2 to "The training objective is derived from a variational lower bound". The objective we train is a reweighted surrogate of the bound in Eq. (4). It differs from the bound in three ways:

  1. Following DDPM [Ho et al., 2020] and most diffusion models optimized for sample quality, we set the weight w_t of the denoising term to 1.
  2. We replace the alignment weight SNR(t)/2β², which grows without bound as t → 0, by its bounded counterpart λ_t = SNR(t)/(SNR(t) + β²).
  3. We add an adaptive per-sample weight on the alignment loss inspired by the literature on few-step models such as MeanFlow [Geng et al., 2025]. This has little effect on FID (8.5 vs. 8.3, Table 6).

Re. the mixed flow objective. We have added a theoretical result in App. A.9 ("Combining the Gaussian and flow policies", Eqs. 43–44), which shows that the flow likelihood term is a principled part of the variational bound and is not derived heuristically. Therefore, our practical objective is a reweighted version of the theoretical variational bound, and we have clarified this in the main text by revising our claims.

We hope our responses have addressed your concerns, and it would be great if you could consider improving your score. We are happy to answer any follow-up questions.

Thank you for the additional results and clarification. The 1M iteration result alleviates my concern that the FID gain is purely an early-training effect. However, RAD’s recall remains much lower than REPA’s, so the gain still comes with reduced sample coverage and diversity.

My main remaining concern is CFG. All ImageNet and text-to-image results are reported without CFG, and the paper does not explain why. This matters because CFG is standard in conditional generation and may further reduce RAD’s already low recall. Could you report RAD and the baselines with CFG, including FID, precision, recall.

Reviewer_SE4YOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces Representation-Augmented Diffusions (RAD), which augments each denoising transition with a latent representation predicted from the current noisy state. A variational formulation connects this design to representation alignment and recovers REPA as a special case under specific parameterizations. Experiments demonstrate improvements in generation quality, training convergence, and few-step sampling, complemented by evaluations of representation quality and text-to-image generation.

Critical strengths
  1. The paper introduces per-step representation feedback and connects it to representation alignment through a variational framework.
  2. The method substantially improves FID and training convergence across model scales and extends to text-to-image generation.
Critical weaknesses
  1. While Table 4 demonstrates substantial gains from representation feedback, it does not isolate the contribution of stochastic latent modeling. A separately trained deterministic-feedback baseline, matched in architecture, alignment supervision, and training budget, is needed to establish the benefit of the probabilistic formulation beyond feature reinjection.
  2. The attribution of improved few-step sampling to greater transition expressiveness remains insufficiently substantiated. Equation (2) motivates a richer transition family, but Figure 4 does not distinguish this effect from representation feedback or improved overall model quality. Controlled comparisons across sampling budgets would strengthen this mechanistic claim.
Questions for authors
  1. Does stochastic feedback outperform a separately trained, matched deterministic-feedback baseline?
  2. What evidence attributes the few-step gains to transition expressiveness rather than improved overall model quality?
3 comments
AuthorsOpenReview ↗

Author Response

We thank the reviewer for their positive feedback. We first list the relevant changes in the revision (highlighted in blue) and then answer each weakness and question.

Relevant changelog:

  1. [W1, Q1] Added Table 5 to the main text (p. 9). It compares deterministic and stochastic policies in RAD with all other settings (architecture, training loss) fixed.
  2. [W1, Q1] Added the paragraph "Deterministic vs. Stochastic Policies" to Sec. 3.5, which discusses Table 5.
  3. [W2, Q2] Updated Fig. 4 (top) to compare RAD-XL/2 at 150K iterations (FID 5.7) with REPA-XL/2 at 4M iterations (FID 5.9) in the speed–quality trade-off.
  4. [W2, Q2] Updated Sec. 3.2 accordingly.
  5. [W2, Q2] Added Fig. 8 and Table 12 with the speed–quality trade-off under the Euler–Maruyama SDE sampler, and extended Table 11 (Euler ODE sampler).

[W1, Q1] Do stochastic policies help?

We trained RAD-B/2 for 400K iterations with either a deterministic or a stochastic representation policy, keeping the loss function, the denoiser architecture, and all other components unchanged, so that any difference in performance is due to the policy alone. The results are in Table R1 below (also see Table 5 and the paragraph "Deterministic vs. Stochastic Policies" in Sec. 3.5 (p. 9, in blue) in the main text).

Key takeaway: stochastic policies improve sample quality substantially over their deterministic counterpart (Table R1 below; Table 5 in the paper).

PolicyTraining controls u_t ∼ q(u_t | z_0)Sampling controls u_t ∼ p_φ(u_t | x_t)FID-50k ↓
Deterministicα_t·z_0α_t·h_φ25.2
Stochastic, Gaussianα_t·z_0 + βσ_t·εα_t·h_φ + β_inf·σ_t·ε10.3
Stochastic, flowα_t·z_0 + βσ_t·εα_t·h_φ + β_inf·σ_t·f_φ(ε; h_φ)8.7

Table R1: More expressive stochastic policies give better sample quality. RAD-B/2 at 400K iterations, Euler–Maruyama SDE with 250 steps, no CFG.

Reinjecting a deterministic prediction does not improve on REPA (FID 22.9, Table 4), while the same pathway with a stochastic policy more than halves the FID.

[W2, Q2] Few-step transition attribution.

We agree that the few-step gains need to be separated from overall model quality. Therefore, we add a controlled comparison between RAD-XL/2 trained for 150K iterations and REPA-XL/2 trained for 4M iterations, which have nearly identical sample quality with the Euler–Maruyama SDE sampler at 250 steps (FID 5.7 vs. 5.9). This rules out better overall model quality as the source of the favorable trade-off. We then measure FID against the number of sampling steps with both the ODE and the SDE sampler.

As Tables R2 and R3 show, RAD has a much better speed–quality trade-off than REPA. With the ODE sampler at 10 steps, REPA's FID degrades to 3.7× its 250-step value, while RAD's degrades only to 1.6×. Since both models have the same quality at a large sampling budget, we attribute RAD's advantage at small budgets to the expressiveness of its transitions.

Euler ODE, FID-50k ↓51015202550100200
REPA-XL/2 (4M)62.321.915.012.210.58.27.47.2
RAD-XL/2 (150K)23.79.17.36.76.46.25.95.9

Table R2: FID against the number of Euler ODE steps for two models of matched 250-step SDE quality.

Euler–Maruyama SDE, FID-50k ↓102550100250
REPA-XL/2 (4M)101.022.010.37.45.9
RAD-XL/2 (150K)46.97.95.95.45.7

Table R3: The same comparison with the Euler–Maruyama SDE sampler.

Representation feedback and transition expressiveness. In the context of RAD, these are not competing mechanisms. The transitions become more expressive precisely due to representation feedback since a stochastic representation u_t ∼ p_φ(u_t | x_t) is fed back to the denoiser: marginalizing over it turns each reverse transition into a continuous mixture of Gaussians (Eq. 2). Feeding back a deterministic prediction (i.e., a Dirac policy) instead leaves each transition a single Gaussian given x_t. The deterministic-feedback ablation in Table 5 (FID 25.2 vs. 10.3 for the Gaussian policy) shows that the feedback pathway helps only when the fed-back representation is stochastic, i.e., when it makes the transition a mixture. The matched-quality comparison above then shows that this richer transition, rather than better overall model quality, is what lets RAD take much larger steps during sampling.

We hope our responses have addressed your concerns and would be happy to answer any follow-up questions.

You have answered my doubts, but I tend to stick to my original decision.

AuthorsOpenReview ↗

Thank you for the response

We thank the reviewer for the response and are glad that our answers resolved their doubts. If there are any remaining concerns that keep the reviewer from giving a higher score, we would be happy to address them during the discussion period.

Reviewer_DywyOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Representation Augmented Diffusion (RAD), which predicts a stochastic semantic representation from the noisy image at each denoising step and conditions the denoiser on it. The framework incorporates representation alignment into a variational likelihood objective, using DINOv2 features as supervision. The experiments report faster training and sampling, competitive image quality, and smooth interpolation in representation space.

I find the framework well motivated and the experimental results strong, and I lean toward acceptance.

Critical strengths

Strong practical results. The reported results in Table 1 show faster training and sampling while maintaining competitive image quality across all scales (B/L/XL). I consider the performance gap where RAD falls behind REG to be marginal (at XL). RAD also improves over REG when the two are combined, suggesting a strong complementary contribution. These gains make RAD practically useful. The smooth interpolation results are cool additions.

The framework provides a coherent probabilistic treatment. I particularly like that semantic representations are explicit latent variables with a defined noising process. The model learns to predict them from noisy images and conditions denoising on them, accounting for where the semantic information comes from and how it enters generation. Representation alignment then has a natural role within a variational likelihood objective. I see this conceptual contribution as valuable even when competing methods achieve similar image quality.

The design is carefully studied. Ablations examine representation policies, injection depths, and the role of alignment, providing useful support for the design choices.

Critical weaknesses

This paper is almost flawless in my opinion. I would ask the two minor points in the Questions sections.

Questions for authors
  1. Benefit of stochastic representations. How does RAD compare with directly feeding back a deterministic prediction of the clean representation, keeping the remaining design as similar as possible? This would help isolate the benefit of stochasticity from representation feedback itself.

  2. Relationship to REDI. Could you clarify how RAD differs from REDI: Boosting Generative Image Modeling via Joint Image-Feature Synthesis, particularly its joint image–feature modeling and representation guidance? A comparison under matched training and sampling budgets would help establish RAD’s additional benefits.

2 comments
AuthorsOpenReview ↗

Author Response

We thank the reviewer for their positive feedback. We first list the relevant changes in the revision (highlighted in blue) and then answer each question.

Relevant changelog:

  1. [Q1] Added Table 5 to the main text (p. 9). It compares deterministic and stochastic policies in RAD with all other settings (architecture, training loss) fixed.
  2. [Q1] Added the paragraph "Deterministic vs. Stochastic Policies" to Sec. 3.5, which discusses Table 5.
  3. [Q2] Added a discussion of how ReDi differs from RAD to the related work (App. A.2).
  4. [Q2] Added ReDi to Table 1.

[Q1] Benefit of stochastic representations.

We trained RAD-B/2 for 400K iterations with either a deterministic or a stochastic representation policy, keeping the loss function, the denoiser architecture, and all other components unchanged, so that any difference in performance is due to the policy alone. The results are in Table R1 below (also see Table 5 and the paragraph "Deterministic vs. Stochastic Policies" in Sec. 3.5 (p. 9, in blue) in the main text).

Key takeaway: stochastic policies improve sample quality substantially over their deterministic counterpart (Table R1 below; Table 5 in the paper).

PolicyTraining controls u_t ∼ q(u_t | z_0)Sampling controls u_t ∼ p_φ(u_t | x_t)FID-50k ↓
Deterministicα_t·z_0α_t·h_φ25.2
Stochastic, Gaussianα_t·z_0 + βσ_t·εα_t·h_φ + β_inf·σ_t·ε10.3
Stochastic, flowα_t·z_0 + βσ_t·εα_t·h_φ + β_inf·σ_t·f_φ(ε; h_φ)8.7

Table R1: More expressive stochastic policies give better sample quality. RAD-B/2 at 400K iterations, Euler–Maruyama SDE with 250 steps, no CFG.

Reinjecting a deterministic prediction does not improve on REPA (FID 22.9, Table 4), while the same pathway with a stochastic policy more than halves the FID.

[Q2] Relationship to ReDi.

RAD and ReDi differ both conceptually and in implementation. We discuss this in the related work (App. A.2, in blue) and added ReDi to Table 1.

  • Conceptually, ReDi (like REG) denoises in the joint space of images and representations, so the representation is part of the diffusion state. RAD instead treats the representation as a latent variable that is inferred at every step by sampling from an explicit policy p_φ(u_t | x_t). This is more flexible since the policy can be any tractable distribution (e.g., a Gaussian or a normalizing flow). Moreover, unlike ReDi or REG, RAD allows controlling the width of this policy, which improves sample quality substantially (Fig. 7b in the main text).
  • The two are complementary. RAD can be applied on top of joint diffusion processes such as ReDi's. Table 2 shows that RAD also speeds up the convergence of REG, which, like ReDi, diffuses in the joint space of images and DINOv2 features (the CLS token).
  • Matched budgets. Under the same training budget (400K iterations, batch size 256) and sampler (Euler–Maruyama SDE, 250 steps, no CFG), RAD has lower FID than ReDi at both scales reported by ReDi: 3.9 vs. 9.4 for SiT-L/2 and 3.7 vs. 7.5 for SiT-XL/2. On the sampling side, ReDi's representation guidance needs two network evaluations per step, whereas RAD needs one.
SiT-XL/2, 400K iter., no CFGFID ↓IS ↑
ReDi7.5129.5
REPA + ReDi5.3149.8
RAD3.7217.8

Table R2: Comparison with ReDi under matched training and sampling budgets. ReDi numbers from its paper (Tables 1 and 9).

We hope our responses have addressed your concerns and would be happy to answer any follow-up questions.

The gain over the deteministic representation is genuinely a surprise to me. And also thanks for running the comparison with ReDi. I'm happy to keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.