acceptodds
Under review as a conference paper at ICLR 2027

A Unified View of Jointly Evolving Representations and Images in Diffusion Models

Abstract

Diffusion models increasingly use pre-trained visual representations to guide image generation, enhancing convergence and generation quality. Existing methods, however, mostly couple representations and images at two endpoint timesteps: RCG first synthesizes a clean representation at and then uses it as a static condition, while ReDi and REG jointly denoise the representation and image from pure noise at . In this paper, we provide a unified view of understanding such contradictions and show that both endpoints are suboptimal. Intuitively, the former prevents representations from receiving image context, while the latter forces image denoising to rely on highly noisy representations. We formalize this trade-off as a minimization problem, where these two setbacks combined, coincide with the divergence between actual diffusion denoising and a hypothetically ideal one, with clean representations or images as conditions. Empirical probing and theoretical analysis show that the optimum lies at neither extreme endpoint, but at an intermediate coupling timestep . Motivated by this insight, we introduce **JERI** (Jointly Evolving Representations and Images), which first denoises representations to this intermediate state and then jointly evolves representations and images. Experiments show that JERI outperforms the extreme designs in prior works across conditional and unconditional generation, model scales, and representation spaces.

Primary area:
generative models
Keywords:
Generative Model, Diffusion Model, Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6KQgOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

In this work, the authors study when to couple an image diffusion process with a separate diffusion process over features extracted by a self-supervised encoder (DINOv2, MoCo v3, MAE). They argue that existing methods lie at two extremes: either conditioning image generation on a clean representation generated beforehand (RCG), or denoising the image and representation jointly from pure noise (ReDi/REG). The main contribution of this work is an analysis and experimental validation showing that starting the coupling at an intermediate timestep (T/2) in an RCG-style approach yields better results than either endpoint.

Critical strengths
  • The paper tackles an important and timely topic in generative modelling
  • The framing of the problem is well prepared with clear initial experiments motivating the work
  • Section 6.4 is a nice addition, showing experimentally that the main observation from the paper’s formal analysis holds.
Critical weaknesses
  • The formal analysis in Section 4 is carefully written, but I am not convinced it provides much insight beyond the paper’s initial intuition. Propositions 4.1 and 4.2 essentially state that when x and z are correlated, conditioning on more information reduces uncertainty. This is expected and holds for almost any pair of correlated variables under Gaussian noising. The same goes with Theorem 4.1, please correct me if I’m wrong, but it boils down to something like: If there’s a timestep where coupling is better than RCG, and a (possibly different) timestep where coupling is better than ReDi, then the best timestep is neither RCG nor ReDi.
  • It seems that gains are mostly visible for smaller and weaker setups. When model is trained for 1M iterations the gains over RCG are marginal (0.6 FID). At the same time, with such small differences the results are reported with just a single run

Small:

  • Section 3.1 recaps the standard DDPM formulation, which will be familiar to most readers at this venue.
Questions for authors
  • I’m lost where CFG was used during evaluation. Section 6.2 says “For a fair comparison, we evaluate JERI without class-label classifier-free guidance”, while in the appendix D4 there is a description of how to combine bi-derctional conditioning with CFG with a description “Unless otherwise specified, all reported quantitative metrics are computed using the corresponding best-performing CFG weights for optimal image FID.“ Could you clearly explain which experiments then use the CFG?
  • Why there are no results for R-FID in 1M setup?
2 comments

We thank the reviewer for their recognition of the importance and timeliness of our research question. We are glad that the reviewer thinks our framing is well prepared and the motivation is clear. Meanwhile, we are especially delighted that the reviewer likes the empirical validation of our formal analysis in Section 6.4. Below, we explain our theoretical contribution, discuss the generation results, and provide the requested evaluation details.

Theoretical Insights Beyond Intuition

Thank you for reading our theoretical analysis in detail! The intuition in the main text indicates that each endpoint, RCG and ReDi/REG, has a drawback. However, it does not say whether an interior timestep is better. Our analysis quantifies this trade-off through the trajectory cost, which depends on the forward process alone and can be probed without training a diffusion model. It also gives sufficient conditions under which an interior timestep has a lower trajectory cost than both endpoints. This turns the intuition into a quantitative prediction that is available before training.

We agree that Propositions 4.1–4.2 are expected on their own. Their role is to establish the two nonzero endpoint costs, which Equation 6 accumulates along each coupling schedule. As the coupling timestep changes, one cost increases and the other decreases. However, their sum can still be lowest at an endpoint, so whether an interior timestep is better depends on the sizes of the two costs.

The reviewer’s summary is correct for the final step of the proof of Theorem 4.1, after the endpoint improvements have been established. The main mathematical step is deriving sufficient conditions for these improvements. Lemma A.7 lower-bounds the cost reduction from leaving each endpoint. Its proof uses monotonicity and telescoping from Lemmas A.5–A.6 to reduce the per-step sums to a few mutual-information terms of the forward process. Conditions (C1)–(C2) require these bounds to remain positive after accounting for JERI’s timestep pairing in Equation 7, thereby guaranteeing the endpoint improvements.

Under class probing on ImageNet, both conditions hold for DINOv2 and MoCo v3, and the trajectory cost has an interior minimum. Section 6.4, which the reviewer also points to, tests this prediction in generation: both tested interior timesteps achieve lower image FID than RCG and fully joint generation.

Improvements beyond smaller or weaker setups

Thanks for raising this concern. Our results also show improvements with our largest model and on datasets with greater semantic diversity.

First, JERI’s improvements extend to DiT-XL, the largest model in our evaluation. Table 1 shows that on full ImageNet class-conditional generation, JERI achieves 3.9 FID compared with 5.9 for ReDi, with both models trained for 400k iterations.

Second, JERI improvements are also more notable on more complex setup. We train unconditional generation on three 100-class ImageNet subsets with increasing semantic diversity. Table 4 shows JERI’s FID improvements over RCG grow from 0.0 to 3.4 and 5.9 as the dataset is growing more diverse.

Our full ImageNet evaluation uses single-run FID-50K and follows RCG and REPA in reporting results without error bars. At 1M iterations, JERI improves both FID and IS over RCG in Table 2, and it matches or improves over RCG in every other comparison we report. Gaps of this size are also used to compare model variants under the same protocol. For example, REPA Table 2 selects its alignment depth with a 0.3 FID gap.

DDPM formulation

We agree that the DDPM formulation is familiar to most readers. The brief recap in Section 3.1 establishes the notation used throughout our theoretical analysis and appendix.

Clarification on CFG

Thank you for noticing the ambiguity. We would clarify that none of our reported results use class-label CFG. Appendix D.4 describes CFG using image conditions for representation denoising (Equation D.2) and representation conditions for image denoising (Equation D.3).

RCG and ReDi also use representation guidance for image denoising. We select guidance weights separately for RCG and JERI to obtain each method’s best image FID, and use ReDi’s reported results with its selected representation guidance. We will label Equations D.2 and D.3 as image guidance and representation guidance, respectively, in the revised manuscript to distinguish them clearly from class-label CFG.

R-FID at 1M iteration

R-FID at 1M was not evaluated at submission time. We have since computed it on the same 1M checkpoints as Table 2:

Model (1M)FID ↓IS ↑R-FID ↓
RCG9.275.410.8
JERI8.679.29.8

JERI improves over RCG on all three metrics at 1M; we will add the R-FID column to Table 2.

Thank you for the detailed response. It clarifies some of my concerns and I think my initial assessment of strong reject was to harsh. Nevertheless I still have some concerns remaining:

  • I acknowledge that the theoretical analysis formalizes the trade-off and provides sufficient conditions for an interior optimum of the proposed cost. However, I still find that it offers limited additional insight beyond the motivating intuition.
  • My main empirical concern is that Table 2 shows modest FID improvements over RCG: 1.0 at 400k iterations and 0.6 at 1M. In this context, the omission of RCG from Table 1 is particularly difficult to understand. RCG is a central baseline motivating the paper and the framework on which JERI is built, yet the DiT-XL experiments do not compare against it. Consequently, Table 1 does not establish whether increasing model capacity yields a more substantial advantage over RCG or whether the gains remain similarly modest. The improvement over ReDi therefore does not directly address this concern.
  • The larger improvements in some subset and representation-space experiments demonstrate benefits in those settings. However, they do not establish whether comparable gains persist with a larger or more extensively trained RCG baseline.

To summarize, introducing partial coupling is a sensible design choice, but I still view it as a modest methodological extension. Together with the limited evidence of substantial gains over RCG in the full-ImageNet experiments, this leaves me unconvinced that the contribution is sufficiently significant for acceptance. Therefore I still think it should not be accepted at ICLR

Reviewer_8Ev4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper used entropy information to characterize the cost of of the image diffusion trajectories from two existing methods about denoising representations without image context. They then proposed a hybrid method called JERI and shown the improvement of the new algorithm over the existing two algorithms. The paper is mostly well written, but some essential information was not clearly provided. The concerns I have are that the paper is not innovative enough with simply combining two existing algorithms. The experiment did not show images reconstructed from the two existing methods they used to compare to their current method.

Critical strengths

This paper used entropy information to characterize the cost of of the image diffusion trajectories from two existing methods about denoising representations without image context. The paper is well written.

Critical weaknesses
  • The proposed a hybrid method, JERI is not innovative enough with simply combining two existing --algorithms.
  • The proof of mutual information that they used has (x0, z0) ∼ N(0, Σ), however, this is typically not true in the diffusion model.
  • The paper did not provide details on how to select the interior coupling.
  • No diagram provided for their algorithm.
  • The lost function was provided as equation [6] that was based on mutual information; however, it was then showed as (D1) without sufficient explanation, and (D1) does not involve mutual information.
Questions for authors

Would the accuracy of the selection of z0 or zt affect the performance of your algorithm? I think it would be helpful if the paper provide performance analysis of this aspect.

1 comment

We appreciate the reviewer’s time and effort in reading our manuscript, and we thank the reviewer’s positive assessment of the paper’s presentation. Below, we clarify JERI’s contribution, theory, and implementation in detail.

Concern on not innovative enough

JERI is not a combination of two algorithms, but a unified framework for studying when representations and images should begin to evolve together during denoising. In this framework, RCG and ReDi/REG represent the two endpoints: coupling only after the representation is clean (s=0s = 0), or from pure noise (s=Ts = T). Between the two endpoints, coupling at an intermediate step lets representations become sufficiently informative before they guide image generation, while later representation states remain adaptive to the generated image.

Our central contribution is to quantify the trade-off between Imageless Cost and Noisy Representation Cost through a trajectory cost and derive sufficient conditions for an interior minimum. In Figure 4, ImageNet class probing finds interior minima of the trajectory cost, and in Figure 6, both tested interior timesteps achieve lower image FID than both endpoints.

Gaussian Assumption

We do not assume Gaussian distributions for clean images or representations in either our theoretical framework, including Propositions 4.1–4.2 and Theorem 4.1, or our empirical validation, including ImageNet probing and ImageNet generation.

Assumption A.3, the joint Gaussian assumption on (x0,z0)(x_0, z_0), is used only for the closed-form examples in Appendices A.7 and A.10. Appendix A.7 derives the trajectory cost for the synthetic Gaussian illustration in Figure 4, while Appendix A.10 evaluates final image KL in the same setting. Therefore, A.3 and the derivation are included only for closed-form illustration. Our main theoretical conclusions and ImageNet results do not depend on A.3.

Details on Selecting Interior Coupling

Section 5.1 sets s=T/2s = T/2, which corresponds to the low trajectory cost range in our ImageNet class probing in Figure 4. The timestep ablation in Figure 6 supports this simple default.

No Diagram

In the original manuscript, Figure 5(a) shows the two branches and their bidirectional conditioning connections. Figure 5(b) illustrates the sampling procedure: representations are first denoised to zsz_s, followed by joint denoising of images and representations along the schedule in Equation 5. If there are specific details the reviewer would like to see in the diagram, we would be glad to add them.

Trajectory Cost and Training Loss

Equation 6 is not a loss function that trains a neural network. It defines the trajectory cost of different diffusion coupling schedules, indexed by the coupling timestep s. We use this cost to compare these schedules, and our ImageNet class probing shows that this cost is lowest at an interior timestep.

After selecting an interior coupling timestep ss, we use Equation D.1, which depends on ss, as our training loss. It applies the standard DDPM noise-prediction loss to the image and representation branches, with ss determining the conditioning inputs to each branch. Therefore this loss does not contain any mutual-information term.

Accuracy of Selection of z0 or zt

We read this question as asking how the choice of representation condition, clean z0z_0 or noisy ztz_t, affects performance. This choice is determined by the coupling schedule: in every image denoising step t, RCG uses z0z_0, fully joint generation uses ztz_t, and JERI uses zτs(t)z_{\tau_s(t)}. Increasing ss gives the image branch noisier representation conditions while allowing the representation branch to receive image context earlier.

Figure 6 evaluates generation under these different coupling schedules. The two tested interior timesteps achieve image FID of 24.7 and 25.0, compared with 28.4 for RCG and 32.0 for fully joint generation. They also achieve lower R-FID than both endpoints, indicating better representation generation quality.

If “accuracy of selection” instead refers to representation errors or additional perturbations under a fixed coupling schedule, could the reviewer clarify the setting they have in mind?

Reviewer_nCqXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when pretrained visual representations and images should begin interacting during diffusion generation. It introduces JERI, which first denoises representations independently and then jointly evolves representations and images from an intermediate coupling timestep. The method is evaluated on conditional and unconditional generation across model scales and representation spaces.

Critical strengths
  • The core idea that representations and images should be coupled neither at the very beginning nor from pure noise, but at an intermediate timestep is highly intuitive and easy to grasp.

  • Experiment results show consistent empirical improvements with little additional cost over RCG.

Critical weaknesses
  • All results are on ImageNet class-conditional/unconditional generation. There is no validation on text-to-image, larger-scale data, or higher resolutions.

  • Latent Forcing already investigates asymmetric schedules that denoise representations before beginning joint generation, including a linear-offset schedule. Could you further explain the essential difference.

Questions for authors
  • Teacher forcing details. Section 5.3 mention stop-gradient on bidirectional conditioning. Is stop-gradient applied in both directions simultaneously? What happens if only one direction is stopped?
3 comments

We appreciate the reviewer’s positive assessment. We are glad that the reviewer found the core idea of representation and images should be coupled from an intermediate timestep highly intuitive. Below, we present new text-to-image results, explain our differences from Latent Forcing, and clarify the teacher-forcing implementation.

Validation beyond ImageNet Generation

We thank the reviewer for pointing out that our experiment focuses on ImageNet. In addition to the class-conditional and unconditional generation results, we extend to a text-to-image setting during the rebuttal. Specifically, we fine-tune Stable Diffusion 1.5 for 8k steps with batch size 128 on 20k images. Our dataset consists of 20k FLUX-generated samples (jackyhate, 2024) used in RecA (Xie et al.).

jackyhate. jackyhate/text-to-image-2m, 2024
Xie et al. Reconstruction Alignment Improves Unified Multimodal Models, ICLR 2026.

To adapt our JERI method to text-to-image generation, which has the text conditioning, we (1) pool the text embeddings into a single feature vector and add it to the U-Net bottleneck feature as the condition for the representation branch, (2) use DINOv2 as the representation space, and (3) feed the synthesized representation back to the SD1.5 U-Net. Crucially, we apply the same partial-joint schedule as all our ImageNet experiments, with the same coupling timestep s=T/2s=T/2. Notably, the JERI representation branch is trained from scratch using only these 20k images. The result is shown in the table below.

MethodGenEval↑
SD1.50.42
SD1.5 w/ Finetuned0.51
SD1.5 w/ JERI0.53

Using the same coupling timestep (s=T/2) as in our ImageNet experiments, JERI raises the GenEval score from 0.51 with standard SD1.5 fine-tuning to 0.53, even though its representation branch is trained from scratch on only 20k images. Together with our unconditional and class-conditional ImageNet results, this gain suggests that our interior coupling design is useful across generation settings and can complement text conditioning in a pretrained generator.

Difference from Latent Forcing

We appreciate the reviewer’s request for a comparison between JERI and Latent Forcing.

Theoretical formulation: Latent Forcing evaluates how denoising order affects generation quality, but does not quantify the trade-off between cleaner representations and earlier image context. JERI formulates this trade-off as a trajectory cost and derives sufficient conditions for an interior coupling timestep to have lower cost than both endpoints.

Empirical method: The reviewer is correct that the Latent Forcing’s linear-offset schedule follows a similar partial-joint structure as JERI. However, the implementations of the two methods are different. Latent Forcing’s Figure 3 compares different inference schedules using a single trained model, so the model is not trained specifically for the schedule it is evaluated on. We first analyze and probe the trajectory cost of different coupling schedules using the forward diffusion process on ImageNet, before training a diffusion model, as shown in Figure 4. We then train and evaluate a separate model for each tested coupling schedule in Figure 6. Both tested interior timesteps outperform RCG and fully joint generation in image FID and R-FID, consistent with our analysis.

Teacher forcing details

We only apply stop-gradient in the image →\to representation direction, because there are no gradients in the representation →\to image direction.

Representation →\to image: the image branch receives ztz_t obtained by adding forward noise to the clean data representation z0z_0. Therefore, the representation branch cannot receive gradients from the image branch.

Image →\to representation: the representation branch receives features from the 8th layer of DiT branch. In our model, we apply stop-gradient to the features before they condition the representation branch. The "w/o Teacher Forcing" ablation in Table 3 removes this stop-gradient. We observed that letting the representation loss update the DiT does not help the representation and hurts the image branch. We will clarify this in the revised manuscript.

Thank you for your response. I will maintain my original positive score.

Thank you for taking the time to read our response and for your continued positive assessment. We are glad that you found the core idea intuitive and the empirical improvements over RCG consistent.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

We thank all three reviewers for their effort and thoughtful feedback. We appreciate the positive comments, and we are glad to learn that the reviewers think that our problem is important and timely, the framing is well prepared, and the core idea is highly intuitive.

The goal of our paper is to unify how existing methods couple representations and images during denoising. RCG gives the image branch a clean representation, while ReDi/REG gives the representation branch image context from the start. Our central contribution is to place these choices on one coupling axis and quantify the trade-off between Imageless Cost and Noisy Representation Cost through a trajectory cost. On ImageNet, class probing shows that this cost has an interior minimum, and both tested interior timesteps outperform the two endpoints in generation.

Clarifying the theoretical assumptions. For Reviewer 8Ev4’s concern about the Gaussian distribution assumption, we clarify that it is not required for the theoretical results. Our main theoretical results hold for general clean data distributions. Only the closed-form illustration examples in Appendices A.7 and A.10 use the joint Gaussian assumption.

The theoretical results beyond intuition. Reviewer 6KQg asked what the theory adds beyond intuition. Although the two costs move in opposite directions as the coupling timestep changes, their sum can still be lowest at an endpoint. Theorem 4.1 gives sufficient conditions for an interior timestep to have a lower trajectory cost than both endpoints.

Comparison with Latent Forcing. As Reviewer nCqX noted, Latent Forcing’s linear-offset schedule has a similar partial-joint structure. Its Figure 3 compares inference schedules using a single trained model. We use the trajectory cost to guide schedule selection, then train and evaluate a separate model for each tested schedule.

Results beyond ImageNet. Following Reviewer nCqX’s suggestion, we extend JERI to text-to-image generation by adapting it to SD1.5 with the same interior coupling schedule used in our ImageNet experiments. Under the same setting, fine-tuning with JERI outperforms the standard fine-tuning baseline on GenEval, providing additional evidence that JERI extends to a more challenging conditional-generation setting.

Training and evaluation details. (i) Regarding Reviewer 8Ev4’s question on the relationship between the network training loss and mutual information, we clarify that the mutual-information-based trajectory cost guides the choice of coupling schedule and that the standard noise-prediction loss trains both branches under that schedule. (ii) In response to Reviewer nCqX’s question, we also clarify that we apply stop-gradient only in the image → representation direction, as the representation → image direction has no gradient flow in our implementation. (iii) Regarding Reviewer 6KQg’s concern about CFG, we clarify that no reported result uses class-label CFG. The guidance in Appendix D.4 uses image conditions for representation denoising and representation conditions for image denoising, and RCG and ReDi also use representation guidance. (iv) As Reviewer 6KQg requested, we report R-FID at 1M iterations, where JERI also improves over RCG.

We hope these clarifications and additional results address the reviewers’ main questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.