- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this work, the authors study when to couple an image diffusion process with a separate diffusion process over features extracted by a self-supervised encoder (DINOv2, MoCo v3, MAE). They argue that existing methods lie at two extremes: either conditioning image generation on a clean representation generated beforehand (RCG), or denoising the image and representation jointly from pure noise (ReDi/REG). The main contribution of this work is an analysis and experimental validation showing that starting the coupling at an intermediate timestep (T/2) in an RCG-style approach yields better results than either endpoint.
Critical strengths
- The paper tackles an important and timely topic in generative modelling
- The framing of the problem is well prepared with clear initial experiments motivating the work
- Section 6.4 is a nice addition, showing experimentally that the main observation from the paper’s formal analysis holds.
Critical weaknesses
- The formal analysis in Section 4 is carefully written, but I am not convinced it provides much insight beyond the paper’s initial intuition. Propositions 4.1 and 4.2 essentially state that when x and z are correlated, conditioning on more information reduces uncertainty. This is expected and holds for almost any pair of correlated variables under Gaussian noising. The same goes with Theorem 4.1, please correct me if I’m wrong, but it boils down to something like: If there’s a timestep where coupling is better than RCG, and a (possibly different) timestep where coupling is better than ReDi, then the best timestep is neither RCG nor ReDi.
- It seems that gains are mostly visible for smaller and weaker setups. When model is trained for 1M iterations the gains over RCG are marginal (0.6 FID). At the same time, with such small differences the results are reported with just a single run
Small:
- Section 3.1 recaps the standard DDPM formulation, which will be familiar to most readers at this venue.
Questions for authors
- I’m lost where CFG was used during evaluation. Section 6.2 says “For a fair comparison, we evaluate JERI without class-label classifier-free guidance”, while in the appendix D4 there is a description of how to combine bi-derctional conditioning with CFG with a description “Unless otherwise specified, all reported quantitative metrics are computed using the corresponding best-performing CFG weights for optimal image FID.“ Could you clearly explain which experiments then use the CFG?
- Why there are no results for R-FID in 1M setup?
2 comments
Thank you for the detailed response. It clarifies some of my concerns and I think my initial assessment of strong reject was to harsh. Nevertheless I still have some concerns remaining:
- I acknowledge that the theoretical analysis formalizes the trade-off and provides sufficient conditions for an interior optimum of the proposed cost. However, I still find that it offers limited additional insight beyond the motivating intuition.
- My main empirical concern is that Table 2 shows modest FID improvements over RCG: 1.0 at 400k iterations and 0.6 at 1M. In this context, the omission of RCG from Table 1 is particularly difficult to understand. RCG is a central baseline motivating the paper and the framework on which JERI is built, yet the DiT-XL experiments do not compare against it. Consequently, Table 1 does not establish whether increasing model capacity yields a more substantial advantage over RCG or whether the gains remain similarly modest. The improvement over ReDi therefore does not directly address this concern.
- The larger improvements in some subset and representation-space experiments demonstrate benefits in those settings. However, they do not establish whether comparable gains persist with a larger or more extensively trained RCG baseline.
To summarize, introducing partial coupling is a sensible design choice, but I still view it as a modest methodological extension. Together with the limited evidence of substantial gains over RCG in the full-ImageNet experiments, this leaves me unconvinced that the contribution is sufficiently significant for acceptance. Therefore I still think it should not be accepted at ICLR
We thank the reviewer for their recognition of the importance and timeliness of our research question. We are glad that the reviewer thinks our framing is well prepared and the motivation is clear. Meanwhile, we are especially delighted that the reviewer likes the empirical validation of our formal analysis in Section 6.4. Below, we explain our theoretical contribution, discuss the generation results, and provide the requested evaluation details.
Theoretical Insights Beyond Intuition
Thank you for reading our theoretical analysis in detail! The intuition in the main text indicates that each endpoint, RCG and ReDi/REG, has a drawback. However, it does not say whether an interior timestep is better. Our analysis quantifies this trade-off through the trajectory cost, which depends on the forward process alone and can be probed without training a diffusion model. It also gives sufficient conditions under which an interior timestep has a lower trajectory cost than both endpoints. This turns the intuition into a quantitative prediction that is available before training.
We agree that Propositions 4.1–4.2 are expected on their own. Their role is to establish the two nonzero endpoint costs, which Equation 6 accumulates along each coupling schedule. As the coupling timestep changes, one cost increases and the other decreases. However, their sum can still be lowest at an endpoint, so whether an interior timestep is better depends on the sizes of the two costs.
The reviewer’s summary is correct for the final step of the proof of Theorem 4.1, after the endpoint improvements have been established. The main mathematical step is deriving sufficient conditions for these improvements. Lemma A.7 lower-bounds the cost reduction from leaving each endpoint. Its proof uses monotonicity and telescoping from Lemmas A.5–A.6 to reduce the per-step sums to a few mutual-information terms of the forward process. Conditions (C1)–(C2) require these bounds to remain positive after accounting for JERI’s timestep pairing in Equation 7, thereby guaranteeing the endpoint improvements.
Under class probing on ImageNet, both conditions hold for DINOv2 and MoCo v3, and the trajectory cost has an interior minimum. Section 6.4, which the reviewer also points to, tests this prediction in generation: both tested interior timesteps achieve lower image FID than RCG and fully joint generation.
Improvements beyond smaller or weaker setups
Thanks for raising this concern. Our results also show improvements with our largest model and on datasets with greater semantic diversity.
First, JERI’s improvements extend to DiT-XL, the largest model in our evaluation. Table 1 shows that on full ImageNet class-conditional generation, JERI achieves 3.9 FID compared with 5.9 for ReDi, with both models trained for 400k iterations.
Second, JERI improvements are also more notable on more complex setup. We train unconditional generation on three 100-class ImageNet subsets with increasing semantic diversity. Table 4 shows JERI’s FID improvements over RCG grow from 0.0 to 3.4 and 5.9 as the dataset is growing more diverse.
Our full ImageNet evaluation uses single-run FID-50K and follows RCG and REPA in reporting results without error bars. At 1M iterations, JERI improves both FID and IS over RCG in Table 2, and it matches or improves over RCG in every other comparison we report. Gaps of this size are also used to compare model variants under the same protocol. For example, REPA Table 2 selects its alignment depth with a 0.3 FID gap.
DDPM formulation
We agree that the DDPM formulation is familiar to most readers. The brief recap in Section 3.1 establishes the notation used throughout our theoretical analysis and appendix.
Clarification on CFG
Thank you for noticing the ambiguity. We would clarify that none of our reported results use class-label CFG. Appendix D.4 describes CFG using image conditions for representation denoising (Equation D.2) and representation conditions for image denoising (Equation D.3).
RCG and ReDi also use representation guidance for image denoising. We select guidance weights separately for RCG and JERI to obtain each method’s best image FID, and use ReDi’s reported results with its selected representation guidance. We will label Equations D.2 and D.3 as image guidance and representation guidance, respectively, in the revised manuscript to distinguish them clearly from class-label CFG.
R-FID at 1M iteration
R-FID at 1M was not evaluated at submission time. We have since computed it on the same 1M checkpoints as Table 2:
JERI improves over RCG on all three metrics at 1M; we will add the R-FID column to Table 2.