- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the limitations of representation alignment in diffusion model training, where alignment objectives rely on attractive forces. To address this limitation, the authors introduce an intra-image contrastive objective using InfoNCE to impose a repulsive force between different spatial patches. The proposed method is evaluated with REPA and iREPA, demonstrating improvements in training convergence and generation quality across SiT, JiT, and MMDiT architectures.
Critical strengths
The paper provides a clear presentation of the proposed method, making the authors' main idea easy to understand. The proposed method is also evaluated across various backbones, including diffusion models in latent and pixel spaces and text-to-image models.
Critical weaknesses
Prior works [1–3] have also proposed dispersion and diversity objectives to improve training convergence and generation quality in diffusion models. While CoRA aims to distinguish spatial patches within an image, it is unclear how this approach differs from what existing representation dispersion methods aim to achieve. Although the experimental results show improvements in generation quality, it is difficult to determine whether these gains arise from intra-image patch discrimination or from the general effects of representation dispersion. The authors should provide further analysis of why intra-image discrimination is necessary and what benefits it offers during training. Direct comparisons with prior works [1–3] are also needed to establish the distinction and effectiveness of the proposed approach.
CoRA is evaluated against REPA and iREPA, but comparisons with other methods for improving diffusion model training efficiency and generation quality [4–9] are missing. Beyond demonstrating improvements over REPA and iREPA, the authors should establish whether the proposed method achieves competitive generation quality compared to existing approaches. Moreover, the main experiments are limited to 400K training iterations, which is substantially fewer than the 4M–7M iterations used in prior works. This training budget is insufficient to assess the generation quality of diffusion models after extended training. While the current results demonstrate improvements in convergence during early training, they do not establish whether the proposed method can achieve competitive generation quality with longer training. The authors should extend the experiments to training budgets comparable to those of prior works and evaluate the final generation quality to support the effectiveness of the proposed method.
References
[1] Wang and He, 2025. Diffuse and Disperse: Image Generation with Representation Regularization.
[2] Yang et al., 2026. DiverseDiT: Towards Diverse Representation Learning in Diffusion Transformers.
[3] Li et al., 2026. DiverseDiT++: Quantifying, Analyzing, and Promoting Representation Diversity in Diffusion Transformers.
[4] Leng et al., 2025. REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion Transformers.
[5] Yao et al., 2025. Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models.
[6] Wang et al., 2025. REPA Works Until It Doesn't: Early-Stopped, Holistic Alignment Supercharges Diffusion Training.
[7] Jiang et al., 2026. No Other Representation Component Is Needed: Diffusion Transformers Can Provide Representation Guidance by Themselves.
[8] Wang et al., 2026. SRA2: Variational Autoencoder Self-Representation Alignment for Efficient Diffusion Training.
[9] Wu et al., 2025. Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Think.
Questions for authors
The authors should provide additional experiments and evidence to address the concerns discussed above. Can the authors compare the proposed method with existing representation dispersion methods and, if possible, visualize their differences? Can the authors provide performance comparisons with advanced REPA methods after training the models for enough iterations to achieve practical generation quality?
Citation Formatting (Minor)
The citation formatting does not follow the required style and should be corrected.
10 comments
Response to Reviewer 3Ar9 (Part 2)
Q2 / W2 (continued). Matched comparison and longer training. We agree that experiments ending at 400K iterations support an early-convergence claim but cannot establish final quality after 4M–7M iterations. Our protocol follows iREPA, whose principal comparisons are concentrated at 100K–400K iterations. Due to limited compute when preparing the original submission, we did not train an iREPA SiT-XL/2 baseline from scratch. We instead evaluated the checkpoint released in the official iREPA repository, while training CoRA separately. We acknowledge that the difference in training provenance weakens that comparison.
During rebuttal, we trained the iREPA-XL/2 baseline from scratch under the same training setup as CoRA and evaluated the newly trained models from 400K through 1M iterations. The table also includes our new SiT-B/2 runs. All 400K figures below come from these new runs: the original XL/2 comparison, 7.15 FID for the released iREPA weights versus 7.49 for our CoRA run, remains a separate result and is not replaced by the new 400K comparison. We did not use Classifier-Free Guidance (CFG) during the sampling process.
| Backbone | Method | Metric | 400K | 600K | 800K | 1M |
|---|---|---|---|---|---|---|
| SiT-B/2 | iREPA | FID ↓ | 19.86 | 17.83 | 16.83 | 16.03 |
| SiT-B/2 | iREPA + CoRA | FID ↓ | 19.57 | 17.74 | 16.78 | 15.96 |
| SiT-XL/2 | iREPA | FID ↓ | 7.42 | 7.07 | 6.59 | 6.42 |
| SiT-XL/2 | iREPA + CoRA | FID ↓ | 7.36 | 6.90 | 6.54 | 6.38 |
CoRA achieves lower FID at every evaluated checkpoint in the matched B/2 and XL/2 runs. The overall reduction in the FID benefit at later budgets is consistent with diminishing returns as training approaches saturation, while the improvement persists through 1M steps.
We additionally evaluated variability using three independent training seeds for B/2 at 400K and three sampling seeds for the fixed seed-0 XL/2 checkpoints at 1M:
| Evaluation | Method | FID ↓, mean ± SD | IS ↑, mean ± SD |
|---|---|---|---|
| B/2, 400K, 3 training seeds | iREPA | 19.79 ± 0.14 | 75.48 ± 0.31 |
| B/2, 400K, 3 training seeds | iREPA + CoRA | 19.58 ± 0.03 | 74.62 ± 0.33 |
| XL/2, 1M, 3 sampling seeds | iREPA | 6.48 ± 0.05 | 145.51 ± 0.39 |
| XL/2, 1M, 3 sampling seeds | iREPA + CoRA | 6.40 ± 0.02 | 143.10 ± 1.42 |
CoRA achieves lower FID in every paired training-seed comparison on B/2 and every paired sampling-seed comparison on XL/2. The mean FID reductions of 0.21 and 0.08 exceed the reported FID SD of either method in their respective evaluations. Computed from the unrounded results, these gains are approximately 7.6 times CoRA’s B/2 training-run SD and 4.1 times CoRA’s XL/2 sampling SD (1.5 and 1.6 times the respective baseline SDs). These results support a consistent FID improvement under the tested setup. IS is lower for CoRA in these summaries, so we distinguish the observed FID benefit from improvement in every metric.
In our experimental environment, training a single iREPA-XL/2 model to 1M iterations takes approximately 3–4 days, and generating evaluation samples for one checkpoint takes approximately six hours. Training and evaluating matched iREPA and CoRA models for 4M–7M iterations is therefore beyond what we can complete with our limited computational resources within the rebuttal window. We have extended our evaluation to 1M iterations and restrict our conclusions to the budgets actually tested.
Response to Reviewer 3Ar9 (Part 3)
Q2 / W2 (continued). Comparison with the methods cited in the review. We have added a discussion of methods [1–9] in Appendix A. To put our results in context, the table below collects representative ImageNet-256 FID-50K results reported in their original papers. We state the backbone, training budget, and CFG setting because these numbers were obtained under different pipelines and are not a controlled head-to-head comparison.
| Method | Reported model | Training budget | CFG | FID ↓ |
|---|---|---|---|---|
| iREPA, our matched retraining | SiT-XL/2 | 400K | No | 7.42 |
| iREPA + CoRA, our matched run | SiT-XL/2 | 400K | No | 7.36 |
| iREPA, our matched retraining | SiT-XL/2 | 1M | No | 6.42 |
| iREPA + CoRA, our matched run | SiT-XL/2 | 1M | No | 6.38 |
| Dispersive Loss [1] | SiT-XL/2 | 80 epochs | No | 15.95 |
| DiverseDiT + REPA [2] | DiverseDiT-XL/2 | 400K | No | 8.09 |
| DiverseDiT++ + REPA [3] | DiverseDiT-XL/2 | 400K | No | 8.06 |
| REPA-E [4] | SiT-XL/2, jointly tuned VAE | 400K | No | 4.07 |
| VA-VAE + LightningDiT [5] | LightningDiT-XL, VA-VAE | 64 epochs | No | 5.14 |
| HASTE [6] | SiT-XL/2 | 100 epochs | No | 5.31 |
| SRA² + REPA [8] | SiT-XL/2 | 400K | No | 6.80 |
| REG [9] | SiT-XL/2 + Entangled Token | 400K | No | 3.40 |
| SRA [7] | SiT-XL/2 | 800 epochs | Yes | 1.58 |
The published 400K results for DiverseDiT + REPA and DiverseDiT++ + REPA are numerically higher than our matched SiT-XL/2 result, whereas REPA-E, SRA² + REPA, and REG report lower FIDs under their respective setups. In particular, REPA-E changes the tokenizer through joint VAE training, DiverseDiT and DiverseDiT++ alter cross-block information flow, and REG changes the generative process. SRA’s value in the table uses CFG and a much longer training budget. We will distinguish these published results from our controlled REPA/iREPA comparisons and limit our claims about final generation quality to the budgets we have tested.
Minor comment. Citation formatting. We have corrected the citation formatting in the revised manuscript. Thank you for pointing this out.
Thank you again for your time and constructive feedback. Your comments have helped us clarify CoRA’s relationship to prior work and strengthen the experimental evaluation and presentation of our results.
Response to Reviewer 3Ar9 (Part 1)
Thank you for your constructive comments. We agree that the distinction between CoRA and existing diversity methods, the scope of our comparisons, and performance beyond 400K iterations need to be clearer.
We separate your compound question into Q1 / W1 (distinction from representation dispersion and the role of intra-image discrimination) and Q2 / W2 (comparisons with recent methods and longer-budget evaluation). We also address the minor citation-formatting comment below.
Q1 / W1. Distinction from representation dispersion methods [1–3]. CoRA and these methods share a broad interest in representation diversity, but they operate on different relationships. Dispersive Loss [1] disperses representations of different images in a minibatch. Its objective is positive-free and does not require a frozen teacher. CoRA’s default objective instead contrasts patches within the same image: each student patch is aligned with its spatially matched frozen-teacher patch and contrasted against other teacher patches. Our question is whether attractive-only representation alignment preserves enough intra-image spatial distinction. The cross-image negative-pool ablation extends the pool of patch negatives; it does not change the objective into sample-level dispersion.
DiverseDiT [2] and DiverseDiT++ [3] study diversity across transformer blocks and introduce long residual connections alongside diversity objectives. CoRA does not modify the generative backbone or inference path; its auxiliary projection heads are discarded after training. Comparing published FID values across these different architectures and training recipes would not isolate the effect of intra-image patch discrimination. This is why our controlled comparisons use REPA and iREPA, the alignment methods that CoRA directly extends.
Q2 / W2. Relation to other efficiency methods [4–9]. REPA-E [4] jointly tunes the VAE and diffusion model, while 5 addresses the quality of the visual tokenizer and latent space. CoRA keeps the tokenizer fixed and changes the objective applied to patch-level teacher alignment. HASTE [6] studies holistic alignment and when to stop applying it; this scheduling question is complementary to CoRA’s spatial discrimination objective. SRA [7] obtains guidance from the diffusion model’s own representations, and SRA² [8] uses VAE representations, whereas CoRA contrasts patches against a frozen external vision teacher. REG [9] introduces global semantic information into the generative process; CoRA applies a local, training-only objective. Because the published results use different model, tokenizer, and training settings, we will not present their reported scores as a controlled comparison with CoRA.
We have revised the manuscript accordingly and added a new section to Appendix A, “Comparison with Recent Diffusion Training Methods.” It compares CoRA with the representation dispersion methods [1–3] and the other diffusion training methods [4–9] cited in your review. The section clarifies the differences in the level of discrimination, source of representation guidance, backbone design, and training schedule.