acceptodds
Under review as a conference paper at ICLR 2027

Attract to Align, Repel to Differentiate: Contrastive Representation Alignment for Diffusion Models

Abstract

Representation alignment effectively accelerates diffusion transformer training by pulling hidden states toward frozen vision model features. However, solving independent point-to-point regression makes this alignment purely attractive. It lacks the repulsive force needed to differentiate spatially distinct patches, causing inefficient structural transfer and feature oversmoothing. We propose Contrastive Representation Alignment (CoRA) to mitigate representation collapse by introducing an intra-image contrastive objective. By explicitly contrasting student patches against distinct teacher patches, CoRA provides the missing repulsive gradient to maximize spatial micro-diversity. To prevent destructive gradient interference, we decouple the alignment and repulsion objectives into parallel pathways. This enables safe scaling of the negative pool size and ensures seamless compatibility with mainstream frameworks, including convolution-based approaches like iREPA. Extensive experiments demonstrate that CoRA consistently accelerates convergence and improves generative quality, adding negligible training overhead and zero inference cost.

Primary area:
generative models
Keywords:
Diffusion Transformers, Representation Alignment, Contrastive Learning, Generative Models, Efficient Training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3Ar9OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the limitations of representation alignment in diffusion model training, where alignment objectives rely on attractive forces. To address this limitation, the authors introduce an intra-image contrastive objective using InfoNCE to impose a repulsive force between different spatial patches. The proposed method is evaluated with REPA and iREPA, demonstrating improvements in training convergence and generation quality across SiT, JiT, and MMDiT architectures.

Critical strengths

The paper provides a clear presentation of the proposed method, making the authors' main idea easy to understand. The proposed method is also evaluated across various backbones, including diffusion models in latent and pixel spaces and text-to-image models.

Critical weaknesses

Prior works [1–3] have also proposed dispersion and diversity objectives to improve training convergence and generation quality in diffusion models. While CoRA aims to distinguish spatial patches within an image, it is unclear how this approach differs from what existing representation dispersion methods aim to achieve. Although the experimental results show improvements in generation quality, it is difficult to determine whether these gains arise from intra-image patch discrimination or from the general effects of representation dispersion. The authors should provide further analysis of why intra-image discrimination is necessary and what benefits it offers during training. Direct comparisons with prior works [1–3] are also needed to establish the distinction and effectiveness of the proposed approach.

CoRA is evaluated against REPA and iREPA, but comparisons with other methods for improving diffusion model training efficiency and generation quality [4–9] are missing. Beyond demonstrating improvements over REPA and iREPA, the authors should establish whether the proposed method achieves competitive generation quality compared to existing approaches. Moreover, the main experiments are limited to 400K training iterations, which is substantially fewer than the 4M–7M iterations used in prior works. This training budget is insufficient to assess the generation quality of diffusion models after extended training. While the current results demonstrate improvements in convergence during early training, they do not establish whether the proposed method can achieve competitive generation quality with longer training. The authors should extend the experiments to training budgets comparable to those of prior works and evaluate the final generation quality to support the effectiveness of the proposed method.

References

[1] Wang and He, 2025. Diffuse and Disperse: Image Generation with Representation Regularization.

[2] Yang et al., 2026. DiverseDiT: Towards Diverse Representation Learning in Diffusion Transformers.

[3] Li et al., 2026. DiverseDiT++: Quantifying, Analyzing, and Promoting Representation Diversity in Diffusion Transformers.

[4] Leng et al., 2025. REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion Transformers.

[5] Yao et al., 2025. Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models.

[6] Wang et al., 2025. REPA Works Until It Doesn't: Early-Stopped, Holistic Alignment Supercharges Diffusion Training.

[7] Jiang et al., 2026. No Other Representation Component Is Needed: Diffusion Transformers Can Provide Representation Guidance by Themselves.

[8] Wang et al., 2026. SRA2: Variational Autoencoder Self-Representation Alignment for Efficient Diffusion Training.

[9] Wu et al., 2025. Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Think.

Questions for authors

The authors should provide additional experiments and evidence to address the concerns discussed above. Can the authors compare the proposed method with existing representation dispersion methods and, if possible, visualize their differences? Can the authors provide performance comparisons with advanced REPA methods after training the models for enough iterations to achieve practical generation quality?

Citation Formatting (Minor)

The citation formatting does not follow the required style and should be corrected.

10 comments
AuthorsOpenReview ↗

Response to Reviewer 3Ar9 (Part 1)

Thank you for your constructive comments. We agree that the distinction between CoRA and existing diversity methods, the scope of our comparisons, and performance beyond 400K iterations need to be clearer.

We separate your compound question into Q1 / W1 (distinction from representation dispersion and the role of intra-image discrimination) and Q2 / W2 (comparisons with recent methods and longer-budget evaluation). We also address the minor citation-formatting comment below.

Q1 / W1. Distinction from representation dispersion methods [1–3]. CoRA and these methods share a broad interest in representation diversity, but they operate on different relationships. Dispersive Loss [1] disperses representations of different images in a minibatch. Its objective is positive-free and does not require a frozen teacher. CoRA’s default objective instead contrasts patches within the same image: each student patch is aligned with its spatially matched frozen-teacher patch and contrasted against other teacher patches. Our question is whether attractive-only representation alignment preserves enough intra-image spatial distinction. The cross-image negative-pool ablation extends the pool of patch negatives; it does not change the objective into sample-level dispersion.

DiverseDiT [2] and DiverseDiT++ [3] study diversity across transformer blocks and introduce long residual connections alongside diversity objectives. CoRA does not modify the generative backbone or inference path; its auxiliary projection heads are discarded after training. Comparing published FID values across these different architectures and training recipes would not isolate the effect of intra-image patch discrimination. This is why our controlled comparisons use REPA and iREPA, the alignment methods that CoRA directly extends.

Q2 / W2. Relation to other efficiency methods [4–9]. REPA-E [4] jointly tunes the VAE and diffusion model, while 5 addresses the quality of the visual tokenizer and latent space. CoRA keeps the tokenizer fixed and changes the objective applied to patch-level teacher alignment. HASTE [6] studies holistic alignment and when to stop applying it; this scheduling question is complementary to CoRA’s spatial discrimination objective. SRA [7] obtains guidance from the diffusion model’s own representations, and SRA² [8] uses VAE representations, whereas CoRA contrasts patches against a frozen external vision teacher. REG [9] introduces global semantic information into the generative process; CoRA applies a local, training-only objective. Because the published results use different model, tokenizer, and training settings, we will not present their reported scores as a controlled comparison with CoRA.

We have revised the manuscript accordingly and added a new section to Appendix A, “Comparison with Recent Diffusion Training Methods.” It compares CoRA with the representation dispersion methods [1–3] and the other diffusion training methods [4–9] cited in your review. The section clarifies the differences in the level of discrimination, source of representation guidance, backbone design, and training schedule.

AuthorsOpenReview ↗

Response to Reviewer 3Ar9 (Part 2)

Q2 / W2 (continued). Matched comparison and longer training. We agree that experiments ending at 400K iterations support an early-convergence claim but cannot establish final quality after 4M–7M iterations. Our protocol follows iREPA, whose principal comparisons are concentrated at 100K–400K iterations. Due to limited compute when preparing the original submission, we did not train an iREPA SiT-XL/2 baseline from scratch. We instead evaluated the checkpoint released in the official iREPA repository, while training CoRA separately. We acknowledge that the difference in training provenance weakens that comparison.

During rebuttal, we trained the iREPA-XL/2 baseline from scratch under the same training setup as CoRA and evaluated the newly trained models from 400K through 1M iterations. The table also includes our new SiT-B/2 runs. All 400K figures below come from these new runs: the original XL/2 comparison, 7.15 FID for the released iREPA weights versus 7.49 for our CoRA run, remains a separate result and is not replaced by the new 400K comparison. We did not use Classifier-Free Guidance (CFG) during the sampling process.

BackboneMethodMetric400K600K800K1M
SiT-B/2iREPAFID ↓19.8617.8316.8316.03
SiT-B/2iREPA + CoRAFID ↓19.5717.7416.7815.96
SiT-XL/2iREPAFID ↓7.427.076.596.42
SiT-XL/2iREPA + CoRAFID ↓7.366.906.546.38

CoRA achieves lower FID at every evaluated checkpoint in the matched B/2 and XL/2 runs. The overall reduction in the FID benefit at later budgets is consistent with diminishing returns as training approaches saturation, while the improvement persists through 1M steps.

We additionally evaluated variability using three independent training seeds for B/2 at 400K and three sampling seeds for the fixed seed-0 XL/2 checkpoints at 1M:

EvaluationMethodFID ↓, mean ± SDIS ↑, mean ± SD
B/2, 400K, 3 training seedsiREPA19.79 ± 0.1475.48 ± 0.31
B/2, 400K, 3 training seedsiREPA + CoRA19.58 ± 0.0374.62 ± 0.33
XL/2, 1M, 3 sampling seedsiREPA6.48 ± 0.05145.51 ± 0.39
XL/2, 1M, 3 sampling seedsiREPA + CoRA6.40 ± 0.02143.10 ± 1.42

CoRA achieves lower FID in every paired training-seed comparison on B/2 and every paired sampling-seed comparison on XL/2. The mean FID reductions of 0.21 and 0.08 exceed the reported FID SD of either method in their respective evaluations. Computed from the unrounded results, these gains are approximately 7.6 times CoRA’s B/2 training-run SD and 4.1 times CoRA’s XL/2 sampling SD (1.5 and 1.6 times the respective baseline SDs). These results support a consistent FID improvement under the tested setup. IS is lower for CoRA in these summaries, so we distinguish the observed FID benefit from improvement in every metric.

In our experimental environment, training a single iREPA-XL/2 model to 1M iterations takes approximately 3–4 days, and generating evaluation samples for one checkpoint takes approximately six hours. Training and evaluating matched iREPA and CoRA models for 4M–7M iterations is therefore beyond what we can complete with our limited computational resources within the rebuttal window. We have extended our evaluation to 1M iterations and restrict our conclusions to the budgets actually tested.

AuthorsOpenReview ↗

Response to Reviewer 3Ar9 (Part 3)

Q2 / W2 (continued). Comparison with the methods cited in the review. We have added a discussion of methods [1–9] in Appendix A. To put our results in context, the table below collects representative ImageNet-256 FID-50K results reported in their original papers. We state the backbone, training budget, and CFG setting because these numbers were obtained under different pipelines and are not a controlled head-to-head comparison.

MethodReported modelTraining budgetCFGFID ↓
iREPA, our matched retrainingSiT-XL/2400KNo7.42
iREPA + CoRA, our matched runSiT-XL/2400KNo7.36
iREPA, our matched retrainingSiT-XL/21MNo6.42
iREPA + CoRA, our matched runSiT-XL/21MNo6.38
Dispersive Loss [1]SiT-XL/280 epochsNo15.95
DiverseDiT + REPA [2]DiverseDiT-XL/2400KNo8.09
DiverseDiT++ + REPA [3]DiverseDiT-XL/2400KNo8.06
REPA-E [4]SiT-XL/2, jointly tuned VAE400KNo4.07
VA-VAE + LightningDiT [5]LightningDiT-XL, VA-VAE64 epochsNo5.14
HASTE [6]SiT-XL/2100 epochsNo5.31
SRA² + REPA [8]SiT-XL/2400KNo6.80
REG [9]SiT-XL/2 + Entangled Token400KNo3.40
SRA [7]SiT-XL/2800 epochsYes1.58

The published 400K results for DiverseDiT + REPA and DiverseDiT++ + REPA are numerically higher than our matched SiT-XL/2 result, whereas REPA-E, SRA² + REPA, and REG report lower FIDs under their respective setups. In particular, REPA-E changes the tokenizer through joint VAE training, DiverseDiT and DiverseDiT++ alter cross-block information flow, and REG changes the generative process. SRA’s value in the table uses CFG and a much longer training budget. We will distinguish these published results from our controlled REPA/iREPA comparisons and limit our claims about final generation quality to the budgets we have tested.

Minor comment. Citation formatting. We have corrected the citation formatting in the revised manuscript. Thank you for pointing this out.

Thank you again for your time and constructive feedback. Your comments have helped us clarify CoRA’s relationship to prior work and strengthen the experimental evaluation and presentation of our results.

Reviewer_fmaTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CoRA adds a patch-level InfoNCE term to REPA-style alignment. Each projected DiT token is pulled toward its matched DINOv3 token and pushed away from other teacher tokens in the same image. Negatives come from a local spatial window. On iREPA, the contrastive term uses a separate MLP head. Experiments cover SiT-B/2 and XL/2 (≤400K iterations), JiT-B/16, and MMDiT on COCO, with FID gains over REPA and iREPA that are largest early in training.

Critical strengths
  1. Clear early gains over REPA at negligible cost. On SiT-XL/2 at 100K iterations, FID drops from 21.40 to 16.40 without CFG and from 7.36 to 5.46 with CFG. Gains are consistent across backbones, and inference is unchanged.
  2. Useful head-routing ablation (Table 6). NCE through the conv head hurts, and the Decoupled Head control rules out extra parameters as the cause. Window negatives beating teacher-guided masking is also a non-obvious finding.
Critical weaknesses
  1. Gains over iREPA are small, mixed, and from a single seed.

    • On XL/2 at 400K without CFG, CoRA is worse (FID 7.49 vs 7.15).
    • With CFG the gap is ~0.1 FID, and IS is lower.
    • No variance is reported.
    • iREPA is re-evaluated from released weights, while CoRA is trained in the authors' own codebase.

    "Significant acceleration" is only supported relative to REPA.

  2. Under-specified method and untested mechanism.

    • Full-image negatives (Eq. 2; Table 2, k=1) barely help (23.49 → 23.15). The headline 21.58 matches the 5×5 window, but the default pool is never stated.
    • "Oversmoothing" is shown only qualitatively (Fig. 3a).
    • Hyperparameters are tuned on the reported FID setting.
    • Novelty is moderate: the method is close to PatchNCE (Park et al. 2020) and CRD (Tian et al. 2020), neither of which is cited.
Questions for authors
  1. Which negative pool is used in Tables 1, 4 and 5?
  2. Can you give multi-seed mean ± std against iREPA, retrained under the same pipeline?
  3. Can you quantify oversmoothing, e.g., the correlation between student and teacher self-similarity matrices?
  4. Do the gains over iREPA hold at ≥1M iterations? What is the wall-clock overhead?
4 comments
AuthorsOpenReview ↗

Response to Reviewer fmaT (Part 1)

Thank you for the detailed review. We agree that the submission should identify the negative pool used for each result, distinguish baseline provenance, and report the limits of the gains over iREPA. We conducted matched retraining, repeated sampling, representation analysis, and training-cost measurements during rebuttal.

We label your two Critical Weaknesses as W1 (evidence and fairness of the comparison with iREPA) and W2 (method specification, mechanism, hyperparameter selection, and novelty). The headings below identify the questions and weaknesses addressed.

Q1 / W2. Negative-pool configuration. The negative-pool ablations were conducted with MLP-based REPA on SiT-B/2. In the main table, the SiT-B/2 MLP-based CoRA result of 21.58 FID at 400K steps without CFG uses the spatial 5×5 pool. All other CoRA main-table results use vanilla_nce, whose negatives are the other teacher patches in the same image. This includes the Conv-based iREPA, JiT, and MM-DiT experiments. We will annotate the 5×5 exception directly in Table 1 and specify the pool used in Tables 4 and 5.

Q2 / W1. Matched retraining and variability. The original XL/2 iREPA numbers were obtained by re-evaluating released weights, whereas CoRA was trained in our pipeline. We agree that this did not provide a matched training comparison. During rebuttal, we trained iREPA and iREPA + CoRA from scratch under the same pipeline and will use these matched runs for the direct comparison in the revision.

For SiT-B/2, three independent training seeds per method gave the following 400K, no-CFG results:

MethodFID ↓, mean ± SD across training seedsIS ↑, mean ± SD across training seeds
iREPA19.79 ± 0.1475.48 ± 0.31
iREPA + CoRA19.58 ± 0.0374.62 ± 0.33

CoRA obtains lower FID for all three independent B/2 training seeds. The mean FID reduction is 0.21 (19.79 → 19.58), exceeding the reported FID SD of either method (0.14 for iREPA and 0.03 for CoRA). This consistent improvement across training seeds supports the effectiveness of CoRA in the matched B/2 comparison. For XL/2, we trained one model per method using training seed 0. At 400K steps without CFG, the matched FID results are 7.42 for iREPA and 7.36 for iREPA + CoRA.

W1. Why can FID improve while IS decreases? The two metrics measure different properties: IS depends on the Inception classifier’s conditional predictions and aggregate class distribution, whereas FID compares generated and real image feature distributions. A lower FID therefore does not imply a higher IS. One possible explanation is that patch-level discrimination changes local image structure without improving, and potentially slightly reducing, the class-level properties rewarded by IS. Our current measurements do not identify which component of IS causes the decrease, so we present this as a hypothesis rather than a demonstrated mechanism. We will report both metrics, and remove claims of uniformly superior generation quality over iREPA.

Q3 / W2. Quantitative representation analysis. We evaluated fixed 400K checkpoints on the same 5,000 validation images at the high-noise timestep t=0.9t=0.9. For each image, we compared the pairwise patch-cosine-similarity matrices of raw student layer-4 features and the frozen DINOv3 teacher.

ComparisonStudent–teacher correlation ↑Similarity RMSE ↓
REPA → REPA + CoRA0.4538 → 0.56990.1705 → 0.2373
iREPA → iREPA + CoRA0.5390 → 0.54560.2431 → 0.2326

CoRA improves the correlation with the teacher’s spatial similarity pattern in both settings. With iREPA, it also reduces the absolute similarity error. For REPA, the higher correlation is accompanied by a higher RMSE, indicating better agreement in the relative pattern of patch relationships but not in their absolute cosine values. Together, these results provide quantitative support that CoRA improves spatial-structure transfer at high noise, while showing that the effect differs between the two alignment backbones.

AuthorsOpenReview ↗

Response to Reviewer fmaT (Part 2)

Q4 / W1. Results through 1M steps. We extended matched seed-0 B/2 and XL/2 training runs to 1M steps. Their no-CFG FID trajectories are:

Model400K600K800K1M
iREPA-B/219.8617.8316.8316.03
iREPA-B/2 + CoRA19.5717.7416.7815.96
iREPA-XL/27.427.076.596.42
iREPA-XL/2 + CoRA7.366.906.546.38

We evaluated each fixed 1M XL/2 checkpoint using three sampling seeds:

Sampling seediREPA FID ↓CoRA FID ↓iREPA IS ↑CoRA IS ↑
06.426.38145.15144.38
16.516.42145.92143.36
26.516.39145.45141.57
Mean ± sample SD6.48 ± 0.056.40 ± 0.02145.51 ± 0.39143.10 ± 1.42

Across the three sampling seeds, CoRA achieves lower FID for each evaluation of the fixed XL/2 1M checkpoints. The mean FID reduction is 0.08 (6.48 → 6.40), exceeding the reported sampling SD of either method (0.05 and 0.02, respectively). These repeated evaluations support a consistent FID benefit for the matched XL/2 checkpoints; the training setup remains one training seed per method.

The FID improvement persists through 1M steps. The overall reduction in the benefit at later budgets is consistent with diminishing returns as training approaches saturation. IS does not improve consistently, as discussed under W1 above. With our limited computational resources, training one XL/2 model to 1M steps takes approximately 3–4 days, and generating evaluation samples for a single checkpoint takes approximately six hours. These costs make it difficult to conduct larger-scale, multi-seed training experiments within the limited rebuttal window. We restrict our conclusions to the evaluated budgets.

Q4. Measured training cost. We benchmarked SiT-B/2 on one GPU with batch size 256, FP16, and compilation, using 100 warm-up and 200 measured steps in each of three repeats:

Measured configurationTime per stepIncreasePeak allocated memoryIncrease
REPA, shared MLP212.87 ms—28.77 GiB—
REPA + vanilla_nce, shared MLP218.44 ms+5.57 ms (+2.62%)28.96 GiB+0.19 GiB
iREPA, Conv head209.17 ms—27.84 GiB—
iREPA + CoRA, separate MLP/NCE head220.24 ms+11.07 ms (+5.29%)29.44 GiB+1.60 GiB

These are full training-step measurements, including the additional head where applicable. Both NCE benchmarks use vanilla_nce. The auxiliary heads are discarded at inference.

W2. Choice of CoRA loss weight. We agree that the concern about hyperparameter tuning applies to the contrastive loss weight λNCE\lambda_{\mathrm{NCE}}. The value 0.2 was selected after examining FID in the reported SiT-B/2, 400K, no-CFG setting. Thus, the 19.15 FID at that setting is not an independent test of the selected weight. We kept the same weight for the other reported architectures and the extended-budget runs rather than retuning it for each setting; those results give some evidence about transfer of the chosen configuration, but do not replace a held-out tuning protocol. We will state this selection procedure and limitation explicitly.

W2. Relationship to PatchNCE and CRD. We agree that PatchNCE and Contrastive Representation Distillation (CRD) are important precedents that the submission should have discussed. We have revised the appendix to compare them directly with CoRA. PatchNCE shares the use of within-image patch negatives, but studies source–output patch correspondence in unpaired image-to-image translation. CRD shares teacher–student contrastive learning, but primarily distinguishes different input instances. CoRA applies contrast to spatial positions within each image for frozen-vision-teacher alignment during diffusion training. When combined with iREPA, it also uses a separate contrastive head, as evaluated in our head-routing ablation. We have clarified that our contribution is this diffusion-alignment design and its measured interaction with the projector, rather than the invention of patch-level contrast or contrastive distillation.

Thank you again. We will revise the configuration notes, baseline provenance, metric discussion, and strength of the claims to reflect these findings.

Thank you for the thorough rebuttal. The clarifications on the negative pool, matched retraining, cost measurements, and positioning relative to PatchNCE/CRD address several of my concerns. Some issues remain:

  1. Mixed configurations. Only the B/2-MLP-400K cell uses the 5×5 window. vanilla_nce in that setting gives 23.15 (Table 2, k=1), barely better than REPA's 23.49. The headline REPA gain there, and "REPA+NCE" in Table 6, therefore mostly reflect the window choice. Please use one pool consistently across Tables 1 and 6, or report both.
  2. Gains over iREPA are marginal, while the IS drop is robust. With matched training, FID improves by 0.21 on B/2 (n=3, Welch p≈0.1) and by 0.04–0.17 on XL/2 (one training seed; sampling-seed SD does not capture training variance). IS decreases consistently: 75.48→74.62 on B/2, about 3× the SD, and 145.5→143.1 on XL/2 at 1M steps.
  3. Representation analysis. The improvement is clear for REPA but small for iREPA (0.539→0.546), consistent with the small FID gains.

Overall, the contribution relative to REPA is clear, but relative to iREPA it is marginal and comes with a consistent IS trade-off. I keep my rating, and would view the paper more favorably if the revision uses a consistent configuration and states the iREPA results and IS trade-off plainly in the abstract and conclusion.

AuthorsOpenReview ↗

Manuscript Revisions Addressing Your Remaining Concerns

Thank you for your helpful follow-up. We have revised the manuscript to clarify the negative-pool configurations in Tables 1 and 6, explicitly state the modest gains over iREPA and the IS trade-off in the abstract and conclusion, and qualify the interpretation of the representation and head-routing analyses. The revised and added text is highlighted in blue for ease of review.

We hope our responses and revisions address your concerns and would appreciate your reconsideration of the paper in light of these changes. Please let us know if any questions remain; we would be happy to clarify further. Thank you again for your time and constructive feedback.

Reviewer_32eCOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper revisits the loss design of representation alignment for diffusion-transformer training (the REPA family). The authors observe that the standard patch-wise cosine alignment objective is purely attractive - it matches each projected diffusion token to its own frozen-encoder target but never compares token i to target j != i - and therefore transfers the teacher's intra-image pairwise patch structure (identified by iREPA as the strongest predictor of generation quality across encoders) only implicitly. They propose CoRA, which adds an intra-image token-level InfoNCE term whose negatives are the frozen teacher's other patch tokens (asymmetric by construction, so joint-encoder collapse cannot occur), and show that when the alignment projector is convolutional (iREPA), the contrastive term must be routed through a parallel position-wise MLP head to avoid gradient interference with the alignment objective (Eq. 2-4; defaults lambda_REPA=1.0, lambda_NCE=0.2, tau=0.07, teacher DINOv3-ViT-B16). The negative-pool design is studied along two axes: pool size across k images is non-monotone (best FID 21.34 at k=4 / 1,023 negatives, degrading to 22.02 at k=32), and spatial-window selection (5x5, 24 negatives, FID 21.58) beats teacher-similarity false-negative masking (rho=0.7, FID 22.25), with the mechanistic argument that the masked "false negatives" are precisely the informative spatially-adjacent hard negatives. On ImageNet-256 with SiT-B/2 and SiT-XL/2 (100K/400K iterations, with and without CFG w=1.5), CoRA improves FID over REPA (e.g., XL/2 at 100K: 21.40 -> 16.40, a 23.4% relative reduction; B/2 with CFG at 100K: 24.48 -> 18.13) and over iREPA (B/2 at 100K: 35.60 -> 34.04; at 400K: 19.59 -> 19.15), and the recipe transfers to pixel-space JiT-B/16 (100ep: 32.56 -> 30.74) and text-to-image MMDiT on MS-COCO (150K, CFG w=2.0: 4.85 -> 4.48). Ablations include head-routing variants with a parameter-matched control, loss-weight sensitivity, global-level versus token-level contrast, a Gram-matching alternative, pair-sample variants, and a DINOv2 replication. Both projectors are discarded at inference, so the method adds no inference cost. Claimed conclusion: an explicit repulsive signal over a deliberately curated negative distribution closes a structural gap in attractive-only alignment, with gains compounding on top of architectural and target-encoder advances.

Critical strengths
  1. The head-routing ablation (Table 6) is unusually well controlled and yields a non-obvious, mechanistically explained finding: adding the same InfoNCE repulsion helps through an MLP head (23.49 -> 21.58) but hurts through iREPA's convolutional head (19.59 -> 21.35); the "Decouple Head" control - identical Conv+MLP architecture with attraction through both heads and no repulsion - lands at 19.64, statistically indistinguishable from plain iREPA (19.59), which cleanly rules out the extra-parameters hypothesis; full CoRA (Conv-attract / MLP-repel) reaches 19.15. This is consequential because it converts a potentially arbitrary engineering choice into a demonstrated interaction between loss geometry and projector inductive bias, and the routing prescription is directly reusable by anyone stacking auxiliary token-level objectives on convolutional alignment heads.
  2. The early-training acceleration is substantial, broad, and comes at zero inference cost: at 100K iterations CoRA cuts FID by 23.4% relative over REPA on SiT-XL/2 (21.40 -> 16.40) and by ~26% under CFG on SiT-B/2 (24.48 -> 18.13), stacks on iREPA (16.20 -> 14.48), and transfers without modification to pixel-space JiT-B/16 and text-to-image MMDiT, with a DINOv2 replication in the appendix (22.75 -> 22.04). The negative-pool study adds a genuinely counterintuitive, well-controlled result (Table 2-3): pool size is non-monotone with a sweet spot, and a 24-negative spatial window matches or beats >1,000 uncurated negatives and decisively beats teacher-guided false-negative masking (21.58 vs 22.25) - reversing a textbook contrastive-learning practice for this dense-prediction setting. Together these make the paper's positive case independent of any single model or benchmark.
Critical weaknesses
  1. The headline table is not fully auditable, and late-budget regressions are omitted from the text. (a) Every result is a single deterministic run: no seeds, no CIs, no significance tests anywhere, while several decisive margins are small - iREPA-stacked B/2 at 400K (19.59 vs 19.15), XL/2 with CFG at 400K (3.04 vs 2.93), JiT 200ep Conv (27.75 vs 27.06) - all plausibly within FID run-to-run noise. (b) On SiT-XL/2 at 400K without CFG, CoRA is worse than iREPA on both metrics (FID 7.15 -> 7.49, IS 132.0 -> 128.1, Table 1), and IS also drops on several Conv rows where FID improves (B/2 400K: 75.71 -> 75.55 without CFG, 202.4 -> 200.0 with; JiT 200ep: 59.08 -> 58.12); the text discusses only the B/2 iREPA stacking and says CoRA "remains highly effective" / "continues to demonstrate superior generation quality" without mentioning these reversals. (c) All Conv-based rows carry a dagger-marked "additional space-normalization method" that is never defined anywhere in the paper or appendix - an undocumented component shared by the iREPA baseline and CoRA rows. (d) Only the XL/2 rows are marked as re-evaluated published weights (*); whether the B/2 REPA/iREPA numbers are the authors' retrainings, under which teacher (DINOv3-B here versus DINOv2 in the original REPA), is never stated. Each item is fixable - add dispersion, report and discuss the regressions (possibly annealing lambda_NCE at late budgets), define the dagger component, and label every row's provenance - but until then the central quantitative claims cannot be independently assessed.
  2. The negative-pool conclusion overstates its own evidence, and the efficiency claim that motivates the default is unmeasured. Uncurated k=4 pooling (1,023 negatives) achieves FID 21.34 (Table 2), better than the paper's chosen spatial-5x5 default (24 negatives, 21.58, matching the Table 1 MLP CoRA row) - so "sample quality decisively outweighs sample quantity" (Sec. 4.3) holds in the efficiency sense only, and the deployed default is not the best-performing configuration. The efficiency half of that argument, "negligible training overhead" (abstract and contributions), is asserted without a single wall-clock or memory measurement for the NCE term at window versus k-scaling; the combination of spatial windowing with multi-image pools is untested; semantic masking is evaluated at a single threshold (rho=0.7); and the InfoNCE temperature (tau=0.07) is never swept despite lambda_NCE receiving a full sensitivity table. In addition, the frozen-teacher asymmetric InfoNCE is essentially token-level contrastive distillation, yet the contrastive-representation-distillation line (e.g., CRD, Tian et al. 2019) is uncited, and the most direct structural alternative - Gram/pairwise-similarity matching - is tested only in Appendix Table 9 (where it indeed loses: 22.82 vs 21.58 on MLP; degrades iREPA to 20.35) rather than discussed as the competing implementation of the paper's own "transfer pairwise structure" thesis. All fixable: quantify overhead, add the window-x-k cell and a tau sweep, cite and promote the Gram comparison.
Questions for authors
  1. (Material) Define the dagger-marked "space-normalization": what operation, applied to which tensors, and is it applied identically to the iREPA baselines and the CoRA Conv rows?
  2. (Material) For every non-starred row of Table 1 (and Tables 2-7): are REPA/iREPA the authors' retrainings under the full CoRA setup, and which teacher encoder (DINOv3-B or DINOv2) does each baseline row use?
  3. (Material) Please provide dispersion for the tight margins - at minimum 2-3 training seeds or bootstrap CIs on FID-50K for iREPA vs CoRA on B/2 at 400K (19.59 vs 19.15) and XL/2 at 400K (7.15 vs 7.49). Are these gaps larger than run-to-run noise?
  4. (Material) Why is the default negative pool the spatial 5x5 window (FID 21.58) rather than uncurated k=4 (21.34)? Please quantify the NCE-term training overhead (wall-clock per iteration and peak memory) for the window versus k-scaling, to substantiate "negligible training overhead."
  5. (Material) Please discuss the SiT-XL/2 400K regression against iREPA (FID 7.15 -> 7.49, IS 132.0 -> 128.1) and the recurring IS drops under FID gains; does lambda_NCE warrant annealing at longer budgets?
  6. (Clarification) Sensitivity to tau? And does combining the spatial window with multi-image pools (window x k) help?
  7. (Clarification) How exactly are the "k images within the mini-batch" chosen (adjacent batch indices, random)? Sec. 1 says "a few neighbouring ones" while Sec. 4.3 says "across k images within the mini-batch."
  8. (Clarification) How does CoRA relate to contrastive representation distillation (CRD, Tian et al., ICLR 2019) and the token-level distillation literature? Will code be released (no repository link appears in the submission)?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 32eC (Part 1)

Thank you for the exceptionally careful review. We appreciate your recognition of the controlled head-routing ablation and the early-training gains. Your questions identified places where our experimental provenance, negative-pool choice, and interpretation of later results needed to be more precise.

We label the two Critical Weaknesses as W1 and W2 below; W1(a–d) follows the subpoints in your review. Each answer identifies the corresponding question and weakness.

Q1 / W1(c). Spatial normalization. The dagger denotes spatial normalization from the original iREPA recipe, not a component introduced by CoRA. For frozen teacher patch tokens x∈RB×T×Dx\in\mathbb{R}^{B\times T\times D}, with mean μT(x)\mu_T(x) and standard deviation σT(x)\sigma_T(x) computed across spatial positions, the operation is

SpNorm⁡(x)=x−γμT(x)σT(x)+ϵ.\operatorname{SpNorm}(x)=\frac{x-\gamma\mu_T(x)}{\sigma_T(x)+\epsilon}.

We follow the official iREPA implementation, with γ=0.6\gamma=0.6 and ϵ=10−6\epsilon=10^{-6}. The iREPA baseline and Conv-based CoRA use the same teacher-feature normalization and convolutional alignment recipe. MLP-based REPA and CoRA do not use spatial normalization.

Q2 / W1(d). Baseline provenance and teacher encoder. All alignment experiments use frozen DINOv3-ViT-B/16 as the default teacher, including both our own training runs and the released checkpoints used for re-evaluation, unless another encoder is explicitly specified. The DINOv2 replication is reported separately in the appendix. The non-starred alignment results were trained in our pipeline, whereas the starred XL/2 REPA and iREPA rows in the submitted Table 1 were obtained by sampling and evaluating released checkpoints.

We agree that the original XL/2 comparison placed released iREPA weights against CoRA trained in our pipeline. It was not a matched training comparison and should not be used to attribute that difference to CoRA. During rebuttal, we trained iREPA-XL/2 and iREPA-XL/2 + CoRA from scratch under the same pipeline. The new matched runs below provide the appropriate direct comparison.

Q3 / W1(a). Repeated-run evidence for the FID improvement. For SiT-B/2 at 400K without CFG, we trained each method with three independent training seeds (0, 1, and 2):

MethodFID ↓, mean ± SDIS ↑, mean ± SD
iREPA19.79 ± 0.1475.48 ± 0.31
iREPA + CoRA19.58 ± 0.0374.62 ± 0.33

CoRA obtains lower FID for all three independent B/2 training seeds. The mean FID reduction is 0.21 (19.79 → 19.58), exceeding the reported FID SD of either method (0.14 for iREPA and 0.03 for CoRA). This consistent improvement across training seeds supports the effectiveness of CoRA in the matched B/2 comparison.

For XL/2, training one model to 1M steps takes approximately 3–4 days in our environment, and generating evaluation samples for one checkpoint takes approximately six hours. Given our limited compute and the short rebuttal window, we trained each method once from scratch (training seed 0), then evaluated each fixed 1M checkpoint with three sampling seeds:

MethodFID ↓, mean ± SDIS ↑, mean ± SD
iREPA6.48 ± 0.05145.51 ± 0.39
iREPA + CoRA6.40 ± 0.02143.10 ± 1.42

Across the three sampling seeds, CoRA achieves lower FID for each evaluation of the fixed XL/2 1M checkpoints. The mean FID reduction is 0.08 (6.48 → 6.40), exceeding the reported sampling SD of either method (0.05 and 0.02, respectively). These repeated evaluations support a consistent FID benefit for the matched XL/2 checkpoints.

AuthorsOpenReview ↗

Response to Reviewer 32eC (Part 2)

Q4 / W2. Choice of negative pool and measured training overhead.

We initially selected the spatial 5×5 window for the MLP-based REPA configuration because it achieved competitive FID using only 24 negatives, compared with 1,023 negatives for grouped k=4k=4 (21.58 vs. 21.34). This motivated a compact, spatially selected pool, although k=4k=4 achieves the better FID. To clarify the scope of this choice, the 5×5 configuration is used for the MLP-based SiT-B/2 main-table result; the other main-table CoRA configurations use full-image vanilla_nce.

We have now measured wall-clock time and peak allocated memory for the spatial windows and all grouped pool sizes. Measurements use SiT-B/2 on one GPU, batch size 256, FP16, and torch.compile, with 100 warmup steps followed by 200 measured steps, averaged over three repeats.

Measured configurationNegatives/queryTime/stepTime increasePeak allocated memoryMemory increaseFID ↓IS ↑
REPA, shared MLP—212.87 ms—28.77 GiB—23.4964.00
REPA + vanilla_nceFull image218.44 ms+5.57 ms (+2.62%)28.96 GiB+0.19 GiB (+0.66%)23.1564.00
Spatial 3×38230.58 ms+16.49 ms (+7.70%)31.97 GiB+3.21 GiB (+11.15%)21.7367.08
Spatial 5×524246.52 ms+33.01 ms (+15.46%)37.98 GiB+9.21 GiB (+32.02%)21.5868.37
Spatial 7×748274.38 ms+61.16 ms (+28.69%)46.98 GiB+18.22 GiB (+63.33%)21.6167.68
Grouped k=2k=2511219.24 ms+5.06 ms (+2.36%)28.96 GiB+0.19 GiB (+0.66%)21.5368.50
Grouped k=4k=41,023219.81 ms+6.18 ms (+2.89%)29.10 GiB+0.33 GiB (+1.15%)21.3468.00
Grouped k=8k=82,047221.39 ms+7.47 ms (+3.49%)29.71 GiB+0.94 GiB (+3.27%)21.6768.05
Grouped k=16k=164,095224.44 ms+10.88 ms (+5.09%)31.21 GiB+2.44 GiB (+8.48%)21.8267.36
Grouped k=32k=328,191230.01 ms+16.21 ms (+7.58%)34.21 GiB+5.44 GiB (+18.91%)22.0267.39

Time and memory measurements are averaged over three benchmark repeats. Increases are computed against each configuration’s paired REPA baseline, whose timings range from 212.87 to 214.18 ms/step; the displayed baseline row corresponds to the vanilla_nce comparison. FID and IS come from separate generation experiments with the corresponding configurations.

These measurements show that fewer negatives do not necessarily imply lower computational cost. In our current implementation, the spatial 5×5 window incurs appreciably greater overhead than grouped k=4k=4, which provides both lower FID and lower time and memory costs. Thus, the original 5×5 choice is supported by its competitive quality with a small negative pool, but the measured trade-off favors k=4k=4.

We will revise the blanket claim of “negligible training overhead” to report configuration-specific costs: the overhead is small for full-image vanilla_nce and grouped k=4k=4, but substantial for the current spatial-window implementation. We will also qualify “sample quality decisively outweighs sample quantity” to reflect the actual evidence: spatial selection can achieve competitive quality with fewer negatives, while the best tested FID is obtained with grouped k=4k=4. All configurations retain zero additional inference cost.

AuthorsOpenReview ↗

Response to Reviewer 32eC (Part 3)

Q5 / W1(b). XL/2 at 400K, later training, IS, and annealing. The submitted XL/2 400K result compared released iREPA weights with a separately trained CoRA model, so the apparent regression could not isolate the effect of CoRA. In the new matched, from-scratch runs, CoRA achieves lower FID at each measured checkpoint without CFG:

ModelMetric400K600K800K1M
iREPA-B/2FID ↓19.8617.8316.8316.03
iREPA-B/2 + CoRAFID ↓19.5717.7416.7815.96
iREPA-B/2IS ↑75.0181.7784.9388.40
iREPA-B/2 + CoRAIS ↑75.0581.4384.4588.44
iREPA-XL/2FID ↓7.427.076.596.42
iREPA-XL/2 + CoRAFID ↓7.366.906.546.38
iREPA-XL/2IS ↑130.31135.58142.03145.15
iREPA-XL/2 + CoRAIS ↑129.73136.20141.29144.38

The checkpoint trajectories above use training seed 0 and sampling seed 0; the three-sampling-seed summary at 1M is reported in Q3. CoRA achieves lower FID at every evaluated checkpoint and for all three XL/2 sampling seeds at 1M. The overall reduction in the FID benefit at later budgets is consistent with diminishing returns as training approaches saturation; the benefit remains positive through 1M steps. We agree that IS declines in some rows despite FID gains, including the original CFG comparisons. FID compares generated and real image feature distributions, whereas IS depends on Inception class predictions; the two need not improve together. The present measurements do not establish a specific cause for the IS decrease. We will report both metrics and avoid claiming uniformly better generation quality.

We also investigated whether reducing the NCE weight during training improves generation quality. These experiments were conducted on SiT-B/2 with iREPA + CoRA, trained for 400K steps, while preparing the initial submission. We compared a constant weight of λNCE=0.2\lambda_{\mathrm{NCE}}=0.2 with cosine decay and schedules that reduce or disable the NCE loss after 100K or 300K steps.

The following results are from the original experimental runs, separate from the new matched runs reported above.

NCE-weight scheduleSteps 0–100KSteps 100K–300KSteps 300K–400KFID ↓IS ↑
Constant weight0.20.20.219.1575.55
Disable after 100K0.20019.2476.66
Cosine decayCosine decayCosine decayCosine decay19.3775.90
Reduce after 300K0.20.20.0519.5875.93
Disable after 300K0.20.2019.5176.14

The constant weight achieves the lowest FID among all tested schedules, supporting our choice of λNCE=0.2\lambda_{\mathrm{NCE}}=0.2 throughout training. Although the annealing schedules yield higher IS, none improves FID over the constant-weight configuration.

Q6 / W2. Sensitivity to temperature and combining spatial windows with multi-image pooling.

We conducted an additional temperature sweep on SiT-B/2 at 400K steps, keeping the other settings fixed:

Temperature τ\tauFID ↓IS ↑
0.0521.9866.84
0.0721.6567.48
0.1021.9866.98

Our default τ=0.07\tau=0.07 achieves the best FID and IS among the three tested values. Both alternatives remain close in FID, suggesting that the result is reasonably stable within this tested range.

Combining spatial windows with multi-image pooling is a useful additional design to investigate. However, we have not evaluated that combination. Our existing experiments vary spatial-window size and multi-image group size separately; they therefore cannot determine whether combining the two improves generation quality. In particular, the non-monotonic results of the kk-sweep show that simply enlarging the pool is insufficient to guarantee improvement, but do not rule out benefits from a carefully constructed combined pool.

Q7 / W2. How are the kk images selected?

The kk images are selected by grouping consecutive indices within the same mini-batch. For example, with k=4k=4, images at batch indices 0–3 form one group, images 4–7 form the next, and so on. Each student patch uses its spatially corresponding teacher patch as the positive and all remaining teacher patches in that group as negatives. With 256 teacher patches per image, this gives 256k−1256k-1 negatives per query.

Thus, “neighbouring images” refers only to adjacency in the mini-batch index order; it does not imply spatial or semantic similarity between images.

AuthorsOpenReview ↗

Response to Reviewer 32eC (Part 4)

Q8 / W2. Relationship to CRD, token-level contrastive learning, Gram matching, and code availability.

We agree that CRD and PatchNCE are relevant precedents and should have been discussed. We have added a comparison in the revised appendix to explain the shared contrastive formulation and the specific contribution of CoRA. CoRA shares teacher–student contrastive representation transfer with CRD and spatially corresponding patch discrimination with PatchNCE. Our setting differs in that the student is a diffusion model processing noisy inputs, while a frozen vision encoder supplies clean-image patch targets. We further study how this contrastive objective interacts with existing attractive alignment and convolutional projectors.

W2. Core contribution and comparison with Gram matching.

Accordingly, our contribution is not a new general contrastive-learning formulation. It is the identification and evaluation of patch discrimination as a complement to attractive alignment in diffusion training, together with the interaction between this objective and the alignment projector. In particular, the controlled head-routing experiments show that applying the contrastive loss through iREPA’s convolutional head harms generation, whereas routing it through a separate position-wise MLP head improves FID. The parameter-matched control further supports the role of objective routing rather than additional projector capacity.

We also agree that Gram matching is a direct alternative to our proposed approach and deserves explicit discussion. The submitted appendix already evaluates this alternative:

Alignment settingAdditional objectiveFID ↓IS ↑
MLP-based REPANone23.4964.00
MLP-based REPAGram matching22.8265.60
MLP-based REPACoRA21.5868.37
Conv-based iREPANone19.5975.71
Conv-based iREPAGram matching20.3573.41
Conv-based iREPACoRA19.1575.55

The main reason we did not adopt Gram matching is that it degrades our strongest alignment baseline, iREPA, increasing FID from 19.59 to 20.35 and reducing IS from 75.71 to 73.41. Although it improves MLP-based REPA, its FID remains worse than CoRA’s. These results favor CoRA among the tested objectives, while also showing that structural objectives must be evaluated in conjunction with the alignment architecture.

Finally, we plan to release the training and evaluation code when possible, including the negative-pool implementations and projector-routing configurations, and will provide the repository link when available.

Thank you again for your time, thoroughness, and constructive feedback, which have helped us significantly strengthen the clarity and rigor of the paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Summary

We thank the reviewers and Area Chair for their careful evaluation and constructive discussion. We have revised the manuscript and highlighted the revised and added text in blue.

Contribution. CoRA introduces explicit patch discrimination into representation-aligned diffusion training. It complements attractive alignment with a contrastive objective and uses a separate contrastive head when combined with iREPA’s convolutional projector. The method improves early-training FID across multiple generative settings without modifying the generative backbone or adding inference-time cost.

Additional evidence and revisions.

  • Matched training and longer evaluation. We retrained iREPA and iREPA + CoRA under the same pipeline and extended SiT-B/2 and SiT-XL/2 evaluations through 1M steps. The appendix reports both FID and IS, including all four requested SiT-XL/2 checkpoints with and without CFG. At 1M steps with matched CFG and guidance-interval sampling, CoRA improves FID from 1.64 to 1.58 and IS from 296.68 to 299.22.
  • Variability and balanced interpretation. We added three independent training seeds for SiT-B/2 and three sampling-seed evaluations of fixed SiT-XL/2 checkpoints, explicitly distinguishing these sources of variability. The abstract and conclusion now acknowledge that incremental gains over iREPA are modest and accompanied by an unguided IS trade-off.
  • Configuration and cost transparency. We clarified the negative pools, spatial normalization, teacher encoder, and relationship between original and newly trained models. We report configuration-specific time and memory costs rather than a blanket claim of negligible overhead.
  • Analysis and positioning. We added quantitative representation analysis, temperature sensitivity, weight-scheduling results, and comparisons with PatchNCE, CRD, and recent diffusion-training methods. We also qualified mechanistic claims and specified sampling protocols in the literature comparison.

The revised paper distinguishes substantial early gains over REPA from smaller additional benefits over iREPA.

We hope these additions and clarifications support a favorable assessment of the revised paper. We remain available to address any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.