acceptodds
Under review as a conference paper at ICLR 2027

Unlocking the Potential of Contrastive Regularization for Diffusion Transformer Training

Abstract

Representation regularization has emerged as an effective way to improve diffusion transformer training. Yet one of the most canonical regularizers–InfoNCE-style contrastive learning–remains largely underexplored in this setting. We naturally ask a question: can standard contrastive learning serve as an effective regularizer for diffusion transformers? Surprisingly, we find that naive InfoNCE regularization brings only marginal gains. Our analysis reveals that the objective is easily dominated by a shortcut: near-orthogonal noise components in diffusion inputs provide a bypass for enlarging negative-pair distances while weakening positive-pair alignment, especially at high noise levels. To unlock the benefit of contrastive regularization, we propose lean epresentation-chored Contrastive () regularization. Specifically, anchors contrastive dispersion with clean-sample representation distances, transferring clean relational semantic structure to noisy diffusion representations to suppress the noise shortcut. Extensive experiments on ImageNet show that  consistently improves diffusion transformer training across model scales and training budgets, outperforming internal feature-alignment and dispersive regularization baselines while matching external DINO-guided REPA. These results re-establish InfoNCE-based contrastive learning as an effective regularizer for diffusion training, once its noise-driven over-dispersion is properly constrained.

Primary area:
generative models
Keywords:
Diffusion models, Representation learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_aHXZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether InfoNCE like contrastive learning can serve as a regularizer for diffusion transformer training, and explains why naive InfoNCE won't work. Based on the explanation, this paper proposes CRAN to clips negative-pair repulsion in InfoNCE using an adaptive anchor derived from clean-sample pairwise distances, transferring clean relational geometry to noisy representations. Experiments show the effectiveness of the proposed method.

Critical strengths
  1. This paper gives a concrete and testable explanation for why prior works fail at applying InfoNCE to diffusion transformer training.
  2. Based on the explanation, the paper proposes a minimal and effectiveness fix. The empirical experiments validate the effectiveness of this fix.
Critical weaknesses
  1. The experiments are limited to ImageNet. Additionally, it would be interesting to explore whether the method can transfer to other tasks such as t2i generation.
Questions for authors

Please see weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer aHXZ

Thank you for your review and we address your concerns below.

1. Evaluation beyond ImageNet and text-to-image generalization (W1 & Q1)


Our evaluation includes two additional image datasets and a motion-generation task. In Appendix C.1, Table 10, CRAN reduces SiT's FID from 6.73 to 5.59 on FFHQ and from 27.6 to 20.5 on WebVision-10-50K. On HumanML3D, CRAN reduces MDM's FID from 0.5320 to 0.3918 for text-to-motion generation (Appendix C.4, Table 13).

To directly address text-to-image generalization, we also evaluate CRAN on MS-COCO using MMDiT.

MethodTraining stepsFID ↓\downarrowPickScore ↑\uparrow
MMDiT150K5.8620.05
MMDiT + REPA150K4.6020.88
MMDiT + SRA150K4.8521.14
MMDiT + CRAN150K4.6921.12

CRAN improves MMDiT on MS-COCO and achieves a lower FID than SRA.

Thank you for responses. I will maintain my positive score.

Thank you very much for your time and effort.

Reviewer_xHMVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Clean Representation-ANchored Contrastive (CRAN) regularization to improve diffusion transformer training. The authors identify a potential limitation of standard InfoNCE regularization, where noisy diffusion representations can exploit noise-induced differences to separate negative pairs without learning useful semantic structure. To address this, CRAN limits negative-pair separation using an adaptive distance threshold estimated from near-clean representations. Experiments demonstrate improvements in generation quality across model scales, training budgets, and several generative modeling settings.

Critical strengths
  1. I find the investigation of contrastive learning for diffusion transformer training interesting and well motivated. In particular, the observation that noise-induced representation differences can interfere with conventional contrastive learning provides a useful perspective on why directly applying InfoNCE may be less effective in diffusion models.

  2. I appreciate the extensive experimental evaluation and ablation studies. The authors compare against several recent representation regularization methods and demonstrate improvements across model scales and training budgets. I also appreciate the additional experiments investigating generalization to other datasets and model formulations, as well as the matched wall-clock comparisons and analysis of training overhead.

Critical weaknesses
  1. I find it unclear whether the improvements come specifically from using clean representations to guide contrastive learning, rather than simply limiting negative-pair separation. The authors argue that standard InfoNCE exploits noise differences between samples, and propose CRAN to address this issue. However, Table 5 shows that using a fixed distance threshold already achieves much of the improvement (27.0 vs. 26.3 FID). Furthermore, CRAN uses a single average distance across the batch, rather than explicitly preserving the relationships between individual clean representations. I would have liked to see stronger evidence that the clean representation anchor offers a meaningful advantage over simpler approaches that limit negative-pair repulsion.

  2. Although the experimental results are encouraging, I find the improvements over the strongest existing methods somewhat limited. For example, CRAN achieves comparable rather than clearly better performance than REPA under classifier-free guidance (1.45 vs. 1.42 FID in Table 4), while some improvements over other representation regularizers are relatively small. The SRA baseline also differs between tables (28.7 vs. 29.1). More importantly, Appendix D.5 demonstrates that combining CRAN with REPA can yield additional improvements, but these experiments are limited to SiT-B/2. I believe evaluating this combination on larger models, particularly SiT-XL/2, would provide stronger evidence of CRAN's practical benefits beyond existing approaches. The absence of this comparison makes it difficult for me to assess the full significance of the proposed regularization.

Questions for authors
  1. Is the adaptive clean anchor computed with stop-gradient? Could the authors also clarify whether anchor statistics are aggregated across GPUs and what the contrastive time interval [0, 0.2) in Table 21 represents? Because the motivation stresses that the shortcut is worst at high noise, and if this means the regularizer only acts at low noise, it is unclear how the high-noise failure mode is addressed.
  2. Could the authors extend the CRAN + REPA experiments in Appendix D.5 to SiT-XL/2 and report the resulting generation performance against REPA alone under matched training and sampling settings? Additionally, REPA-E (Leng et al., ICCV 2025) demonstrates substantial improvements through end-to-end VAE and diffusion transformer training. Could the authors discuss whether CRAN could complement REPA-E and whether its benefits might extend to such stronger training pipelines?
  3. Could the authors evaluate CRAN on a text-to-image diffusion transformer, such as an MMDiT-based architecture? Given the promising results on class-conditional ImageNet generation, it would be interesting to understand whether these improvements extend to more challenging text-conditioned generation settings.
2 comments
AuthorsOpenReview ↗

Response to Reviewer xHMV

Thank you for your constructive feedback. We address your points below.

1. Why use a clean adaptive anchor? (W1)


We agree that limiting excessive negative-pair separation accounts for part of the gain. Table 5 shows, however, that the benefit depends on the threshold: cos⁡70∘\cos 70^\circ does not improve on the unclipped result, and cos⁡60∘\cos 60^\circ, the best fixed threshold we tested, yields 27.0 FID compared with 26.3 for CRAN. Choosing a fixed threshold requires additional training and evaluation trials, and a threshold selected in one setting may not transfer to another model scale or task.

CRAN uses clean representation statistics to calibrate an upper bound on separation; it does not aim to preserve each clean pairwise relationship. Table 8 also shows that, with adaptive clipping, estimating the threshold from noisier features worsens FID from 26.3 to 28.2. This supports the use of clean features as a better reference in the tested setting.

2. Complementarity with REPA and REPA-E (W2 & Q2)


REPA on SiT-XL/2. We compare REPA and REPA + CRAN on SiT-XL/2 using Imagenet 256.

MethodTraining stepsFID ↓\downarrowsFID ↓\downarrowIS ↑\uparrowPrecision ↑\uparrowRecall ↑\uparrow
REPA200K10.735.04104.30.690.64
REPA + CRAN200K9.084.87112.60.700.64
REPA400K7.675.07128.60.710.65
REPA + CRAN400K6.744.92132.70.710.66

Adding CRAN reduces FID from 10.73 to 9.08 at 200K (15.4%) and from 7.67 to 6.74 at 400K (12.1%). It also improves sFID and IS at both budgets.

REPA-E. REPA-E uses external feature alignment to jointly train the VAE and the diffusion model. CRAN regularizes internal diffusion features and can be added to the diffusion-model update directly.

We follow the REPA-E setup with SiT-B/2, ImageNet-256. The effectiveness of CRAN on REPA-E is shown below:

MethodTraining stepsFID ↓\downarrowsFID ↓\downarrowIS ↑\uparrowPrecision ↑\uparrowRecall ↑\uparrow
REPA-E100K34.86.3139.10.570.59
REPA-E + CRAN100K27.95.5248.00.610.59

Compared with the reported REPA-E result, REPA-E + CRAN reduces FID from 34.8 to 27.9 (19.8%) and improves sFID, IS, and precision, with essentially unchanged recall. This shows that CRAN can complement REPA-E in end-to-end training.

3. SRA baseline results (W2)


The 28.7 FID in Table 1 is from our SRA-style, positive-only alignment baseline, which uses cosine distance in our common ablation setup. The 29.1 FID in Table 3 is the result reported in the SRA paper. We will distinguish the two results in the revision.

4. Anchor gradients, cross-GPU aggregation, and timestep range (Q1)


We stop gradients through the adaptive anchor. In multi-GPU training, we all-reduce the sums of within-GPU pairwise angles and the corresponding pair counts, then use the resulting mean to update the anchor shared across GPUs.

The interval [0,0.2)[0,0.2) in Table 21 is the sampling range of the student–teacher timestep difference Δt\Delta t, with tteacher=max⁡(0,tstudent−Δt)t_{\mathrm{teacher}}=\max(0,t_{\mathrm{student}}-\Delta t). It is not a threshold for estimating the clean anchor. The CRAN loss applies across the entire noise range. We will clarify this distinction in the revision.

5. Text-to-image generation with MMDiT (Q3)


To test whether CRAN extends from class-conditional to text-conditioned generation, we evaluate it on MS-COCO using MMDiT.

MethodTraining stepsFID ↓\downarrowPickScore ↑\uparrow
MMDiT150K5.8620.05
MMDiT + REPA150K4.6020.88
MMDiT + SRA150K4.8521.14
MMDiT + CRAN150K4.6921.12

CRAN improves MMDiT on MS-COCO and achieves a lower FID than SRA.

Reviewer_xHMVOpenReview ↗

Official Comment by Reviewer xHMV

I thank the authors for their detailed responses, which have addressed my concerns, especially around CRAN+REPA, CRAN+REPA-E and additional T2I results. I believe that the paper could be a valuable contribution to the field, and I retain my original positive score.

Reviewer_fwXPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the limited effectiveness of standard InfoNCE contrastive regularization in diffusion transformer (DiT) training. The authors identify a potential noise shortcut, where InfoNCE exploits random noise differences to separate negative pairs, leading to excessive feature dispersion and weakened positive-pair alignment.

To address this issue, the paper proposes Clean Representation-ANchored Contrastive (CRAN) regularization, which limits negative-pair separation using an adaptive distance threshold estimated from low-noise representations. The method builds on internal representation alignment without requiring an external pretrained visual encoder.

The authors evaluate CRAN on ImageNet using SiT models across different scales and training budgets, demonstrating improvements over standard InfoNCE and existing internal representation regularizers. They further provide diagnostic analyses, ablation studies, and experiments on additional datasets, resolutions, and generation tasks to validate the effectiveness and generalizability of the proposed method.

Critical strengths

1. A meaningful investigation of contrastive learning in diffusion transformers.

The paper identifies an interesting limitation of directly applying InfoNCE to DiT training and provides systematic analyses of the noise shortcut. Figure 1 and the additional diagnostics in Appendix B offer useful evidence that InfoNCE amplifies noise-induced feature differences and that CRAN alleviates this behavior. These analyses provide a reasonable motivation for the proposed method and offer useful insights into contrastive representation learning for diffusion models.

2. Strong empirical improvements supported by extensive experiments.

CRAN consistently improves generation quality across different model scales and training budgets. The paper includes comprehensive ablations on the positive/negative components, anchor design, and temperature, as well as evaluations across different datasets and generation tasks. Importantly, the improvements remain meaningful under matched wall-clock budgets, and combining CRAN with REPA provides further gains (FID 24.4 → 21.0). These results demonstrate the practical effectiveness and potential generality of the proposed regularization.

Critical weaknesses

1. Potential timestep shortcut and insufficient clarification of the core optimization mechanism.

The paper attributes InfoNCE's limited effectiveness to a Gaussian noise shortcut, but does not rule out a potential timestep shortcut. Positive pairs have correlated noise levels, whereas in-batch negatives may come from different timesteps. Since DiT explicitly incorporates timestep information, the model may exploit timestep differences to distinguish positive and negative pairs. The current diagnostics do not isolate this factor. A comparison using timestep-matched negatives would help determine whether the reported failure is specifically caused by Gaussian noise differences.

Furthermore, Eq. (5) does not clarify whether the adaptive anchor is detached from gradient computation. If gradients propagate through the anchor, clipped negatives can still influence the optimization indirectly, potentially contradicting the stated mechanism of stopping excessive negative-pair repulsion. The authors should clarify the implementation and gradient behavior.

These issues matter because they directly affect the interpretation and validity of the paper's central mechanism.

2. Missing comparison with a strong recent self-supervised DiT method.

The paper claims the best performance among methods without external pretrained visual encoders but omits Self-Flow (ICML 2026), a closely related self-supervised diffusion training method. On ImageNet-256 with SiT-XL/2 at 4M training steps without CFG, Self-Flow reports an FID of 5.70, compared with 6.2 for CRAN.

Although architectural and computational differences prevent a strictly direct comparison, the omission leaves the claimed state-of-the-art performance insufficiently substantiated. A comparison under matched training and evaluation settings, including computational costs, is needed to establish CRAN's competitiveness against recent methods.

Questions for authors

1. Could timestep differences provide an alternative shortcut to Gaussian noise differences?

The paper attributes the failure of InfoNCE to noise-induced feature separation. However, positive pairs have correlated timesteps, while in-batch negatives may come from different timesteps. Since DiT explicitly incorporates timestep information, could the model also exploit these differences to distinguish samples?

Could the authors clarify the timestep sampling strategy and compare InfoNCE and CRAN using timestep-matched negatives? This would help distinguish the proposed Gaussian noise shortcut from a potential timestep shortcut.

2. What is the exact implementation of the CRAN loss and its gradient flow?

In Eq. (4), distance clipping appears to apply to all terms in the denominator, including the positive pair, whereas the text describes clipping only negative pairs. Additionally, Eq. (5) does not specify whether the adaptive anchor is detached from gradient computation.

Could the authors clarify both implementation details and provide the actual loss formulation? These choices directly affect whether negative pairs stop contributing repulsive gradients after reaching the anchor.

3. How does CRAN's negative-pair regularization behave across different noise levels?

Figure 1 shows that negative-pair distances increase with noise strength, while CRAN limits their separation. However, the paper does not quantify how frequently negative pairs exceed the adaptive anchor at different timesteps.

Could the authors report the fraction of clipped negative pairs and the relative positive/negative gradient contributions across noise levels? This would clarify whether CRAN maintains meaningful negative-pair regularization at high noise levels or primarily relies on positive-pair alignment there.

4. Could the authors clarify the experimental settings behind the layer-pairing ablation?

Table 9 reports an FID of 42.9 for student layer 3 and teacher layer 8, whereas Table 21 identifies this layer pair as the default configuration and the main results report an FID of 26.3 on SiT-B/2.

Were these experiments conducted with different training budgets or evaluation settings? Please specify the configurations to clarify how the layer-pairing ablation should be interpreted.

5. How does CRAN compare with Self-Flow under matched experimental settings?

The paper claims the best performance among methods without external pretrained visual encoders but does not compare with Self-Flow (ICML 2026). Self-Flow reports an FID of 5.70 on ImageNet-256 with SiT-XL/2 at 4M steps, compared with CRAN's 6.2.

Could the authors provide a comparison under matched training and evaluation settings, including computational costs? This comparison is important for assessing the claimed state-of-the-art performance.

4 comments
AuthorsOpenReview ↗

Response to Reviewer fwXP

Thank you for your careful review and for your specific suggestions. We address your points below.

1. Timestep shortcuts (W1 & Q1)


In our original setting, we sample student timesteps independently and uniformly across images and derive each teacher timestep from its paired student timestep. To test the potential timestep shortcut, we sample one student timestep and one teacher timestep per training step and share each across the batch. This removes the timestep cue, but the noise shortcut remains uncontrolled because Gaussian noise is still sampled independently across images.

We train both methods for 400K steps on ImageNet-256 with SiT-B/2.

MethodFID ↓\downarrowsFID ↓\downarrowIS ↑\uparrowPrecision ↑\uparrowRecall ↑\uparrow
InfoNCE30.06.1748.60.5610.632
CRAN28.36.1852.30.5700.637

Under this control, CRAN still outperforms InfoNCE, reducing FID from 30.0 to 28.3. Thus, InfoNCE's poorer performance relative to CRAN cannot be attributed to a timestep shortcut.

2. Loss formulation, clipping, and gradient flow (W1 & Q2)


The implementation clips only negative pairs and stops gradients through the adaptive anchor in Eq. (5). The CRAN loss is:

= -\mathbb E_i\left[ \log\frac{\exp(-d_t^{ii}/\tau)}{ \sum_j\exp\left(-\min\left(d_t^{ij},m_{\mathrm{cran}}\right)/\tau\right) } \right], m_{\mathrm{cran}} = \mathrm{sg}\left(\mathbb E_{i\ne j}\left[d(z_0^i,z_0^j)\right]\right).$$ Here, $\mathrm{sg}$ denotes stop-gradient. In practice, the positive-pair distance $d_t^{ii}$ is below $m_{\mathrm{cran}}$, so $\min(d_t^{ii},m_{\mathrm{cran}})=d_t^{ii}$. In this case, leaving the positive term inside the minimum gives the same result as writing it separately without clipping: $$\mathcal L_{\mathrm{CRAN}}(\theta,\psi) = -\mathbb E_i\left[ \log\frac{\exp(-d_t^{ii}/\tau)}{ \exp(-d_t^{ii}/\tau) +\sum_{j\ne i}\exp\left(-\min\left(d_t^{ij},m_{\mathrm{cran}}\right)/\tau\right) } \right].$$ Once $d_t^{ij}$ exceeds $m_{\mathrm{cran}}$, that negative pair no longer contributes a repulsive gradient, either directly or through the anchor. ### 3. Negative-pair clipping and gradients across noise levels (Q3) --- To test whether CRAN retains a negative-pair gradient at high noise, we measure the fraction of clipped negative pairs and the ratio of negative- to positive-term gradient norms by student timestep. | Student timestep $t$ | Clipped negative pairs | Negative/positive gradient norm | |:---:|:---:|:---:| | $[0.0,0.2)$ | 38.7% | 90.6% | | $[0.2,0.4)$ | 30.4% | 90.7% | | $[0.4,0.6)$ | 28.3% | 88.8% | | $[0.6,0.8)$ | 30.7% | 86.4% | | $[0.8,1.0)$ | 42.8% | 81.9% | In the highest-noise bin, 42.8% of negative pairs are clipped, yet the negative-term gradient norm is still 81.9% of the positive-term norm. CRAN therefore retains a substantial negative-pair signal even at high noise, rather than reducing to positive-only alignment. ### 4. Layer-pairing ablation and training budget (Q4) --- For the default pairing (student block 3, teacher block 8), Table 9 reports the result after 200K training steps (42.9 FID), whereas the main result uses 400K steps (26.3 FID). All other training and evaluation settings are the same for this pairing. We will make this distinction explicit in the revision. In Table 9, moving the student from block 3 to block 2 or 4 worsens FID. Moving the teacher from block 8 to block 6 also worsens FID, while moving it to block 10 yields no clear gain. The results support the default 3-to-8 pairing among the tested configurations. ### 5. Comparison with Self-Flow (W2 & Q5) --- Thank you for pointing out this omission. We compare CRAN with Self-Flow below. Both methods use SiT-XL/2 on ImageNet-256 and train for 4M steps. | Method | Training steps | FID without CFG $\downarrow$ | FID with CFG $\downarrow$ | |:---:|:---:|:---:|:---:| | Self-Flow | 4M | **5.7** | 1.47 | | CRAN | 4M | 6.2 | **1.45** | Self-Flow has lower FID without CFG, whereas CRAN has slightly lower FID with CFG. The reported results show that CRAN remains competitive with Self-Flow. We also compare the two methods in terms of training time per step, GFLOPs per sample, and peak memory per GPU on SiT-XL/2. | Method | Time/step | GFLOPs/sample | Peak GPU memory | |:---:|:---:|:---:|:---:| | Self-Flow | 165.67 ms | 1322.3 | 30.89 GiB | | CRAN | 156.11 ms | 893.6 | 26.32 GiB | Self-Flow assigns different timesteps to different tokens during training. To support this, its official implementation computes adaLN modulation per token, whereas CRAN shares the modulation across all tokens of each image. The extra matrix multiplications and intermediate activations account for Self-Flow's higher compute, memory use, and time per step in this comparison.
AuthorsOpenReview ↗

A Gentle Follow-up on Our Rebuttal

Dear Reviewer fwXP,

Thank you again for your thoughtful and detailed review. Your comments have been very helpful in improving our paper.

We have posted a point-by-point response covering each of the issues you raised, including the possible timestep shortcut, the gradient flow through the adaptive anchor, the behavior of negative-pair clipping across noise levels, the settings of Tables 9 , and the comparison with Self-Flow.

We know this is a busy period, so we truly appreciate any time you can spare to look at our response. If anything remains unclear or you have further questions, we would be happy to provide additional clarification or results. If our response has addressed your concerns, we would be grateful if you would consider updating your assessment.

Thank you very much for your time and effort.

Best regards, The Authors

The authors’ clarifications and additional experiments address most of my concerns and strengthen the empirical support for CRAN, so I am raising my recommendation from Weak Reject to Weak Accept.

Thank you for taking the time to revisit our submission and consider our clarifications. We are glad to see that our new results addressed most of your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.