acceptodds
Desk rejected at ICLR 2027

dREPA: Difficulty-Aware Representation Alignment Accelerates Diffusion Training

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

Representation alignment (REPA) accelerates diffusion transformer training by aligning intermediate features with a frozen encoder, but it averages both the denoising and the alignment loss uniformly over spatial tokens, although denoising difficulty concentrates on boundaries and fine structure. We show that the per-token alignment loss, which REPA already computes and discards, is a free spatial difficulty signal: tokens that are hard to align are consistently hard to denoise. Turning a difficulty signal into useful weights is not automatic, however. Reweighting the denoising loss by its own per-token residual hurts, and a matched comparison of signals through one pipeline points to two properties a useful signal needs: it should rank tokens consistently across noise draws, and it should stay current with the model. The online residual lacks the first, a frozen signal lacks the second, and the alignment loss has both at no extra cost. We propose dREPA, which converts the alignment loss into bounded, per-sample-normalized weights for the denoising loss in five lines of code with no additional parameters. dREPA is a training-acceleration method: on SiT-XL/2 it reaches at 1M steps the FID that iREPA reaches at 4M, and with guidance tuned independently for each model it improves FID from 2.21 to 2.04. The gain is consistent across model scales, teacher encoders, latent and pixel-space backbones, resolutions, and class-conditional and text-to-image generation, and on SiT-XL/2 it exceeds the run-to-run variation measured in a matched-seed study. Code: https://anonymous.4open.science/r/dREPA-17C2/README.md

Primary area:
generative models
Keywords:
Representation alignment, Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Px7wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes dREPA, which further improves REPA based on iREPA. The authors argue that the REPA alignment loss in diffusion models is correlated with denoising difficulty. Based on this observation, the authors use the REPA alignment loss to reweight tokens during training, placing more emphasis on difficult-to-learn regions. Experiments across various REPA teachers and diffusion models of different sizes demonstrate that dREPA can accelerate model convergence and achieve better performance at the same number of training steps.

Critical strengths
  1. The authors observe a correlation between the REPA alignment loss and denoising difficulty, which is an insightful finding.
  2. The proposed method is simple. It directly redistributes token weights during training based on the alignment loss, requires no additional architecture, and can be easily integrated into existing methods.
  3. The experiments are comprehensive. The authors conduct extensive experiments using SiT, JiT, and MMDiT across different model sizes and visual encoders, consistently achieving performance improvements.
Critical weaknesses
  1. The paper states that the REPA alignment loss tracks denoising difficulty, but also emphasizes that the method is not hard-token mining. This description is somewhat confusing. Moreover, if the improvement is not due to denoising difficulty (Figure 2 also shows that the correlation between the two is not very strong), the explanation for the effectiveness of dREPA becomes less convincing.The paper only demonstrates some correlations without providing causal evidence or experimental validation.
  2. The effectiveness of dREPA diminishes noticeably under stronger CFG. However, CFG is commonly used in practical applications, which limits the applicability of the proposed method to some extent.
Questions for authors

Clarification is needed. Please refer to Weakness 1.

1 comment

Response to Reviewer Px7w

We thank the reviewer for the comments. Changes in the revised paper are marked in blue.

Weakness 1 and Question 1: difficulty, hard-token mining, and causal evidence

dREPA does not emphasize difficult tokens. It weights each token by its cosine similarity to the teacher feature, normalized within the image, so tokens whose features are already close to the teacher receive larger weights. Eq. (5) of the submission stated the cosine distance, but the implementation behind all reported results uses cosine similarity (see the response to Reviewer q2o7). The revision corrects the equation and removes the difficulty-based explanation. No reported number changes.

This removes the contradiction between "tracks difficulty" and "not hard-token mining". The second statement is the accurate one. Figure 2 shows only a weak association between alignment distance and denoising error; it is kept as a descriptive statistic and is no longer used to explain the gain.

The causal evidence is an intervention on the ordering of the weights. For each sample,

ui=zscore(cos⁡i),wi=1+βui,u_i=\mathrm{zscore}(\cos_i),\qquad w_i=1+\beta u_i,

followed by symmetric clipping and exact mean-one normalization. Positive and negative β\beta give the same weight mean, standard deviation, quantiles and clipping rate. They differ only in which tokens receive the larger weights. FID at 100K steps (SiT-B/2, 50K samples):

β\betaSeed 0Seed 1Seed 2Mean ± std.
-0.642.3442.9842.3842.57 ± 0.29
-0.338.9540.4538.5439.31 ± 0.82
028.2630.0528.4328.92 ± 0.81
+0.322.5423.4221.7722.58 ± 0.67
+0.621.6721.7120.6821.35 ± 0.48

The order +0.6<+0.3<0<−0.3<−0.6+0.6<+0.3<0<-0.3<-0.6 holds in every seed, and the differences between settings are much larger than the variation across seeds. Up-weighting well-aligned tokens improves FID. Up-weighting poorly aligned tokens degrades it. The dREPA rule itself improves all three paired seeds: 28.26 to 24.78, 30.05 to 26.00, 28.43 to 24.63.

Weighting by difficulty does not work. Using the current denoising error as the weight is worse than uniform weighting in all three seeds (30.96 vs. 28.26, 32.06 vs. 30.05, 30.37 vs. 28.43). A moving average of that error is worse in both paired seeds (31.78 vs. 30.05, 30.17 vs. 28.43). Random weights have no effect (18.86 vs. 18.43). The tokens that dREPA emphasizes are on average the easier ones: mean cosine falls from 0.631 in the easiest denoising-error quintile to 0.582 in the hardest at t=0.4t=0.4.

The explanation of effectiveness in the revision follows from these results. The denoising loss improves when more of its weight is placed on tokens whose representation is already aligned with the teacher, and it degrades when the weight is placed on high-error tokens. The alignment signal costs nothing, because REPA computes it at every step, and it ranks tokens more consistently across noise draws than the residual (rank correlation 0.853 vs. 0.569).

Weakness 2: the effect under strong guidance

The FID margin narrows under a shared guidance scale, and the paper states this as a limitation. The other metrics do not narrow. dREPA minus iREPA on SiT-XL/2 at 400K steps:

wwFIDISPrecisionRecall
1.0-1.65+15.9+0.023-0.026
1.35-0.41+18.9+0.020-0.016
1.5-0.02+16.4+0.020-0.008

The IS and precision gains are the same size with and without guidance. FID stops improving with further precision near its best ww, which is why the FID values converge. The two methods also act on different tokens. On paired SiT-B/2 checkpoints at 100K steps (512 images), the per-token error reductions from dREPA and from guidance at w=1.5w=1.5 are negatively correlated (rank correlation -0.20 to -0.22), and the dREPA reduction keeps its spatial pattern under guidance (rank correlation 0.94 to 0.97).

Three results concern practical use.

  • With scale and interval tuned independently on the same grid, iREPA reaches 5.71 and dREPA 4.38 on SiT-B/2, and 2.21 and 2.04 on SiT-XL/2. The tuned dREPA setting on XL/2 guides 60% of the trajectory instead of 80%, about 11% fewer network evaluations.
  • On MS-COCO text-to-image generation with MMDiT, dREPA improves FID at every tested guidance scale from 1.0 to 2.0.
  • dREPA reduces training cost, and guidance is applied after training in either case. dREPA at 1M steps reaches FID 5.75, close to iREPA at 4M steps (5.91), and it is ahead at every evaluated horizon from 400K to 4M.
Reviewer_1NfgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces dREPA, a simple recipe that can be added in REPA-style training, which reuses the per-token alignment loss as a spatial difficulty signal for the denoising loss. Through a matched comparison of weighting signals, the paper shows that reweighting by the model’s own denoising residual hurts, and argue that a useful signal must rank tokens consistently across noise draws and stay current with the model, both of which the alignment loss satisfies. On top of iREPA, dREPA shows consistent FID improvement without CFG across different size of models, teacher encoders, and in more applications (e.g., T2I).

Critical strengths
  • The suggested method is simple and practical which shows improvement over REPA-style training across different setup (e.g., model scale, encoders). The idea of reusing the alignment loss as a difficulty signal for weighting the denoising loss, based on its correspondence with denoising difficulty, can be applied to other similar recipes.

  • The experiments are well organized and comprehensive. They are structured around clearly stated questions and cover a broad range of settings, including different architectures, resolutions, T2I image generation, etc. The figures are also clear - e.g., Fig. 1 effectively summarizes both the acceleration and the comparison of weighting signals.

Critical weaknesses
  • The benefit at the practical operating point appears limited. The gains are largest without CFG, while under a shared guidance scale (w = 1.5) the difference becomes negligible for SiT-L/2 and XL/2. The margin also narrows with longer training (1.65 → 0.31 FID from 400K to 4M), and recall consistently decreases. The authors attribute this to an overlap between dREPA and guidance, but this explanation is not directly tested. If this can be further clarified or analyzed, it would help strengthen the paper.

  • The motivation remains largely empirical. Reweighting the loss by difficulty is a well-established idea (e.g., hard example mining), and the main contribution lies in using the alignment loss as the spatial difficulty signal. However, the correspondence between the alignment loss and denoising difficulty is weak and is shown mainly through binned statistics. A clearer account of why the alignment loss tracks denoising difficulty would be helpful.

Questions for authors
  • Could the authors compare dREPA with simply tuning iREPA’s guidance scale and interval to reach a similar precision–recall operating point?

  • Could the authors show whether the tokens improved by dREPA overlap with those improved by guidance, for example by reporting the error reduction by difficulty bin with and without CFG?

  • Could the authors further characterize which tokens the alignment loss emphasizes and explain why it tracks denoising difficulty?

2 comments

Response to Reviewer 1Nfg

We thank the reviewer for the detailed comments. Changes in the revised paper are marked in blue.

Weakness 2 and Question 3: dREPA is not difficulty reweighting

dREPA weights each token by its cosine similarity to the teacher feature, normalized within the image. Better-aligned tokens receive larger weights. Eq. (5) of the submission stated the cosine distance, but the implementation behind all reported results uses cosine similarity (see the response to Reviewer q2o7). The revision corrects the equation and removes the difficulty-based motivation. No reported number changes.

The weak correspondence between alignment loss and denoising difficulty is real, and the method does not rely on it.

  1. Weighting by difficulty hurts. Up-weighting tokens by their denoising error is worse than uniform weighting in all three seeds at 100K steps (30.96 vs. 28.26, 32.06 vs. 30.05, 30.37 vs. 28.43) and at every strength at 400K steps (best 18.92 vs. 18.43). A moving average of the error is worse in both paired seeds.

  2. The direction of the weights determines the outcome. We set wi=1+β zscore(cos⁡i)w_i=1+\beta\,\mathrm{zscore}(\cos_i) with symmetric clipping and mean-one normalization. Opposite signs of β\beta give identical weight distributions and differ only in which tokens get the larger weights. Three-seed mean FID at 100K steps (SiT-B/2):

β\beta-0.6-0.30 (uniform)+0.3+0.6
FID42.5739.3128.9222.5821.35

The order is the same in every seed. The dREPA rule itself improves every paired seed: 28.26 to 24.78, 30.05 to 26.00, 28.43 to 24.63.

  1. The emphasized tokens are the easier ones. On 2048 images (SiT-B/2, 400K steps), the 20% of tokens with the highest similarity in each image have a denoising error of 0.94 / 0.88 / 0.87 times the image average at t=0.2/0.4/0.6t=0.2/0.4/0.6; the lowest 20% have 1.06 / 1.13 / 1.15. The overlap between the highest-similarity 20% and the highest-error 20% is 0.16 / 0.14 / 0.14 (chance 0.20). High-similarity tokens are also slightly less often edge tokens (22.7% vs. 28.7% for the lowest 20% at t=0.6t=0.6; 25% overall). This is a statistical tendency toward smoother, lower-error regions, not a semantic category.

The contribution is therefore not a new difficulty signal. It is the finding that the denoising loss should be weighted toward well-aligned tokens, using a quantity that REPA already computes at every step.

Weakness 1: operating point and guidance

The submission attributed the shrinking margin to an overlap between dREPA and guidance without testing it. Neither the guidance sweep nor a token-level test (Question 2) supports that account. dREPA minus iREPA on SiT-XL/2 at 400K steps:

wwFIDISPrecisionRecall
1.0-1.65+15.9+0.023-0.026
1.35-0.41+18.9+0.020-0.016
1.5-0.02+16.4+0.020-0.008

The IS and precision gains stay the same size under guidance. Only the FID margin shrinks. L/2 and B/2 behave the same way (IS +14.0 to +16.8 and +11.3 to +15.5). Both raise precision and lower recall, and FID stops improving with further precision near its best ww. The revision reports both tables and drops the overlap explanation.

Question 1: iREPA tuned to a matched precision-recall point

At low guidance the two can be matched. iREPA at w=1.1w=1.1 gives FID / Precision / Recall of 14.98 / 0.654 / 0.634, and dREPA without guidance gives 14.93 / 0.659 / 0.626. This comparison is a tie.

The best operating points are different. With scale and interval tuned independently on the same grid, iREPA reaches 5.71 and dREPA 4.38 on B/2, and 2.21 and 2.04 on XL/2. The tuned dREPA setting on XL/2 guides 60% of the trajectory instead of 80%, which is about 11% fewer network evaluations.

Question 2: do dREPA and guidance improve the same tokens

They do not. We measured the per-token denoising error on 512 images with paired SiT-B/2 checkpoints (uniform and dREPA, same seed, 100K steps, identical noise). Gains are error reductions: uniform to dREPA, and w=1w=1 to w=1.5w=1.5.

ttRank corr., dREPA vs. guidance gainTop-20% overlap (chance 0.20)Rank corr., dREPA gain at w=1w=1 vs. 1.51.5Top-20% kept
0.2-0.1950.1550.9690.878
0.4-0.2170.1680.9480.849
0.6-0.2140.1780.9400.842

The two gains are negatively correlated and overlap below chance. The dREPA gain keeps its spatial pattern when guidance is applied. By difficulty bin, dREPA matches or exceeds iREPA in every bin, including the hardest (appendix table).

Long training and recall

dREPA at 400K steps (6.19) matches iREPA at 1M (6.28), and dREPA at 1M (5.75) matches iREPA at 4M (5.91). The margin without guidance narrows from 1.65 at 400K to 0.31 at 4M, and recall decreases modestly at every scale. Both are stated as limitations.

Thank you for the response. However, since the method turns out not to emphasize difficult tokens, the motivation and analysis are reversed, and the reason for the gain remains unclear. I will keep my score.

Reviewer_q2o7OpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes to use REPA loss to estimate denoising difficulty across tokens, and then uses this signal to reweight the denoising loss during training. The method improves the training efficiency of diffusion transformers with different REPA variants. The experiments show that it works across different model architectures, representation encoders, and both class-conditional and text-to-image generation tasks.

Critical strengths
  1. Table 2 shows that persistence is indeed correlated with FID. The reported results show promising gains across several settings.
  2. Based on the implementation and experiments, the proposed method is simple and effective. I think it offers a useful idea for further work in this area.
Critical weaknesses

I will focus on two weaknesses, with a few examples for each.

  1. Some claims lack evidence or are not supported by the experiments. There are also inconsistencies and errors in the paper. For example:

    (a) On lines 238-241, I do not think experiment (iii) in Table 1 supports the claim about "currency." First, freezing the projector also affects REPA. Second, the alignment signal still comes from the current DiT backbone, so it is not accurate to say that it "no longer tracks the model."

    (b) Fig. 2(c) is confusing to me. Under the stated normalization, the average of the five bin means, weighted by the number of tokens in each bin, should be 1. However, Q1 is at 1 and the other four bin means are above 1, so their weighted average would be greater than 1.

    (c) Some experimental numbers do not match. Fig. 3(a) reports 3.2% and 3.3% for iREPA and dREPA, respectively, but the main text and page 23 report 2.2%.

    (d) The images and text do not match. On pages 29-30, the images do not match their class labels. Lines 1466-1467 also refer to a "coffee mug" when describing the improvement.

    (e) About the code. In the anonymous repository, the weights seem to be computed from −cos⁡i-\cos_i rather than the 1−cos⁡i1 - \cos_i defined in the paper. This directly affects the normalized weights.

Questions for authors
  1. Did the code implementation used to produce the reported results use 1−cos⁡i1 - \cos_i?
  2. The paper describes a linear warm-up for λ\lambda, but "si_loss.py" does not seem to increase it linearly.
  3. Table 1 says it reports the "best strength" for each signal and gives 21.09 for Online MSE. However, Table 2 reports 18.92 at λ=0.1\lambda = 0.1. Why is 21.09 used in Table 1?
  4. Regarding the citation on line 116: DDT does not seem to make a specific improvement to REPA.
2 comments

Response to Reviewer q2o7

We thank the reviewer for checking the code against the paper. We audited the code and all training records. Every reported model was trained with the released code, so no reported number changes. Eq. (5), Algorithm 1 and the warm-up description did not match that code; they are corrected in the revision, and all changes are marked in blue.

(e) and Question 1: implemented signal

The reported results did not use the cosine distance of Eq. (5). The code uses si=−cos⁡(ϕ(hi),zi)s_i=-\cos(\phi(h_i),z_i). With γ=1\gamma=1 and per-sample mean normalization,

−cos⁡iN−1∑j(−cos⁡j)=cos⁡iN−1∑jcos⁡j.\frac{-\cos_i}{N^{-1}\sum_j(-\cos_j)}=\frac{\cos_i}{N^{-1}\sum_j\cos_j}.

In the observed training regime the mean cosine is positive, so the rule gives larger denoising weights to tokens with higher cosine similarity. This is the reverse of what Eq. (5) stated. The revision corrects the equation, the algorithm, the method text and the appendix, drops the "difficulty-aware" name and the hard-token interpretation, and removes the γ=0.5\gamma=0.5 result, which is undefined for a signed signal.

To test this direction we use wi=1+β zscore(cos⁡i)w_i=1+\beta\,\mathrm{zscore}(\cos_i) with symmetric clipping and exact mean-one normalization. Opposite signs of β\beta give the same weight mean, standard deviation, quantiles and clipping rate; only the token ordering differs. FID at 100K steps (SiT-B/2, 50K samples):

β\betaSeed 0Seed 1Seed 2Mean ± std.
-0.642.3442.9842.3842.57 ± 0.29
-0.338.9540.4538.5439.31 ± 0.82
028.2630.0528.4328.92 ± 0.81
+0.322.5423.4221.7722.58 ± 0.67
+0.621.6721.7120.6821.35 ± 0.48

The order +0.6<+0.3<0<−0.3<−0.6+0.6<+0.3<0<-0.3<-0.6 holds in every seed. The released dREPA rule improves all three paired seeds: 28.26 to 24.78, 30.05 to 26.00, 28.43 to 24.63.

Question 2: warm-up

The code uses a threshold, not a linear ramp. Weighting is off before 10K steps and on at full strength afterwards. The method, algorithm and appendix now describe this. Moving the threshold to 0, 10K, 20K or 50K gives FID 15.20, 14.93, 14.94 and 15.09, so the result is not sensitive to it.

(a): the "currency" claim

Freezing the projector changes the alignment objective as well as the weighting signal, and the current backbone still enters the signal. The ablation therefore does not isolate currency. The claim is removed, and the row is kept only as an ablation that changes the training objective.

The same audit affects the persistence argument. Cosine similarity ranks tokens more consistently across noise draws than online MSE (rank correlation 0.853 vs. 0.569), but the reversed ordering has the same consistency and is far worse. Consistency is now reported as a descriptive statistic and not as the cause of the gain.

(b): Figure 2(c)

The panel plots the mean alignment distance in each denoising-error quintile divided by the Q1Q_1 value at the same timestep. It is not normalized by the per-image mean, so the five values do not have to average to one. The caption described the wrong normalization and is corrected. The panel is used only as a descriptive association.

(c): 2.2% vs. 3.2% / 3.3%

Figure 3(a) has the right values: 3.2% for iREPA and 3.3% for dREPA on the hardest bin at t=0.4t=0.4. The two occurrences of 2.2% were transcription errors and are fixed.

(d): images and class labels

The images are correct and the class names were wrong. The ImageNet captions are corrected and the "coffee mug" sentence is removed.

Question 3: 21.09 vs. 18.92

Table 1 compares all signals at the same strength, λ=0.5\lambda=0.5, where Online-MSE gives 21.09. Its best value over the sweep is 18.92 at λ=0.1\lambda=0.1. Both are worse than uniform weighting (18.43). The caption said "best strength" and now says "matched strength". In the new controlled comparison, Online-MSE weighting is also worse than uniform in all three seeds: 30.96 vs. 28.26, 32.06 vs. 30.05, 30.37 vs. 28.43.

Question 4: DDT

DDT uses representation alignment inside a new architecture and does not propose an improvement to REPA. The sentence is corrected.

Thank you for the rebuttal and revision.

First, I apologize for leaving out the second point under Weaknesses, which was mainly about presentation. I will not revisit those comments here since the paper has now been revised.

Most of my original concerns have now been addressed. However, I still have a few questions:

  1. In the released code, setting γ=0.5\gamma=0.5 takes the square root of negative alignment values. I'm confused about how you obtain FID 14.95 reported in the original submission.

  2. Table 17 now reports 3.2%/3.3%, but Fig. 5 still appears to show approximately 2.2%.

The revision reverses the claimed weighting direction compared with the original submission, which is the central design of this work. It also changes important claims about difficulty-aware weighting, consistency, and currency. Together, these issues leave me concerned about the reliability of this work. I think using REPA signal to weight the denoising loss is a promising direction. However, I do not think this work is sufficiently prepared for ICLR.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.