- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes dREPA, which further improves REPA based on iREPA. The authors argue that the REPA alignment loss in diffusion models is correlated with denoising difficulty. Based on this observation, the authors use the REPA alignment loss to reweight tokens during training, placing more emphasis on difficult-to-learn regions. Experiments across various REPA teachers and diffusion models of different sizes demonstrate that dREPA can accelerate model convergence and achieve better performance at the same number of training steps.
Critical strengths
- The authors observe a correlation between the REPA alignment loss and denoising difficulty, which is an insightful finding.
- The proposed method is simple. It directly redistributes token weights during training based on the alignment loss, requires no additional architecture, and can be easily integrated into existing methods.
- The experiments are comprehensive. The authors conduct extensive experiments using SiT, JiT, and MMDiT across different model sizes and visual encoders, consistently achieving performance improvements.
Critical weaknesses
- The paper states that the REPA alignment loss tracks denoising difficulty, but also emphasizes that the method is not hard-token mining. This description is somewhat confusing. Moreover, if the improvement is not due to denoising difficulty (Figure 2 also shows that the correlation between the two is not very strong), the explanation for the effectiveness of dREPA becomes less convincing.The paper only demonstrates some correlations without providing causal evidence or experimental validation.
- The effectiveness of dREPA diminishes noticeably under stronger CFG. However, CFG is commonly used in practical applications, which limits the applicability of the proposed method to some extent.
Questions for authors
Clarification is needed. Please refer to Weakness 1.
Response to Reviewer Px7w
We thank the reviewer for the comments. Changes in the revised paper are marked in blue.
Weakness 1 and Question 1: difficulty, hard-token mining, and causal evidence
dREPA does not emphasize difficult tokens. It weights each token by its cosine similarity to the teacher feature, normalized within the image, so tokens whose features are already close to the teacher receive larger weights. Eq. (5) of the submission stated the cosine distance, but the implementation behind all reported results uses cosine similarity (see the response to Reviewer q2o7). The revision corrects the equation and removes the difficulty-based explanation. No reported number changes.
This removes the contradiction between "tracks difficulty" and "not hard-token mining". The second statement is the accurate one. Figure 2 shows only a weak association between alignment distance and denoising error; it is kept as a descriptive statistic and is no longer used to explain the gain.
The causal evidence is an intervention on the ordering of the weights. For each sample,
ui=zscore(cosi),wi=1+βui,
followed by symmetric clipping and exact mean-one normalization. Positive and negative β give the same weight mean, standard deviation, quantiles and clipping rate. They differ only in which tokens receive the larger weights. FID at 100K steps (SiT-B/2, 50K samples):
The order +0.6<+0.3<0<−0.3<−0.6 holds in every seed, and the differences between settings are much larger than the variation across seeds. Up-weighting well-aligned tokens improves FID. Up-weighting poorly aligned tokens degrades it. The dREPA rule itself improves all three paired seeds: 28.26 to 24.78, 30.05 to 26.00, 28.43 to 24.63.
Weighting by difficulty does not work. Using the current denoising error as the weight is worse than uniform weighting in all three seeds (30.96 vs. 28.26, 32.06 vs. 30.05, 30.37 vs. 28.43). A moving average of that error is worse in both paired seeds (31.78 vs. 30.05, 30.17 vs. 28.43). Random weights have no effect (18.86 vs. 18.43). The tokens that dREPA emphasizes are on average the easier ones: mean cosine falls from 0.631 in the easiest denoising-error quintile to 0.582 in the hardest at t=0.4.
The explanation of effectiveness in the revision follows from these results. The denoising loss improves when more of its weight is placed on tokens whose representation is already aligned with the teacher, and it degrades when the weight is placed on high-error tokens. The alignment signal costs nothing, because REPA computes it at every step, and it ranks tokens more consistently across noise draws than the residual (rank correlation 0.853 vs. 0.569).
Weakness 2: the effect under strong guidance
The FID margin narrows under a shared guidance scale, and the paper states this as a limitation. The other metrics do not narrow. dREPA minus iREPA on SiT-XL/2 at 400K steps:
The IS and precision gains are the same size with and without guidance. FID stops improving with further precision near its best w, which is why the FID values converge. The two methods also act on different tokens. On paired SiT-B/2 checkpoints at 100K steps (512 images), the per-token error reductions from dREPA and from guidance at w=1.5 are negatively correlated (rank correlation -0.20 to -0.22), and the dREPA reduction keeps its spatial pattern under guidance (rank correlation 0.94 to 0.97).
Three results concern practical use.