- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces an alternative to REPA-style representation alignment for accelerating diffusion transformer training. Instead of aligning intermediate diffusion features to fixed DINOv2 patch features, a set of learnable tokens extracts information from the external representation via cross-attention. The resulting tokens are injected into the backbone via a second cross-attention, so that the denoising loss determines which information is extracted. A third cross-attention learns to predict these tokens from the backbone’s own intermediate features. At inference, this makes the model can denoise by its won without any prior information. Experiments on ImageNet 256×256 and 512×512 with SiT-B/L/XL and JiT-B/L, all at 400K iterations without CFG, report lower FID than REPA and iREPA.
Critical strengths
-
The paper is clearly written and easy to follow. The method offers a simple way to selectively align representations, requiring only a few lightweight cross-attention modules on top of REPA-style training.
-
The method improves over REPA across various setups, including three SiT scales, two resolutions, a pixel-space backbone (JiT), and in combination with REG. These results are complemented by fairly rich ablations (e.g., module design) and analyses (linear probing, LDS), which offer meaningful insights into the method’s behavior.
Critical weaknesses
-
The method appears to be a variation of existing alignment methods, combining several familiar ingredients. While the idea of letting the denoising objective decide what to extract is interesting, the selectivity claim could be supported more directly. Further analysis of what the selected tokens capture, and how this differs from the original encoder features, would enrich the paper. In addition, the alignment and injection block does not seem to be reported or ablated, although depth is known to affect REPA-style methods. If the selection is adaptive, one might expect lower sensitivity to depth than REPA, which would be a direct test of the claim.
-
Main comparisons are conducted at 400K iterations without CFG, which makes it difficult to tell whether the method mainly accelerates convergence or also improves final quality. This seems particularly relevant here, since the paper’s motivation builds on the observation that rigid alignment becomes restrictive later in training. This would suggest that the advantage persists or even grows with longer training, which would be valuable to verify.
Questions for authors
-
At which block are f_φ and h_ω applied? A depth sweep compared with REPA would help verify whether the selection (and the information through selection) is adaptive.
-
Could the authors provide an analysis of what the selected tokens capture and how this differs from the original encoder features?
-
Does the advantage over REPA persist, grow, or diminish with longer training (e.g., 1–2M steps on B/2 or L/2)?
Thank you for the response. Still, my main concerns remain - no direct evidence for the selectivity claim and no results with longer training or CFG. Hence, I am keeping my original score.