acceptodds
Under review as a conference paper at ICLR 2027

Learning What to Align for Faster Diffusion Transformer Training

Abstract

Semantic emergence in Diffusion Transformers motivates alignment with external semantic representations to accelerate training convergence. However, observations suggest that such rigid alignment is often suboptimal, creating an inherent paradox: higher semantic accuracy does not necessarily lead to better generation fidelity. In this paper, we propose allowing the denoising objective itself to ascertain the specific facets of semantic information most beneficial to the generative process. Furthermore, we introduce a semantic internalization mechanism coupled with stage-wise training to estimate and reintegrate those task-relevant signals into the model’s intrinsic representation. Our acceleration regime prioritizes generation-critical features over rigid alignment with fixed-target semantic representations. Extensive experiments demonstrate that our approach consistently further accelerates training and improves generation quality across various model scales and resolutions, outperforming existing alignment-based baselines.

Primary area:
generative models
Keywords:
Diffusion transformer, Representation alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bnH4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces an alternative to REPA-style representation alignment for accelerating diffusion transformer training. Instead of aligning intermediate diffusion features to fixed DINOv2 patch features, a set of learnable tokens extracts information from the external representation via cross-attention. The resulting tokens are injected into the backbone via a second cross-attention, so that the denoising loss determines which information is extracted. A third cross-attention learns to predict these tokens from the backbone’s own intermediate features. At inference, this makes the model can denoise by its won without any prior information. Experiments on ImageNet 256×256 and 512×512 with SiT-B/L/XL and JiT-B/L, all at 400K iterations without CFG, report lower FID than REPA and iREPA.

Critical strengths
  • The paper is clearly written and easy to follow. The method offers a simple way to selectively align representations, requiring only a few lightweight cross-attention modules on top of REPA-style training.

  • The method improves over REPA across various setups, including three SiT scales, two resolutions, a pixel-space backbone (JiT), and in combination with REG. These results are complemented by fairly rich ablations (e.g., module design) and analyses (linear probing, LDS), which offer meaningful insights into the method’s behavior.

Critical weaknesses
  • The method appears to be a variation of existing alignment methods, combining several familiar ingredients. While the idea of letting the denoising objective decide what to extract is interesting, the selectivity claim could be supported more directly. Further analysis of what the selected tokens capture, and how this differs from the original encoder features, would enrich the paper. In addition, the alignment and injection block does not seem to be reported or ablated, although depth is known to affect REPA-style methods. If the selection is adaptive, one might expect lower sensitivity to depth than REPA, which would be a direct test of the claim.

  • Main comparisons are conducted at 400K iterations without CFG, which makes it difficult to tell whether the method mainly accelerates convergence or also improves final quality. This seems particularly relevant here, since the paper’s motivation builds on the observation that rigid alignment becomes restrictive later in training. This would suggest that the advantage persists or even grows with longer training, which would be valuable to verify.

Questions for authors
  • At which block are f_φ and h_ω applied? A depth sweep compared with REPA would help verify whether the selection (and the information through selection) is adaptive.

  • Could the authors provide an analysis of what the selected tokens capture and how this differs from the original encoder features?

  • Does the advantage over REPA persist, grow, or diminish with longer training (e.g., 1–2M steps on B/2 or L/2)?

Reviewer_KS5NOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies representation alignment for accelerating diffusion transformer (SiT/JiT) training. Starting from the observation that rigid alignment of intermediate diffusion features to pretrained semantic encoders (REPA) is suboptimal -- higher linear-probing accuracy does not reliably improve generation fidelity -- the authors let the denoising objective decide which semantic information to use: M=32 learnable tokens attend to the external encoder output y through a cross-attention module g, and the infused tokens are injected into the backbone through a second cross-attention module h (Stage 1, steps s < r). A stage-wise schedule then trains a predictor f that estimates the infused tokens from internal diffusion features (aligned to stop-gradiented infused tokens via a cosine similarity loss Lsim); in Stage 2 (s >= r) the backbone is driven only by self-predicted tokens with g and the learnable tokens frozen, and at inference g and the external encoder are removed entirely while f and h remain. Experiments on ImageNet at 400K iterations without classifier-free guidance (NFE=250, SDE Euler-Maruyama, FID-50K) report FID 17.7 vs REPA 24.4 and iREPA 21.4 on SiT-B/2, 8.2 vs 9.7/9.4 on SiT-L/2, and 7.1 vs 7.9/7.5 on SiT-XL/2 (Table 1); 19.3/8.2/7.0 vs REPA 25.2/9.6/7.6 at 512x512 (Table 2); 33.4/16.1 vs REPA 37.2/16.8 on JiT-B/16 and JiT-L/16 (Table 3); robustness across eight pretrained encoders (Fig. 3); ablations over fusion-module design and stage-wise training (Table 4) and over r, alpha, M (Table 5); linear probing showing the method can beat REPA's FID with lower probing accuracy (59.83% vs 61.86% on B/2, Table 6); and an LDS-based analysis of training dynamics (Fig. 4). The paper concludes that denoising-guided selection plus internalization consistently accelerates training and improves generation quality relative to alignment-based baselines.

Critical strengths

S1. Broad, directionally consistent gains over REPA/iREPA at a fixed 400K-iteration budget with only modest inference-time parameter overhead: Tables 1-3 (SiT-B/2 24.4 -> 17.7, SiT-L/2 9.7 -> 8.2, SiT-XL/2 7.9 -> 7.1; 512x512 SiT-B/2 25.2 -> 19.3; JiT-B/16 37.2 -> 33.4), and the advantage holds across eight different teacher encoders (Fig. 3). This matters for the acceptance decision because the breadth of settings makes selective reporting unlikely and, if the numbers stand, establishes selection-plus-internalization as a drop-in, encoder-agnostic replacement for rigid alignment rather than a single-config trick. S2. The ablation design in Table 4 directly attacks the obvious "the gain is just extra parameters / a stronger fusion module" confound by including REPA+linear (20.8), REPA+MLP (18.7) and cross-attention variants (19.8/19.9) before the full method (17.7 at +5M inference params), and Table 6 supplies the paper's own evidence for its motivating hypothesis (better FID with lower linear-probing accuracy than REPA on SiT-B/2, 59.83% vs 61.86%). This matters because these are exactly the controls that separate a genuine mechanism contribution from an architecture-capacity artifact, and they raise the potential ceiling of the paper above "yet another REPA variant".

Critical weaknesses

W1. Pervasive numerical inconsistencies between the text/figures and the paper's own tables (Sec. 4.2, Tables 1-4, Figs. 1, 3). The 512x512 paragraph reports REPA->Ours as 31.7->26.1 (B/2), 14.1->12.8 (L/2), 11.6->10.5 (XL/2), none of which appear anywhere in Table 2 (which gives 25.2->19.3, 9.6->8.2, 7.6->7.0); the JiT paragraph reports 15.7 for JiT-L/16 while Table 3 reports 16.1; Table 4's first row is described as "identical to REPA" at FID 23.0 while Table 1 reports REPA SiT-B/2 = 24.4 under the same stated protocol; Fig. 1's endpoint labels (B/2+REPA approx. 18.7 at 400K, "SiT-XL/2 (Baseline)" approx. 24.4) and Fig. 3's value ranges (20-60 FID) are incompatible with Table 1's 400K values. Fixable (correct text/figures and state protocols), but as submitted it undermines trust in every reported number, which is decisive for a purely empirical paper and caps the score. W2. Claims outrun the evidence on two fronts (Abstract, Sec. 4.2, Table 1). (i) REG -- an alignment-based baseline included in the authors' own Table 1 -- beats the proposed method alone at every scale (B/2 15.2 vs 17.7; L/2 4.6 vs 8.2; XL/2 3.4 vs 7.1 at nearly identical parameter counts, 677M vs 682M), and adding the method on top of REG buys only 0.4/0.2 FID at L/XL (4.6->4.2, 3.4->3.2), yet the abstract claims "outperforming existing alignment-based baselines" and the conclusion claims superiority over "state-of-the-art alignment methods" without a single sentence discussing REG (or the RAE row). (ii) Every result is a single run: a full-text search finds no occurrence of seeds, standard deviations, or confidence intervals, so the headline XL-scale margins (Ours 7.1 vs iREPA 7.5; REG+Ours 3.2 vs REG 3.4) sit inside typical run-to-run FID-50K noise. Fixable (calibrate claims, discuss REG, report multi-seed statistics), but as written the central superiority claim is not established, which drives the weak-reject rating.

Questions for authors
  1. (material) The 512x512 paragraph of Sec. 4.2 quotes 31.7->26.1 (B/2), 14.1->12.8 (L/2), 11.6->10.5 (XL/2), and the JiT paragraph quotes 15.7 for JiT-L/16, while Tables 2-3 report 25.2->19.3, 9.6->8.2, 7.6->7.0 and 16.1. Which values are correct, and from which runs/configurations do the paragraph numbers come?
  2. (clarification) REPA SiT-B/2 at 400K without CFG appears as 24.4 in Table 1, 23.0 in Table 4 ("identical to REPA"), and approximately 18.7 as the endpoint of the B/2+REPA curve in Fig. 1. Please reconcile these three values (alignment depth? seed? iteration?), and define exactly how the "2.0x faster" and "1.9x faster" annotations in Fig. 1 were computed (which FID target, which pair of curves).
  3. (material) REG alone outperforms your method alone at all three scales in Table 1, and REG+Ours improves REG by only 0.4 (L/2) and 0.2 (XL/2) FID. How do you reconcile this with the abstract's "outperforming existing alignment-based baselines", and are these deltas above run-to-run variance -- i.e., how many seeds were run per configuration and can you report std for the Table 1 headline rows?
  4. (clarification) Please give a formal definition of the LDS score (equation, which features/tokens are compared, neighborhood sizes, citation), and the evidence or reference supporting "higher LDS generally correlates with better structural coherence".
  5. (clarification) Fig. 3: at how many training iterations were these runs evaluated (the 20-60 FID ranges are incompatible with the 400K values in Table 1), and in the SiT-B/2 panel the Ours markers for mocov3-vit-l and mocov3-vit-b appear to the right of (worse than) the REPA markers -- please confirm and comment on the caption's "consistently achieves lower FID".
  6. (material) For reproduction, please specify: which diffusion block provides h_t and at which block g/f/h are attached; cross-attention head counts and dimensions; the default r, alpha, M used for Tables 1-3 and for the JiT and 512x512 runs (Table 5's optima r=75K and alpha=1.75 lie at the edge of the tested grid -- were larger values tried?); and the linear-probing protocol of Table 6 (which features -- h_t, infused, or predicted tokens -- which layer, which checkpoint).
  7. (clarification) In Table 4, adding h(.) plus stage-wise training to the MLP configuration degrades FID from 18.7 to 20.0, while the same addition improves the CA configuration from 19.9 to 17.7. Why does internalization help only in the CA configuration, and in rows 4-5 (CA as f without h) what gradient trains g, given the stop-gradient on the infused tokens in Eq. 10?
  8. (clarification) Since f and h remain in the inference graph at every sampling step (NFE=250), unlike REPA which has zero inference-time cost, please quantify the added sampling latency of your method vs vanilla SiT and REPA at B/2 and XL/2, and the added training cost (e.g., GPU-hours per 400K run).
Reviewer_fkGYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper aims to improve the generation performance of diffusion transformers by improving the representation alignment framework. Specifically, learnable queries extract tokens from pretrained visual encoders, which are injected into the DiT through cross-attention. A stage-wise strategy then replaces these tokens with predictions from internal DiT features at inference. Experiments on ImageNet show improved unguided FID over REPA across model scales, resolutions, and visual encoders, with additional results on pixel-space JiT.

Critical strengths

The unguided FID improvements are consistent, spanning latent-space DiT-B/L/XL, two image resolutions, and pixel-space JiT.

Critical weaknesses
  1. The evaluation is insufficient to establish the method's performance relative to related work. The evaluation stops at 400K steps (80 epochs) and provides no quantitative results with CFG. Contemporary comparisons commonly include unguided FID at 400K/1M steps (80/200 epochs), alongside best guided performance after longer training (e.g., after 200 epochs). It remains unclear how the method compares with modern approaches after full convergence.
  2. Section 4.3 does not sufficiently explain what information the method retains, and the interpretation of LDS is somewhat problematic.
  • The paper claims that the method retains "only the information that is most beneficial for the denoising process", but does not examine what information is retained or discarded. Achieving a lower linear probing accuracy does not mean the inessential information is discarded.
  • Figure 4 may conflate spatial structure with alignment to pretrained semantics: "While REPA shows increasing LDS (closer alignment to DINO), our method exhibits a decreasing trend, suggesting selective deviation from pretrained features" and "REPA which continues to increase ... excessive alignment to pretrained semantics". However, according to iREPA, LDS measures spatial self-similarity within a representation, rather than its alignment with DINO or any semantic information.
  • iREPA identifies spatial structure (LDS) as a stronger indicator of generation performance. The paper invokes this insight to explain the advantage of its higher LDS, but then associates REPA's increasing LDS with "excessive alignment" and its own decreasing LDS with "recovering fine-grained details". Aren't these interpretations somewhat contradictory?
Questions for authors

Please refer to the weaknesses.

Comments on the paper

1 comment

Thank you for the response. Still, my main concerns remain - no direct evidence for the selectivity claim and no results with longer training or CFG. Hence, I am keeping my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.