- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes REPI, a training strategy that temporarily injects pretrained visual encoder representations into a diffusion transformer and subsequently internalizes them. During an initial scaffold phase, projected keys and values from the encoder's final attention layer replace the denoiser's native K/V at one layer, while native queries and residual connections are preserved. The scaffold is then removed, and an internalization loss trains the native K/V to match the projected targets. Inference uses the original diffusion architecture without the encoder or projection heads. Experiments demonstrate faster convergence across model scales, backbones, resolutions, and visual encoders, with further improvements when REPI is combined with REPA.
Critical strengths
-
The approach is simple and produces substantial, broadly supported improvements. On SiT-XL at 100K steps, FID improves from REPA's 19.40 to 14.51 with REPI and to 11.78 with their combination. The evaluation across ten encoders, multiple diffusion backbones, 512-resolution generation, and text-to-image generation provides useful evidence that the effect is not confined to one encoder/backbone pairing. Discarding the scaffold at inference also preserves the original model architecture.
-
The component studies are informative. Figure 6 shows a lasting benefit from temporary injection even without the internalization loss, and further improvement when that loss is added. The injection-target, duration, loss-weight, and layer-order ablations clarify practical design choices. The paper correctly distinguishes the oracle experiment from standard generation and reports training time in addition to optimizer-step comparisons.
Critical weaknesses
-
The evidence for the specific scaffold-to-internalization mechanism remains incomplete. Figure 6 establishes that scaffold-only training helps and that internalization improves it, but does not include a no-injection K/V-alignment control. Thus, it remains unclear how much of the full method's benefit specifically requires temporary representation injection, beyond changing the alignment direction and target. Such a control should account for how the target projections are learned and frozen, rather than use arbitrary untrained targets. Similarly, K/V is the best empirical choice in Table 5, but the explanation that it is more suitable for internalization is still a hypothesis: V-only is better in the oracle study, whereas K/V is better after scaffold removal. The results establish effectiveness more clearly than they establish the proposed mechanism.
-
The positioning relative to the closest representation-based methods needs greater precision and comparative evidence. Section 6 cites REG[3] and joint image-feature synthesis[4], but its subsequent characterization of prior methods as treating external representations only as alignment targets is too broad: these works already place representations in the generative computation through joint denoising. REPI's temporary clean K/V injection and unchanged inference architecture are meaningful differences, and I do not view the algorithms as identical. However, these differences should be articulated more precisely and supported by a relevant comparison. The evaluation also concentrates on external-encoder alignment methods, leaving its relation to self-supervised representation guidance underexplored.
Notably, I would consider raising my score if the rebuttal provides convincing targeted controls and clarifies the method's advantage and distinction relative to the closest alternatives.
Questions for authors
-
How does REPI compare with a no-scaffold baseline using the same K/V internalization targets? Please specify a suitable control for learning/freezing the target projections, and clarify when the internalization loss becomes active. This would help isolate the benefit of actually using the scaffold during denoising.
-
Why does jointly injecting K/V favor internalization more than V-only, despite the opposite ranking in the oracle experiment? Can you provide direct evidence, such as the generation-quality change at removal, recovery during subsequent training, or matching errors across noise levels? The current attention-routing explanation is plausible, but the target ablation alone does not identify this mechanism.
-
How does REPI compare with or complement SRA[1] and Self-Flow[2] under comparable settings? More broadly, does the scaffold depend on a fixed pretrained encoder, or could lower-noise EMA-teacher representations support injection/internalization? A comparison would inform the current contribution; the latter is a scope/mechanism question, not a requirement to develop a complete new variant during rebuttal.
-
Can you clarify the distinction from REG[3] and joint image-feature synthesis[4], qualify the broad alignment-target description in Section 6, and provide a matched comparison or explain the relevant protocol differences? Answers to these questions could materially change my assessment.
References
[1] Jiang D, Wang M, Li L, et al. No other representation component is needed: Diffusion transformers can provide representation guidance by themselves[J]. arXiv preprint arXiv:2505.02831, 2025.
[2] Chefer H, Esser P, Lorenz D, et al. Self-supervised flow matching for scalable multi-modal synthesis[J]. arXiv preprint arXiv:2603.06507, 2026.
[3] Wu G, Zhang S, Shi R, et al. Representation entanglement for generation: Training diffusion transformers is much easier than you think[J]. arXiv preprint arXiv:2507.01467, 2025.
[4] Kouzelis T, Karypidis E, Kakogeorgiou I, et al. Boosting generative image modeling via joint image-feature synthesis[J]. arXiv preprint arXiv:2504.16064, 2025.
4 comments
Response to Reviewer ANN8 — Part 2 of 2
Weakness 2(a) & Question 4: [...] REG[3] and joint image-feature synthesis[4] already place representations in the generative computation through joint denoising. REPI's temporary clean K/V injection and unchanged inference architecture are meaningful differences [...] Can you clarify the distinction from REG[3] and joint image-feature synthesis[4], qualify the broad alignment-target description in Section 6, and provide a matched comparison or explain the relevant protocol differences?
We thank the reviewer for pointing this out. We agree that our original wording in Section 6 was too broad: REG [3] and joint image-feature synthesis (ReDi) [4] already incorporate semantic representations into the generative computation through joint denoising of image latents and semantic features.
Distinction from REG and ReDi. REG and ReDi jointly denoise image latents and semantic representations, which requires extending the transformer’s input embeddings and prediction heads. Both therefore carry additional semantic denoising states and modality-specific input/output mappings into inference, which adds some inference overhead. In contrast, our approach injects projected clean encoder representations only during training, as a temporary scaffold, and internalizes this guidance into the model’s native representations. The inference architecture and sampling state thus remain identical to those of the original diffusion transformer.
We have updated Section 6 (L475-480) accordingly:
[...] Most closely related, REG (Wu et al., 2025a) and ReDi (Kouzelis et al., 2025) also incorporate external representations by jointly denoising image latents and semantic representations. However, both methods extend the transformer's input embeddings and prediction heads to generate semantic representations, and keep these modifications at inference. In contrast, we inject projected clean encoder representations only during training, as a temporary scaffold, and internalize this guidance into the model's native representations, leaving the inference architecture and sampling state unchanged.
Weakness 2(b) & Question 3: The evaluation also concentrates on external-encoder alignment methods, leaving its relation to self-supervised representation guidance underexplored. How does REPI compare with or complement SRA[1] and Self-Flow[2] under comparable settings? Does the scaffold depend on a fixed pretrained encoder, or could lower-noise EMA-teacher representations support injection/internalization? [...] the latter is a scope/mechanism question, not a requirement to develop a complete new variant during rebuttal.
We thank the reviewer for suggesting methods based on self-supervised representations.
Comparison with SRA and Self-Flow. The table below compares reported guided-generation results on ImageNet 256×256 with SiT-XL/2. Detailed evaluation settings and results are provided in Appendix E of the revised paper. REPA + REPI reaches an FID of 1.47 at 1M steps, matching Self-Flow's reported 1.47 at 4M steps and outperforming SRA's 1.58 at 4M steps. We note, however, that SRA and Self-Flow do not rely on external encoders, which is a clear advantage when strong pretrained encoders are unavailable. We have added this comparison in Appendix E (L945-951) of the revised paper.
| Method | Training steps | FID ↓ |
|---|---|---|
| SRA | 4M | 1.58 |
| Self-Flow | 4M | 1.47 |
| REPA + REPI (ours) | 1M | 1.47 |
| REPA + REPI (ours) | 2M | 1.42 |
Lower-noise EMA-teacher representations for injection and internalization. Technically, EMA-teacher features can also be projected into the student's K/V spaces. We believe this is a promising direction for removing the dependence on external encoders. One possible challenge is that the teacher's features may not be sufficiently informative early in training. A natural remedy is to first train the student and update its EMA teacher without injection, then introduce the scaffold after a warm-up period while the EMA updates continue. Due to the limited rebuttal time, we leave this extension for future work and will discuss it in the revised paper.
References
[1] Jiang D, Wang M, Li L, et al. No other representation component is needed: Diffusion transformers can provide representation guidance by themselves[J]. arXiv preprint arXiv:2505.02831, 2025.
[2] Chefer H, Esser P, Lorenz D, et al. Self-supervised flow matching for scalable multi-modal synthesis[J]. arXiv preprint arXiv:2603.06507, 2026.
[3] Wu G, Zhang S, Shi R, et al. Representation entanglement for generation: Training diffusion transformers is much easier than you think[J]. arXiv preprint arXiv:2507.01467, 2025.
[4] Kouzelis T, Karypidis E, Kakogeorgiou I, et al. Boosting generative image modeling via joint image-feature synthesis[J]. arXiv preprint arXiv:2504.16064, 2025.
I thank the authors for their detailed response. Most of my concerns have been addressed in the rebuttal, and I will accordingly raise my score to 3.
I would like to offer one additional suggestion. I understand that the rebuttal period is limited and that new experiments may not be feasible within this timeframe. Nevertheless, I would encourage the authors to consider a small-scale text-to-image experiment to further demonstrate the generality of the proposed method. Including such results in the future version would strengthen the paper.
Dear Reviewer ANN8,
Thank you for confirming that our rebuttal addressed most of your concerns. We sincerely appreciate your constructive feedback, which has helped us strengthen the manuscript. We will incorporate the additional results and discussions above into the revised version.
We agree that text-to-image experiments are valuable for demonstrating the generality of REPI. We would like to point out that our current manuscript already includes a small-scale text-to-image experiment, using MM-DiT on MS-COCO trained for 150K steps (Section 5.2, Table 4). Without classifier-free guidance, REPI improves FID over REPA from 10.40 to 9.47, and combining REPA and REPI further reduces it to 9.08. We will make these results more prominent in the revised manuscript.
Best regards,
The Authors
Response to Reviewer ANN8 — Part 1 of 2
We thank Reviewer ANN8 for the careful reading of our paper and the constructive comments.
Below, we address the reviewer's concerns point by point. Due to the comment length limit, our response is split into two parts.
We thank the reviewer for this suggestion and agree that our original ablation in Figure 6 does not include a no-injection K/V-alignment control. We have run the requested control experiment and report the results below.
No-injection K/V-alignment control. As in REPI, we obtain the encoder-to-diffusion projectors from a 20K-step scaffold phase on a separate SiT-XL/2, which is then discarded. We freeze these projectors and train a freshly initialized SiT-XL/2 with both the denoising and internalization objectives (same layer and λ as REPI). The student is never trained with injection, and its targets are identical to those used by REPI. The updated ablation on SiT-XL/2 (FID, w/o CFG) is:
How does REPI compare with this no-scaffold baseline? As shown in the table above, the control improves substantially over vanilla SiT-XL/2, from 39.40 to 18.09 FID at 100K steps and from 17.20 to 7.71 at 400K steps. However, a gap to full REPI remains at every checkpoint: 3.58 at 100K (18.09 vs. 14.51), 1.15 at 200K (10.08 vs. 8.93), and 0.50 at 400K (7.71 vs. 7.21). Notably, the control performs at roughly the level of REPA (7.71 vs. 7.90 at 400K), so REPI's advantage over REPA comes mainly from the scaffold phase.
We summarize three observations showing that the scaffold phase makes a meaningful contribution:
In the revised paper, we have added this new control to Figure 6 and updated Section 5.4 (L407-413) to discuss the result.
We thank Reviewer ANN8 for raising this issue. In the revised paper, we have added Figure 10 (L1009-1025), which shows the denoising loss curves of the K/V and V-only variants before and after scaffold removal. During the scaffold phase (0–20K steps), V-only attains a lower loss than K/V throughout, consistent with the oracle results in Table 1. After scaffold removal, the ranking reverses, and K/V has a lower loss than V-only for the rest of training. This supports our hypothesis that jointly injecting K/V is better suited to scaffold removal and internalization. The oracle setting, by contrast, only measures how effectively an external representation can be consumed, which is why V-only performs best there. We have added this observation to Section 5.4 (L418-423) of the revised paper.