acceptodds
Under review as a conference paper at ICLR 2027

Scaffold Then Internalize: Representation Injection for Diffusion Transformers

Abstract

Recent representation alignment (REPA) methods accelerate diffusion transformer training by aligning projections of the transformer's hidden states with representations from pretrained visual encoders. In this work, we explore a reverse and complementary direction to REPA: rather than projecting diffusion representations into the encoder's space, we inject encoder representations into the diffusion transformer, allowing them to actively participate in the denoising process. To this end, we introduce REPresentation Injection (REPI), a training framework based on a scaffold-to-internalization strategy, in which projected encoder representations initially serve as a temporary scaffold and are then progressively internalized by the diffusion transformer. REPI not only outperforms REPA across a wide range of backbones but is also highly complementary to it, and combining the two yields substantial gains over either alone. Notably, with only 160K training steps, REPI + REPA surpasses vanilla SiT trained for 7M steps, a speedup of over . Code is provided in the supplementary material and will be made publicly available.

Primary area:
generative models
Keywords:
Diffusion Transformer, Representation Alignment, REPA

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ANN8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes REPI, a training strategy that temporarily injects pretrained visual encoder representations into a diffusion transformer and subsequently internalizes them. During an initial scaffold phase, projected keys and values from the encoder's final attention layer replace the denoiser's native K/V at one layer, while native queries and residual connections are preserved. The scaffold is then removed, and an internalization loss trains the native K/V to match the projected targets. Inference uses the original diffusion architecture without the encoder or projection heads. Experiments demonstrate faster convergence across model scales, backbones, resolutions, and visual encoders, with further improvements when REPI is combined with REPA.

Critical strengths
  1. The approach is simple and produces substantial, broadly supported improvements. On SiT-XL at 100K steps, FID improves from REPA's 19.40 to 14.51 with REPI and to 11.78 with their combination. The evaluation across ten encoders, multiple diffusion backbones, 512-resolution generation, and text-to-image generation provides useful evidence that the effect is not confined to one encoder/backbone pairing. Discarding the scaffold at inference also preserves the original model architecture.

  2. The component studies are informative. Figure 6 shows a lasting benefit from temporary injection even without the internalization loss, and further improvement when that loss is added. The injection-target, duration, loss-weight, and layer-order ablations clarify practical design choices. The paper correctly distinguishes the oracle experiment from standard generation and reports training time in addition to optimizer-step comparisons.

Critical weaknesses
  1. The evidence for the specific scaffold-to-internalization mechanism remains incomplete. Figure 6 establishes that scaffold-only training helps and that internalization improves it, but does not include a no-injection K/V-alignment control. Thus, it remains unclear how much of the full method's benefit specifically requires temporary representation injection, beyond changing the alignment direction and target. Such a control should account for how the target projections are learned and frozen, rather than use arbitrary untrained targets. Similarly, K/V is the best empirical choice in Table 5, but the explanation that it is more suitable for internalization is still a hypothesis: V-only is better in the oracle study, whereas K/V is better after scaffold removal. The results establish effectiveness more clearly than they establish the proposed mechanism.

  2. The positioning relative to the closest representation-based methods needs greater precision and comparative evidence. Section 6 cites REG[3] and joint image-feature synthesis[4], but its subsequent characterization of prior methods as treating external representations only as alignment targets is too broad: these works already place representations in the generative computation through joint denoising. REPI's temporary clean K/V injection and unchanged inference architecture are meaningful differences, and I do not view the algorithms as identical. However, these differences should be articulated more precisely and supported by a relevant comparison. The evaluation also concentrates on external-encoder alignment methods, leaving its relation to self-supervised representation guidance underexplored.

Notably, I would consider raising my score if the rebuttal provides convincing targeted controls and clarifies the method's advantage and distinction relative to the closest alternatives.

Questions for authors
  1. How does REPI compare with a no-scaffold baseline using the same K/V internalization targets? Please specify a suitable control for learning/freezing the target projections, and clarify when the internalization loss becomes active. This would help isolate the benefit of actually using the scaffold during denoising.

  2. Why does jointly injecting K/V favor internalization more than V-only, despite the opposite ranking in the oracle experiment? Can you provide direct evidence, such as the generation-quality change at removal, recovery during subsequent training, or matching errors across noise levels? The current attention-routing explanation is plausible, but the target ablation alone does not identify this mechanism.

  3. How does REPI compare with or complement SRA[1] and Self-Flow[2] under comparable settings? More broadly, does the scaffold depend on a fixed pretrained encoder, or could lower-noise EMA-teacher representations support injection/internalization? A comparison would inform the current contribution; the latter is a scope/mechanism question, not a requirement to develop a complete new variant during rebuttal.

  4. Can you clarify the distinction from REG[3] and joint image-feature synthesis[4], qualify the broad alignment-target description in Section 6, and provide a matched comparison or explain the relevant protocol differences? Answers to these questions could materially change my assessment.

References

[1] Jiang D, Wang M, Li L, et al. No other representation component is needed: Diffusion transformers can provide representation guidance by themselves[J]. arXiv preprint arXiv:2505.02831, 2025.

[2] Chefer H, Esser P, Lorenz D, et al. Self-supervised flow matching for scalable multi-modal synthesis[J]. arXiv preprint arXiv:2603.06507, 2026.

[3] Wu G, Zhang S, Shi R, et al. Representation entanglement for generation: Training diffusion transformers is much easier than you think[J]. arXiv preprint arXiv:2507.01467, 2025.

[4] Kouzelis T, Karypidis E, Kakogeorgiou I, et al. Boosting generative image modeling via joint image-feature synthesis[J]. arXiv preprint arXiv:2504.16064, 2025.

4 comments
AuthorsOpenReview ↗

Response to Reviewer ANN8 — Part 1 of 2

We thank Reviewer ANN8 for the careful reading of our paper and the constructive comments.

Below, we address the reviewer's concerns point by point. Due to the comment length limit, our response is split into two parts.

Weakness 1(a) & Question 1: The evidence for the specific scaffold-to-internalization mechanism remains incomplete. Figure 6 establishes that scaffold-only training helps and that internalization improves it, but does not include a no-injection K/V-alignment control. [...] How does REPI compare with a no-scaffold baseline using the same K/V internalization targets?

We thank the reviewer for this suggestion and agree that our original ablation in Figure 6 does not include a no-injection K/V-alignment control. We have run the requested control experiment and report the results below.

No-injection K/V-alignment control. As in REPI, we obtain the encoder-to-diffusion projectors from a 20K-step scaffold phase on a separate SiT-XL/2, which is then discarded. We freeze these projectors and train a freshly initialized SiT-XL/2 with both the denoising and internalization objectives (same layer and λ\lambda as REPI). The student is never trained with injection, and its targets are identical to those used by REPI. The updated ablation on SiT-XL/2 (FID, w/o CFG) is:

ScaffoldInternalization100K200K400K
✗✗39.4026.4917.20Vanilla SiT-XL/2
✓✗22.4513.699.49Scaffold-only
✗✓18.0910.087.71No-injection K/V-alignment control
✓✓14.518.937.21Full REPI

How does REPI compare with this no-scaffold baseline? As shown in the table above, the control improves substantially over vanilla SiT-XL/2, from 39.40 to 18.09 FID at 100K steps and from 17.20 to 7.71 at 400K steps. However, a gap to full REPI remains at every checkpoint: 3.58 at 100K (18.09 vs. 14.51), 1.15 at 200K (10.08 vs. 8.93), and 0.50 at 400K (7.71 vs. 7.21). Notably, the control performs at roughly the level of REPA (7.71 vs. 7.90 at 400K), so REPI's advantage over REPA comes mainly from the scaffold phase.

We summarize three observations showing that the scaffold phase makes a meaningful contribution:

  1. The scaffold alone is beneficial. In the scaffold-only setting, the projectors are discarded after 20K steps and encoder-derived targets are never used as supervision. Even so, FID improves from 17.20 (vanilla SiT-XL/2) to 9.49 at 400K steps.
  2. The scaffold helps even when good targets are available. In the new control, meaningful projected targets are already available. Adding the scaffold phase still improves FID from 18.09 to 14.51 at 100K steps and from 7.71 to 7.21 at 400K steps.
  3. The projected targets themselves are produced by the scaffold. The encoder-to-diffusion projectors used in the new control are learned during the scaffold phase, and they must be learned separately for each diffusion transformer architecture.

In the revised paper, we have added this new control to Figure 6 and updated Section 5.4 (L407-413) to discuss the result.

Weakness 1(b) & Question 2: K/V is the best empirical choice in Table 5, but the explanation that it is more suitable for internalization is still a hypothesis: V-only is better in the oracle study, whereas K/V is better after scaffold removal. [...] Why does jointly injecting K/V favor internalization more than V-only, despite the opposite ranking in the oracle experiment?

We thank Reviewer ANN8 for raising this issue. In the revised paper, we have added Figure 10 (L1009-1025), which shows the denoising loss curves of the K/V and V-only variants before and after scaffold removal. During the scaffold phase (0–20K steps), V-only attains a lower loss than K/V throughout, consistent with the oracle results in Table 1. After scaffold removal, the ranking reverses, and K/V has a lower loss than V-only for the rest of training. This supports our hypothesis that jointly injecting K/V is better suited to scaffold removal and internalization. The oracle setting, by contrast, only measures how effectively an external representation can be consumed, which is why V-only performs best there. We have added this observation to Section 5.4 (L418-423) of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer ANN8 — Part 2 of 2

Weakness 2(a) & Question 4: [...] REG[3] and joint image-feature synthesis[4] already place representations in the generative computation through joint denoising. REPI's temporary clean K/V injection and unchanged inference architecture are meaningful differences [...] Can you clarify the distinction from REG[3] and joint image-feature synthesis[4], qualify the broad alignment-target description in Section 6, and provide a matched comparison or explain the relevant protocol differences?

We thank the reviewer for pointing this out. We agree that our original wording in Section 6 was too broad: REG [3] and joint image-feature synthesis (ReDi) [4] already incorporate semantic representations into the generative computation through joint denoising of image latents and semantic features.

Distinction from REG and ReDi. REG and ReDi jointly denoise image latents and semantic representations, which requires extending the transformer’s input embeddings and prediction heads. Both therefore carry additional semantic denoising states and modality-specific input/output mappings into inference, which adds some inference overhead. In contrast, our approach injects projected clean encoder representations only during training, as a temporary scaffold, and internalizes this guidance into the model’s native representations. The inference architecture and sampling state thus remain identical to those of the original diffusion transformer.

We have updated Section 6 (L475-480) accordingly:

[...] Most closely related, REG (Wu et al., 2025a) and ReDi (Kouzelis et al., 2025) also incorporate external representations by jointly denoising image latents and semantic representations. However, both methods extend the transformer's input embeddings and prediction heads to generate semantic representations, and keep these modifications at inference. In contrast, we inject projected clean encoder representations only during training, as a temporary scaffold, and internalize this guidance into the model's native representations, leaving the inference architecture and sampling state unchanged.

Weakness 2(b) & Question 3: The evaluation also concentrates on external-encoder alignment methods, leaving its relation to self-supervised representation guidance underexplored. How does REPI compare with or complement SRA[1] and Self-Flow[2] under comparable settings? Does the scaffold depend on a fixed pretrained encoder, or could lower-noise EMA-teacher representations support injection/internalization? [...] the latter is a scope/mechanism question, not a requirement to develop a complete new variant during rebuttal.

We thank the reviewer for suggesting methods based on self-supervised representations.

Comparison with SRA and Self-Flow. The table below compares reported guided-generation results on ImageNet 256×256 with SiT-XL/2. Detailed evaluation settings and results are provided in Appendix E of the revised paper. REPA + REPI reaches an FID of 1.47 at 1M steps, matching Self-Flow's reported 1.47 at 4M steps and outperforming SRA's 1.58 at 4M steps. We note, however, that SRA and Self-Flow do not rely on external encoders, which is a clear advantage when strong pretrained encoders are unavailable. We have added this comparison in Appendix E (L945-951) of the revised paper.

MethodTraining stepsFID ↓
SRA4M1.58
Self-Flow4M1.47
REPA + REPI (ours)1M1.47
REPA + REPI (ours)2M1.42

Lower-noise EMA-teacher representations for injection and internalization. Technically, EMA-teacher features can also be projected into the student's K/V spaces. We believe this is a promising direction for removing the dependence on external encoders. One possible challenge is that the teacher's features may not be sufficiently informative early in training. A natural remedy is to first train the student and update its EMA teacher without injection, then introduce the scaffold after a warm-up period while the EMA updates continue. Due to the limited rebuttal time, we leave this extension for future work and will discuss it in the revised paper.

References

[1] Jiang D, Wang M, Li L, et al. No other representation component is needed: Diffusion transformers can provide representation guidance by themselves[J]. arXiv preprint arXiv:2505.02831, 2025.

[2] Chefer H, Esser P, Lorenz D, et al. Self-supervised flow matching for scalable multi-modal synthesis[J]. arXiv preprint arXiv:2603.06507, 2026.

[3] Wu G, Zhang S, Shi R, et al. Representation entanglement for generation: Training diffusion transformers is much easier than you think[J]. arXiv preprint arXiv:2507.01467, 2025.

[4] Kouzelis T, Karypidis E, Kakogeorgiou I, et al. Boosting generative image modeling via joint image-feature synthesis[J]. arXiv preprint arXiv:2504.16064, 2025.

I thank the authors for their detailed response. Most of my concerns have been addressed in the rebuttal, and I will accordingly raise my score to 3.

I would like to offer one additional suggestion. I understand that the rebuttal period is limited and that new experiments may not be feasible within this timeframe. Nevertheless, I would encourage the authors to consider a small-scale text-to-image experiment to further demonstrate the generality of the proposed method. Including such results in the future version would strengthen the paper.

Dear Reviewer ANN8,

Thank you for confirming that our rebuttal addressed most of your concerns. We sincerely appreciate your constructive feedback, which has helped us strengthen the manuscript. We will incorporate the additional results and discussions above into the revised version.

We agree that text-to-image experiments are valuable for demonstrating the generality of REPI. We would like to point out that our current manuscript already includes a small-scale text-to-image experiment, using MM-DiT on MS-COCO trained for 150K steps (Section 5.2, Table 4). Without classifier-free guidance, REPI improves FID over REPA from 10.40 to 9.47, and combining REPA and REPI further reduces it to 9.08. We will make these results more prominent in the revised manuscript.

Best regards,
The Authors

Reviewer_Adz6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

REPI uses clean-image representations from a frozen encoder to temporarily replace keys and values in a diffusion transformer. After removing this scaffold, an internalization loss encourages the transformer's native K/V representations to match the projected targets. The encoder and projection layers are discarded at inference. The paper reports faster convergence across several encoders, backbones, and generation settings, with further gains when combining REPI with REPA.

Critical strengths
  1. The scaffold has a measurable benefit.

  2. Gains are huge while the overhead is modest.

Critical weaknesses
  1. Figure 6 tests scaffold-only training and then adds internalization, but it does not compare the full method with direct K/V distillation without scaffolded student training. That control would help determine how much the proposed sequence adds beyond the choice of K/V supervision. Since the projection targets are learned during scaffolding, a fair comparison would need comparably obtained targets and include their training cost. The existing ablation shows that injection helps; it leaves this further attribution unresolved.

  2. The closest guided comparisons need more detail. Table 3 reports FIDs of 1.79 for REPI, 1.80 for Stable Velocity, and 1.73 for REPA+REPI with guidance scheduling. These point estimates lack run-to-run uncertainty, and the PDF does not explicitly specify the quantitative guidance scales, interval bounds, or selection procedure. The small differences are therefore less convincing than the much larger early-training improvements.

Questions for authors
  1. How does REPI compare with direct K/V distillation without scaffolded student training, using comparably trained projection targets and a matched total training budget?

  2. What guidance scales, interval bounds, and selection procedure were used for Table 3? Are the closest improvements larger than the observed variation across independent runs or evaluations?

1 comment
AuthorsOpenReview ↗

Response to Reviewer Adz6

We thank Reviewer Adz6 for the careful reading of our paper and the constructive comments.

Below, we address the reviewer’s concerns point by point.

Weakness 1 & Question 1: Figure 6 tests scaffold-only training and then adds internalization, but it does not compare the full method with direct K/V distillation without scaffolded student training. [...] Since the projection targets are learned during scaffolding, a fair comparison would need comparably obtained targets and include their training cost. [...] How does REPI compare with direct K/V distillation without scaffolded student training, using comparably trained projection targets and a matched total training budget?

We thank the reviewer for this suggestion and agree that our original ablation did not include a comparison with direct K/V distillation without scaffolded student training. We have run this experiment and report the results below.

New variant (direct K/V distillation, no scaffolded student training). As in REPI, we obtain the encoder-to-diffusion projectors from a 20K-step scaffold phase on a separate SiT-XL/2, which is then discarded. We freeze the projectors and train a freshly initialized SiT-XL/2 with the denoising objective plus the K/V internalization loss (same layer and λ\lambda as REPI). Following the reviewer's suggestion, the step counts for this variant include the 20K projector-training steps, so all comparisons are made under a matched total training budget. The updated ablation on SiT-XL/2 (FID, w/o CFG) is:

ScaffoldInternalization100K200K400K
✗✗39.4026.4917.20Vanilla SiT-XL/2
✓✗22.4513.699.49Scaffold-only
✗✓18.0910.087.71New variant (no scaffolded training)
✓✓14.518.937.21Full REPI

How does REPI compare with direct K/V distillation? REPI is better at every checkpoint under a matched total budget, by 3.58 at 100K (18.09 vs. 14.51), 1.15 at 200K (10.08 vs. 8.93), and 0.50 at 400K (7.71 vs. 7.21). Notably, the new variant is on par with REPA at 400K (7.71 vs. 7.90), so REPI's advantage over REPA comes mainly from the injection component.

In the revised paper, we have added this new variant to Figure 6 and updated Section 5.4 (L407-413) to discuss the result.

Weakness 2 & Question 2: [...] Table 3 reports FIDs of 1.79 for REPI, 1.80 for Stable Velocity, and 1.73 for REPA+REPI with guidance scheduling. These point estimates lack run-to-run uncertainty, and the PDF does not explicitly specify the quantitative guidance scales, interval bounds, or selection procedure. The small differences are therefore less convincing than the much larger early-training improvements. What guidance scales, interval bounds, and selection procedure were used for Table 3? Are the closest improvements larger than the observed variation across independent runs or evaluations?

We thank Reviewer Adz6 for pointing out these missing details. We provide them below and have updated the paper accordingly.

Guidance settings and selection procedure. We follow the evaluation protocol of REPA. Specifically, we sweep the guidance scale and the guidance interval and select the configuration with the lowest FID. For both REPI and REPA + REPI, the selected settings are a scale of 1.3 without the guidance interval, and a scale of 1.7 with the guidance interval [0, 0.7]. As is standard in this line of work, each method uses its own best-performing scale. Full details of the guidance settings are given in Tables 12 and 13 of the revised paper.

On the small FID differences. We agree that the FID gap narrows later in training. We attribute this mainly to FID saturating in the low-FID regime: REPA itself improves only from 1.96 to 1.80 between 200 and 800 epochs. With the guidance interval, REPA + REPI reaches an FID of 1.42 at 400 epochs, which matches REPA’s 1.42 at 800 epochs while using half the training (400 fewer epochs, i.e., 2M fewer iterations). We therefore regard the training cost required to reach a given FID as the more informative measure of the gain.

Run-to-run variation. Prior work in this line, including REPA, reports single-run point estimates, and our original Table 3 followed this practice. As Reviewer Adz6 suggested, we now report mean ± standard deviation over three evaluation runs of the FIDs. The results are 1.797 ± 0.021 for REPI and 1.733 ± 0.015 for REPA + REPI. Due to limited compute and rebuttal time, we could only repeat the evaluation to measure its variation, and could not run independent training runs.

Are the closest improvements larger than the observed variation? REPA + REPI vs. Stable Velocity (1.733 vs. 1.80). The difference of 0.067 is more than four times the standard deviation we observe. REPA + REPI vs. REPA (1.733 vs. 1.98). The difference of 0.25 is more than ten times the standard deviation. For the baselines, only point estimates are available.

Reviewer_Dw7dOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes REPI, a training framework that temporarily injects projected features from a frozen vision encoder into the key/value pathways of a diffusion transformer. An internalization objective encourages native representations to reproduce the projected features, allowing the scaffold to be removed without adding inference-time encoder computation. The approach is complementary to REPA and is evaluated across model sizes, architectures, encoder choices, and generation settings.

Critical strengths
  1. The method is conceptually clear and addresses a practical limitation of relying exclusively on representation alignment. The scaffold provides a direct path for useful encoder features to influence denoising, while the eventual model retains its native inference architecture.

  2. The empirical evidence is broad. At 400K steps, SiT-XL obtains unguided FID 7.21 with REPI and 6.33 with REPA + REPI, compared with 7.90 for REPA (Table 2). Figure 6 shows improvements from adding the scaffold and then internalization. The encoder and architecture experiments strengthen the case that this is more than a single-configuration improvement.

  3. The appendix reports measured training cost, not just iteration counts: 100K steps take 4.63 hours for REPA, 4.78 for REPI, and 4.81 for their combination on the reported hardware (Appendix A, Table 7). This supports the practical relevance of faster convergence.

Critical weaknesses
  1. The component analysis does not fully isolate the contribution of injection. Figure 6 compares the baseline, scaffold only, and scaffold plus internalization. These comparisons demonstrate the benefit of adding components, but do not fully separate the benefit of direct injection from that of learning suitable encoder-to-model targets. A control using comparable, meaningfully initialized projected targets without injection would help; using arbitrary frozen random projectors would not be an informative substitute. This is a limitation of the mechanistic explanation rather than a refutation of the reported performance.

  2. The benefit under longer training remains unclear. The reported REPI results extend only to 200 epochs (Table 3), leaving open whether its advantage over REPA persists as training approaches convergence. Under the same CFG setting without a guidance interval, the FID gap between REPA and REPI decreases from 0.17 at 80 epochs (2.39 versus 2.22) to 0.06 at 200 epochs (1.96 versus 1.90). The combined REPA + REPI model also retains a smaller advantage at 200 epochs: its FID is 1.85, compared with 1.96 for REPA. This does not establish that the gains will disappear, but motivates a longer-training comparison. I would like to see REPA, REPI, and REPA + REPI trained substantially longer under matched settings, ideally reaching the sub-1.5 FID regime, to establish whether REPI improves final sample quality in addition to accelerating early training. Training curves and matched sampling/guidance settings are essential for this comparison.

Questions for authors
  1. Can a controlled no-injection variant retain most of the benefit when its projected targets are suitably initialized? Which evidence most directly establishes the need for the scaffold phase?

  2. Does the advantage over REPA persist beyond 200 epochs, particularly in the sub-1.5 FID regime? Please provide longer-training curves for REPA, REPI, and their combination with matched training and evaluation settings.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Dw7d — Part 1 of 2

We thank Reviewer Dw7d for the careful reading of our paper and the constructive comments.

Below, we address the reviewer's concerns point by point. Due to the comment length limit, our response is split into two parts.

Weakness 1 & Question 1: The component analysis does not fully isolate the contribution of injection. [...] A control using comparable, meaningfully initialized projected targets without injection would help; using arbitrary frozen random projectors would not be an informative substitute. [...] Can a controlled no-injection variant retain most of the benefit when its projected targets are suitably initialized? Which evidence most directly establishes the need for the scaffold phase?

We thank the reviewer for this suggestion and agree that our original component analysis did not fully isolate the contribution of injection. We have run the requested control experiment and report the results below.

New control (same targets, no injection). As in REPI, we obtain the encoder-to-diffusion projectors from a 20K-step scaffold phase on a separate SiT-XL/2, which is then discarded. We freeze these projectors and train a freshly initialized SiT-XL/2 with both the denoising and internalization objectives (same layer and λ\lambda as REPI). The diffusion transformer is therefore never exposed to injection. The updated ablation on SiT-XL/2 (FID, w/o CFG) is:

ScaffoldInternalization100K200K400K
✗✗39.4026.4917.20Vanilla SiT-XL/2
✓✗22.4513.699.49Scaffold-only
✗✓18.0910.087.71New control (no injection)
✓✓14.518.937.21Full REPI

Without injection, the model still improves substantially over vanilla SiT-XL/2, reducing FID from 39.40 to 18.09 at 100K steps and from 17.20 to 7.71 at 400K steps. However, a gap to full REPI remains at every checkpoint: 3.58 at 100K (18.09 vs. 14.51), 1.15 at 200K (10.08 vs. 8.93), and 0.50 at 400K (7.71 vs. 7.21). At 400K, the no-injection variant performs on par with REPA (7.71 vs. 7.90), suggesting that REPI's advantage over REPA comes from injection.

Which evidence most directly establishes the role of the scaffold phase? We point to three observations:

  1. The scaffold alone is beneficial. In the scaffold-only setting, the projectors are discarded after 20K steps and encoder-derived targets are never used as supervision. Even so, FID improves from 17.20 (vanilla SiT-XL/2) to 9.49 at 400K steps.
  2. The scaffold helps even when good targets are available. In the new control, meaningful projected targets are already available. Adding the scaffold phase still improves FID from 18.09 to 14.51 at 100K steps and from 7.71 to 7.21 at 400K steps.
  3. The projected targets are themselves a product of the scaffold phase. The encoder-to-diffusion projectors used in the new control are learned during the scaffold phase, and they must be learned separately for each diffusion transformer architecture.

In the revised paper, we have added this new control to Figure 6 and updated Section 5.4 (L407-413) to discuss the result.

AuthorsOpenReview ↗

Response to Reviewer Dw7d — Part 2 of 2

Weakness 2 & Question 2: The benefit under longer training remains unclear. The reported REPI results extend only to 200 epochs (Table 3) [...] The combined REPA + REPI model also retains a smaller advantage at 200 epochs: its FID is 1.85, compared with 1.96 for REPA. This does not establish that the gains will disappear, but motivates a longer-training comparison. [...] Does the advantage over REPA persist beyond 200 epochs, particularly in the sub-1.5 FID regime? Please provide longer-training curves for REPA, REPI, and their combination with matched training and evaluation settings.

Following the reviewer’s suggestion, we extended the training of REPA + REPI on SiT-XL/2 to 400 epochs (2M iterations). The results are shown below and have been added to Table 3 of the revised paper, and the training curves are provided in Figure 9 (L918-937). For guidance settings, we follow REPA’s evaluation protocol: we sweep the guidance interval and the CFG scale and select the best-performing setting. This yields a guidance interval of [0, 0.7] and a CFG scale of 1.7. The details of guidance settings are provided in Appendix D.1 (L939-943) of the revised paper.

MethodEpochs (Steps)FID (CFG) ↓\downarrowFID (CFG with guidance interval) ↓\downarrow
REPA200 (1M)1.961.66
REPA800 (4M)1.801.42
REPA + REPI (ours)200 (1M)1.851.47
REPA + REPI (ours)400 (2M)1.791.42

Does the advantage persist beyond 200 epochs, particularly in the sub-1.5 FID regime? Yes, in terms of training efficiency. With the guidance interval, REPA + REPI reaches 1.42 FID at 400 epochs, matching REPA's 1.42 at 800 epochs with half the training iterations. Without the guidance interval, REPA + REPI's 1.79 at 400 epochs likewise matches REPA's 1.80 at 800 epochs. Notably, REPA + REPI already reaches sub-1.5 FID (1.47) at 200 epochs. This 200-epoch result was absent from the original Table 3 because the baselines do not report 200-epoch numbers, and it is now included.

On the narrowing FID gap. We agree that absolute FID gaps shrink later in training. We believe this mainly reflects the compression of FID in the low-FID regime. REPA itself improves only from 1.96 to 1.80 between 200 and 800 epochs, so our 0.11 advantage at 200 epochs (1.85 vs. 1.96) amounts to roughly two-thirds of REPA's entire improvement over 600 additional epochs. For this reason, we consider the training required to reach a given FID the more informative measure.

Due to limited compute and time, we prioritized the combined model (REPA + REPI), our strongest configuration. The long-training run for REPI alone is in progress. We will post its results in this thread if it completes within the discussion period, and will include them in the final version of the paper.

Reviewer_Dw7dOpenReview ↗

Response

I thank the author for the new experiments and results. Weakness 1 is resolved with positive evidence and 2 is resolved with slightly negative evidence. It's good to know that REPI serves more as an convergence acceleration methods where FID gap narrows in longer training. I'll therefore keep my positive score.

Dear Reviewer Dw7d,

Thank you for your thoughtful follow-up and continued support. We appreciate your view of REPI as primarily a convergence accelerator, and we agree that the narrowing FID gap under longer training calls for a measured interpretation of the gains.

As promised, the long-training run for REPI alone has finished. Results on SiT-XL/2 are shown below.

MethodEpochs (Steps)FID ↓ (CFG with guidance interval)
REPA200 (1M)1.66
REPA800 (4M)1.42
REPI (ours)200 (1M)1.51
REPI (ours)400 (2M)1.44
REPA + REPI (ours)200 (1M)1.47
REPA + REPI (ours)400 (2M)1.42

At 200 epochs it reaches an FID of 1.51, compared with 1.66 for REPA. At 400 epochs it reaches 1.44, within 0.02 of REPA's 1.42 at 800 epochs, using half the training iterations. We have added these results to Table 3 of the revised paper.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.