acceptodds
Under review as a conference paper at ICLR 2027

An End-to-End Latent-Rollout Approach for Pushing Few-Step ImageNet- Generation to FID without Fr\'echet Losses

Abstract

Iterative generation poses a joint optimization problem across steps, as intermediate predictions shape subsequent computations and ultimately determine the final output distribution. Few-step generators distilled from pretrained diffusion and flow-matching models make such optimization computationally practical end to end. We build on this opportunity with a distill-then-refine approach that uses teacher imitation to establish a strong initialization for a few-step rollout in latent space, then shifts to end-to-end refinement of the complete latent rollout against real data. We introduce FiST (Flow-in-Stage Transformer), an architecture that composes learned latent-state transitions in a few stages using a shared Transformer, with optional cross-stage hidden communication. Distillation applies teacher-forced regression to selected states along teacher trajectories; refinement replaces this supervision with adversarial and auxiliary classification objectives on the final latent output. A trainable discriminator module operates on semantically rich features extracted from clean real and generated latents by a frozen SiT backbone pretrained with REPA. All training takes place in latent space, without image decoding. During refinement, FiST consumes its own intermediate predictions, and endpoint gradients pass through every generation stage. For class-conditional generation on ImageNet at , our approach achieves FID 1.11 (IS 282) with three stages and FID 1.15 (IS 280) with two. These results demonstrate competitive few-step generation through learned distribution-level supervision, without explicit Fr\'echet-distance minimization. Ablations characterize how distillation, pretrained checkpoint choices, refinement supervision, and cross-stage hidden communication affect generation quality.

Primary area:
generative models
Keywords:
Image Generation, Few-Step Generation, End-to-End, Generative Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_v9juOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FiST, a few-step generator initialized through teacher-forced trajectory distillation and then refined with adversarial and classification losses. During refinement, the model uses its own intermediate predictions, and gradients pass through all generation stages. A frozen REPA-SiT backbone provides features for the discriminator. The method achieves FID 1.15 with two stages and 1.11 with three on ImageNet-256. I see a main question as whether additional distillation provides a better initialization for GAN fine-tuning.

Critical strengths
  1. The paper achieves strong few-step generation results on ImageNet-256. It also reports precision and recall, giving a broader view of generation quality beyond FID.

  2. The ablations on distillation duration, teacher guidance, pretrained checkpoints, and classification loss are useful. In particular, the results show the importance of REPA features for the discriminator.

  3. The cross-stage hidden router gives a small improvement with little additional inference cost.

Critical weaknesses
  1. The choice of initialization is not well motivated. A good initialization may help GAN fine-tuning, but this does not explain why the authors choose PD-style trajectory regression. Why not use distribution-level methods such as DMD, SiD, or TDM, or consistency-based methods such as CD and CTM? The paper also discusses APT, which uses consistency-based initialization before adversarial post-training. The current ablations vary distillation duration, teacher guidance, and pretrained checkpoints, but do not compare different initialization methods under the same refinement setup. Without this comparison, simply combining trajectory regression with GAN fine-tuning seems to offer limited novelty.

  2. A fair comparison with direct GAN fine-tuning is missing. The authors should keep the pretrained checkpoint, generator architecture, discriminator, and refinement losses unchanged, remove trajectory distillation, and give direct GAN fine-tuning the compute budget saved from distillation. Simply removing distillation while keeping refinement at 20K iterations would give this baseline less total compute. The current ablation gives every initialization 20K refinement iterations, so longer distillation also means more total training compute. It does not show whether distillation is a better use of the budget than longer GAN fine-tuning. A stronger direct fine-tuning baseline could also consider partial freezing to preserve pretrained representations.

  3. The benefit of full backpropagation through the rollout is unclear. The paper emphasizes passing the final loss through every stage, but does not compare this with stopping gradients between stages while keeping the same forward rollout. The stage-count ablation shows that multiple stages help under this recipe, but does not isolate the contribution of cross-stage gradient propagation.

Questions for authors
  1. How do different distillation methods compare as initializations when followed by the same GAN fine-tuning?

  2. If the compute spent on distillation is instead used for additional GAN fine-tuning, does the proposed two-stage recipe still perform better?

  3. How much does full backpropagation through all stages improve over stopping gradients between stages?

7 comments
AuthorsOpenReview ↗

Response to Reviewer v9ju

1. Regarding your question, “How do different distillation methods compare as initializations when followed by the same GAN fine-tuning?”

We chose teacher-trajectory regression because it gives each stage a direct target and keeps initialization simple. Our main interest is how low the final FID can go after refining a two- or three-step generator on ImageNet 256×256256\times256. We have made this motivation clearer in Section 3.2. DMD, SiD, TDM, CD, and CTM are reasonable alternatives, but we have not tested them under the same refinement setup.

2. Regarding your question, “If the compute spent on distillation is instead used for additional GAN fine-tuning, does the proposed two-stage recipe still perform better?”

We did try skipping distillation in early FiST experiments, but direct adversarial training failed to produce meaningful images. The pretrained backbone alone does not provide trained stage transitions: FiST introduces new stage embeddings and a state-prediction head. Distillation gives these components direct supervision before switching to endpoint losses.

We also found that extending adversarial training beyond roughly 20K iterations brought little improvement in the settings we tested, and sometimes reduced sample quality. We have added these observations to Section 4.3. They explain our training choices, though they are not a strictly compute-matched comparison under the final configuration.

3. Regarding your question, “How much does full backpropagation through all stages improve over stopping gradients between stages?”

With only two or three stages, full backpropagation is practical. If gradients are computed only through the last stage, the update still changes earlier stages because their weights are shared, but it does not account for how those earlier changes affect the final output. We retain the full gradient path to include these effects when updating the generator. We have not measured the improvement over truncation with an otherwise identical forward rollout.

Thank you for the response. My concerns remain. Two key ablations are still missing. The first would compare initialization methods under the same GAN fine-tuning setup. The second would compare full backpropagation against a variant that stops gradients between stages while keeping the forward rollout identical. More importantly, the reported failure of direct adversarial training to produce meaningful samples is at odds with the results of DMD2’s four-step GAN-only baseline (Yin et al., 2024, Table 4 and Fig. 7). Given the current experimental evidence, I will maintain my original score.

Reference

Yin, T., Gharbi, M., Park, T., Zhang, R., Shechtman, E., Durand, F., & Freeman, W. T. (2024). Improved distribution matching distillation for fast image synthesis. arXiv. https://arxiv.org/abs/2405.14867v2

AuthorsOpenReview ↗

Response to Reviewer v9ju (2)

Thank you for pointing out this baseline. We agree that DMD2 demonstrates successful GAN-only training. Our phrase “no meaningful samples” was intended to describe unsatisfactory sample quality in our preliminary direct-training trials, rather than a general failure of GAN-only fine-tuning.

The starting parameterizations differ. DMD2 retains a pretrained time-conditioned denoiser, whereas FiST removes time condition and introduces totally new stage embeddings and reinitializes the output layer to predict next-stage states. Reusing the backbone therefore does not provide the same initialized generative mapping.

Our earlier FiST experiments offer some context. We tried 1K distillation steps (compared against our reported 4K,10K,40K), the two- and three-stage FiST had FIDs of 163.75 and 215.0, which subsequent refinement reduced to 3.03 and 3.51. These earlier configurations are close to GAN-only training. They nevertheless show that refinement could succeed despite poor initial sample quality, suggesting that brief trajectory regression helps prepare the stage transitions for endpoint optimization.

Reviewer_PNMnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper propose the method named FiST, Flow-in-Stage Transformer. Core idea is that they distill the pretrained diffusion model into few-step generation, then fine-tune it with a end-to-end adversarial loss on rollout samples to learn end-to-end generation. Based on the REPA-SiT checkpoint, they achieve FID of 1.11 when they with 3 stage rollout examples, achieving lower FID comparable to the FD-loss post-trained diffusion models.

Critical strengths
  • Straight forward algorithm to extend the end-to-end generation with unrolled diffusion models.

  • Strong quantitative performance. It achieves FID of 1.11 with 3 step unrolling, comparable to FD-loss trained model which uses ground-truth feature extractor such as inception-v3.

Critical weaknesses
  • My major concern is its novelty. Since the adversarial objective and unrolling process is largely similar to typical adversarial post-training such as AFM, it seems the only difference is its usage of REPA-SiT and additional distillation process. What is the major difference against AdvFlow? I wonder that the difference to AFMstart with the pretrained diffusion model. Also, in this sense, I personally think AFM is more interesting, as they show that the feasibility of GAN training from scratch in transformer and ImageNet dataset.

  • Usage of VFM-distilled diffusion model. As noted in Table 4, the proposed method largely leverages the pretrained feature from REPA. Thus, I wonder that the importance of contribution, the fact that proposed method does not use FD-loss. It still significantly depends on the pretrained feature from REPA, thus for me, this means that its effectiveness is caused by the feature space where the supervision occurs, not the other components.

Questions for authors

First of all, please refer to weaknesses.

  • Why do you use mse-driven distillation? There are many few-step distillation method such as DMD family or models learning trajectory such as MeanFlow. In case of meanflow and improved meanflow, they inherently support few-step inference, so what we need is just adversarial post training with rollout. I think this does not hurt the performance lot, as table 4 show that the quality of features is important for discriminator.

  • It is better to more carefully denote the main difference towards AFM, without the usage of pretrained models with REPA. Also, it is good to show that the performance of AFM with REPA-SiT.

  • I wonder that the performance of FiST with vision encoder discriminator. For example, drifting model suggest the features extractor with SD-VAE, so it would be good to test this model as encoder for discriminator.

If my concerns are fully resolved after rebuttal, I am willing to increase my rating.

2 comments
AuthorsOpenReview ↗

Response to Reviewer PNMn

1. Regarding your question, “What is the major difference against AdvFlow?” and your suggestion to “show the performance of AFM with REPA-SiT.”

AFM’s multi-step formulation learns transitions between interpolated states using adversarial supervision and OT regularization. FiST starts with direct distillation of a two- or three-stage rollout, then trains all stages together using adversarial and auxiliary classification losses applied only to the final output. The intermediate states can therefore change during refinement. AFM also explores end-to-end training through Transformer repetition, so backpropagating through the full generator is not itself our distinction.

Our focus is on achieving competitive many-step quality through 2,3-step generation. We therefore use direct distillation and reuse existing checkpoints to keep the formulation simple and limit training cost. AFM with REPA-SiT would be a useful controlled comparison, but remains untested; our comparison uses AFM’s published results.

2. Regarding your concern that “its effectiveness is caused by the feature space where the supervision occurs, not the other components”.

REPA features are an important part of our approach. However, the representation space and the optimization objective are distinct: FiST learns a discriminator in this space to provide adaptive supervision, without directly minimizing Fréchet distances. Our claim concerns the refinement objective, not independence from pretrained representations.

The ablations also show contributions beyond the feature space. With the REPA discriminator unchanged, longer distillation improves final FID from 1.69 to 1.11 under the same refinement budget, while auxiliary classification improves FID from 1.22 to 1.11. Our study examines whether these components together enable two- and three-stage generators to achieve quality competitive with many-step models without Fréchet losses; we do not attribute the results to any single component.

3. Regarding your question, “Why do you use mse-driven distillation?” and your suggestion to use DMD-family or MeanFlow models for initialization.

Our focus is on end-to-end refinement of a short rollout. Teacher-trajectory regression provides direct targets for initializing each stage’s latent-state transition, keeping the formulation simple and limiting training cost, though DMD-family and MeanFlow models may be reasonable alternatives to investigate. During refinement, this regression objective is replaced entirely by endpoint adversarial and auxiliary classification losses, allowing the rollout to adapt beyond teacher imitation.

Also, the importance of discriminator features does not imply that initialization has little effect: our distillation-duration ablation shows that initialization affects the final FID even with the same REPA discriminator and refinement budget. The reported results demonstrate that direct trajectory regression already provides an effective initialization for refinement.

4. Regarding your question about “the performance of FiST with vision encoder discriminator.”

We use a frozen REPA-SiT backbone because it directly processes real and generated latents, keeping refinement entirely in latent space without image decoding. Pixel-space vision encoders would typically require decoding generated latents and backpropagating through the VAE decoder, adding training cost.

Our refinement formulation can also accommodate other differentiable feature extractors. In particular, the latent-MAE used by Drifting provides a decoder-free alternative, distinct from its SD-VAE tokenizer. These are reasonable alternatives to investigate, but their performance within FiST remains untested.

Thank you for the response. However, my main concerns remain.

  • Novelty vs. AFM: As you acknowledged, end-to-end backpropagation is not itself a distinction from AFM. The remaining differences (distillation-based init, REPA-SiT) are not isolated by a controlled comparison.
  • REPA dependency: Your reply to Reviewer VvTA supports this concern. Vanilla SiT gives FID > 8, and replacing only the discriminator with REPA gives 1.21. Thus, most of the gain comes from the REPA feature space, which weakens the "without Fréchet losses" claim.

Therefore, I will keep my score.

Reviewer_VvTAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on further refining a distilled few-step model by treating it as a differentiable generator, an approach the authors term distill-then-refine. Teacher imitation first provides a good initialization. The full few-step rollout is then optimized end to end against real data with an adversarial objective, rather than only imitating the teacher, and without Fréchet-distance losses. On ImageNet 256×256, the method reaches FID 1.11 with 3 steps and 1.15 with 2 steps.

Technically, the generator (FiST) is a pretrained SiT-XL/2 (REPA) applied K = 2–3 times with shared weights, with an optional cross-stage router that passes hidden features between stages. Distillation uses teacher-forced regression onto states along the teacher’s trajectories. Refinement applies a GAN loss with an auxiliary classifier to the final output only, with gradients flowing through all stages. The discriminator is a frozen REPA backbone with trainable heads.

Critical strengths
  • The paper addresses an important problem. Few-step generation is key to making diffusion models practical, and the idea of treating a short rollout as a differentiable generator and refining it against real data is natural and well motivated.

  • The empirical results are strong. FiST reaches FID 1.11 with 3 steps and 1.15 with 2 steps on ImageNet 256×256, comparable to many-step models such as REPA-E and RAE, and clearly surpasses its own teacher, without inference-time CFG or EMA.

  • The refinement stage appears to work well with a purely adversarial objective, without relying on the teacher, while maintaining good sample diversity (recall 0.69) which actually suprises me a bit.

Critical weaknesses

-Limited novelty and missing discussion of closely related work. Each component of the method has clear precedents. Adversarial training on real data as a way to go beyond teacher imitation already appears in DMD2 (Yin et al., 2024) and APT (Lin et al., 2025). Discriminators that attach lightweight trainable heads to pretrained backbone features appear in Projected GANs (Sauer et al., 2021), LADD (Sauer et al., 2024), and Soft-Di[M]O (Zhu et al., 2026). Refinement without a distillation loss appears in APT (Lin et al., 2025) and Reward-Instruct (Luo et al., 2025). Supervising only the endpoint of a multi-step rollout, with gradients through all steps, appears in DRaFT (Clark et al., 2024) and, for few-step generators, in Reward-Instruct (Luo et al., 2025). The paper seems to find its niche by combining several conditions (few steps + pure GAN + gradients through all stages + no Fréchet losses) into a setting not yet covered. Several closely related works are missing or only briefly mentioned: Soft-Di[M]O (Zhu et al., 2026), whose discriminator design closely similar to FiST’s and which targets the same ImageNet-256 benchmark; the RL line of work on few-step generators, e.g., AdvDMD (Wang et al., 2026) and TDM-R1 (Luo et al., 2026); direct optimization through few-step generators, such as ReNO (Eyring et al., 2024) and Reward-Instruct (Luo et al., 2025); and the GAN component of DMD2 (Yin et al., 2024). The differences from DMD2 and LADD should be discussed explicitly.

  • The core claim is not validated. The main selling point, emphasized throughout the paper including the title, is that gradients flow through all generation stages, yet no ablation tests it. Notably, DRaFT (Clark et al., 2024), which the paper cites, found that truncating backpropagation to the last few steps, even a single step, outperforms full backpropagation. The stage-count ablation does not answer this question, since it conflates more computation with end-to-end gradients. I would like to see, under the same setting: (a) full backpropagation; (b) backpropagation through the last stage(s) only; (c) gradients truncated between stages, with supervision applied at each stage, as in DMD2 (Yin et al., 2024) or Reward-Instruct+ (Luo et al., 2025) if applicable (I do understand there's x0^\hat{x_0} and xtx_t problem).

  • The architectural contribution is overstated. The “FiST architecture” is essentially SiT with the timestep embedding replaced by a stage embedding. The only new module is the cross-stage router, and its gain is limited (FID 1.16 → 1.11 at 3 stages). Moreover, because the hidden memory stays in the computation graph, the augmented variant already receives cross-stage gradients during distillation, while the basic variant does not, so it is unclear where the improvement comes from. An ablation that applies stop-gradient to the router’s memory would help separate its forward role (passing information) from its backward role (providing a gradient path).

Minor:

  • Why does pure adversarial refinement on clean inputs remain stable? In theory and my experience, real and generated distributions in high dimensions tend to have nearly common supports (usually when the generator is not good enough, e.g., starting from the init status, it would too easy to differentiate the real and the fake), so the discriminator can separate them perfectly and its gradients become uninformative or unstable. This is why many methods add noise to the discriminator inputs. FiST instead feeds clean latents (t = 0) and uses no teacher regularization. What prevents training from collapsing? Is it the frozen backbone, spectral normalization, the strong initialization, or something else? This part actually is quite interesting and I am very curious to know why this is happening? It would be appreciated if the authors could provide (i) an ablation with noised discriminator inputs, (ii) training curves beyond 20K iterations (multiple seeds preferred of course), and (iii) an explanation of why refinement fails when vanilla SiT is used for both distillation and the discriminator.

  • Depiste nice performance under IN-256, it would be also very interesting to see this method to be extended to T2I settings.

  • I also find the writing quality and clarity is a bit subpar.

Questions for authors

Addressing my major concerns would help me to re-evaluate the paper.

1 comment
AuthorsOpenReview ↗

Response to Reviewer VvTA

1. Regarding your comment that “Several closely related works are missing or only briefly mentioned…”

Thank you for these suggestions. We have added Soft-Di[M]O, Reward-Instruct, ReNO, AdvDMD, and TDM-R1 to Related Work.

We now discuss the closest connections directly: Soft-Di[M]O trains discriminator heads on a frozen teacher backbone, while Reward-Instruct backpropagates endpoint rewards through the full few-step generator. ReNO instead optimizes input noise at inference time while keeping the generator fixed. AdvDMD combines distribution matching and adversarial training with discriminator-reward-driven GRPO updates, and TDM-R1 learns surrogate rewards for non-differentiable feedback.

2. Regarding your comment that “The differences from DMD2 and LADD should be discussed explicitly.”

DMD2 already uses real-data adversarial training, together with teacher-score distribution matching and an online fake-score model. FiST uses teacher-trajectory regression for initialization, then refines the generator using only endpoint adversarial and auxiliary classification losses. The gradient paths differ as well: DMD2’s released multi-step code generates inputs through backward simulation without gradients and updates a sampled denoising step. FiST backpropagates the final-output losses through all stages of the rollout.

LADD is also closely related: it uses frozen diffusion features for adversarial training in latent space, without decoding images. In its main setup, positive samples come from the teacher, and discriminator features are extracted after adding noise to the inputs. FiST uses encoded real images as positive samples and extracts features from clean real and generated latents with a frozen REPA-pretrained SiT backbone. LADD also reports adversarial-only training on synthetic data, so we do not claim that dropping the distillation loss is itself a distinguishing feature.

We have added both comparisons to Related Work.

3. Regarding your comment that “gradients flow through all generation stages, yet no ablation tests it.”

FiST has only two or three stages, so backpropagating through the full rollout is practical. Since the stages share weights, backpropagating through only the final stage would still change how earlier stages behave, while ignoring how those changes affect the final output. Keeping the earlier stages fixed would require separate frozen and trainable weights, turning the rollout into a fixed prefix followed by a trainable final stage.

DRaFT studies how far to backpropagate through a 50-step sampling chain. Truncation can help in that setting, but the same may not hold for a rollout of just two or three stages. A matched comparison with truncated backpropagation would still be useful to assess the benefit of retaining all cross-stage gradients.

4. Regarding your comments that “The architectural contribution is overstated” and that “An ablation that applies stop-gradient to the router’s memory would help ...”

Our main contribution is the distill-then-refine approach, implemented with a staged version of SiT. The router adds a small improvement: the basic three-stage rollout achieves FID 1.16, and adding the router brings it to 1.11 at roughly 1% more inference computation per stage. We describe this gain as modest in the paper.

We allow gradients to flow through memory so that later stages can influence what earlier stages pass forward. The ablation measures the combined benefit of passing information and adapting the features being passed. It does not separate their effects; a stop-gradient comparison would help do so.

5. Regarding your question, “Why does pure adversarial refinement on clean inputs remain stable?” and your request for “an ablation with noised discriminator inputs.”

FiST uses distilled initialization, frozen REPA features, spectral normalization, and TTUR-style optimization with warmup and gradient clipping to help stabilize training.

In early experiments, we tried adding input noise, both with and without dropout. Training was stable, but FID was worse for both two- and three-stage FiST. Input noise may smooth the discriminator’s response while weakening the signal needed to improve the generator. We therefore chose clean inputs.

6. Regarding your request for “an explanation of why refinement fails when vanilla SiT is used for both distillation and the discriminator.”*

When vanilla SiT is used for both distillation and the discriminator, FID remains above 8 after refinement. Replacing only the discriminator backbone with REPA brings FID to 1.21. This suggests that the main limitation is in the frozen discriminator features, since the same SiT initialization can still be refined successfully. REPA’s alignment with DINOv2 may make discriminator training on clean latents more stable. Using REPA for distillation as well further improves FID to 1.11. We have added this explanation to the pretrained-checkpoint ablation discussion.

Comments on the paper

1 comment

Dear AC and authors,

I appreciate authors answered most of the questions, yet barely without any experimental support. My concerns about the novelty (Ans1), ablation (Ans3/4) and other questions remains.

I have to maintain my current rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.