- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes FiST, a few-step generator initialized through teacher-forced trajectory distillation and then refined with adversarial and classification losses. During refinement, the model uses its own intermediate predictions, and gradients pass through all generation stages. A frozen REPA-SiT backbone provides features for the discriminator. The method achieves FID 1.15 with two stages and 1.11 with three on ImageNet-256. I see a main question as whether additional distillation provides a better initialization for GAN fine-tuning.
Critical strengths
-
The paper achieves strong few-step generation results on ImageNet-256. It also reports precision and recall, giving a broader view of generation quality beyond FID.
-
The ablations on distillation duration, teacher guidance, pretrained checkpoints, and classification loss are useful. In particular, the results show the importance of REPA features for the discriminator.
-
The cross-stage hidden router gives a small improvement with little additional inference cost.
Critical weaknesses
-
The choice of initialization is not well motivated. A good initialization may help GAN fine-tuning, but this does not explain why the authors choose PD-style trajectory regression. Why not use distribution-level methods such as DMD, SiD, or TDM, or consistency-based methods such as CD and CTM? The paper also discusses APT, which uses consistency-based initialization before adversarial post-training. The current ablations vary distillation duration, teacher guidance, and pretrained checkpoints, but do not compare different initialization methods under the same refinement setup. Without this comparison, simply combining trajectory regression with GAN fine-tuning seems to offer limited novelty.
-
A fair comparison with direct GAN fine-tuning is missing. The authors should keep the pretrained checkpoint, generator architecture, discriminator, and refinement losses unchanged, remove trajectory distillation, and give direct GAN fine-tuning the compute budget saved from distillation. Simply removing distillation while keeping refinement at 20K iterations would give this baseline less total compute. The current ablation gives every initialization 20K refinement iterations, so longer distillation also means more total training compute. It does not show whether distillation is a better use of the budget than longer GAN fine-tuning. A stronger direct fine-tuning baseline could also consider partial freezing to preserve pretrained representations.
-
The benefit of full backpropagation through the rollout is unclear. The paper emphasizes passing the final loss through every stage, but does not compare this with stopping gradients between stages while keeping the same forward rollout. The stage-count ablation shows that multiple stages help under this recipe, but does not isolate the contribution of cross-stage gradient propagation.
Questions for authors
-
How do different distillation methods compare as initializations when followed by the same GAN fine-tuning?
-
If the compute spent on distillation is instead used for additional GAN fine-tuning, does the proposed two-stage recipe still perform better?
-
How much does full backpropagation through all stages improve over stopping gradients between stages?
7 comments
Thank you for the response. My concerns remain. Two key ablations are still missing. The first would compare initialization methods under the same GAN fine-tuning setup. The second would compare full backpropagation against a variant that stops gradients between stages while keeping the forward rollout identical. More importantly, the reported failure of direct adversarial training to produce meaningful samples is at odds with the results of DMD2’s four-step GAN-only baseline (Yin et al., 2024, Table 4 and Fig. 7). Given the current experimental evidence, I will maintain my original score.
Reference
Yin, T., Gharbi, M., Park, T., Zhang, R., Shechtman, E., Durand, F., & Freeman, W. T. (2024). Improved distribution matching distillation for fast image synthesis. arXiv. https://arxiv.org/abs/2405.14867v2
Response to Reviewer v9ju (2)
Thank you for pointing out this baseline. We agree that DMD2 demonstrates successful GAN-only training. Our phrase “no meaningful samples” was intended to describe unsatisfactory sample quality in our preliminary direct-training trials, rather than a general failure of GAN-only fine-tuning.
The starting parameterizations differ. DMD2 retains a pretrained time-conditioned denoiser, whereas FiST removes time condition and introduces totally new stage embeddings and reinitializes the output layer to predict next-stage states. Reusing the backbone therefore does not provide the same initialized generative mapping.
Our earlier FiST experiments offer some context. We tried 1K distillation steps (compared against our reported 4K,10K,40K), the two- and three-stage FiST had FIDs of 163.75 and 215.0, which subsequent refinement reduced to 3.03 and 3.51. These earlier configurations are close to GAN-only training. They nevertheless show that refinement could succeed despite poor initial sample quality, suggesting that brief trajectory regression helps prepare the stage transitions for endpoint optimization.
Response to Reviewer v9ju
1. Regarding your question, “How do different distillation methods compare as initializations when followed by the same GAN fine-tuning?”
We chose teacher-trajectory regression because it gives each stage a direct target and keeps initialization simple. Our main interest is how low the final FID can go after refining a two- or three-step generator on ImageNet 256×256. We have made this motivation clearer in Section 3.2. DMD, SiD, TDM, CD, and CTM are reasonable alternatives, but we have not tested them under the same refinement setup.
2. Regarding your question, “If the compute spent on distillation is instead used for additional GAN fine-tuning, does the proposed two-stage recipe still perform better?”
We did try skipping distillation in early FiST experiments, but direct adversarial training failed to produce meaningful images. The pretrained backbone alone does not provide trained stage transitions: FiST introduces new stage embeddings and a state-prediction head. Distillation gives these components direct supervision before switching to endpoint losses.
We also found that extending adversarial training beyond roughly 20K iterations brought little improvement in the settings we tested, and sometimes reduced sample quality. We have added these observations to Section 4.3. They explain our training choices, though they are not a strictly compute-matched comparison under the final configuration.
3. Regarding your question, “How much does full backpropagation through all stages improve over stopping gradients between stages?”
With only two or three stages, full backpropagation is practical. If gradients are computed only through the last stage, the update still changes earlier stages because their weights are shared, but it does not account for how those earlier changes affect the final output. We retain the full gradient path to include these effects when updating the generator. We have not measured the improvement over truncation with an otherwise identical forward rollout.