- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates how critic capacity and conditioning affect distribution matching distillation, highlighting the importance of scaling the critic alongside the teacher. The authors also introduce a step-aware critic (STAC), which conditions on the student’s generation step to estimate the step-wised distribution. Experiments with SiT on ImageNet, SDXL, FLUX.2, and Wan2.1 demonstrate improvements in generation quality and training stability.
Critical strengths
-
The controlled scaling experiments show that increasing teacher capacity alone can degrade student quality or cause training collapse, providing practical evidence for scaling the critic alongside the teacher.
-
STAC offers a simple conditioning change whose benefits are supported by targeted ablations and experiments across image and video generation, without increasing student inference cost.
Critical weaknesses
- Although Figure 4 reports computational cost per student update, it remains unclear whether the advantage of larger critics persists under the same total training budget. Figure 13 also omits an XL critic with five updates per student update, leaving the tradeoff between critic capacity, update frequency, and training duration insufficiently characterized.
Questions for authors
-
Under the same total computational budget, how does an XL critic using five updates per student update compare with an XL critic using one update and a smaller critic using more updates or a longer training run?
-
How does STAC compare with TDM in generation quality and training stability when the teacher, student, critic capacity, and total training budget are held fixed?
We thank Reviewer ZA2K for the positive evaluation and for raising important and insightful questions.
W1 & Q1: Iso-compute comparison of critic size, update frequency, and training duration. Evaluation under similar compute is indeed a very important comparison. Following your suggestion, we added an explicit iso-compute view in Figure 13d,e,f, re-plotting the runs from Figure 13a,b,c against total training FLOPs, counting all forward and backward passes of the student, teacher, and critic with the statistics in Table 6.
Q2: Controlled comparison of STAC vs. TDM at fixed capacity and training budget
As summarized in the general response (and also Table 20, Appendix F.4 in the paper), we added a controlled comparison on FLUX.2. The measured training times per step on 8×H100 GPUs are 1.623 s for the DMD2 baseline, 1.634 s for the TDM-style variant, and 1.630 s for STAC (Table 12).
Regarding training stability, all three configurations train smoothly without collapse under the matched 9B/9B capacity. Regarding generation quality, TDM's disjoint score-query intervals yield a lower FID (12.24 vs. 13.93), whereas STAC's full-range step-k conditioning achieves higher text alignment and human preference across CLIP-S, ImageReward, HPSv3, and GenEval.