- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work investigates that whether the improvements we have in single sample generation performance (pass@1) can translate into better performance at larger sampling budgets. It uses pass@k and best-of-k and compare five diffusion distillation families across four T2I benchmarks. It finds that although the distilled students show superior results at k=1, there is usually a crossover pattern observed and teacher models overtake them as we increase the sampling budget. It then proposes a controlled analysis using FLUX and compares DMD2, ADD, and MeanFlow and finds that DMD2 and ADD show coverage degradation while MeanFlow preserves the coverage. The work finally extends to video generation and observes a similar trend, which indicates the importance of such evaluation that is overlooked in previous works.
Critical strengths
- The paper has a very interesting finding that pass@1 that previous approaches are all focusing at, does not necessarily mean improvement when increasing sampling budget. And then it focuses on distilled diffusion models and compares five distillation families and four t2i benchmarks and shows a teacher-student crossover pattern that as we increase sampling budgets we see students usually get lower score than teachers even though they all show superior result with k=1 as the budget.
- The work further proposes an isolated study where it uses FLUX as shared teacher and student and compares DMD2, ADD, and MeanFlow. This controlled design has interesting findings that they show (i) Distribution matching and adversarial objectives lead to coverage collapse and (ii) degredation in pass@k happens with reduced diversity. They also evaluate different guidance scales and show that guidance alone does not lead to coverage degredation.
- There are interesting figures showing how coverage and diversity are related. Figure 4 investigates CLIP-space diversity of samples from both teacher and student and figure 6 shows how diversity changes throughout training and DMD2's coverage degredation happens with reduced diversity in representation space.
Critical weaknesses
- Although the controlled study in sec. 5.2 give us good insight in terms of how DMD2 and ADD show coverage degradation but MeanFlow preserves that, it is not fully clear from the distribution-matching objective that if the adversial loss term in DMD2 is causing the coverage degradation or not. a more controlled analysis such as varying the weight of adversial loss (and also discarding it) could clarify how it affects the coverage.
Questions for authors
- Is coverage collapse we see in sec. 5.2 an inherent property of distribution-matching objectives or is it because of additional loss?
- Can we optimize pass@k using differentiable verifier approximiations or using policy-gradient methods? If yes, can we use a large-k-aware training objective and combining them with DMD to preserve the student's improvement that it has at pass@1 while at same time recover larger-k performace?
2 comments
Thank you for the thorough answer and explanations.
I am grateful to the authors for their clarification of the fact that the controlled experiments are based on comparing full recipes of distillations and not individual components related to losses. The elaboration on the ablations on DMD2 also helps me understand how difficult it is to separate the adversarial objective.
Still, my main question has not been completely answered. Although the results show that different distillation recipes lead to different levels of coverage, the authors have not provided a clear conclusion on the reason of coverage degradation – distribution matching, adversarial objective, or both.
As for the second question, I am grateful to the authors for discussing the possibility of direct optimization of pass@k.
In general, I still find the experimental results interesting and informative, especially taking into account the systematic study under different sampling budgets. Thus, I will maintain my score of 3 (Weak Acceptance).
Thanks!
We thank the reviewer for your detailed review and recognition. Here, we address your concerns and questions.
W1, Q1 – Role of individual loss components– Good question! We are actually comparing complete recipes instead of single lossOur experiments in Sec. 5.2 compare complete training recipes rather than individual loss components. They neither isolate the contribution of the adversarial loss nor establish coverage collapse as an inherent property of distribution matching alone.
Diffusion distillation can be divided into two broad paradigms based on training objectives. Distribution matching trains a few-step student whose generations match the teacher's along the diffusion forward process. Trajectory-based consistency distillation instead shortens the teacher's sampling path by enforcing consistency along its trajectory.
We evaluate DMD2 and ADD as representative recipes of the first paradigm, and MeanFlow as a popular method of the second. DMD2 optimizes a VSD-style distribution-matching loss together with an adversarial objective, LVSD+Ladv. Our ADD implementation optimizes LSDS+Ladv. Component-only variants are not necessarily competitive practical baselines. In the original DMD2 ablation, removing the adversarial loss worsens FID from 19.32 to 26.90 and produces oversaturated or overly smoothed samples. This result motivates evaluating the complete DMD2 recipe at a quality-producing operating point. We love the idea of varying the loss weights of the adversarial loss in DMD2, but we cannot get meaningful results from most of the weights due to the sensitivity of DMD2 training.
Our controlled study compares two practically used distillation families. Both evaluated distribution-level recipes (DMD2 and ADD) exhibit coverage contraction under pass@k, whereas the trajectory-based MeanFlow recipe preserves coverage. Thus, our result identifies a coverage limitation of commonly used distribution-matching distillation recipes, rather than attributing the effect to an individual VSD, SDS, or adversarial term.
Table 4 of the DMD2 paper: Ablation studies with the SDXL backbone on 10K prompts from COCO 2014. Lower FID and Patch FID indicate better image quality, while higher CLIP scores indicate better text-image alignment. Bold values denote the best results.
Q2 – Optimizing pass@k– Thank you! We have added this as a future direction in the revised manuscript.We thank the reviewer for suggesting this promising future direction. Recent work on LLM reinforcement learning with verifiable rewards has already explored directly optimizing pass@k, including Pass@K Policy Optimization [1] and Pass@k Training [2]. In particular, [1] investigates annealing k during training to improve both pass@1 and pass@k. Extending such budget-aware objectives to diffusion distillation and combining them with DMD is a promising avenue for retaining the student’s single-sample advantage while improving large-budget performance. We appreciate the reviewer highlighting this connection and have discussed it as a future direction in the revised manuscript.
[1] Walder and Karkhanis. Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems. NeurIPS, 2025.
[2] Chen et al. Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models. arXiv:2508.10751, 2025.