acceptodds
Under review as a conference paper at ICLR 2027

Visualizing Distribution Coverage in Diffusion Models

Abstract

Modern diffusion pipelines rely on distribution-shaping and acceleration techniques such as classifier-free guidance, few-step distillation, and video forcing. However, improvements in sample quality do not reveal how these interventions reshape distribution coverage. We study this question through pass@, a sampling-budget lens on distribution coverage. Pass@ reports mean one-draw success, whereas the pass@ curve provides nonlinear summaries of how prompt-wise success probabilities are distributed and whether an early advantage persists as additional samples are drawn. Using CFG as a controlled proof of concept, we find that high guidance improves pass@, but its advantage shrinks and can reverse at larger . Across few-step diffusion families, several students similarly lead at small budgets but trail their multi-step teachers at larger budgets. A controlled ablation reveals an objective-level divide: distribution-matching distillation improves early-hit performance by concentrating the output distribution but systematically reduces large-budget distribution coverage, whereas consistency-style distillation better preserves coverage. Finally, video-forcing recipes exhibit different marginal best-of- gains as the sampling budget increases, extending the same analysis to video generation. Together, these results establish pass@ as a practical diagnostic of distribution coverage in diffusion model generations.

Primary area:
generative models
Keywords:
diffusion models, mode collapse, few-step distillation, video generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_jZZrOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This work investigates that whether the improvements we have in single sample generation performance (pass@1) can translate into better performance at larger sampling budgets. It uses pass@k and best-of-k and compare five diffusion distillation families across four T2I benchmarks. It finds that although the distilled students show superior results at k=1, there is usually a crossover pattern observed and teacher models overtake them as we increase the sampling budget. It then proposes a controlled analysis using FLUX and compares DMD2, ADD, and MeanFlow and finds that DMD2 and ADD show coverage degradation while MeanFlow preserves the coverage. The work finally extends to video generation and observes a similar trend, which indicates the importance of such evaluation that is overlooked in previous works.

Critical strengths
  • The paper has a very interesting finding that pass@1 that previous approaches are all focusing at, does not necessarily mean improvement when increasing sampling budget. And then it focuses on distilled diffusion models and compares five distillation families and four t2i benchmarks and shows a teacher-student crossover pattern that as we increase sampling budgets we see students usually get lower score than teachers even though they all show superior result with k=1 as the budget.
  • The work further proposes an isolated study where it uses FLUX as shared teacher and student and compares DMD2, ADD, and MeanFlow. This controlled design has interesting findings that they show (i) Distribution matching and adversarial objectives lead to coverage collapse and (ii) degredation in pass@k happens with reduced diversity. They also evaluate different guidance scales and show that guidance alone does not lead to coverage degredation.
  • There are interesting figures showing how coverage and diversity are related. Figure 4 investigates CLIP-space diversity of samples from both teacher and student and figure 6 shows how diversity changes throughout training and DMD2's coverage degredation happens with reduced diversity in representation space.
Critical weaknesses
  • Although the controlled study in sec. 5.2 give us good insight in terms of how DMD2 and ADD show coverage degradation but MeanFlow preserves that, it is not fully clear from the distribution-matching objective that if the adversial loss term in DMD2 is causing the coverage degradation or not. a more controlled analysis such as varying the weight of adversial loss (and also discarding it) could clarify how it affects the coverage.
Questions for authors
  • Is coverage collapse we see in sec. 5.2 an inherent property of distribution-matching objectives or is it because of additional loss?
  • Can we optimize pass@k using differentiable verifier approximiations or using policy-gradient methods? If yes, can we use a large-k-aware training objective and combining them with DMD to preserve the student's improvement that it has at pass@1 while at same time recover larger-k performace?
2 comments

We thank the reviewer for your detailed review and recognition. Here, we address your concerns and questions.


W1, Q1 – Role of individual loss components – Good question! We are actually comparing complete recipes instead of single loss

Our experiments in Sec. 5.2 compare complete training recipes rather than individual loss components. They neither isolate the contribution of the adversarial loss nor establish coverage collapse as an inherent property of distribution matching alone.

Diffusion distillation can be divided into two broad paradigms based on training objectives. Distribution matching trains a few-step student whose generations match the teacher's along the diffusion forward process. Trajectory-based consistency distillation instead shortens the teacher's sampling path by enforcing consistency along its trajectory.

We evaluate DMD2 and ADD as representative recipes of the first paradigm, and MeanFlow as a popular method of the second. DMD2 optimizes a VSD-style distribution-matching loss together with an adversarial objective, LVSD+Ladv. Our ADD implementation optimizes LSDS+Ladv. Component-only variants are not necessarily competitive practical baselines. In the original DMD2 ablation, removing the adversarial loss worsens FID from 19.32 to 26.90 and produces oversaturated or overly smoothed samples. This result motivates evaluating the complete DMD2 recipe at a quality-producing operating point. We love the idea of varying the loss weights of the adversarial loss in DMD2, but we cannot get meaningful results from most of the weights due to the sensitivity of DMD2 training.

Our controlled study compares two practically used distillation families. Both evaluated distribution-level recipes (DMD2 and ADD) exhibit coverage contraction under pass@k, whereas the trajectory-based MeanFlow recipe preserves coverage. Thus, our result identifies a coverage limitation of commonly used distribution-matching distillation recipes, rather than attributing the effect to an individual VSD, SDS, or adversarial term.

Table 4 of the DMD2 paper: Ablation studies with the SDXL backbone on 10K prompts from COCO 2014. Lower FID and Patch FID indicate better image quality, while higher CLIP scores indicate better text-image alignment. Bold values denote the best results.

MethodFID (↓)Patch FID (↓)CLIP (↑)
w/o GAN26.9027.660.328
w/o Distribution Matching13.7727.960.307
w/o Backward Simulation20.6624.210.332
DMD2 (Ours)19.3220.860.332

Q2 – Optimizing pass@k – Thank you! We have added this as a future direction in the revised manuscript.

We thank the reviewer for suggesting this promising future direction. Recent work on LLM reinforcement learning with verifiable rewards has already explored directly optimizing pass@k, including Pass@K Policy Optimization [1] and Pass@k Training [2]. In particular, [1] investigates annealing k during training to improve both pass@1 and pass@k. Extending such budget-aware objectives to diffusion distillation and combining them with DMD is a promising avenue for retaining the student’s single-sample advantage while improving large-budget performance. We appreciate the reviewer highlighting this connection and have discussed it as a future direction in the revised manuscript.


[1] Walder and Karkhanis. Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems. NeurIPS, 2025.

[2] Chen et al. Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models. arXiv:2508.10751, 2025.

Thank you for the thorough answer and explanations.

I am grateful to the authors for their clarification of the fact that the controlled experiments are based on comparing full recipes of distillations and not individual components related to losses. The elaboration on the ablations on DMD2 also helps me understand how difficult it is to separate the adversarial objective.

Still, my main question has not been completely answered. Although the results show that different distillation recipes lead to different levels of coverage, the authors have not provided a clear conclusion on the reason of coverage degradation – distribution matching, adversarial objective, or both.

As for the second question, I am grateful to the authors for discussing the possibility of direct optimization of pass@k.

In general, I still find the experimental results interesting and informative, especially taking into account the systematic study under different sampling budgets. Thus, I will maintain my score of 3 (Weak Acceptance).

Reviewer_4Zf9OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether few-step diffusion students preserve their teachers' distribution coverage. The authors use pass@k for binary evaluations and expected best-of-k for continuous scores to examine performance as the number of independent samples per prompt increases.The main finding is that several students' single-sample advantages disappear or reverse at larger sampling budgets. The authors associate these differences with distillation objectives, reporting better coverage preservation for consistency- and trajectory-based methods in the evaluated settings.

Critical strengths
  • Systematic evaluation across sampling budgets.
  • Controlled comparisons and complementary diversity measurements.
Critical weaknesses
  • Diversity loss in diffusion distillation is already an established research problem. One-step Diffusion Models with f-Divergence Distribution Matching (Xu et al., 2025) explicitly analyzes the mode-seeking behavior of reverse-KL-based distillation and investigates alternative divergences with different coverage properties. Distilling Diversity and Control in Diffusion Models (Gandikota and Bau, 2025) directly studies reduced within-prompt diversity, investigates its relationship to generation dynamics, and proposes an intervention to restore it. More directly, rCM already discusses the differing quality–diversity tendencies of consistency and score distillation, and combines these objectives in both image and video generation. DP-DMD likewise introduces a training strategy specifically designed to preserve diversity within distribution matching distillation. These closely related studies are not adequately discussed in the manuscript. The authors should therefore clarify what their findings establish beyond these uncited existing results. The most defensible contribution appears to be the systematic characterization of how teacher–student rankings change with sampling budget.

  • For a prompt (x), let (q_x) denote the probability that one generated sample satisfies the evaluator. The population statistic is E_x[1-(1-q_x)^k]. This depends only on the prompt-wise success probabilities. It contains no information about how probability mass is distributed among distinct successful outputs. For example, a model that always generates the same valid image and a model that generates many different valid images can have identical pass@k curves for every (k), despite having substantially different within-prompt diversity. Consequently, a crossover in binary pass@k demonstrates differences in the distribution of success probabilities across benchmark prompts; it does not, by itself, identify within-prompt mode collapse. Similarly, best-of-k characterizes the upper tail of evaluator scores without directly measuring the diversity of the corresponding outputs.

Questions for authors
  • Beyond the settings in Figures 4 and 6, what evidence supports interpreting the observed crossovers as reduced within-prompt diversity? Could the abstract and conclusion distinguish this interpretation from benchmark-relative success coverage?
  • Do the DMD2 and ADD runs differ only in the distribution-matching term? Please clarify their loss weights and update schedules, together with the exact teacher-supervised MeanFlow objective. Is there evidence that the degradation persists without DMD2's adversarial component?
  • Could the authors assess whether model-dependent evaluator errors could change the teacher–student ordering? Appendix D provides useful checks, but its null-pair calibration does not directly measure errors on near-miss outputs from each model. Blinded human checks of representative outputs, an independent evaluator, or sensitivity bounds would help address this concern.
4 comments
AuthorsOpenReview ↗

Thank you! (1/3)

We thank the reviewer for your observant review and recognition. Here, we address your concerns and questions.


W1 – Missing related work – Sure, we expanded the related work in Section 2 to cover these studies.

Thank you for highlighting these closely related studies. We agree that diversity loss in diffusion distillation is an established problem, and we have expanded the related-work discussion (Section 2, in blue) to explicitly cover f-distill, Distilling Diversity and Control, rCM, and DP-DMD.

Specifically, f-distill [1] generalizes reverse-KL-based distribution matching to alternative f-divergences with different mode-seeking properties, evaluating diversity through recall and DINO-based in-batch similarity. Distilling Diversity and Control [2] investigates the role of early denoising steps in determining diversity and proposes hybrid inference that uses the base model for the first step before switching to the distilled model, with within-prompt diversity quantified using pairwise DreamSim distances. rCM [3] combines continuous-time consistency and score distillation to balance quality and diversity in large-scale image and video generation, demonstrating diversity differences through multi-seed visual examples. DP-DMD [4] uses teacher-derived supervision for the first student step to preserve diversity and DMD for subsequent refinement, measuring within-prompt diversity through DINO and CLIP feature distances.

Our contribution is complementary: we examine how a fixed sampling budget translates into benchmark-level success. Rather than asking only whether generated samples are perceptually diverse, we ask: given k independently generated candidates per prompt, how likely is a model to produce at least one successful output, and does the student still outperform its teacher? Pass@k measures the expected fraction of benchmark prompts successfully addressed within this budget, while best-of-k measures the expected highest evaluator score. This directly addresses multi-candidate generation and selection, where the relevant question is not merely whether outputs differ, but whether additional attempts yield a satisfactory result.

Reduced diversity alone does not determine the answer to this question: a less diverse student could still outperform its teacher at every sampling budget. Our results instead reveal that single-sample advantages frequently reverse as the budget increases. Across five teacher–student families and four image benchmarks, we observe reversals in 15 of 20 comparisons under the main evaluation protocols (Fig. 3), with analogous behavior in causal video generation (Fig. 7). Thus, this is a widespread pattern across the evaluated model families and benchmarks, rather than an isolated example.

We have revised the manuscript to acknowledge these prior contributions and foreground our distinct finding: single-sample benchmark rankings can give a different picture of teacher–student performance from evaluation at larger sampling budgets.

[1] Xu, Yilun, Weili Nie, and Arash Vahdat. "One-step diffusion models with f-divergence distribution matching." arXiv preprint arXiv:2502.15681 (2025).

[2] Gandikota, Rohit, and David Bau. "Distilling diversity and control in diffusion models." 2026 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE, 2026.

[3] Zheng, Kaiwen, et al. "Large scale diffusion distillation via score-regularized continuous-time consistency." International Conference on Learning Representations. Vol. 2026. 2026.

[4] Wu, Tianhe, et al. "Diversity-preserved distribution matching distillation for fast visual synthesis." arXiv preprint arXiv:2602.03139 (2026).


AuthorsOpenReview ↗

Thank you! (2/3)

W2, Q1 – Benchmark success and within-prompt diversity – Fair point; we now distinguish the two in the abstract and conclusion, and also provide Venti score measurement.

We thank the reviewer for highlighting this distinction and apologize for the ambiguity in our presentation. Our primary claim concerns benchmark-relative success coverage: whether a model’s single-sample advantage persists as the sampling budget increases. This is directly relevant to practical settings where users generate multiple candidates and select among them. Accordingly, pass@k measures the probability of obtaining a successful generation.

Figures 4 and 6 provide complementary evidence connecting these benchmark-level findings to within-prompt diversity. Figure 4 shows that the distilled student produces less diverse CLIP representations even on prompts where teacher and student achieve identical correctness. Figure 6 shows that DMD2’s declining large-budget performance during training is accompanied by decreasing within-prompt CLIP diversity, whereas MeanFlow preserves diversity while improving performance. These analyses support a connection in the examined settings; we do not claim that every crossover, by itself, establishes within-prompt diversity loss.

Table R4: Vendi score of teachers and students, higher means more diverse.

Model familyTeacher VendiStudent Vendi
SD3.5-L / Turbo2.9051.762
Qwen / Lightning1.9281.333
Sana / Sprint2.7261.695
SDXL / LCM3.7922.533
Z-Image / Z-Turbo5.4561.576

Beyond Figures 4 and 6, we additionally evaluate the Vendi score based on our generation. We use the same 50 GenEval prompts for every model. For each prompt, we take 48 images from each model and compute the Vendi score using CLIP ViT-L/14 features. We report the average across prompts. All five students have lower Vendi scores than their teachers. However, as shown by the pass@k results in the paper, lower Vendi does not always lead to lower benchmark coverage: LCM has lower Vendi than SDXL (2.53 vs. 3.79) yet higher pass@256 on GenEval (Fig. 3). Pass@k measures whether drawing more samples helps solve more benchmark prompts, while Vendi measures how spread out the generated images are in CLIP feature space; that spread can reflect semantic diversity as well as differences in image quality or artifacts.

We have revised the abstract and conclusion to clearly distinguish the benchmark-relative success coverage measured by our evaluation from the within-prompt diversity interpretation supported by these complementary analyses, stating that they are closely related and serve as complementary.

AuthorsOpenReview ↗

Thank you (3/3)

Q2 – Role of individual loss components – Good question! We are actually comparing complete recipes instead of single loss.

Our experiments compare complete training recipes rather than individual loss components. They neither isolate the contribution of the adversarial loss nor establish coverage collapse as an inherent property of distribution matching alone.

Diffusion distillation can be divided into two broad paradigms based on the way of training. Distribution matching trains a few-step students whose generations match the teacher's along the diffusion forward process. Trajectory distillation instead shortens the teacher's sampling path by enforcing consistency along its trajectory.

We evaluate DMD2 and ADD as representative recipes of the first paradigm, and MeanFlow as a popular method of the second. DMD2 optimizes a VSD-style distribution-matching loss together with an adversarial objective, LVSDL_{VSD}+LadvL_{adv}. Our ADD implementation optimizes LSDSL_{SDS}+LadvL_{adv}. Component-only variants are not necessarily competitive practical baselines. In the original DMD2 ablation, removing the adversarial loss worsens FID from 19.32 to 26.90 and produces oversaturated or overly smoothed samples. This result motivates evaluating the complete DMD2 recipe at a quality-producing operating point.

Table 4 of the DMD2 paper: Ablation studies with the SDXL backbone on 10K prompts from COCO 2014. Lower FID and Patch FID indicate better image quality, while higher CLIP scores indicate better text-image alignment. Bold values denote the best results.

MethodFID (↓)Patch FID (↓)CLIP (↑)
w/o GAN26.9027.660.328
w/o Distribution Matching13.7727.960.307
w/o Backward Simulation20.6624.210.332
DMD2 (Ours)19.3220.860.332

Q2 – Clarify loss weights and update schedules

SettingDMD2ADDMeanFlow
Teacher/base backboneFLUXFLUXFLUX
Data / resolutionGPIC / 512²GPIC / 512²GPIC / 512²
Embedded guidance1.0/3.5/7.03.53.5
Global batch size128128128
Training seed000
Evaluated iteration4,5004,5004,500
Student learning rate10⁻⁵10⁻⁵10⁻⁵
Student objectiveL_VSD+0.03L_advL_adv+2.5L_SDSJVP-corrected MeanFlow loss
Auxiliary objectivesFake-score DSM + discriminatorDiscriminator + 0.1L_R1None
Auxiliary learning rateFake score: 10⁻⁵; discriminator: 10⁻⁵Discriminator: 4×10⁻⁶N/A
Training-time samplingUniform tShifted t, fixed four-step gridShifted logit-normal t,r
Student samplerFour-step SDEFour-step SDEFour-step ODE
Adversarial discriminatorYesYesNo

Our study controls the teacher, student backbone, data, resolution, guidance, batch size, and nominal training budget, but it does not isolate individual loss components or match every optimization detail.

Under the complete recipes evaluated here, both DMD2 and ADD exhibit coverage contraction, whereas teacher-supervised MeanFlow does not exhibit the same late-training contraction. We therefore identify a coverage limitation of these representative, practically used distribution-level distillation recipes, without attributing it specifically to VSD, SDS, or the adversarial objective.


Q3 – Model-dependent evaluator errors – Sure, we will add a blinded human inspection to Appendix D.

We thank the reviewer for this suggestion. We will supplement Appendix D with a blinded human inspection of representative teacher and student generations. Model identities will be hidden during inspection, and we will report agreement with human judgments separately for teachers and students to assess potential model-dependent biases.

Importantly, we want to highlight that the observed sampling-budget-dependent ranking reversals are not confined to a single model or benchmark. Our image evaluation covers five teacher–student families across four benchmarks, with reversals in 15 of 20 comparisons (Fig. 3), and analogous behavior appears in video generation (Fig. 7). This cross-model and cross-benchmark consistency supports the usefulness of pass@k and best-of-k as diagnostics of whether single-sample advantages persist as the sampling budget increases. The human inspection will complement this existing evidence by directly examining the near-miss evaluator errors.


Reviewer_4Zf9OpenReview ↗

After rebuttal

Thanks for the thorough rebuttal and supplementary experiments.

I maintain my weak acceptance.

Reviewer_hKMxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper argues that single-sample text-to-image benchmarks can hide coverage loss in few-step distilled diffusion models, and proposes pass@k (and best-of-k) over existing benchmark verifiers to measure how success grows with more samples. The authors validate this on classifier-free guidance, where stronger guidance usually wins on one draw but loses its lead as more samples are drawn. Across five teacher-student pairs and four benchmarks, every student leads on one draw, but the teacher dominates in most comparisons. A controlled FLUX.1-dev study links this to the training objective: DMD2 and ADD fall below the teacher in coverage and diversity, while MeanFlow improves coverage with stable diversity. Following Table 1 in Appendix D, The same split appears for few-step video models, and an appendix checks robustness to evaluator false positives on a subset of comparisons.

Critical strengths

(i) Applying pass@k and best-of-k to existing benchmark verifiers showcases the coverage loss that single-draw scores hide, demonstrated across five distillation families and four benchmarks.

(ii) By distilling the same FLUX.1-dev teacher into the same student, while varying only the loss (DMD2, ADD, MeanFlow), and additionally sweeping guidance, the paper isolates the effect of the distillation objective from model, data and guidance confounds.

Critical weaknesses

(i) The controlled study uses one teacher, one consistency-style method, one benchmark, but I could could not find seeds and std error-bars. Hence the statistical significance of the study is unclear. It would be nice to have more empirical evidence to support the author's findings.

(ii) The paper does not show what pass@k adds over existing diversity and coverage metrics. The paper itself acknowledges that distilled models are often observed to be less diverse, yet pass@k is never compared empirically against measures such as the Vendi score or precision-recall.

Questions for authors

I am concerned with the following inconsistencies and ambiguities regarding the presentation of your results.

(1) Sec. 5.1 says the teacher overtakes in 16 of 20 comparisons, but Figs. 3 shows 15. Which comparison is the 16th?

(2) In Table 1 for SDXL/LCM on GenAI-Bench I read a crossover at k=256, but Fig. 3 shows no clear crossover. Which is correct, and does LCM-SDXL stay ahead on all four benchmarks?

(3) Sec. 3 says all reported comparisons survive the false-positive checks, but Table 1 covers only 9. Do the remaining comparisons, including all of DPG-Bench, also survive? (Fig. 3 has 20 teacher-student comparisons, hence my confusion.)

(4) For the FLUX study, what are the training details (batch size, learning rate, total iterations), and did you run multiple seeds?

(5) To the best of my understanding, the FLUX teacher is sampled "without CFG", while the students use an embedded guidance of 3.5. What embedded guidance does the teacher use, and how do the teacher-student comparisons look when guidance is matched?

5 comments
AuthorsOpenReview ↗

Thanks! (1/3)

We thank the reviewer for your observant review and recognition. Here, we address your concerns and questions.


W1 – Statistical uncertainty – Sure, we provide bootstrap confidence intervals and seed-block analyses.

We thank the reviewer for raising the question of statistical uncertainty. We added uncertainty estimates to the broader teacher–student study using the retained per-image scores. For each of the 800 GenEval2 prompts, we compute the expected best-of-k score and use 10,000 paired prompt-bootstrap replicates to estimate standard errors and 95% confidence intervals (Table R2). The student advantage at k=1 changes into a statistically significant teacher advantage at the largest evaluated k for SD3.5/Turbo, Qwen/Lightning, and Sana/Sprint. In contrast, LCM remains ahead of SDXL at both small and large k.

Table R2: Teacher–student uncertainty on GenEval2.

Model pairkTeacherStudentStudent − Teacher ± SE95% CI
SD3.5-L / Turbo116.2319.75+3.52 ± 0.44[2.66, 4.40]
SD3.5-L / Turbo6465.0061.80−3.20 ± 0.80[−4.76, −1.62]
Qwen / Lightning133.1038.93+5.83 ± 0.60[4.68, 7.02]
Qwen / Lightning25692.1678.21−13.95 ± 0.90[−15.67, −12.20]
Sana / Sprint111.9915.31+3.31 ± 0.40[2.54, 4.10]
Sana / Sprint25667.1764.68−2.49 ± 0.80[−4.01, −0.93]
SDXL / LCM14.267.86+3.61 ± 0.34[2.97, 4.27]
SDXL / LCM25643.0151.95+8.94 ± 0.82[7.34, 10.56]

We also partitioned the generation samples into four disjoint seed blocks and observed the same large-budget direction in every block (Table R3).

Table R3: Generation-seed sensitivity.

Model pairSeed blocksBlock budgetStudent − Teacher, mean ± block SERange
SD3.5-L / Turbo4 × 1616−1.67 ± 0.30[−2.04, −0.78]
Qwen / Lightning4 × 6464−13.83 ± 0.45[−14.93, −12.75]
Sana / Sprint4 × 6464−0.98 ± 0.19[−1.26, −0.45]
SDXL / LCM4 × 6464+8.55 ± 0.17[8.26, 8.90]

All evaluations use distinct deterministic generation seeds, 42+10000p+j, for prompt p and sample j. These analyses quantify uncertainty across prompts and generation samples conditional on fixed checkpoints.

The controlled FLUX study complements this evidence by probing how the distillation objective shapes these trends, and each controlled model was trained once. The broader study goes beyond one teacher, one consistency-style method, and one benchmark: Fig. 3 covers five model families on four benchmarks. In the final revised manuscript, we will evaluate the controlled study across multiple seeds and report the corresponding error bars.

AuthorsOpenReview ↗

Thanks! (2/3)

W2 – Complementarity with existing metrics – Good question, we clarify how pass@k complements diversity metrics and provide Vendi scores.

We understand the reviewer’s concern. Our goal is not to replace diversity or distributional coverage metrics, but to answer a complementary question: how does additional sampling translate into successful prompt completion? Vendi measures output diversity and precision–recall measures fidelity and coverage relative to a reference set, whereas pass@k measures the probability of obtaining at least one verifier-accepted generation within a budget of k samples. Reduced diversity alone does not establish whether a student's single-sample advantage persists, disappears, or reverses as the sampling budget increases.

This distinction is demonstrated by our existing results in Fig. 3: distilled students can outperform their teachers at k=1 yet be overtaken at larger budgets. Our contribution is therefore not simply another observation that distilled models are less diverse, but a systematic characterization of sampling-budget-dependent model rankings that single-sample scores and diversity measurements alone do not directly provide. We additionally report within-prompt CLIP diversity in Figs. 4 and 6 to examine how these changes in benchmark success relate to output diversity. These analyses are complementary: pass@k measures budget-dependent task success, while representation-based metrics characterize the diversity of the generated outputs.

We additionally evaluate the Vendi score based on our generation. We use the same 50 GenEval prompts for every model. For each prompt, we take 48 images from each model and compute the Vendi score using CLIP ViT-L/14 features. We report the average across prompts.

Table R4: Vendi score of teachers and students (higher means more diverse).

Model familyTeacher VendiStudent Vendi
SD3.5-L / Turbo2.9051.762
Qwen / Lightning1.9281.333
Sana / Sprint2.7261.695
SDXL / LCM3.7922.533
Z-Image / Z-Turbo5.4561.576

All five students have lower Vendi scores than their teachers. However, as shown by the pass@k results in the paper, lower Vendi does not always lead to lower benchmark coverage: LCM has lower Vendi than SDXL (2.53 vs. 3.79) yet higher pass@256 on GenEval (Fig. 3). Pass@k measures whether drawing more samples helps solve more benchmark prompts, while Vendi measures how spread out the generated images are in CLIP feature space; that spread can reflect semantic diversity as well as differences in image quality or artifacts.

We present 5 selected models' results due to the rebuttal time limit; we will present the full table in the final revised version. We thank the reviewer for the advice.


Q1 – Comparison count – True; we corrected "16" to "15".

We thank the reviewer for catching this typo and apologize for the confusion. The correct count is 15 out of 20 comparisons, consistent with Fig. 3. We corrected "16" to "15" in Section 5.1 of the revised manuscript.


Q2 – GenAI-Bench in Fig. 3 vs. Table 1 – Both are correct: they use different statistics.

Thank you for pointing out this ambiguity. Both are correct: Fig. 3 and Table 1 use different statistics for GenAI-Bench. VQAScore is continuous, so Fig. 3 reports expected best-of-k on the original scores, which requires no success threshold. Table 1 is a false-positive check that counts how many samples succeed per prompt, which requires a pass/fail decision, so there we define success as VQAScore ≥ 0.8 and report binary pass@k for the simplicity of the problem. The two are directly related: best-of-k equals thresholded pass@k averaged over all thresholds, so Table 1 examines one threshold of the same score distribution.

LCM-SDXL remains ahead on all four benchmarks under the main evaluation protocols in Fig. 3, throughout the evaluated range k ≤ 256, and this is what the statement in Section 5.1 refers to. Under the thresholded GenAI-Bench metric in Table 1, SDXL overtakes at k=256 by 0.4 percentage points. Under the two-success variant in the same table, no crossover occurs and LCM leads by 0.9 percentage points at k=256. This is still a valid false-positive check: a single false positive cannot produce two successes, and the k=256 crossover disappears once two successes are required, which agrees with Fig. 3.


AuthorsOpenReview ↗

Thanks! (3/3)

Q3 – Number of the false-positive checks – True; we report partial experiments due to limited resources and report additional comparisons in Table R1.

Thank you for pointing out this ambiguity. The false-positive checks in Appendix D were performed on a subset of the comparisons, rather than all 20 teacher–student comparisons in Fig. 3. We apologize that the wording in Section 3 did not make this scope clear. We have been working to extend the false-positive robustness analysis, and Table R1 reports results for additional comparisons. The remaining comparisons require additional generation and scoring that could not be completed within the rebuttal period. We will provide the full table in the revised manuscript and clarify the corresponding text.

Table R1: False-positive robustness of additional comparisons, computed as in Table 1. k× is the first budget at which the first model overtakes the second, rounded up to a power of two.

BenchmarkComparisonStandard k×Standard ΔRequire ≥2 k×Require ≥2 Δ
GenEvalSDXL / LCM—−.060—−.090
GenEval2Qwen-Image / Lightning4+.1218+.144
GenEval2Sana / Sprint16+.05932+.054
GenEval2SDXL / LCM—−.085—−.084
GenEval2CFG 1 vs. 7 (SD3.5-L)32+.05564+.043

Q4 – Training details and random seeds – Sure, we list the configurations and describe our seed setup.

The configurations for our controlled FLUX.1-dev study are summarized below.

Method (embedded guidance)Global batch sizeStudent learning rateAuxiliary learning ratesIterations used
DMD2 (1.0 / 3.5 / 7.0)1281×10⁻⁵Fake-score network: 1×10⁻⁵; Discriminator: 1×10⁻⁵4,500
ADD (3.5)1281×10⁻⁵Discriminator: 4×10⁻⁶4,500
MeanFlow (3.5)1281×10⁻⁵None4,500

We use a single fixed training seed per run, following common practice in generative modeling. For evaluation, each checkpoint is assessed using at least 64 independently generated samples per prompt with different sampling seeds on GenEval2. Thus, our evaluation incorporates multiple sampling seeds, while training is not repeated across multiple seeds.


Q5 – Guidance settings – Fair point; we explain our guidance choice and add matched-guidance comparisons in Appendix C.2.

We thank the reviewer for pointing out this ambiguity. The FLUX.1-dev teacher uses embedded guidance 1.0 with no external classifier-free guidance branch. Our design principle is to compare each student with the teacher's original distribution. For distillation, we follow common practice in the literature [1, 2]: students use embedded guidance 3.5 unless otherwise specified. To further investigate the affection of guidance on distillation, DMD2 is additionally trained and evaluated at guidance 1.0 and 7.0.

To address the reviewer's concern on matched guidance comparison, we additionally compare every student with its teacher at matched guidance, as is shown in the added Appendix C.2 in the revised version of our paper.

Taken together, Appendices C.1 and C.2 make the main conclusion clear: distribution-matching distillation concentrates the student's output distribution, improving single-draw performance with an advantage that shrinks as k grows, whereas the trajectory distillation better preserves the teacher's coverage. On the interaction between guidance and distillation, Fig. 16(a)–(c) shows that lowering the embedded guidance used in distillation postpones the crossover against the guidance 1.0 teacher, but the student's advantage still shrinks as k grows. Fig. 17(a)–(c) shows the same shrinking advantage at every scale when the teacher and DMD2 use the same guidance.


[1] Yin, Tianwei, et al. "Improved distribution matching distillation for fast image synthesis." Advances in Neural Information Processing Systems 37 (2024): 47455-47487.

[2] Zhou, Mingyuan, Yi Gu, and Zhendong Wang. "Few-step diffusion via score identity distillation." arXiv preprint arXiv:2505.12674 (2025).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We are grateful to receive an all-positive review. Here, we provide a summary of our work and our rebuttals for the convenience of the AC and the general public.


TL;DR of Our Work

Our work investigates a practical question: do the single-sample improvements of few-step diffusion models persist when users generate multiple candidates? We use pass@k to evaluate how benchmark success changes with the sampling budget. Across five teacher–student families and four image benchmarks, teachers overtake initially stronger students in 15 of 20 comparisons. We also observe analogous behavior in causal video generation.

To investigate these differences, we compare DMD2, ADD, and MeanFlow using a shared FLUX backbone and controlled experimental settings. The evaluated DMD2 and ADD recipes exhibit coverage contraction, while MeanFlow better preserves coverage. Complementary CLIP-space analyses connect these trends to within-prompt diversity. Together, our results show that single-sample rankings can give a different picture of teacher–student performance from evaluation at larger sampling budgets, motivating evaluation across budgets when assessing diffusion distillation.


Reviewers’ Recognition

We appreciate the reviewers’ recognition of several aspects of this work:

  • The value of evaluating beyond single-sample performance. hKMx recognizes that our evaluation “showcases the coverage loss that single-draw scores hide.” 4Zf9 highlights our “Systematic evaluation across sampling budgets,” while jZZr describes the finding that single-sample improvements need not persist at larger budgets as “very interesting.”
  • The controlled comparisons and complementary analyses. hKMx appreciates the shared-backbone FLUX study and guidance sweep. 4Zf9 highlights “Controlled comparisons and complementary diversity measurements,” and jZZr recognizes the insights from our training analysis and the figures connecting coverage with representation diversity.
  • The usefulness of the empirical findings. In the discussion, jZZr reiterates that the results are “interesting and informative,” particularly the systematic study under different sampling budgets.

Reviewers’ Concerns and Our Responses

The reviewers’ constructive feedback helped us strengthen the paper. Below, we summarize the key points raised and the additional experiments and analyses we provided in response.

  1. hKMx and 4Zf9 ask what our evaluation adds beyond existing diversity metrics and prior work—we clarified the contribution and added Vendi measurements.

  2. hKMx requests statistical uncertainty estimates and repeated training runs—we added both.

  3. hKMx questions the guidance settings, while hKMx and 4Zf9 request clearer training details—we added matched-guidance comparisons and detailed configurations.

  4. 4Zf9 and jZZr ask which loss component causes coverage degradation—we clarified that our experiments compare complete training recipes and specified the conclusions supported by these comparisons.

  5. hKMx and 4Zf9 raise concerns about evaluator errors—we extended the robustness checks and clarified their current scope.

  6. hKMx identifies inconsistencies in the presentation—we corrected the comparison count and clarified the evaluation protocols.

  7. jZZr suggests directly optimizing pass@k—we added this as a future research direction.


All three reviewers engaged with our rebuttal and now recommend weak acceptance. hKMx stated that most concerns were addressed and raised their score; 4Zf9 thanked us for the thorough rebuttal and supplementary experiments and maintained their recommendation; jZZr maintained their recommendation and reiterated that the results are “interesting and informative.” Their question about what causes coverage degradation points to an interesting direction for future exploration.

We thank the reviewers for helping us strengthen the empirical evidence and sharpen the interpretation of our findings. We thank the AC for their time and thoughtful consideration of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.