acceptodds
Under review as a conference paper at ICLR 2027

Revisiting the Score Difference in Distribution Matching Distillation

Abstract

Distribution Matching Distillation (DMD) trains a few-step student generator from the difference between a frozen teacher's score and an online critic's estimate of the student's score. Its quality hinges on how well the critic estimates that score, yet the critic's design is largely left to defaults. We revisit two choices—what the critic is told, and how the critic and teacher are sized—each improving few-step quality while leaving the student, and thus inference, unchanged. First, the critic sees the score-evaluation timestep but not the source noise level of each student prediction, so it cannot distinguish an expected early-step blur from a genuine error and fits a score averaged across steps. Conditioning it on the source level makes it step-aware in a few lines of code and adds no training or inference cost. Second, the student, teacher, and critic need not share one size, but this asymmetry has a limit—the critic must scale with the teacher. An undersized critic loses quality and destabilizes training; with the critic small, even a larger teacher hurts rather than helps. The critic should therefore match the teacher's size, while only the student stays small, keeping inference cheap. We validate both changes on class-conditional ImageNet, SDXL text-to-image, and WAN text-to-video in both bidirectional and autoregressive regimes, spanning -prediction and flow-matching models. In every setting, they consistently improve few-step distillation quality.

Primary area:
generative models
Keywords:
diffusion model, distillation, Distribution Matching Distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ZA2KOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates how critic capacity and conditioning affect distribution matching distillation, highlighting the importance of scaling the critic alongside the teacher. The authors also introduce a step-aware critic (STAC), which conditions on the student’s generation step to estimate the step-wised distribution. Experiments with SiT on ImageNet, SDXL, FLUX.2, and Wan2.1 demonstrate improvements in generation quality and training stability.

Critical strengths
  1. The controlled scaling experiments show that increasing teacher capacity alone can degrade student quality or cause training collapse, providing practical evidence for scaling the critic alongside the teacher.

  2. STAC offers a simple conditioning change whose benefits are supported by targeted ablations and experiments across image and video generation, without increasing student inference cost.

Critical weaknesses
  1. Although Figure 4 reports computational cost per student update, it remains unclear whether the advantage of larger critics persists under the same total training budget. Figure 13 also omits an XL critic with five updates per student update, leaving the tradeoff between critic capacity, update frequency, and training duration insufficiently characterized.
Questions for authors
  1. Under the same total computational budget, how does an XL critic using five updates per student update compare with an XL critic using one update and a smaller critic using more updates or a longer training run?

  2. How does STAC compare with TDM in generation quality and training stability when the teacher, student, critic capacity, and total training budget are held fixed?

1 comment

We thank Reviewer ZA2K for the positive evaluation and for raising important and insightful questions.

W1 & Q1: Iso-compute comparison of critic size, update frequency, and training duration. Evaluation under similar compute is indeed a very important comparison. Following your suggestion, we added an explicit iso-compute view in Figure 13d,e,f, re-plotting the runs from Figure 13a,b,c against total training FLOPs, counting all forward and backward passes of the student, teacher, and critic with the statistics in Table 6.

  • Small critics with more updates or longer training do not outperform large critic. Under the same total compute budget, a 1-update XL critic still outperforms a student-sized critic using 5 updates per step or trained for a longer duration. While 5 critic updates accelerate early convergence and help prevent collapse (Figure 13a), DMD runs converge within tens of thousands of student updates and then plateau (or slightly drift upward, as also seen in Figure 5a). Because a small critic hits an expressivity ceiling rather than an optimization lag, spending more total FLOPs on extra critic updates or longer runs does not close the FID gap to the 1-update XL critic.
  • 1-update XL critic vs. 5-update XL critic under fixed total compute: Because the XL critic dominates per-step FLOPs (Table 6), increasing an XL critic from 1 to 5 updates raises the total compute per student update by ∼3.3×\sim 3.3\times. Under a fixed total compute budget, a 5-update XL critic therefore receives ∼3.3×\sim 3.3\times fewer student updates, whereas, on ImageNet experiments, a single update already suffices for the matched XL critic to track the student stably and reach strong FID. Pairing a teacher-sized critic with a 1:11:1 update ratio thus offers more favorable compute-quality trade-off.

Q2: Controlled comparison of STAC vs. TDM at fixed capacity and training budget
As summarized in the general response (and also Table 20, Appendix F.4 in the paper), we added a controlled comparison on FLUX.2. The measured training times per step on 8×H1008\times\text{H100} GPUs are 1.623 s for the DMD2 baseline, 1.634 s for the TDM-style variant, and 1.630 s for STAC (Table 12).

Method (T=9B,F=9BT=9\text{B}, F=9\text{B}, 4B Student)FID ↓\downarrowCLIP-S ↑\uparrowImageReward ↑\uparrowHPSv3 ↑\uparrowGenEval ↑\uparrow
DMD2 Baseline15.2028.2379.211.6471.4
+ TDM-style interval conditioning12.2428.3477.010.9474.0
+ STAC (Ours)13.9328.5686.811.9174.5

Regarding training stability, all three configurations train smoothly without collapse under the matched 9B/9B capacity. Regarding generation quality, TDM's disjoint score-query intervals yield a lower FID (12.24 vs. 13.93), whereas STAC's full-range step-kk conditioning achieves higher text alignment and human preference across CLIP-S, ImageReward, HPSv3, and GenEval.

Reviewer_UqVzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how critic capacity and conditioning affect distribution matching distillation (DMD), and shows that 1. scaling the critic with the teacher generally improves performance, and 2. conditions the critic on student's generation steps alleviates degradation too. Experiments cover ImageNet with seven SiT model sizes, as well as SDXL, FLUX.2, and Wan2.1.

Critical strengths
  1. The paper's study on critic scaling is systematic and provides useful insights to the community. The scaling effect on the critic & teacher side is cleanly demonstrated and effective, and the paper also includes substantial controls involving learning rates, critic update frequency, critic warm-up, initialization, and LoRA adaptation.
  2. The step-aware critic also offers novel insights, that under different generation steps, the student's output distribution can be modeled more precisely.
  3. The proposed method generalizes well across different architectures and tasks including ImageNet and T2I / T2V.
Critical weaknesses
  1. The step-aware critic seems related to the critic conditioning proposed in Luo et al., 2025. The authors only include the discussions in the related work, but including it as a baseline would strengthen the effectiveness of STAC.
  2. The ablation axes can be more complete, i.e. asymmetric fake_score & teacher model (L / XL on ImageNet, 4B / 9B on FLUX.2), and more evidence on the effectiveness of STAC, that if it could reduce the need for a larger critic.
Questions for authors
  1. I mainly want to raise the question on AR diffusion generation in video, since this is the field DMD is mostly used in now. Do the authors envision STAC is transferrable to AR video generation too? Can we condition the critic not only on generation steps, but also the physical time in the AR rollout too?
1 comment

We thank Reviewer UqVz for the positive assessment and insightful suggestions.

W1: Empirical comparison with TDM (Luo et al., 2025). As explained in the general response, and also Table 20, Appendix F.4 in the paper, we added a controlled comparison on FLUX.2 (T=9B,F=9BT=9\text{B}, F=9\text{B}, 4B student) isolating TDM's non-overlapping score-query intervals against STAC's step-kk conditioning over the full score-query range. Both methods improve over the step-agnostic DMD2 baseline, with a TDM-style interval conditioning achieves a lower FID, whereas STAC achieves stronger results for all other metrics.

W2: Complete teacher/critic grid and STAC with smaller critics. Thanks for your suggestion! On ImageNet, the full 7×77\times 7 teacher–critic grids in Figure 5b,c and Figure 10 already include the asymmetric L/XL and XL/L pairs alongside B/B + STAC in Table 2. To further complete these axes on FLUX.2 (Table 3) and Wan2.1 (Table 5), we added the asymmetric 4B/9B pair on FLUX.2 as well as STAC on the small-critic configurations (4B/4B on FLUX.2 and 1.3B/1.3B on Wan2.1):

FLUX.2 (4B Student)STACFID ↓\downarrowCLIP-S ↑\uparrowImageReward ↑\uparrowHPSv3 ↑\uparrowGenEval ↑\uparrow
T=4B,F=4BT=4\text{B}, F=4\text{B}w/o17.7128.2478.612.6570.9
T=4B,F=4BT=4\text{B}, F=4\text{B}w/16.7528.1882.212.6871.1
T=9B,F=4BT=9\text{B}, F=4\text{B}w/o20.0327.8555.18.1170.5
T=4B,F=9BT=4\text{B}, F=9\text{B}w/o19.2928.0286.213.0072.1
T=9B,F=9BT=9\text{B}, F=9\text{B}w/o15.2028.2379.211.6471.4
T=9B,F=9BT=9\text{B}, F=9\text{B}w/13.9328.5686.811.9174.5
Wan2.1 (1.3B Student)VBench Total ↑\uparrowQuality ↑\uparrowSemantic ↑\uparrowDynamic ↑\uparrow
T=1.3B,F=1.3BT=1.3\text{B}, F=1.3\text{B}83.9685.1479.2681.39
T=1.3B,F=1.3BT=1.3\text{B}, F=1.3\text{B} + STAC84.0985.3878.9183.33
T=14B,F=14BT=14\text{B}, F=14\text{B}84.2685.0481.1657.52
T=14B,F=14BT=14\text{B}, F=14\text{B} + STAC84.9085.9380.7973.48

Across ImageNet, FLUX.2, and Wan2.1, STAC consistently improves generation quality at a fixed small critic size, while combining joint scaling with STAC yields the strongest overall performance.

Q1: Transferability of STAC to AR video generation and conditioning on physical rollout time. Yes. In fact, our Wan2.1 experiments in Section 4.2 (Figure 8, bottom and Table 5) already evaluate STAC directly on autoregressive video generation using the online Self Forcing and Causal Forcing AR rollout recipes, where STAC consistently improves VBench Total and Quality and mitigates degradation over longer training. We appreciate the suggestion to condition the critic on physical rollout time and think it would be worth trying in future experiments.

Reviewer_w7byOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how critic capacity and conditioning affect distribution matching distillation. With student capacity fixed, the authors systematically vary teacher and critic sizes, finding that scaling the teacher alone can degrade generation quality or destabilize training, while jointly scaling the critic can mitigate these effects. They also introduce a step-aware critic, STAC, which receives the current generation step to distinguish the distributions of predictions produced at different steps. The approach is evaluated on ImageNet with SiT, text-to-image generation with FLUX.2 and SDXL, and video generation with Wan2.1.

Critical strengths
  1. A systematic study of teacher–critic capacity interactions at fixed student capacity. The seven-size SiT study goes beyond evaluating a few isolated configurations. The additional analyses of learning rates, critic update frequency, initialization, and pretraining help distinguish capacity effects from straightforward optimization explanations. The finding that teacher-only scaling can be counterproductive is informative for practitioners.

  2. A simple critic modification with controlled evidence of benefits independent of scaling. STAC changes only the training-time critic and adds no inference overhead to the student. The fixed-size comparisons and token controls in Table 2, together with the equal-length continuation experiment in Table 4, support the value of step information beyond merely adding parameters or training longer.

Critical weaknesses
  1. The relationship to the closest prior work needs a more precise treatment. The paper already cites TDM and Phased DMD in Sec. 2, and Sec. 3.2 contrasts STAC with TDM’s restricted score-evaluation intervals. However, TDM also conditions its fake-score network on the total sampling budget KK. This should be explicitly distinguished from STAC’s conditioning on the current generation step kk. The comparison with Phased DMD (CVPR'26) could likewise clarify the differences in parameter sharing, phase-specific experts, and supervision targets.

  2. The additional STAC gain in the final video experiment is not fully controlled. In Table 5, adding STAC to the 14B/14B teacher–critic configuration increases VBench from 84.26 to 84.32. However, Table 12 specifies that EMA starts at iteration 200 for the size controls and at iteration 0 for the final STAC run. This prevents attributing the small difference solely to STAC. This concern is specific to that comparison and does not invalidate the separate evidence in Figure 8 or the image-generation experiments.

Questions for authors
  1. The experiments initialize students from existing checkpoints of the corresponding smaller size. Is access to compatible large and small pretrained models an assumption of the joint-scaling recommendation? What applicability claims are supported when only a large-model checkpoint is available?

  2. Table 3 already evaluates both scaling and STAC on FLUX.2, but STAC is tested only with the 9B/9B teacher–critic pair. Could the authors add 4B teacher / 4B critic + STAC under the same protocol, and report training time and peak memory for this and the existing configurations? This would quantify the benefit available from a single 4B model size and clarify the additional quality–cost tradeoff of joint scaling once STAC is enabled.

  3. Does the improvement from 84.26 to 84.32 in Table 5 persist under the same EMA schedule and checkpoint-selection rule?

1 comment

We thank Reviewer w7by for the thorough reading and constructive feedback! Below we address each point in detail.

W1: Distinction from TDM and Phased DMD. We have expanded Section 2 and 3.2 to make these architectural and objective differences explicit:

  • TDM (Luo et al., 2025): The unified TDM variant conditions its fake-score network on the total sampling budget KK (which is constant across all steps of a KK-step student) and restricts each generation step to a non-overlapping score-evaluation interval with a corresponding trajectory target. By contrast, STAC conditions a single shared critic on the current generation step index k∈{1,…,K}k \in \{1, \dots, K\} (via rkr_k) while retaining the standard DMD objective and the full score-evaluation range tt for every step. As shown in the controlled FLUX.2 comparison in the General Response (Table 20, Appendix F.4), TDM's interval restriction achieves a lower FID (12.2412.24 vs. 13.9313.93), whereas STAC's full-range step-specific supervision performs better on CLIP-S (28.5628.56 vs. 28.3428.34), ImageReward (86.886.8 vs. 77.077.0), HPSv3 (11.9111.91 vs. 10.9410.94), and GenEval (74.574.5 vs. 74.074.0).
  • Phased DMD (Fan et al., 2026): Phased DMD partitions the SNR range into sub-intervals and trains separate phase-specific generator and critic experts under truncated phase targets. STAC shares a single student and a single critic across all steps, adding only 4 (or 32) lightweight in-context step tokens (<0.2%<0.2\% parameter overhead, Table 21) to disambiguate the step distributions.

W2 & Q3: Matched EMA schedule and checkpoint selection on Wan2.1. Thank you for catching this discrepancy. We re-trained the 14B/14B + STAC run using the same EMA schedule and training protocol, and also added a matched 1.3B/1.3B + STAC control (updated in Tables 5 and 13):

Wan2.1 Configuration (1.3B Student)VBench Total ↑\uparrowQuality ↑\uparrowSemantic ↑\uparrowDynamic ↑\uparrow
T=1.3B,F=1.3BT=1.3\text{B}, F=1.3\text{B}83.9685.1479.2681.39
T=1.3B,F=1.3BT=1.3\text{B}, F=1.3\text{B} + STAC84.0985.3878.9183.33
T=14B,F=14BT=14\text{B}, F=14\text{B}84.2685.0481.1657.52
T=14B,F=14BT=14\text{B}, F=14\text{B} + STAC84.9085.9380.7973.48

Under this matched protocol, adding STAC to the 14B/14B pair improves VBench Total by +0.64+0.64 (84.26→84.9084.26 \rightarrow \mathbf{84.90}), Quality by +0.89+0.89 (85.04→85.9385.04 \rightarrow \mathbf{85.93}), and Dynamic by +15.96+15.96 (57.52→73.4857.52 \rightarrow 73.48), while Semantic decreases from 81.1681.16 to 80.7980.79.

Q1: Pretrained checkpoint assumptions for joint scaling. Thank you for the insightful question. On the critic side, joint scaling (F=TF=T) does not require access to an extra model size. On the student side, our experiments do assume a smaller student, but it can be initialized from ODE trajectory distillation from the teacher (as in our Wan2.1 causal student initialized from Causal Forcing Stage-2A ODE trajectories, Table 12). Extending our study to random initialized student is an interesting direction that we leave to future work.

Q2: FLUX.2 complete comparison grid and training cost profiles. We added 4B/4B + STAC and the 4B/9B pair under the same FLUX.2 protocol and profiled training time and peak memory on 8×H1008\times\text{H100} GPUs.

TTFFVariantFID ↓\downarrowCLIP-S ↑\uparrowImageReward ↑\uparrowHPSv3 ↑\uparrowGenEval ↑\uparrowTime / step (s)Peak Mem (GB)
4B4Bw/o17.7128.2478.612.6570.91.08937.12
4B4Bw/ STAC16.7528.1882.212.6871.11.11237.09
9B4Bw/o20.0327.8555.18.1170.51.25049.74
4B9Bw/o19.2928.0286.213.0072.11.59764.67
9B9Bw/o15.2028.2379.211.6471.41.62367.36
9B9Bw/ STAC13.9328.5686.811.9174.51.63067.80

Comments on the paper

1 comment

We thank all three reviewers for their reviewing effort and constructive feedback! We are glad the reviewers acknowledge our systematic teacher-critic scaling study informative and our step-aware critic (STAC) effective across image and video generation without adding student inference cost.

As all three reviewers asked how STAC relates to TDM (Luo et al., 2025), we added a controlled comparison on FLUX.2 (Table 20, Appendix F.4). We keep the 4B student, 9B teacher, 9B critic, 10k updates, and DMD settings fixed, comparing TDM's non-overlapping score-query intervals and trajectory targets against STAC's explicit step-kk conditioning over the full score-query range:

FID ↓\downarrowCLIP-S $ \uparrow $ImageReward ↑\uparrowHPSv3 ↑\uparrowGenEval ↑\uparrow
DMD2 baseline15.2028.2379.211.6471.4
+ TDM-style interval conditioning12.2428.3477.010.9474.0
+ STAC (Ours)13.9328.5686.811.9174.5

While TDM-style interval restriction lowers FID, retaining full-range score supervision with STAC yields higher text alignment, human preference (ImageReward, HPSv3), and compositional accuracy (GenEval). We address all reviewer-specific questions and additional ablations in the individual comments below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.