- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes DMAD, a method for distilling diffusion models into few-step student model using synthetic teacher-generated samples and real data. The method trains two discriminator heads on a shared backbone: one distinguishes teacher samples from student samples, while the other distinguishes real samples from student samples. The authors show that, at the optimal discriminator, the logit gradients recover the score differences required to minimize the corresponding reverse KL divergences. This motivates training the student with a linear loss on the discriminator logits. The method further adjusts teacher supervision across noise levels using the real-data head’s measured gap between real and teacher samples. The experiments cover class-conditional image generation with EDM on ImageNet-64, text-to-image generation with SDXL, text-to-video generation with Wan2.1-1.3B and 14B, and joint audio-video generation with MiniMax-H3. DMAD achieves competitive performance against baselines including DMD2, rCM, SDXL-Turbo, and SDXL-Lightning in their respective settings.
Critical strengths
-
Interesting formulation with a clear theoretical connection. The paper derives how a linear generator loss on optimal discriminator logits recovers the distribution-matching gradient. This provides a principled basis for replacing auxiliary score fitting with discriminative learning.
-
Broad experimental coverage. The method is evaluated across class-conditional images, text-to-image generation, text-to-video generation, and joint audio-video generation, including large video backbones. This provides evidence that the formulation can be applied beyond a single model or modality.
-
Useful component ablations. The experiments examine the teacher head, real-data head, adaptive weighting, and linear generator loss. The results support the usefulness of these components within the proposed framework, with training curves further illustrating their effects on convergence.
-
Promising quality and online efficiency results. DMAD achieves competitive image and video quality while removing online teacher evaluation and auxiliary score fitting. The human evaluation on MiniMax-H3 also provides evidence of perceptual benefits beyond automated metrics.
Critical weaknesses
Although I find the loss formulation interesting and the theoretical connection to distribution matching reasonable, several experimental questions remain:
-
Incomplete accounting of training cost. The reported speedup measures online training steps and excludes the cost of generating the cached teacher samples. Since DMAD moves teacher computation offline, an end-to-end comparison should include teacher-sample generation and report the total GPU-hours required to reach comparable generation quality. Furthermore, the comparison against DMD2 uses five critic updates per generator update.
-
**Insufficient comparison with existing adversarial distillation objectives. ** While the reverse-KL interpretation is theoretically interesting, the experiments do not isolate the practical benefit of the proposed objective over established adversarial distillation. ADD already uses a linear generator adversarial loss; consequently, the non-saturating-loss ablation in Table 4 establishes the benefit of this choice within DMAD but does not demonstrate an advantage over ADD. The SDXL-Turbo comparison is useful, yet does not control for differences in discriminator architecture, training data, and optimization. A matched ADD-style baseline would help assess the benefit of replacing online teacher distillation with teacher-sample discrimination. Separately, a BCE-versus-hinge ablation with fixed teacher/real weights would clarify whether the discriminator objective underlying the reverse-KL interpretation provides an empirical advantage.
-
Missing comparison with naive data mixing. The ablations establish the benefit of combining teacher and real-data supervision, but do not establish the advantage of using separate discriminator heads and adaptive loss weights over simply mixing real and teacher-generated samples as positives for one discriminator. A comparison using the same discriminator backbone, linear generator loss, data exposure, and training budget, with a tuned mixing ratio, would help isolate the contribution of the proposed design.
-
Limited video baseline coverage. The Wan2.1-14B evaluation compares generation quality only against rCM, omitting even DMD2. AnyFlow [1] and PDD [2] evaluate the same Wan2.1 backbones under comparable few-step settings. These methods are cited but not experimentally compared. Given the small VBench gains reported for DMAD, matched comparisons would help establish its advantage in quality, motion, diversity, and training efficiency.
[1] Gu et al. “AnyFlow: Any-Step Video Diffusion Model with On-Policy Flow Map Distillation.” 2026. https://arxiv.org/abs/2605.13724
[2] Shaul et al. “Parallel Decoding Distillation for Fast Image and Video Generation.” 2026. https://arxiv.org/abs/2607.26004
Questions for authors
As discussed in Weaknesses
3 comments
W1. End-to-end training cost and the critic update ratio & W2/W3. ADD-style distillation, discriminator loss, and single-head data mixing
W1. End-to-end training cost and the critic update ratio
We agree that the online timings in Table 5 capture only part of the training cost. We now include the cost of generating the teacher samples and report the total GPU-hours required to reach comparable FID on ImageNet-64 and four-step SDXL.
The one-time generation cost is 229 H200 GPU-hours for one million EDM samples on ImageNet-64 and 289 H200 GPU-hours for 480,000 SDXL latents. We charge the full cache cost to a single DMAD run in the table below. The same cache can be reused across different training experiments.
| Setting | Target FID | DMAD online | DMAD total | DMAD + PG online | DMAD + PG total | DMD2 total |
|---|---|---|---|---|---|---|
| ImageNet-64 | 1.51 | 220 | 449 | 116 | 345 | 136 |
| ImageNet-64 | 1.28 | 434 | 663 | 204 | 433 | 439 |
| ImageNet-64 | 1.24 | 454 | 683 | 231 | 460 | N/A |
| ImageNet-64 | 1.04 | N/A | N/A | 303 | 532 | N/A |
| SDXL | 20.00 | 97 | 386 | N/A | N/A | 126 |
| SDXL | 19.3 | 103 | 392 | N/A | N/A | 324 |
| SDXL | 14.47 | 123 | 412 | N/A | N/A | N/A |
All costs are in H200 GPU-hours. PG denotes the projected discriminator. On ImageNet-64, when all methods reach FID 1.51, the standard DMD2 configuration has the lowest total cost. At FID 1.28, DMAD with PG and longer-training DMD2 have similar total costs of 433 and 439 GPU-hours, respectively. On SDXL, DMAD reduces the online cost of reaching approximately the same FID as DMD2 from 324 to 103 GPU-hours. Including the full cache raises the single-run total to 392 GPU-hours. DMAD then reaches FID 14.47 at a total cost of 412 GPU-hours. We report the online and total costs together so the effect of cache reuse is clear. Table 5 uses DMD2's standard ratio of five critic updates per generator update and DMAD's ratio of one. We also benchmarked DMD2 at 1:1 on ImageNet-64 with the same hardware and batch size. Its time per generator update falls from 2.321 s at 5:1 to 0.717 s at 1:1, compared with 0.496 s for DMAD at 1:1. Thus, matching the update ratio narrows the timing gap, and DMAD still takes about 31% less time per generator update.
W2/W3. ADD-style distillation, discriminator loss, and single-head data mixing
We ran three additional comparisons on ImageNet-64 to examine how DMAD differs from ADD-style distillation, how the discriminator loss affects performance, and whether pooling real and teacher samples into one positive class is sufficient.
| Method | FID ↓ |
|---|---|
| ADD-style baseline | 1.32 |
| DMAD (fixed weights, hinge) | 1.42 |
| DMAD (fixed weights, BCE) | 1.33 |
| Single-head data mixing | 1.53 |
| DMAD | 1.24 |
W2. ADD-style distillation and BCE versus hinge. The ADD-style baseline uses the same discriminator backbone, real-data head, optimizer, and real-data source as DMAD. It combines a linear adversarial generator loss with online regression to the teacher's predictions. This baseline reaches 1.32 FID. Full DMAD reaches 1.24 FID using cached teacher samples, with no teacher evaluations during student training. In a separate comparison, we keep both heads and their weights fixed and change only the discriminator loss. BCE improves FID from 1.42 to 1.33, supporting its use in our two-head formulation.
W3. Single-head data mixing. We match DMAD's discriminator backbone, data sources, linear generator loss, and training budget. We train a single head with mixed real and teacher-generated samples as positives and student samples as negatives. We test real-sample fractions of 25%, 50%, and 75% in the positive class. The best result comes from the 75% ratio, which achieves an FID of 1.53, compared with 1.24 for full DMAD. The single head gives the generator one training signal for the pooled target. DMAD's separate heads provide a signal for each target, so the teacher weight can vary across noise bands while the real-data weight stays fixed. The FID comparison supports this combination of separate heads and adaptive weighting over the best fixed mixture we tested.
W4. Wan2.1-14B comparison with DMD2, rCM, AnyFlow, and PDD
Thank you for suggesting these baselines. We extend the Wan2.1-14B comparison to include DMD2, AnyFlow, and the concurrent work PDD in the four-step setting. The table reports VBench quality and semantics, two VBench motion metrics, diversity, and online training cost. Following PDD, we measure diversity using mean pairwise cosine and L2 distances in V-JEPA 2 and VideoMAE V2 feature spaces. Training costs are expressed in H200 GPU-hours for the reported checkpoints.
| Method | VBench Total ↑ | Quality ↑ | Semantic ↑ | Dynamic degree ↑ | Motion smoothness ↑ | V-JEPA 2 cos. ↑ | V-JEPA 2 L2 ↑ | VideoMAE V2 cos. ↑ | VideoMAE V2 L2 ↑ | Training GPU-h ↓ |
|---|---|---|---|---|---|---|---|---|---|---|
| DMD2 | 84.40 | 85.16 | 81.34 | 0.7633 | 0.9741 | 0.0568 | 17.67 | 0.00945 | 1.710 | 3,328 |
| rCM | 84.92 | 85.43 | 82.88 | 0.7139 | 0.9803 | 0.0779 | 21.10 | 0.01253 | 1.983 | 3,742 |
| AnyFlow | 84.95 | 85.70 | 81.92 | 0.7042 | 0.9827 | 0.0786 | 20.99 | 0.01297 | 1.992 | 3,374 |
| PDD-short | 84.92 | 85.71 | 81.77 | 0.6934 | 0.9814 | 0.0791 | 21.27 | 0.01247 | 2.027 | 760 |
| PDD-long | 84.69 | 85.69 | 80.71 | 0.7069 | 0.9783 | 0.0846 | 22.13 | 0.01264 | 2.058 | 10,700 |
| DMAD | 85.15 | 85.96 | 81.90 | 0.7444 | 0.9856 | 0.0790 | 21.38 | 0.01463 | 2.683 | 2,890 |
DMAD has the highest reported VBench Total and Quality scores, with a 0.20-point Total margin over AnyFlow. It also has the highest motion smoothness and the second-highest dynamic degree, behind DMD2. For diversity, DMAD leads on both VideoMAE V2 measures, with a mean pairwise cosine distance of 0.01463 and an L2 distance of 2.683. The next-highest scores are 0.01297 for cosine distance (AnyFlow) and 2.058 for L2 distance (PDD-long). In V-JEPA 2 feature space, DMAD is close to PDD-short, and PDD-long leads on both measures. PDD-short reaches a VBench Total of 84.92 at the lowest online training cost of 760 GPU-hours. DMAD uses fewer online training GPU-hours than DMD2, rCM, AnyFlow, and PDD-long.
We thank the reviewer again for the thoughtful review and concrete suggestions, which helped us give a fuller account of training costs and strengthen the experimental comparisons. We hope the additional cost analysis, objective ablations, and video comparisons address the reviewer's concerns. We would be grateful if the reviewer would consider raising the score to 4.
Response to Reviewer ysVn
Thank you for your thoughtful review and specific suggestions. We appreciate your recognition of the connection to distribution matching and the breadth of our evaluation.
Your comments helped us account for offline teacher-sample generation, distinguish the effects of the discriminator objective and the two-target design, and broaden the Wan2.1-14B comparison. We address these points in the replies below.