acceptodds
Under review as a conference paper at ICLR 2027

DMAD: Distribution Matching as Adversarial Distillation for Fast Visual Generation

13 views

Abstract

Distribution Matching Distillation (DMD) trains a few-step student from the difference between separately estimated target and student scores, so it must keep an auxiliary diffusion model fitted to the student's evolving distribution at extra memory and computation cost. We introduce DMAD, Distribution Matching as Adversarial Distillation, which recasts distribution matching as classification and learns the required log-density ratios directly. Two discriminator heads on a shared backbone distinguish real data and teacher samples from the student's, and linear losses on their logits train the student without auxiliary score fitting. We prove that at the discriminator optimum these losses recover the distribution-matching gradient underlying DMD, through the classical identity linking discriminator logits to log-density ratios. We further introduce gap-based reweighting, which adapts teacher supervision across noise levels from the real-data head's empirical logit gap between real and teacher samples. DMAD reaches a Fréchet Inception Distance (FID) of 1.04 with one-step generation on ImageNet-64×64, 14.47 with four-step SDXL on COCO-10K, and a VBench total score of 85.15 with four-step Wan2.1-T2V-14B, the best values among the compared few-step methods and the multi-step teachers. On MiniMax-H3-33B, our four-step student achieves overall human preference rates of 79.1% over DMD2 and 84.6% over rCM for joint audio-video generation, excluding ties.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Generation, Video Distillation, Distribution Matching Distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ysVnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DMAD, a method for distilling diffusion models into few-step student model using synthetic teacher-generated samples and real data. The method trains two discriminator heads on a shared backbone: one distinguishes teacher samples from student samples, while the other distinguishes real samples from student samples. The authors show that, at the optimal discriminator, the logit gradients recover the score differences required to minimize the corresponding reverse KL divergences. This motivates training the student with a linear loss on the discriminator logits. The method further adjusts teacher supervision across noise levels using the real-data head’s measured gap between real and teacher samples. The experiments cover class-conditional image generation with EDM on ImageNet-64, text-to-image generation with SDXL, text-to-video generation with Wan2.1-1.3B and 14B, and joint audio-video generation with MiniMax-H3. DMAD achieves competitive performance against baselines including DMD2, rCM, SDXL-Turbo, and SDXL-Lightning in their respective settings.

Critical strengths
  • Interesting formulation with a clear theoretical connection. The paper derives how a linear generator loss on optimal discriminator logits recovers the distribution-matching gradient. This provides a principled basis for replacing auxiliary score fitting with discriminative learning.

  • Broad experimental coverage. The method is evaluated across class-conditional images, text-to-image generation, text-to-video generation, and joint audio-video generation, including large video backbones. This provides evidence that the formulation can be applied beyond a single model or modality.

  • Useful component ablations. The experiments examine the teacher head, real-data head, adaptive weighting, and linear generator loss. The results support the usefulness of these components within the proposed framework, with training curves further illustrating their effects on convergence.

  • Promising quality and online efficiency results. DMAD achieves competitive image and video quality while removing online teacher evaluation and auxiliary score fitting. The human evaluation on MiniMax-H3 also provides evidence of perceptual benefits beyond automated metrics.

Critical weaknesses

Although I find the loss formulation interesting and the theoretical connection to distribution matching reasonable, several experimental questions remain:

  • Incomplete accounting of training cost. The reported speedup measures online training steps and excludes the cost of generating the cached teacher samples. Since DMAD moves teacher computation offline, an end-to-end comparison should include teacher-sample generation and report the total GPU-hours required to reach comparable generation quality. Furthermore, the comparison against DMD2 uses five critic updates per generator update.

  • **Insufficient comparison with existing adversarial distillation objectives. ** While the reverse-KL interpretation is theoretically interesting, the experiments do not isolate the practical benefit of the proposed objective over established adversarial distillation. ADD already uses a linear generator adversarial loss; consequently, the non-saturating-loss ablation in Table 4 establishes the benefit of this choice within DMAD but does not demonstrate an advantage over ADD. The SDXL-Turbo comparison is useful, yet does not control for differences in discriminator architecture, training data, and optimization. A matched ADD-style baseline would help assess the benefit of replacing online teacher distillation with teacher-sample discrimination. Separately, a BCE-versus-hinge ablation with fixed teacher/real weights would clarify whether the discriminator objective underlying the reverse-KL interpretation provides an empirical advantage.

  • Missing comparison with naive data mixing. The ablations establish the benefit of combining teacher and real-data supervision, but do not establish the advantage of using separate discriminator heads and adaptive loss weights over simply mixing real and teacher-generated samples as positives for one discriminator. A comparison using the same discriminator backbone, linear generator loss, data exposure, and training budget, with a tuned mixing ratio, would help isolate the contribution of the proposed design.

  • Limited video baseline coverage. The Wan2.1-14B evaluation compares generation quality only against rCM, omitting even DMD2. AnyFlow [1] and PDD [2] evaluate the same Wan2.1 backbones under comparable few-step settings. These methods are cited but not experimentally compared. Given the small VBench gains reported for DMAD, matched comparisons would help establish its advantage in quality, motion, diversity, and training efficiency.

[1] Gu et al. “AnyFlow: Any-Step Video Diffusion Model with On-Policy Flow Map Distillation.” 2026. https://arxiv.org/abs/2605.13724

[2] Shaul et al. “Parallel Decoding Distillation for Fast Image and Video Generation.” 2026. https://arxiv.org/abs/2607.26004

Questions for authors

As discussed in Weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer ysVn

Thank you for your thoughtful review and specific suggestions. We appreciate your recognition of the connection to distribution matching and the breadth of our evaluation.

Your comments helped us account for offline teacher-sample generation, distinguish the effects of the discriminator objective and the two-target design, and broaden the Wan2.1-14B comparison. We address these points in the replies below.

AuthorsOpenReview ↗

W1. End-to-end training cost and the critic update ratio & W2/W3. ADD-style distillation, discriminator loss, and single-head data mixing

W1. End-to-end training cost and the critic update ratio

We agree that the online timings in Table 5 capture only part of the training cost. We now include the cost of generating the teacher samples and report the total GPU-hours required to reach comparable FID on ImageNet-64 and four-step SDXL.

The one-time generation cost is 229 H200 GPU-hours for one million EDM samples on ImageNet-64 and 289 H200 GPU-hours for 480,000 SDXL latents. We charge the full cache cost to a single DMAD run in the table below. The same cache can be reused across different training experiments.

SettingTarget FIDDMAD onlineDMAD totalDMAD + PG onlineDMAD + PG totalDMD2 total
ImageNet-641.51220449116345136
ImageNet-641.28434663204433439
ImageNet-641.24454683231460N/A
ImageNet-641.04N/AN/A303532N/A
SDXL20.0097386N/AN/A126
SDXL19.3103392N/AN/A324
SDXL14.47123412N/AN/AN/A

All costs are in H200 GPU-hours. PG denotes the projected discriminator. On ImageNet-64, when all methods reach FID 1.51, the standard DMD2 configuration has the lowest total cost. At FID 1.28, DMAD with PG and longer-training DMD2 have similar total costs of 433 and 439 GPU-hours, respectively. On SDXL, DMAD reduces the online cost of reaching approximately the same FID as DMD2 from 324 to 103 GPU-hours. Including the full cache raises the single-run total to 392 GPU-hours. DMAD then reaches FID 14.47 at a total cost of 412 GPU-hours. We report the online and total costs together so the effect of cache reuse is clear. Table 5 uses DMD2's standard ratio of five critic updates per generator update and DMAD's ratio of one. We also benchmarked DMD2 at 1:1 on ImageNet-64 with the same hardware and batch size. Its time per generator update falls from 2.321 s at 5:1 to 0.717 s at 1:1, compared with 0.496 s for DMAD at 1:1. Thus, matching the update ratio narrows the timing gap, and DMAD still takes about 31% less time per generator update.

W2/W3. ADD-style distillation, discriminator loss, and single-head data mixing

We ran three additional comparisons on ImageNet-64 to examine how DMAD differs from ADD-style distillation, how the discriminator loss affects performance, and whether pooling real and teacher samples into one positive class is sufficient.

MethodFID ↓
ADD-style baseline1.32
DMAD (fixed weights, hinge)1.42
DMAD (fixed weights, BCE)1.33
Single-head data mixing1.53
DMAD1.24

W2. ADD-style distillation and BCE versus hinge. The ADD-style baseline uses the same discriminator backbone, real-data head, optimizer, and real-data source as DMAD. It combines a linear adversarial generator loss with online regression to the teacher's predictions. This baseline reaches 1.32 FID. Full DMAD reaches 1.24 FID using cached teacher samples, with no teacher evaluations during student training. In a separate comparison, we keep both heads and their weights fixed and change only the discriminator loss. BCE improves FID from 1.42 to 1.33, supporting its use in our two-head formulation.

W3. Single-head data mixing. We match DMAD's discriminator backbone, data sources, linear generator loss, and training budget. We train a single head with mixed real and teacher-generated samples as positives and student samples as negatives. We test real-sample fractions of 25%, 50%, and 75% in the positive class. The best result comes from the 75% ratio, which achieves an FID of 1.53, compared with 1.24 for full DMAD. The single head gives the generator one training signal for the pooled target. DMAD's separate heads provide a signal for each target, so the teacher weight can vary across noise bands while the real-data weight stays fixed. The FID comparison supports this combination of separate heads and adaptive weighting over the best fixed mixture we tested.

AuthorsOpenReview ↗

W4. Wan2.1-14B comparison with DMD2, rCM, AnyFlow, and PDD

Thank you for suggesting these baselines. We extend the Wan2.1-14B comparison to include DMD2, AnyFlow, and the concurrent work PDD in the four-step setting. The table reports VBench quality and semantics, two VBench motion metrics, diversity, and online training cost. Following PDD, we measure diversity using mean pairwise cosine and L2 distances in V-JEPA 2 and VideoMAE V2 feature spaces. Training costs are expressed in H200 GPU-hours for the reported checkpoints.

MethodVBench Total ↑Quality ↑Semantic ↑Dynamic degree ↑Motion smoothness ↑V-JEPA 2 cos. ↑V-JEPA 2 L2 ↑VideoMAE V2 cos. ↑VideoMAE V2 L2 ↑Training GPU-h ↓
DMD284.4085.1681.340.76330.97410.056817.670.009451.7103,328
rCM84.9285.4382.880.71390.98030.077921.100.012531.9833,742
AnyFlow84.9585.7081.920.70420.98270.078620.990.012971.9923,374
PDD-short84.9285.7181.770.69340.98140.079121.270.012472.027760
PDD-long84.6985.6980.710.70690.97830.084622.130.012642.05810,700
DMAD85.1585.9681.900.74440.98560.079021.380.014632.6832,890

DMAD has the highest reported VBench Total and Quality scores, with a 0.20-point Total margin over AnyFlow. It also has the highest motion smoothness and the second-highest dynamic degree, behind DMD2. For diversity, DMAD leads on both VideoMAE V2 measures, with a mean pairwise cosine distance of 0.01463 and an L2 distance of 2.683. The next-highest scores are 0.01297 for cosine distance (AnyFlow) and 2.058 for L2 distance (PDD-long). In V-JEPA 2 feature space, DMAD is close to PDD-short, and PDD-long leads on both measures. PDD-short reaches a VBench Total of 84.92 at the lowest online training cost of 760 GPU-hours. DMAD uses fewer online training GPU-hours than DMD2, rCM, AnyFlow, and PDD-long.

We thank the reviewer again for the thoughtful review and concrete suggestions, which helped us give a fuller account of training costs and strengthen the experimental comparisons. We hope the additional cost analysis, objective ablations, and video comparisons address the reviewer's concerns. We would be grateful if the reviewer would consider raising the score to 4.

Reviewer_SHg5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper itroduces DMAD, which recasts DMD's distribution matching as joint binary classification. Two discriminator heads on a shared feature backbone are trained with balanced BCE, and the generator is trained with a linear loss on the logits. A "gap-based reweighting" adapts the teacher's weight per noise band from the real head's EMA mean-logit gap between real and teacher samples. The experiments are done on Z-Image-Turbo 6B and Qwen-Image 20B, and ablation experiment shows the benefits in a shared training setting.

Critical strengths
  1. The same two-class objective replacing DMD's critic machinery is a neat idea
  2. The training-time savings are clear.
  3. The ablation experiments on the dual-target design is a good demonstration of the two targets’ complementary role.
  4. Strong results are shown even for big test sets.
  5. Gap reweighting is a lightweight, adaptive mechanism with improvements across images and video.
Critical weaknesses
  1. Objective-family novelty appears limited. The method could recover the DMD gradient, but in practice the discriminator is a trained, moving classifier. The paper's "distribution matching" framing may be overstating the connection to the DMD objective. The author could discuss more on the contributions from other components, eg. trainable pretrained backbone, dual targets, linear loss, etc.
  2. The intuition behind the gap reweighting (low real-vs-teacher gap -> weight teacher more) is not really analyzed. There is no discussion of failure modes (e.g., real-head saturation, band sparsity with batch size 8), and τ/band counts are hand-tuned for each setting.
Questions for authors
  1. Please report training memory/compute comparisons vs DMD2 and SiDA (the core efficiency claim), for SDXL and Wan at least.
  2. What if the backbone is frozen, with only the two heads trained? This would separate the dual-target formulation's benefit from a well-trained discriminator.
4 comments
AuthorsOpenReview ↗

Response to Reviewer SHg5

Thank you for your thoughtful review and constructive suggestions. We appreciate your recognition of the complementary roles of the two targets and the efficiency of the shared discriminator.

We address the scope of the distribution-matching interpretation together with the frozen-backbone experiment in W1/Q2, discuss gap-based reweighting in W2, and report training time and memory comparisons with DMD2 and SiDA in Q1.

AuthorsOpenReview ↗

W1/Q2. Distribution-matching interpretation and frozen-backbone ablation

Scope of the distribution-matching interpretation. We agree that the optimal-discriminator result and the behavior of a learned discriminator during training should be distinguished more clearly. For a fixed target and noise level, the optimal balanced-BCE logit is the target-to-student log-density ratio. Holding that discriminator fixed for the generator update, the linear logit loss recovers the idealized reverse-KL gradient underlying DMD. During training, the student distribution changes, and the learned discriminator provides an approximation to this gradient. We will clarify this distinction in Section 3.1.

The contribution is to estimate the score difference directly with classifiers, using one head for teacher samples and one for real data. This lets a shared discriminator provide both training signals without online teacher evaluation or auxiliary score fitting. Figure 2 illustrates the gradient estimation idea in a controlled setting. With a fixed student distribution and the same teacher pretraining, the learned logit gradient gives a normalized mean squared error of 0.144 against the analytic reference, compared with 1.539 for the separately estimated DMD scores. This experiment isolates single-target gradient estimation.

Roles of the two targets and the linear loss. Table 4 shows that the targets contribute to different aspects of generation quality. On SDXL, removing the real-data head raises FID from 14.47 to 21.37. Removing the teacher head lowers global FID to 13.85, while patch FID worsens from 19.88 to 26.03 and CLIP decreases from 0.328 to 0.312. The linear generator loss uses the logit gradient directly. Replacing it with the non-saturating loss, which applies a sample-dependent multiplier to that gradient, raises ImageNet FID from 1.24 to 1.31.

Q2. What happens when the discriminator backbone is frozen? On ImageNet-64, we froze the shared discriminator backbone at its pretrained EDM teacher weights and trained only the two discriminator heads. As the student distribution evolves, each head continues to learn to distinguish student samples from its own target, using the same fixed feature representation. The frozen-backbone variant achieves a best FID of 1.33, compared with 1.24 for the trainable-backbone model reported in the paper.

The frozen-backbone result shows that the two heads can provide useful distillation gradients using fixed, in-domain features. This complements the head-removal ablations in Table 4, which examine what each target contributes to the full model. The lower FID of the trainable configuration suggests that adapting the shared features can further improve generation quality in this setting.

AuthorsOpenReview ↗

W2. Motivation and limitations of gap-based reweighting & Q1. Training time and memory compared with DMD2 and SiDA

Thank you for asking us to explain the weighting rule and its failure modes. We use the real-data head to score both noisy real and teacher samples. When the teacher receives a mean logit closer to that of real data in a given noise band, we assign more weight to teacher supervision in that band. The gap is a relative signal that depends on the current student and discriminator. Figure 7(a) shows that it varies across noise bands and over training, which motivates adapting the weights rather than using the same teacher coefficient at every noise level.

Real-head saturation. We measure the gap in logits, which can retain differences even when sigmoid probabilities are close to zero or one. If the real-data head itself produces nearly indistinguishable gaps across bands, the normalized weights approach uniform weighting. Conversely, a temperature that is too small relative to the gap variation can make the weighting nearly binary. We will clarify how the temperature controls this tradeoff.

Sparse bands and hyperparameters. With a batch size of eight, many bands have no observations on a given update. For Wan, we use ten equal-probability bands and pool the per-band statistics across devices. We update a band's EMA only when both real and teacher samples are present and retain its previous estimate otherwise. We also normalize the teacher weights over the measured bands to avoid making the normalization depend on which bands happen to appear in a minibatch. Sparsely observed bands can still adapt more slowly. We use a temperature of 0.5 for ImageNet and SDXL and 2 for Wan and MiniMax-H3. The band boundaries follow each model's noise schedule or sampling distribution.

The fixed-weight ablation in Table 4 measures the benefit of reweighting at the reported settings. Reweighting improves FID from 1.33 to 1.24 on ImageNet and from 16.62 to 14.47 on SDXL, and increases the Wan2.1-1.3B VBench total from 83.56 to 84.70. We will add the rationale for the weighting rule and discuss how head quality, band occupancy, and temperature affect it.

Q1. Training time and memory compared with DMD2 and SiDA We extended the training measurements in Table 5 to include SiDA and four-step SDXL. Each table entry gives seconds per generator update, including its associated critic updates, and peak allocated memory in GiB per GPU. The critic-to-generator update ratios are 5:1 for DMD2 and 1:1 for SiDA and DMAD.

SettingH200 GPUsDMD2 time (s)SiDA time (s)DMAD time (s)DMD2 peak memory (GiB/GPU)SiDA peak memory (GiB/GPU)DMAD peak memory (GiB/GPU)
ImageNet-6412.320.840.5040.353.734.0
SDXL, four-step15.281.941.5698.7103.982.5
Wan2.1-14B4162.939.7038.50131.0130.7124.6

ImageNet uses batch size 40. The SDXL benchmark uses BF16, gradient checkpointing, and batch size one. The 14B comparison uses the four-GPU configuration from Table 5. We use the official SiDA implementation on ImageNet and our ports of the SiDA training objective for SDXL and Wan.

On SDXL, DMAD reduces the time per generator update from 5.28 s to 1.56 s relative to DMD2 and from 1.94 s to 1.56 s relative to SiDA. Its peak memory is also lower than both baselines. On Wan, DMAD uses the least time per generator update and less peak memory under the profiled setup.

We thank the reviewer again for the thoughtful review and specific suggestions, which helped us clarify the scope of our theoretical result and report training costs more fully. We hope the frozen-backbone comparison, the discussion of gap-based reweighting, and the expanded training benchmarks address the reviewer's concerns. We would be grateful if the reviewer would consider raising the score to 4.

Thanks for the clarifications and the new results. Yet I feel that the current score reflects well how the paper compares to the other submissions I've seen.

Reviewer_6vf1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper builds on the Distribution Matching Distillation (DMD) family. DMD requires a fake diffusion model that tracks the student's distribution. This paper, DMAD, replaces it with binary classifiers using a shared backbone and two heads. The method introduces gap-based reweighting to adapt to the noise band. The method reaches strong performance on 1-Step ImageNet and 4-step SDXL-based COCO generation on images, and 4-step Wan-2.1-based generation on video. The paper also shows relevant baselines, and an ablation study in Table 4, which shows that performance degrades without each of the proposed components.

Critical strengths

Overall, the method is a solid contribution. Rather than having 3 networks (student generator, frozen teacher, and tuned critic), the method has 2 networks (student generator, one discriminator with shared backbone and 2 heads).

The attached video is informative on the quality of the results. The related works are well-organized and current. The comparisons are thorough, with clear improvements over the DMD2, sCM, and rCM baselines in the videos. There are comparisons in two image settings as well as video. Not only is there a quantative study, but also a user study, which is appreciated.

Critical weaknesses

One concern is that some of the improvements are overstated. The paper presents the ImageNet result as 1.04, but digging into Table 1 we see that this is done wiht the benefit of a projected discriminator, with that method being labeled as DMAD+projected discriminator. A projected discriminator takes advantage of pretrained features, and ImageNet FID is known to be sensitive to ImageNet-class features (Kynkäänniemi, ICLR 2023). Just using DMAD achieves 1.24, which is worse than several baselines (SiD2A 1.11 and D2O-F 1.16), and notably is close to DMD2+longer training of 1.28. This calls into question whether the other methods could have benefited from the projected discriminator as well. An evaluating with FDR6 (Yang 2026) would also be more convincing, at least with respect to FID hacking.

While the work discusses the efficiency of not having to carry the auxiliary diffusion model, the training dynamics are not presented in the paper. One can see Table 5 in the appendix about training time per generator update, which is interesting. It would be further informative to get a sense of the quality vs wall clock training time.

Questions for authors

Please address the questions regarding the ImageNet results above.

  • Do performance gains hold with FDr6?
  • Does having DMD2+projected discriminator close the gap to DMAD+projected discriminator?
  • Do FID gains hold over various seeds, as they can vary between training runs (FID lottery https://kyutai.org/fid-lottery/)

Please comment about the training dynamics.

  • What is the difference in training efficiency in wall clock time? Are there gains relative to the baselines?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 6vf1

Thank you for your thoughtful review and for describing DMAD as a “solid contribution.” We appreciate your recognition of the thorough comparisons across image and video generation and the value of the human evaluation.

We conducted additional experiments to address your concerns about the ImageNet results, their robustness, and training efficiency. We address each weakness together with its related questions in two replies below: W1/Q1–Q3 covers the ImageNet results, and W2/Q4 covers training dynamics and efficiency.

AuthorsOpenReview ↗

W1/Q1–Q3. ImageNet results, FDr6, projected discriminator, and training seeds

Thank you for suggesting these additional comparisons. DMAD without a projected discriminator achieves the best FDr6 among the evaluated methods. With the same projected discriminator, DMAD also retains its FID advantage over DMD2.

We agree that the results with and without the projected discriminator should be clearly distinguished. The reported FID of 1.04 uses the projected discriminator, while base DMAD achieves 1.24. We will make this distinction explicit wherever the headline ImageNet result is summarized.

Q1. Do performance gains hold with FDr6?

DMAD achieves the lowest FDr6 among the evaluated methods. Below, we report conventional FID together with FDr6 and its six component scores, using the normalization prescribed by FDr6. Lower is better for all columns. PG denotes the projected discriminator.

MethodFIDIncep.ConvNeXtDINOv2MAE-HSigLIPCLIPFDr6
DMD21.511.4372.0664.4338.5097.7823.1084.555
DMD2 + PG1.591.4842.7166.94013.11210.8034.4916.591
DMD2, longer training1.281.1981.7453.5127.2756.9202.8483.917
DMD2, longer training + PG1.151.0731.7724.3209.9098.5293.0794.780
D2O-F1.161.0891.4263.7729.4837.7932.6754.373
SiD2A1.111.0391.5083.9477.1386.9423.2553.972
DMAD1.241.1572.1083.7526.9226.8252.2783.840
DMAD + PG1.040.9721.6854.3069.6878.4033.0134.678

DMAD achieves an FDr6 of 3.840, compared with 3.917 for DMD2 with longer training, 3.972 for SiD2A, and 4.373 for D2O-F. It also achieves the lowest MAE-H, SigLIP, and CLIP component scores. These results provide evidence for DMAD’s quality beyond Inception-based FID, without requiring a projected discriminator.

Q2. Does DMD2 with a projected discriminator close the gap to DMAD with a projected discriminator?

We trained DMD2 with the same projected discriminator under both its original and longer-training settings. Under longer training, the projected discriminator improves DMD2’s FID from 1.28 to 1.15, confirming that DMD2 can benefit from it. However, this remains above the 1.04 achieved by DMAD with the projected discriminator.

The additional evaluation also reveals a distinction between the metrics. Adding the projected discriminator increases FDr6 from 3.917 to 4.780 for longer-training DMD2 and from 3.840 to 4.678 for DMAD. Thus, DMAD’s best FDr6 is obtained without PG, while its best FID is obtained with projected discriminator.

Q3. Do FID gains hold across training seeds?

We follow DMD2 to use seed 10 in our experiments. We additionally trained DMAD and the longer-training DMD2 baseline with seeds 1 and 100. The table reports FID for each training seed and the average results across the three runs.

MethodSeed 1Seed 10Seed 100Mean
DMD2, longer training1.311.281.291.293
DMAD1.261.241.231.243

DMAD achieves lower FID for all three training seeds, showing that the observed advantage persists across the evaluated training seeds.

AuthorsOpenReview ↗

W2/Q4. Training dynamics and wall-clock efficiency

Thank you for requesting a comparison beyond time per generator update. We address your concern about training dynamics together with your question about wall-clock efficiency by reporting the training compute required to reach several FID thresholds.

These measurements account for both the cost of each update and the number of updates needed to reach a given quality. All GPU-hour costs are measured on H200 GPUs.

ImageNet-64: quality versus online training compute.

The table reports online training GPU-hours. PG denotes the projected discriminator, and “N/A” indicates that the threshold was not reached within the reported run.

Target FID ≤DMADDMAD + PGDMD2DMD2, longer training
2.00536154256
1.75808280258
1.51220116136266
1.40270150N/A280
1.35300170N/A298
1.30414197N/A405
1.28434204N/A439
1.24454231N/AN/A
1.19N/A272N/AN/A
1.04N/A303N/AN/A

Without PG, DMAD and longer-training DMD2 require comparable compute near FID 1.3. They reach FID ≤ 1.28 in 434 and 439 GPU-hours, respectively. DMAD subsequently reaches FID 1.24 at 454 GPU-hours.

With PG, DMAD reaches FID ≤ 1.28 in 204 GPU-hours, compared with 439 for longer-training DMD2, a 53.5% reduction in training compute. It further reaches FID 1.04 at 303 GPU-hours. Thus, base DMAD achieves better final FID at a comparable training cost, while DMAD with PG substantially reduces the compute needed to reach the reported FID thresholds.

SDXL: quality versus online training compute.

DMAD uses 8 GPUs with batch size 8 per GPU. DMD2 follows its official configuration of 64 GPUs with batch size 2 per GPU.

Target FIDDMAD GPU-hoursDMD2 GPU-hours
303854
2097126
19.3103324
15112N/A
14.47123N/A

DMAD reaches approximately DMD2’s final FID of 19.3 using 103 versus 324 GPU-hours, a 68.2% reduction in training compute. DMAD further reaches FID 14.47 after 123 GPU-hours.

These measurements complement Table 5 by quantifying the compute required to reach comparable quality and making the effect of GPU allocation on elapsed time explicit.

We thank the reviewer again for the thoughtful review and specific suggestions, which helped us clarify our ImageNet results and present the relationship between generation quality and training cost more clearly. We hope the additional FDr6 evaluation, the comparison with DMD2 using the same projected discriminator, the results across training seeds, and the new training efficiency analysis address the reviewer’s concerns. In light of these additions, we would be grateful if the reviewer would consider raising the score to 4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.