- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Velocity Adversarial Distillation, a framework for few-step diffusion and flow-model distillation that uses a vector-valued critic to match student and teacher velocity fields. It introduces two constructions: A-form captures conditional-mean discrepancies, and B-form captures score discrepancies for the student’s actual perturbed distribution. Under specified assumptions and gradient conventions, the framework connects moment-matching distillation, reverse-KL updates, and Fisher-based matching. VA-TDM applies both forms to intermediate student transitions and combines transition-aware auxiliary-field estimation, mixed pathwise and flow-product updates, and regularization of the auxiliary fields toward the teacher velocity. Experiments report competitive four-step text-to-image and text-to-video generation on SD3.5-Medium, FLUX.2-Klein-Base 4B, and MiniMax-H3.
Critical strengths
1. Principled analysis and a unified perspective. The paper identifies a meaningful mismatch between perturbing an intermediate student state and independently perturbing its clean prediction, which can induce different score targets. This analysis motivates supervising the actual transition distribution and transition-aware auxiliary-field estimation. VAD further connects conditional-moment matching, reverse-KL updates, and Fisher-based matching under explicit assumptions, while clarifying MMD’s heuristic update.
2. Empirical gains. Table 1 shows that VA-TDM improves over few-step baselines on most evaluation metrics across SD3.5-Medium and FLUX.2-Klein-Base 4B. The MiniMax-H3 user study further reports a 61.33% win rate against Fast-H3 and a 45.53% preference rate against the 100-NFE teacher. Together, these results support the method’s practical value across backbones and modalities without increasing inference cost beyond four steps.
Critical weaknesses
1. Unclear compatibility of dual-form supervision. Theorem 5 establishes different matching conditions for A-form and B-form, but does not show that they are jointly compatible. Since the clean anchor only parameterizes the student transition, matching the intermediate distribution need not eliminate the A-form residual. Dual-form mixing may therefore introduce bias rather than simply provide complementary supervision, a trade-off that remains insufficiently analyzed.
2. Limited video evaluation beyond user preferences. The video results rely on qualitative examples and a user study covering only 10 prompts, without complementary benchmark-based quantitative evaluation of visual quality, temporal consistency, or prompt alignment. While human preferences provide valuable evidence, the limited evaluation makes it difficult to determine which aspects of video generation improve and how broadly the reported gains generalize.
Questions for authors
-
Under exact auxiliary-field estimation, under what conditions can A-form and B-form simultaneously attain their respective matching conditions for intermediate student transitions with ? When these conditions are incompatible, does combining the two forms introduce bias relative to the distribution-matching objective, and how does this trade-off depend on the mixing coefficient ?
-
The current video comparison relies mainly on the user study and selected frames, lacking a more comprehensive quantitative evaluation. Could the authors supplement objective metrics such as consistency and text alignment?
-
The training details in Appendix D.1 are insufficient for reproducibility. Could the authors provide more detailed experimental configurations?
4 comments
Reply to Reviewer k7Xs (2/2)
More Qualitative and Quantitative Evaluation on Text-to-Video Generation (W2 & Q2)
Great suggestion. We provide more video samples in the supplementary material, and we scaled our anonymized user study on 33B MiniMax-H3 from 10 prompts to 100 prompts, expanding the number of valid human votes per model pair from around 300 to nearly 1,000 (nearly 3,000 votes in total):
| Model A | Model B | Win rate of A |
|---|---|---|
| VA-TDM-H3 (4 NFE) | Fast-H3 (4 NFE) | 62.89% |
| VA-TDM-H3 (4 NFE) | Teacher (100 NFE) | 47.23% |
| Teacher (100 NFE) | Fast-H3 (4 NFE) | 70.62% |
VA-TDM-H3 (4 NFE) achieves a 62.89% win rate against Fast-H3 (4 NFE) and a 47.23% win rate against the 100-NFE teacher, while the teacher achieves a 70.62% win rate against Fast-H3.
We also provide quantitative results on VBench [e]—a widely adopted, comprehensive benchmark suite evaluating video generation across 16 dimensions of visual quality, temporal consistency, and prompt alignment—as shown below:
| Method | NFE | Total | Quality | Semantic | Subject consistency | Motion smoothness | Dynamic degree | Imaging quality | Multiple objects | Spatial relationship |
|---|---|---|---|---|---|---|---|---|---|---|
| MiniMax-H3 (Base) | 100 | 84.97 | 85.72 | 81.94 | 95.95 | 99.13 | 76.94 | 68.91 | 90.79 | 80.44 |
| Fast-H3 | 4 | 84.15 | 84.83 | 81.42 | 95.35 | 98.74 | 76.67 | 69.69 | 86.25 | 84.42 |
| VA-TDM (Ours) | 4 | 85.03 | 85.85 | 81.75 | 95.80 | 98.92 | 80.83 | 71.09 | 86.29 | 85.03 |
At 4 NFE, VA-TDM consistently outperforms Fast-H3 across all overall scores (Total 85.03 vs. 84.15, Quality 85.85 vs. 84.83, Semantic 81.75 vs. 81.42) and representative dimensions, and even matches or slightly exceeds the 100-NFE base model in Total and Quality scores (it is worth noting that automated benchmarks might be less sensitive to perceptual artifacts like oversaturation on frontier models like H3).
Detailed Experimental Configurations for Reproducibility (Q3)
We provide more experimental configurations here (and will add them to Appendix D.1): the global batch size is ; we use AdamW with and LoRA rank ; the learning rate is for the generator and for the online field model (updated ); we set (path/flow-product mixing), (teacher regularization), (dual-form mixing), and . We will open-source the code upon publication to ensure reproducibility.
References
- [a] Multistep Distillation of Diffusion Models via Moment Matching, NeurIPS 2024.
- [b] Simpler Diffusion: 1.5 FID on ImageNet512 with Pixel-space Diffusion, CVPR 2025.
- [c] Diffusion Fine-tuning with Rewarded Moment Matching Distillation, NeurIPS 2026.
- [d] Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMD, arXiv.
- [e] VBench: Comprehensive Benchmark Suite for Video Generative Models, CVPR 2024.
Thanks for the authors’ responses and work. My concerns are resolved, and I will raise my score.
Thank you for acknowledging our work and for raising the score! Thank you again for your time and effort in reviewing our paper.
Reply to Reviewer k7Xs (1/2)
We sincerely thank the reviewer for the constructive comments. We believe that as flow matching becomes increasingly prevalent in pretraining for frontier visual generation models, establishing a unified few-step distillation framework natively in velocity space is a timely and critical problem. Unlike score-based distillation—which relies on adding independent Gaussian noise and thus faces either high variance or distribution mismatch at intermediate steps—working directly with velocities removes this Gaussian constraint, allowing us to flexibly construct low-variance, coupled supervision targets directly along the student's actual sampling chain.
To our knowledge, our work is the first to unify Reverse-KL (DMD/Diff-Instruct), Fisher divergence, and Moment Matching by revealing their shared adversarial nature within a single velocity adversarial game (VAD), while deriving stable new objectives (VA-TDM) that scale to 4-step text-to-image (SD3.5-M, FLUX.2-Klein 4B) and further scale to 33B audio-video generation (MiniMax-H3) with state-of-the-art performance. We address each concern below.
Both forms supervise distribution-level quantities: A-form matches the conditional moment, while B-form matches the perturbed marginal distribution. Intuitively, exact distribution matching naturally entails moment matching, suggesting that B-form and A-form are compatible.
Therefore, the dual-form would not introduce bias into the learning of the generator.
In practice, however, a finite-capacity few-step student cannot converge to the optimal solution, requiring a practical trade-off between which distributional information to preserve and the corresponding learning variance. A-form provides a more relaxed, lower-variance distributional supervision than B-form (which suffers from high online target variance when s>0). Mixing them via λm∈(0,1) in dual-form balances learning variance and distributional information preservation. Because finite-capacity distillation ultimately hinges on how well the trade-off translates into optimization quality, we believe practical effectiveness is the most important criterion: our experiments show that A-form alone is effective, the A/B-form hybrid further improves performance, and MMD (recovered by our A-form detached pathwise update) [a] and its variants have also proven effective across a wide range of scenarios [b, c, d].
Moreover, to further investigate the standalone effectiveness of the dual-form, we conduct an ablation without the transition-aware field (on 4-step SD3.5-Medium) as shown below:
Here, our dual-form (λm=0.8) exhibits clear, consistent gains across all metrics. Interestingly, even though standalone B-form (λm=0) performs worse than A-form (λm=1) on GenEval and OCR, mixing a small weight of this empirically weaker B-form (0.2) into A-form (0.8) actually improves upon A-form across the board. This aligns well with our analysis above.