acceptodds
Under review as a conference paper at ICLR 2027

VA-TDM: A Unified View for Improving Few-Step Diffusion Models by Velocity Adversarial Distillation

Abstract

Few-step distillation offers efficient generation with diffusion and flow models, but preserving quality with only a few sampling steps remains challenging. Distribution-matching methods via score distillation provide a strong foundation, yet they rely on accurately tracking the student’s changing distribution. At intermediate generation steps in few-step sampling, student-score estimation can be difficult, and commonly used estimators may describe a different distribution from the one used to update the student. Flow matching allows a more flexible approach to supervision through velocity fields constructed from the student’s sampling process. Building on this perspective, we introduce Velocity Adversarial Distillation (VAD), which uses a learned critic to compare student and teacher fields and turn their disagreement into feedback for learning. VAD unifies score distillation, Fisher-based matching, and moment matching, also explaining existing heuristic MMD updates and delivering new training objectives. We further develop VA-TDM, which follows trajectory distribution matching: providing supervision for partial denoising along the student trajectory. Beyond this, VA-TDM combines transition-aware online velocity learning and dual-form VAD supervision across intermediate student transitions. Extensive results show that VA-TDM is powerful for learning few-step generators, achieving state-of-the-art four-step performance in both text-to-image and text-to-video generation across various backbones and scaling effectively to 33B MinMAX-H3. In particular, our VA-TDM-H3 notably outperforms the few-step Fast-H3 and is closer to the original many-step H3 according to human preference.

Primary area:
generative models
Keywords:
Diffusion Models, Video Geneartion, Few Step Diffusion Models, Image Generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_k7XsOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Velocity Adversarial Distillation, a framework for few-step diffusion and flow-model distillation that uses a vector-valued critic to match student and teacher velocity fields. It introduces two constructions: A-form captures conditional-mean discrepancies, and B-form captures score discrepancies for the student’s actual perturbed distribution. Under specified assumptions and gradient conventions, the framework connects moment-matching distillation, reverse-KL updates, and Fisher-based matching. VA-TDM applies both forms to intermediate student transitions and combines transition-aware auxiliary-field estimation, mixed pathwise and flow-product updates, and regularization of the auxiliary fields toward the teacher velocity. Experiments report competitive four-step text-to-image and text-to-video generation on SD3.5-Medium, FLUX.2-Klein-Base 4B, and MiniMax-H3.

Critical strengths

1. Principled analysis and a unified perspective. The paper identifies a meaningful mismatch between perturbing an intermediate student state and independently perturbing its clean prediction, which can induce different score targets. This analysis motivates supervising the actual transition distribution and transition-aware auxiliary-field estimation. VAD further connects conditional-moment matching, reverse-KL updates, and Fisher-based matching under explicit assumptions, while clarifying MMD’s heuristic update.

2. Empirical gains. Table 1 shows that VA-TDM improves over few-step baselines on most evaluation metrics across SD3.5-Medium and FLUX.2-Klein-Base 4B. The MiniMax-H3 user study further reports a 61.33% win rate against Fast-H3 and a 45.53% preference rate against the 100-NFE teacher. Together, these results support the method’s practical value across backbones and modalities without increasing inference cost beyond four steps.

Critical weaknesses

1. Unclear compatibility of dual-form supervision. Theorem 5 establishes different matching conditions for A-form and B-form, but does not show that they are jointly compatible. Since the clean anchor only parameterizes the student transition, matching the intermediate distribution need not eliminate the A-form residual. Dual-form mixing may therefore introduce bias rather than simply provide complementary supervision, a trade-off that remains insufficiently analyzed.

2. Limited video evaluation beyond user preferences. The video results rely on qualitative examples and a user study covering only 10 prompts, without complementary benchmark-based quantitative evaluation of visual quality, temporal consistency, or prompt alignment. While human preferences provide valuable evidence, the limited evaluation makes it difficult to determine which aspects of video generation improve and how broadly the reported gains generalize.

Questions for authors
  1. Under exact auxiliary-field estimation, under what conditions can A-form and B-form simultaneously attain their respective matching conditions for intermediate student transitions with s>0s>0? When these conditions are incompatible, does combining the two forms introduce bias relative to the distribution-matching objective, and how does this trade-off depend on the mixing coefficient λm\lambda_m?

  2. The current video comparison relies mainly on the user study and selected frames, lacking a more comprehensive quantitative evaluation. Could the authors supplement objective metrics such as consistency and text alignment?

  3. The training details in Appendix D.1 are insufficient for reproducibility. Could the authors provide more detailed experimental configurations?

4 comments
AuthorsOpenReview ↗

Reply to Reviewer k7Xs (1/2)

We sincerely thank the reviewer for the constructive comments. We believe that as flow matching becomes increasingly prevalent in pretraining for frontier visual generation models, establishing a unified few-step distillation framework natively in velocity space is a timely and critical problem. Unlike score-based distillation—which relies on adding independent Gaussian noise and thus faces either high variance or distribution mismatch at intermediate steps—working directly with velocities removes this Gaussian constraint, allowing us to flexibly construct low-variance, coupled supervision targets directly along the student's actual sampling chain.

To our knowledge, our work is the first to unify Reverse-KL (DMD/Diff-Instruct), Fisher divergence, and Moment Matching by revealing their shared adversarial nature within a single velocity adversarial game (VAD), while deriving stable new objectives (VA-TDM) that scale to 4-step text-to-image (SD3.5-M, FLUX.2-Klein 4B) and further scale to 33B audio-video generation (MiniMax-H3) with state-of-the-art performance. We address each concern below.

Compatibility of A-form and B-form Supervision (W1 & Q1)

Both forms supervise distribution-level quantities: A-form matches the conditional moment, while B-form matches the perturbed marginal distribution. Intuitively, exact distribution matching naturally entails moment matching, suggesting that B-form and A-form are compatible.

  • Exact Equivalence for the optimal desired ultimate sampling distribution. At the final step (s=0s=0), our few-step generator directly predicts x^0\hat{\mathbf{x}}_0, where A-form and B-form are equivalent (Theorem 5(ii)); thus, the optimal final sample distribution (our ultimate desired target) is to match the teacher distribution.
  • Common Global Optimum for Intermediate Transitions. For general intermediate transitions t→st \to s (s>0s>0), as shown in MMD ([a], Appendix D), under sufficient student capacity, moment matching (connected with A-form) and score matching (connected with B-form) share a common global optimum at the target teacher distribution.

Therefore, the dual-form would not introduce bias into the learning of the generator.

In practice, however, a finite-capacity few-step student cannot converge to the optimal solution, requiring a practical trade-off between which distributional information to preserve and the corresponding learning variance. A-form provides a more relaxed, lower-variance distributional supervision than B-form (which suffers from high online target variance when s>0s>0). Mixing them via λm∈(0,1)\lambda_m \in (0,1) in dual-form balances learning variance and distributional information preservation. Because finite-capacity distillation ultimately hinges on how well the trade-off translates into optimization quality, we believe practical effectiveness is the most important criterion: our experiments show that A-form alone is effective, the A/B-form hybrid further improves performance, and MMD (recovered by our A-form detached pathwise update) [a] and its variants have also proven effective across a wide range of scenarios [b, c, d].

Moreover, to further investigate the standalone effectiveness of the dual-form, we conduct an ablation without the transition-aware field (on 4-step SD3.5-Medium) as shown below:

ConfigurationGenEval ↑\uparrowOCR ↑\uparrowHPSv3 ↑\uparrowLPIPS ↑\uparrow
λm=1\lambda_m = 1 (A-form) w/o transition-aware field0.600.5411.090.55
λm=0\lambda_m = 0 (B-form) w/o transition-aware field0.580.5311.130.56
λm=0.8\lambda_m = 0.8 (Dual-form) w/o transition-aware field0.620.5611.380.57

Here, our dual-form (λm=0.8\lambda_m = 0.8) exhibits clear, consistent gains across all metrics. Interestingly, even though standalone B-form (λm=0\lambda_m = 0) performs worse than A-form (λm=1\lambda_m = 1) on GenEval and OCR, mixing a small weight of this empirically weaker B-form (0.20.2) into A-form (0.80.8) actually improves upon A-form across the board. This aligns well with our analysis above.

AuthorsOpenReview ↗

Reply to Reviewer k7Xs (2/2)

More Qualitative and Quantitative Evaluation on Text-to-Video Generation (W2 & Q2)

Great suggestion. We provide more video samples in the supplementary material, and we scaled our anonymized user study on 33B MiniMax-H3 from 10 prompts to 100 prompts, expanding the number of valid human votes per model pair from around 300 to nearly 1,000 (nearly 3,000 votes in total):

Model AModel BWin rate of A
VA-TDM-H3 (4 NFE)Fast-H3 (4 NFE)62.89%
VA-TDM-H3 (4 NFE)Teacher (100 NFE)47.23%
Teacher (100 NFE)Fast-H3 (4 NFE)70.62%

VA-TDM-H3 (4 NFE) achieves a 62.89% win rate against Fast-H3 (4 NFE) and a 47.23% win rate against the 100-NFE teacher, while the teacher achieves a 70.62% win rate against Fast-H3.

We also provide quantitative results on VBench [e]—a widely adopted, comprehensive benchmark suite evaluating video generation across 16 dimensions of visual quality, temporal consistency, and prompt alignment—as shown below:

MethodNFETotal ↑\uparrowQuality ↑\uparrowSemantic ↑\uparrowSubject consistencyMotion smoothnessDynamic degreeImaging qualityMultiple objectsSpatial relationship
MiniMax-H3 (Base)10084.9785.7281.9495.9599.1376.9468.9190.7980.44
Fast-H3484.1584.8381.4295.3598.7476.6769.6986.2584.42
VA-TDM (Ours)485.0385.8581.7595.8098.9280.8371.0986.2985.03

At 4 NFE, VA-TDM consistently outperforms Fast-H3 across all overall scores (Total 85.03 vs. 84.15, Quality 85.85 vs. 84.83, Semantic 81.75 vs. 81.42) and representative dimensions, and even matches or slightly exceeds the 100-NFE base model in Total and Quality scores (it is worth noting that automated benchmarks might be less sensitive to perceptual artifacts like oversaturation on frontier models like H3).

Detailed Experimental Configurations for Reproducibility (Q3)

We provide more experimental configurations here (and will add them to Appendix D.1): the global batch size is 256256; we use AdamW with (β1,β2)=(0,0.95)(\beta_1, \beta_2) = (0, 0.95) and LoRA rank 6464; the learning rate is 4×10−54 \times 10^{-5} for the generator and 2×10−42 \times 10^{-4} for the online field model (updated 1:11:1); we set λb=0.7\lambda_b = 0.7 (path/flow-product mixing), λr=1.0\lambda_r = 1.0 (teacher regularization), λm=0.8\lambda_m = 0.8 (dual-form mixing), and η=1\eta = 1. We will open-source the code upon publication to ensure reproducibility.

References

  • [a] Multistep Distillation of Diffusion Models via Moment Matching, NeurIPS 2024.
  • [b] Simpler Diffusion: 1.5 FID on ImageNet512 with Pixel-space Diffusion, CVPR 2025.
  • [c] Diffusion Fine-tuning with Rewarded Moment Matching Distillation, NeurIPS 2026.
  • [d] Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMD, arXiv.
  • [e] VBench: Comprehensive Benchmark Suite for Video Generative Models, CVPR 2024.

Thanks for the authors’ responses and work. My concerns are resolved, and I will raise my score.

Thank you for acknowledging our work and for raising the score! Thank you again for your time and effort in reviewing our paper.

Reviewer_miUYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces velocity adversarial distillation - a technique for few-step distillation for diffusion/flow-based models that builds on trajectory distribution matching (TDM). Authors argue that TDM is relying on a mismatched estimate of the student score and suggest to work in velocity space instead. The key idea is to align student and teacher velocity fields through an adversarial game with a critic. The critic minimizes alignment with student and maximizes it with teacher, while student maximizes it while keeping critic fixed. The method is validated on several backbones (FLUX, SD, MiniMax) for t2i and t2v tasks and seems to be competitive with established techniques (e.g. DMD2).

Critical strengths
  • Competitive quantitative performance. At 4 NFE, proposed method outperforms all baselines on most metrics on several backbones (SD, FLUX). It also scales to SOTA video model MiniMax H3. This is consequential because this suggests this method generalizes well and is highly useful in practice.
Critical weaknesses
  • There is a bit of over-statement of unification that this method provides, and similarly a mismatch between the theory and the proposed method. For example, in practice critic output is detached, so derivation and actual implementation differ significantly. Ablation study actually indicates that most of the gains are not originating from the novel formulation, but from additional components. This is consequential since theory is claimed as one of the key contributions.
  • Limited qualitative evaluation. There are very few qualitative results provided - there are just ~4 videos in supplementary, and the user study is very limited. This is critical because scaling to the video model is a major claim.
Questions for authors
  • Could you provide details on relationship to DMD-family approaches? The adversarial part seems to be sharing a lot of similarities with that work, so maybe a dedicated section discussing this would be helpful.
  • Why limited number of visual samples? More samples and some reader / viewer guidance on qualitative benefits would be useful.
  • Please clarify the theory/practice mismatch. Ideally ablation should separate the novel formulation from additional components.
3 comments
AuthorsOpenReview ↗

Reply to Reviewer miUY (1/3)

We sincerely thank the reviewer for the constructive comments. We believe that as flow matching becomes increasingly prevalent in pretraining for frontier visual generation models, establishing a unified few-step distillation framework natively in velocity space is a timely and critical problem. Unlike score-based distillation—which relies on adding independent Gaussian noise and thus faces either high variance or distribution mismatch at intermediate steps—working directly with velocities removes this Gaussian constraint, allowing us to flexibly construct low-variance, coupled supervision targets directly along the student's actual sampling chain.

To our knowledge, our work is the first to unify Reverse-KL (DMD/Diff-Instruct), Fisher divergence, and Moment Matching by revealing their shared adversarial nature within a single velocity adversarial game (VAD), while deriving stable new objectives (VA-TDM) that scale to 4-step text-to-image (SD3.5-M, FLUX.2-Klein 4B) and further scale to 33B audio-video generation (MiniMax-H3) with state-of-the-art performance. We address each concern below.

Relationship to DMD-Family Approaches (Q1)

B-form with a detached pathwise update recovers reverse KL—the core distillation loss of the DMD family. Specifically, VAD models a general single-step transition t→st \to s: if t=1t = 1 and s=0s = 0, B-form with a detached critic is equivalent to DMD1 without ODE regression (and Diff-Instruct); if extended to a few-step student via a consistency sampler performing tk→0t_k \to 0 at each step, it recovers DMD2 without its extra GAN loss.

More importantly, VAD offers a new conceptual insight into why these methods work. Estimating the reverse-KL gradient via diffusion scores originated in Diff-Instruct (DI) / Variational Score Distillation. However, DI explicitly positioned itself as a non-adversarial alternative to GANs (avoiding a density-ratio discriminator), and subsequent works such as DMD2 treated reverse KL as non-adversarial and added a separate GAN discriminator on top.

In contrast, our VAD framework directly provides a unified adversarial perspective that explains these methods and brings a new conceptual understanding to the community: your reverse KL / DMD is inherently performing adversarial training in velocity space (where the score/velocity residual is the optimal critic f⋆\mathbf{f}^\star). Recognizing this intrinsic connection opens the door to adapting established GAN stabilization and game-balancing techniques (such as our path/flow-product mixing) to further advance score and velocity distillation. We will add a dedicated discussion section in the Appendix.

AuthorsOpenReview ↗

Reply to Reviewer miUY (2/3)

Expanded Qualitative and Quantitative Video Evaluation (W2 & Q2)

Thanks for your suggestion. We provide more qualitative video samples in the supplementary material, where VA-TDM-H3 (4 NFE) consistently avoids the severe color oversaturation and dark artifacts of Fast-H3 while preserving sharper details and multi-scene temporal consistency. Compared with Fast-H3 at 4 NFE, VA-TDM consistently delivers (1) natural lighting and color balance without oversaturation or harsh contrast artifacts, and (2) sharper high-frequency details and text/UI legibility without motion ghosting or texture smudging.

Furthermore, we extended the anonymized user study on 33B MiniMax-H3 from 10 prompts to 100 prompts, expanding the number of valid human votes per model pair from around 300 to nearly 1,000 (nearly 3,000 votes in total):

Model AModel BWin rate of A
VA-TDM-H3 (4 NFE)Fast-H3 (4 NFE)62.89%
VA-TDM-H3 (4 NFE)Teacher (100 NFE)47.23%
Teacher (100 NFE)Fast-H3 (4 NFE)70.62%

VA-TDM-H3 (4 NFE) achieves a 62.89% win rate against Fast-H3 (4 NFE) and a 47.23% win rate against the 100-NFE teacher, while the teacher achieves a 70.62% win rate against Fast-H3.

We also provide supplementary quantitative results on VBench [a]—a widely adopted, comprehensive benchmark suite evaluating video generation across 16 dimensions of visual quality, temporal consistency, and prompt alignment—as shown below (best in bold, second-best italicized):

MethodNFETotal ↑\uparrowQuality ↑\uparrowSemantic ↑\uparrowSubject consistencyMotion smoothnessDynamic degreeImaging qualityMultiple objectsSpatial relationship
MiniMax-H3 (Base)10084.9785.7281.9495.9599.1376.9468.9190.7980.44
Fast-H3484.1584.8381.4295.3598.7476.6769.6986.2584.42
VA-TDM (Ours)485.0385.8581.7595.8098.9280.8371.0986.2985.03

At 4 NFE, VA-TDM consistently outperforms Fast-H3 across all overall scores (Total 85.03 vs. 84.15, Quality 85.85 vs. 84.83, Semantic 81.75 vs. 81.42) and representative dimensions, and even matches or slightly exceeds the 100-NFE base model in Total and Quality scores (it is worth noting that automated benchmarks might be less sensitive to perceptual artifacts like oversaturation on frontier models like H3).

AuthorsOpenReview ↗

Reply to Reviewer miUY (3/3)

Clarifying the Unification Theory and the Detached Critic (W1 & Q3)

Thank you for the opportunity to clarify the role of stop-gradient. The core of VAD is providing a unified adversarial perspective in velocity space, where velocity construction, generator-loss formulation, and critic gradient flow are explicit design choices that both unify existing methods (Theorem 5) and inspire practical new losses. Our VAD:

  • Recovers the reverse-KL gradient (DMD/Diff-Instruct) via B-form with a detached pathwise update (Lpath,sgB\mathcal{L}_{\mathrm{path,sg}}^{B}).
  • Recovers the practical MMD update via A-form with a detached pathwise update (Lpath,sgA\mathcal{L}_{\mathrm{path,sg}}^{A}).
  • Recovers the exact gradients of MMD's intractable objective and Fisher divergence via A/B-form flow-product updates with a non-detached critic (GfpA/B\mathcal{G}_{\mathrm{fp}}^{A/B}).
  • Motivates a practical new loss: A/B-form flow-product updates with a detached critic (Lfp,sgA/B\mathcal{L}_{\mathrm{fp,sg}}^{A/B}, Eq. (19)), adopted in VA-TDM.

We agree that detaching f(xτ)\mathbf{f}(x_\tau) in Lfp,sgA/B\mathcal{L}_{\mathrm{fp,sg}}^{A/B} is indeed a practical stabilization choice: while a non-detached critic is needed to reproduce exact Fisher/moment-matching gradients for theoretical unification, backpropagating through potentially noisy spatial Jacobians of multi-billion-parameter DiTs (across both the CFG teacher and the online velocity field) can make optimization much more sensitive to hyperparameters and harder to tune. Empirically, under the same hyperparameter setup widely used in previous work (e.g., TDM and DMD2) without extensive tuning, the non-detached variant is prone to training instability and collapse (Fig. 3), whereas detaching the critic offers a simpler, more robust alternative that trains stably out of the box (distilling the 33B DiT directly follows the same hyperparameters as distilling SD3.5-M).

Crucially, under VAD's unified adversarial game, detaching f\mathbf{f} treats the critic as a black-box directional guide without altering the ideal equilibrium: since the optimal critic vanishes when the induced student and target teacher velocities match (fF⋆≡0\mathbf{f}_F^\star \equiv \mathbf{0}, Proposition 3), both detached and non-detached updates vanish at the exact same equilibrium.

We will make this distinction explicit: the non-detached configuration provides the exact Fisher/moment-gradient recovery results, while the detached configuration provides an equilibrium-preserving practical update within the same adversarial formulation. This flexibility supports the practical design of VA-TDM, which stably scales up to 33B SOTA DiT-based audio-video models.

Clarifying Contributions and Isolating VAD's Improvements (W1 & Q3)

We highlight that our contributions are twofold:

  • 1) A unified adversarial view and equilibrium-preserving practical loss: VAD unifies Reverse-KL, Fisher divergence, and Moment Matching within one velocity adversarial game (Theorem 5) while motivating new practical updates (Lfp,sgA/B\mathcal{L}_{\mathrm{fp,sg}}^{A/B}) that can stably scale up to 33B DiT. Although detaching the critic is a practical choice (to improve training stability, Fig. 3), under VAD's adversarial formulation, it preserves the ideal equilibrium, illustrating the flexibility of our framework.
  • 2) Improvements directly driven by VAD formulations and VAD-guided designs: Our ablations isolate clear standalone gains from VAD itself:
    • i) Dual-form game: Without transition-aware estimation (on 4-step SD3.5-Medium), VAD's dual-form game alone raises GenEval from 0.580.58 (B-form/reverse-KL) to 0.620.62 and OCR from 0.530.53 to 0.560.56:

      ConfigurationGenEval ↑\uparrowOCR ↑\uparrowHPSv3 ↑\uparrowLPIPS ↑\uparrow
      λm=1\lambda_m = 1 (A-form) w/o transition-aware field0.600.5411.090.55
      λm=0\lambda_m = 0 (B-form) w/o transition-aware field0.580.5311.130.56
      λm=0.8\lambda_m = 0.8 (Dual-form) w/o transition-aware field0.620.5611.380.57

      Interestingly, even though standalone B-form (λm=0\lambda_m=0) performs worse than A-form (λm=1\lambda_m=1) on GenEval and OCR, mixing a small weight of this empirically weaker B-form (0.20.2) into A-form (0.80.8) actually improves upon A-form across the board. This highlights the effectiveness of the proposed dual-form.

    • ii) Path/flow-product mixing: In Tab. 2, incorporating Lfp,sgA/B\mathcal{L}_{\mathrm{fp,sg}}^{A/B} via path/flow-product mixing further improves GenEval from 0.630.63 to 0.650.65;

    • iii) Regularization-corrected critic: Analytically derived from VAD's formulation rather than an ad-hoc add-on, the regularization-corrected critic further boosts GenEval to 0.670.67 and HPSv3 from 11.4411.44 to 11.8911.89, directly showing how our formulation guides practical gains.

[a] VBench: Comprehensive Benchmark Suite for Video Generative Models, CVPR 2024.

Reviewer_xCHUOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes VA-TDM, which combines the idea of Velocity Adversarial Distillation and trajectory distribution matching.

VA-TDM address several theoretical issues of few-step l in prior works, including TDM and DMD and validates the solution in several base models, including SD3.5, Flux2, and Minimax-H3.

The experiments covers image and video generation, and both qualitative and quantitative results demonstrate the effectiveness of the results.

Critical strengths

1, solid experiments. A good few-step distillation method should work on both image and video generation and this work provides extensive experiments on different base models, of which the size range from several billion to 33B.

2, This paper addresses several critic theoretical issues in prior works.

a, score matching within a subinterval by regressing onto ∇xτlog⁡q(xτ∣xs)\nabla_{\mathbf{x}_\tau} \log q(\mathbf{x}_\tau \mid \mathbf{x}_s) is hard due to the large variance.

b, the critic model needs to track different distribution (in training, the roll-out of generator ends at different step ) in few-step DMD, so there are theoretical flaws in few-step DMD.

Critical weaknesses

1, A-form does not generally estimate the score needed by DMD, so adding it means the combined update is no longer simply minimizing reverse KL. My concern is that even when the student distribution already matches the teacher, A-form may still push the student away from that solution. The paper should explain why combining the two forms is appropriate despite this potential conflict.

2,it is not clear whether minimizing the moment-matching objective leads to the right student distribution. Even when the student distribution already matches the teacher, the moment loss may still be nonzero and push the student away. The final update also uses stop-gradient operations, so it no longer follows the exact gradient of that objective.

3, Using only B-form with the detached pathwise update gives the same reverse-KL gradient as earlier methods such as DMD. The paper adds other training improvements, but Tab2 (the first 3 rows) shows only small gains from combining A-form and B-form. This makes me question how much the dual-form design itself contributes.

Questions for authors

The theoretical derivations is sound under the stated assumptions, but they do not fully justify the final algorithm. In particular, the practical detached update differs from the update in the theory, and the compatibility of the A-form and B-form objectives is not established. The theoretical results therefore explain parts of the method, rather than provide a complete justification for their combination.

7 comments
AuthorsOpenReview ↗

Reply to Reviewer xCHU (1/3)

We sincerely thank the reviewer for the constructive comments. We believe that as flow matching becomes increasingly prevalent in pretraining for frontier visual generation models, establishing a unified few-step distillation framework natively in velocity space is a timely and critical problem. Unlike score-based distillation—which relies on adding independent Gaussian noise and thus faces either high variance or distribution mismatch at intermediate steps—working directly with velocities removes this Gaussian constraint, allowing us to flexibly construct low-variance, coupled supervision targets directly along the student's actual sampling chain.

To our knowledge, our work is the first to unify Reverse-KL (DMD/Diff-Instruct), Fisher divergence, and Moment Matching by revealing their shared adversarial nature within a single velocity adversarial game (VAD), while deriving stable new objectives (VA-TDM) that scale to 4-step text-to-image (SD3.5-M, FLUX.2-Klein 4B) and further scale to 33B audio-video generation (MiniMax-H3) with state-of-the-art performance. We address each concern below.

Compatibility of A-form and B-form (W1, W2 & Questions)

Both forms supervise distribution-level quantities: A-form matches the conditional moment, while B-form matches the perturbed marginal distribution. Intuitively, exact distribution matching naturally entails moment matching, suggesting that B-form and A-form are compatible.

  • Exact Equivalence for the optimal desired ultimate sampling distribution. At the final step (s=0s=0), our few-step generator directly predicts x^0\hat{\mathbf{x}}_0, where A-form and B-form are equivalent (Theorem 5(ii)); thus, the optimal final sample distribution (our ultimate desired target) is to match the teacher distribution.
  • Common Global Optimum for Intermediate Transitions. For general intermediate transitions t→st \to s (s>0s>0), as shown in MMD ([a], Appendix D in [a]), under sufficient student capacity, moment matching (connected with A-form) and score matching (connected with B-form) share a common global optimum at the target teacher distribution.
  • Practical Trade-off under Finite Capacity. In practice, however, a finite-capacity few-step student cannot converge to the optimal solution, requiring a practical trade-off between which distributional information to preserve and the corresponding learning variance. A-form provides a more relaxed, lower-variance distributional supervision than B-form (which suffers from high online target variance when s>0s>0). Mixing them via λm∈(0,1)\lambda_m \in (0,1) in dual-form balances learning variance and distributional information preservation. Because finite-capacity distillation ultimately hinges on how well the trade-off translates into optimization quality, we believe practical effectiveness is the most important criterion: our experiments show that A-form alone is effective, the A/B-form hybrid further improves performance, and MMD (recovered by our A-form detached pathwise update) [a] and its variants have also proven effective across a wide range of scenarios [b, c, d].
AuthorsOpenReview ↗

Reply to Reviewer xCHU (2/3)

Clarifying the Detached Critic (W2 & Questions)

Thank you for the opportunity to clarify the role of stop-gradient. The core of VAD is providing a unified adversarial perspective in velocity space, where velocity construction, generator-loss formulation, and critic gradient flow are explicit design choices that both unify existing methods (Theorem 5) and inspire practical new losses. Our VAD:

  • Recovers the reverse-KL gradient (DMD/Diff-Instruct) via B-form with a detached pathwise update (Lpath,sgB\mathcal{L}_{\mathrm{path,sg}}^{B}).
  • Recovers the practical MMD update via A-form with a detached pathwise update (Lpath,sgA\mathcal{L}_{\mathrm{path,sg}}^{A}).
  • Recovers the exact gradients of MMD's intractable objective and Fisher divergence via A/B-form flow-product updates with a non-detached critic (GfpA/B\mathcal{G}_{\mathrm{fp}}^{A/B}).
  • Motivates a practical new loss: A/B-form flow-product updates with a detached critic (Lfp,sgA/B\mathcal{L}_{\mathrm{fp,sg}}^{A/B}, Eq. (19)), adopted in VA-TDM.

We agree that detaching the critic $\mathbf{f}(x_\tau) $ in Lfp,sgA/B\mathcal{L}_{\mathrm{fp,sg}}^{A/B} is indeed a practical stabilization choice: while a non-detached critic is needed to reproduce exact Fisher/moment-matching gradients for theoretical unification, backpropagating through potentially noisy spatial Jacobians of multi-billion-parameter DiTs (across both the CFG teacher and the online velocity field) can make optimization much more sensitive to hyperparameters and harder to tune. Empirically, under the same hyperparameter setup widely used in previous work (e.g., TDM and DMD2) without extensive tuning, the non-detached variant is prone to training instability and collapse (Fig. 3), whereas detaching the critic offers a simpler, more robust alternative that trains stably out of the box (distilling the 33B DiT directly follows the same hyperparameters as distilling SD3.5-M).

Crucially, under VAD's unified adversarial game, detaching f\mathbf{f} treats the critic as a black-box directional guide without altering the ideal equilibrium: since the optimal critic vanishes when the induced student and target teacher velocities match (fF⋆≡0\mathbf{f}_F^\star \equiv \mathbf{0}, Proposition 3), both detached and non-detached updates vanish at the exact same equilibrium.

We will make this distinction explicit: the non-detached configuration provides the exact Fisher/moment-gradient recovery results, while the detached configuration provides an equilibrium-preserving practical update within the same adversarial formulation. This flexibility supports the practical design of VA-TDM, which stably scales up to 33B SOTA DiT-based audio-video models.

Specifically, on the 33B MiniMax-H3 model, we further validated VA-TDM (4 NFE) with an expanded 100-prompt human study (extending valid human votes per model pair from around 300 to nearly 1,000, totaling nearly 3,000 votes, and achieving a 62.89% win rate over 4-NFE Fast-H3 and 47.23% against the 100-NFE teacher) and new VBench quantitative evaluations, where VA-TDM outperforms Fast-H3 across all overall scores (Total 85.03 vs. 84.15, Quality 85.85 vs. 84.83, Semantic 81.75 vs. 81.42) and even slightly exceeds the 100-NFE teacher in Total (84.97) and Quality (85.72) scores. Please refer to our General Response for the detailed tables and results.

AuthorsOpenReview ↗

Reply to Reviewer xCHU (3/3)

Clarifying the Improvement of Dual-Form Supervision (W3)

We highlight that both dual-form supervision (λm∈(0,1)\lambda_m \in (0,1)) and transition-aware field estimation (hψ(xτ,τ,s)h_\psi(\mathbf{x}_\tau, \tau, s)) are principled core designs of VA-TDM, jointly motivated by accurately learning online velocity critics across intermediate transitions (t→s,s>0t \to s, s>0): (i) Transition-aware conditioning resolves cross-step distribution confusion when supervision intervals overlap—different few-step transitions induce distinct noisy distributions even at the same τ\tau; and (ii) Dual-form supervision reconciles learning variance and distributional information preservation at s>0s>0—pairing A-form's low-variance moment learning with B-form's full marginal information, since neither form enjoys both benefits alone (Sec. 3.3).

Because both designs improve online critic tracking from complementary angles, the first two rows in Tab. 2 already include transition-aware estimation, bringing single-form tracking close to saturation.

To isolate the standalone effect of dual-form supervision, we conduct an ablation without the transition-aware field as shown below:

ConfigurationGenEval ↑\uparrowOCR ↑\uparrowHPSv3 ↑\uparrowLPIPS ↑\uparrow
λm=1\lambda_m = 1 (A-form) w/o transition-aware field0.600.5411.090.55
λm=0\lambda_m = 0 (B-form) w/o transition-aware field0.580.5311.130.56
λm=0.8\lambda_m = 0.8 (Dual-form) w/o transition-aware field0.620.5611.380.57

Here, our dual-form (λm=0.8\lambda_m = 0.8) exhibits clear, consistent gains across all metrics. Interestingly, even though standalone B-form (λm=0\lambda_m = 0) performs worse than A-form (λm=1\lambda_m = 1) on GenEval and OCR, mixing a small weight of this empirically weaker B-form (0.20.2) into A-form (0.80.8) actually improves upon A-form across the board. This aligns well with our analysis above.

References

  • [a] Multistep Distillation of Diffusion Models via Moment Matching, NeurIPS 2024.
  • [b] Simpler Diffusion: 1.5 FID on ImageNet512 with Pixel-space Diffusion, CVPR 2025.
  • [c] Diffusion Fine-tuning with Rewarded Moment Matching Distillation, NeurIPS 2026.
  • [d] Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMD, arXiv.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.