- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes Data-Forcing Distillation (DFD), a post-training modification of distribution matching distillation for four-step video generation. On randomly selected updates, the frozen teacher scores a noised reference video rather than the student's generated video, while the auxiliary fake-score model continues to score generated samples. DFD uses curated synthetic reference videos and removes the auxiliary GAN loss. Experiments cover Wan2.1 text-to-video and autoregressive generation and Cosmos image-to-video generation, with ablations on GAN supervision, update mixing, initialization, and batch size. The reported results show improved quality and partial recovery of diversity after a short continuation of an already distilled model.
Critical strengths
-
The reported quality and diversity gains across text-to-video, image-to-video, and autoregressive generation suggest practical utility (Tables 1–4).
-
The teacher-input substitution is straightforward to implement in an existing DMD pipeline and adds no learned component.
Despite the diverse experiments and broad coverage, I regret to find that the proposed method suffers from notable theoretical flaws and that the proofs given in the paper contain several errors.
Critical weaknesses
-
Zero-mean score discrepancy does not imply a zero parameter update at the target. Appendix D.1 establishes , whereas Eq. (4) uses $ \mathbb{E} [J_\theta^\top\Delta] $. However, does not indicate . For and , Eq. (15) gives at the exact match , including the chain-rule factor. Thus gradient descent increases variance away from the target, even with exact scores and the prescribed shared noise. Mixing with RKL loss retains this bias for any .
-
The reference-score contribution vanishes in the idealized expected update -- DFD loss does not distill. Under the independent sampling in Eqs. (13)-(15), is independent of the generator Jacobian. An exact reference score has zero mean under its own distribution, so for every . Pure DFD therefore follows a negative-entropy gradient of the noised student distribution, and is not connected to distillation.
-
DMD2 initialization does not establish the claimed small-error regime. Section 3.3 asserts that DMD2 pretraining implicitly satisfies Eq. (6) by naturally keeping the independent-pair MSE small. Distribution matching does not minimize this loss: its minimizer is the deterministic conditional mean, not the reference distribution. Better distribution matching even increases this MSE. The initialization ablation (Appendix C, Fig. 6(c)) reports convergence failure without pretraining, this is because the DFD loss cannot serve as a distillation loss—it is merely pure entropy maximization.
Questions for authors
See Weaknesses.
6 comments
New Proposition 2 (Appendix D.3). Assume exact denoisers without CFG. Let and let be its Bregman divergence. Write , and . Then is convex, , and for every
Proof.
(i) Decomposition. Adding and subtracting gives . So the left-hand side equals .
(ii) First term. By Tweedie's formula, . Since , the standard DMD identity gives .
(iii) Second term. By Tweedie's formula, , and . So is convex and . Since does not depend on , . The distribution of does not depend on , so gradient and expectation commute, which gives the second term.
The proof is pointwise in . It holds for the paired of our pipeline, and also for independent .
What the two terms mean.
- Distribution matching (standard DMD). The first term is the reverse KL of DMD, kept in full for every . It is an expectation over the student's own samples , and it is mode-seeking: it moves each sample toward a high-density region of the teacher.
- Paired discrepancy. The second term is an expectation over the joint distribution , . It compares each generation with the reference video its own input was re-noised from, and it vanishes only when the generation reproduces that reference. By the mean-value theorem, with . So it is a regression of onto in the teacher's metric, and it is mean-seeking: its minimizer is .
DFD therefore adds a mean-seeking term to the mode-seeking DMD objective, and sets the balance between them; is DMD.
The reviewer's Gaussian example, for the method we run. We refer to this part as the Gaussian example below; it is Remark 1(iii) of Appendix D.3. Take reference data , a teacher , and a student , where is the student input standardized to unit variance. Define
the strength of the pairing: when is independent of , and when determines . With , the paired term of Proposition 2 is exactly .
Mean. The KL term pulls the mean toward the teacher mean , and the paired term pulls it toward the data mean . Setting the gradient to zero gives
In the reviewer's example , both terms point to 0, so and the mean is correct. When the teacher is shifted (for example by CFG), DFD moves the mean from the teacher toward the data, which DMD cannot do.
Scale. Take , , . The combined update is
which we checked by Monte Carlo. The DMD term alone is stationary at the data scale . The paired term alone is stationary at , the scale of . The combined stationary point lies in , and sets where it lies: gives DMD. The balance also depends on . Averaged over with , the stationary variance is 0.54 of the data variance at our last student step (, ) and 0.11 at the first (, ). This is why we use DFD as a short post-training stage from DMD2 and not as a standalone objective.
Where diversity comes from. The reverse-KL term is an expectation over the student's own samples, so a mode the student has dropped receives no gradient from it. The paired term is an expectation over : a reference mode with mass contributes with weight whether or not the student covers it. For student inputs that still carry information about the mode of , this term pulls the corresponding generations toward that mode. Diversity therefore enters through the sampling of from the diverse reference set.
Thank you for the careful reading of our theory. Your three derivations are correct for the equations as we wrote them. The problem is that our written equations did not describe the method we actually run, in two places. We first correct these two places, and then answer each weakness for the corrected method. All changes are marked in red in the revised paper, and the new videos are in the updated supplementary material.
Two corrections to our presentation
C1. The student input and the reference sample are paired. In our experiments, one student update is:
So the student input is a re-noised reference video, not a backward simulation from pure noise, and x and z are not independent: z∼qs(⋅∣x). The submission wrote z∼N(0,I) independent of x, which does not match our implementation; we thank the reviewer for prompting us to correct this. Eq. 4 and Algorithm 1 of the revision now state the re-noising, and every expectation below is over the joint distribution of (x,z).
C2. The update uses x0-predictions, not raw scores. As in other DMD implementations, the update direction is a difference of denoiser outputs D(⋅). Let x~t=αtGθ(z,c)+σtϵ and xt=αtx+σtϵ with shared ϵ. Then
ΔDMD=Dfake(x~t)−Dref(x~t),ΔDFD=Dfake(x~t)−Dref(xt).
By Tweedie's formula, D(y)=(y+σt2s(y))/αt, so
ΔDFD=αtσt2(sfake(x~t)−sref(xt))+(Gθ(z,c)−x).
For a DMD step, the x0 form and the score form agree up to a positive factor. For a DFD step, the x0 form has one more term, the paired residual Gθ(z,c)−x. Our Eq. (15) left this term out.