acceptodds
Under review as a conference paper at ICLR 2027

Data-Forcing Distillation: Restoring Diversity and Fidelity in Few-Step Video Generation

Abstract

Recent progress has shown promise in distilling multi-step video diffusion models into efficient few-step students. Among them, Distribution Matching Distillation (DMD) and its successor DMD2 achieve strong generation quality and fast convergence. However, due to the nature of the reverse Kullback–Leibler (KL) objective, these methods can exhibit two persistent failure modes: reduced sample diversity and visibly over-saturated outputs that deviate from real-video appearance. In this work, we propose *Data-Forcing Distillation* (DFD), a simple post-training framework that improves DMD-based students *with only a single line of code change*. At its core is the *teacher score discrepancy*, which guides the student toward the reference-data distribution by pulling probability mass toward underrepresented modes while suppressing problematic modes that are inconsistent with the reference data. We provide an in-depth theoretical analysis of our framework and evaluate it on text-to-video, image-to-video, and autoregressive video generation. With only *100–300 steps* of finetuning, DFD improves both sample diversity and visual quality in text-to-video and autoregressive generation, while improving visual quality and temporal coherence in image-to-video generation. These results show that DFD can mitigate key failure modes of few-step video generators across different generation settings. We refer readers to the supplementary material for full video comparisons.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video diffusion model, distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tnv4OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Data-Forcing Distillation (DFD), a post-training modification of distribution matching distillation for four-step video generation. On randomly selected updates, the frozen teacher scores a noised reference video rather than the student's generated video, while the auxiliary fake-score model continues to score generated samples. DFD uses curated synthetic reference videos and removes the auxiliary GAN loss. Experiments cover Wan2.1 text-to-video and autoregressive generation and Cosmos image-to-video generation, with ablations on GAN supervision, update mixing, initialization, and batch size. The reported results show improved quality and partial recovery of diversity after a short continuation of an already distilled model.

Critical strengths
  • The reported quality and diversity gains across text-to-video, image-to-video, and autoregressive generation suggest practical utility (Tables 1–4).

  • The teacher-input substitution is straightforward to implement in an existing DMD pipeline and adds no learned component.

Despite the diverse experiments and broad coverage, I regret to find that the proposed method suffers from notable theoretical flaws and that the proofs given in the paper contain several errors.

Critical weaknesses
  • Zero-mean score discrepancy does not imply a zero parameter update at the target. Appendix D.1 establishes E[Δ]=0\mathbb{E}[\Delta]=0, whereas Eq. (4) uses $ \mathbb{E} [J_\theta^\top\Delta] $. However, E[Δ]=0\mathbb{E}[\Delta]=0 does not indicate E[Jθ⊤Δ]=0\mathbb{E}[J_\theta^\top\Delta]=0. For prefer=N(0,1)p_{\rm refer}=\mathcal{N}(0,1) and Ga(z)=azG_a(z)=az, Eq. (15) gives gDFD=−αt2/(αt2+σt2)≠0g_{\rm DFD}=-\alpha_t^2/(\alpha_t^2+\sigma_t^2)\ne0 at the exact match a=1a=1, including the chain-rule factor. Thus gradient descent increases variance away from the target, even with exact scores and the prescribed shared noise. Mixing with RKL loss retains this bias for any w>0w>0.

  • The reference-score contribution vanishes in the idealized expected update -- DFD loss does not distill. Under the independent sampling in Eqs. (13)-(15), xtx_t is independent of the generator Jacobian. An exact reference score has zero mean under its own distribution, so E[Jθ⊤srefer(xt)]=E[Jθ]⊤E[srefer(xt)]=0\mathbb{E}[J_\theta^\top s_{\rm refer}(x_t)]=\mathbb{E}[J_\theta]^\top\mathbb{E}[s_{\rm refer}(x_t)]=0 for every θ\theta. Pure DFD therefore follows a negative-entropy gradient of the noised student distribution, and is not connected to distillation.

  • DMD2 initialization does not establish the claimed small-error regime. Section 3.3 asserts that DMD2 pretraining implicitly satisfies Eq. (6) by naturally keeping the independent-pair MSE small. Distribution matching does not minimize this loss: its minimizer is the deterministic conditional mean, not the reference distribution. Better distribution matching even increases this MSE. The initialization ablation (Appendix C, Fig. 6(c)) reports convergence failure without pretraining, this is because the DFD loss cannot serve as a distillation loss—it is merely pure entropy maximization.

Questions for authors

See Weaknesses.

6 comments

Thank you for the careful reading of our theory. Your three derivations are correct for the equations as we wrote them. The problem is that our written equations did not describe the method we actually run, in two places. We first correct these two places, and then answer each weakness for the corrected method. All changes are marked in red in the revised paper, and the new videos are in the updated supplementary material.


Two corrections to our presentation

C1. The student input and the reference sample are paired. In our experiments, one student update is:

  1. Take a reference video x∼pref(⋅∣c)x\sim p_{\mathrm{ref}}(\cdot\mid c).
  2. Re-noise it to a student timestep s∈0.999,0.937,0.833,0.624s\in\\{0.999,0.937,0.833,0.624\\}: z=αsx+σsηz=\alpha_sx+\sigma_s\eta with η∼N(0,I)\eta\sim\mathcal{N}(0,I). We write z∼qs(⋅∣x)z\sim q_s(\cdot\mid x).
  3. Run the student once: Gθ(z,c)G_\theta(z,c).
  4. Sample a DMD noise level tt and one noise ϵ\epsilon. Form x~t=αtGθ(z,c)+σtϵ\tilde{x}_t=\alpha_tG_\theta(z,c)+\sigma_t\epsilon and xt=αtx+σtϵx_t=\alpha_tx+\sigma_t\epsilon.
  5. Evaluate the fake score at x~t\tilde{x}_t. Evaluate the teacher at x~t\tilde{x}_t in a DMD step, and at xtx_t in a DFD step, which is chosen with probability ww.

So the student input is a re-noised reference video, not a backward simulation from pure noise, and xx and zz are not independent: z∼qs(⋅∣x)z\sim q_s(\cdot\mid x). The submission wrote z∼N(0,I)z\sim\mathcal{N}(0,I) independent of xx, which does not match our implementation; we thank the reviewer for prompting us to correct this. Eq. 4 and Algorithm 1 of the revision now state the re-noising, and every expectation below is over the joint distribution of (x,z)(x,z).

C2. The update uses x0x_0-predictions, not raw scores. As in other DMD implementations, the update direction is a difference of denoiser outputs D(⋅)D(\cdot). Let x~t=αtGθ(z,c)+σtϵ\tilde{x}_t=\alpha_tG_\theta(z,c)+\sigma_t\epsilon and xt=αtx+σtϵx_t=\alpha_tx+\sigma_t\epsilon with shared ϵ\epsilon. Then

ΔDMD=Dfake(x~t)−Dref(x~t),ΔDFD=Dfake(x~t)−Dref(xt).\Delta^{\mathrm{DMD}}=D_{\mathrm{fake}}(\tilde{x}_t)-D_{\mathrm{ref}}(\tilde{x}_t),\qquad \Delta^{\mathrm{DFD}}=D_{\mathrm{fake}}(\tilde{x}_t)-D_{\mathrm{ref}}(x_t).

By Tweedie's formula, D(y)=(y+σt2s(y))/αtD(y)=(y+\sigma_t^2s(y))/\alpha_t, so

ΔDFD=σt2αt(sfake(x~t)−sref(xt))+(Gθ(z,c)−x).\Delta^{\mathrm{DFD}}=\frac{\sigma_t^2}{\alpha_t}\big(s_{\mathrm{fake}}(\tilde{x}_t)-s_{\mathrm{ref}}(x_t)\big)+\big(G_\theta(z,c)-x\big).

For a DMD step, the x0x_0 form and the score form agree up to a positive factor. For a DFD step, the x0x_0 form has one more term, the paired residual Gθ(z,c)−xG_\theta(z,c)-x. Our Eq. (15) left this term out.

New Proposition 2 (Appendix D.3). Assume exact denoisers without CFG. Let ϕt(y)=(12∣y∣2+σt2log⁡pref,t(y))/αt\phi_t(y)=\big(\tfrac12\\|y\\|^2+\sigma_t^2\log p_{\mathrm{ref},t}(y)\big)/\alpha_t and let Bt(u,v)=ϕt(u)−ϕt(v)−⟨∇ϕt(v),u−v⟩\mathcal{B}_t(u,v)=\phi_t(u)-\phi_t(v)-\langle\nabla\phi_t(v),u-v\rangle be its Bregman divergence. Write J=∇θGθ(z,c)J=\nabla_\theta G_\theta(z,c), gDMD=E[J⊤ΔDMD]g_{\mathrm{DMD}}=\mathbb{E}[J^\top\Delta^{\mathrm{DMD}}] and gDFD=E[J⊤ΔDFD]g_{\mathrm{DFD}}=\mathbb{E}[J^\top\Delta^{\mathrm{DFD}}]. Then ϕt\phi_t is convex, Bt≥0\mathcal{B}_t\ge0, and for every w∈[0,1]w\in[0,1]

(1−w),gDMD+w,gDFD=∇θ[σt2αt2DKL(pθ,t,∣,pref,t)⏟distribution matching (DMD)+wαt,Ex,z,ϵ,Bt(x~t,xt)⏟paired discrepancy].(1-w)\\,g_{\mathrm{DMD}}+w\\,g_{\mathrm{DFD}}=\nabla_\theta\Big[\underbrace{\frac{\sigma_t^2}{\alpha_t^2}D_{\mathrm{KL}}\big(p_{\theta,t}\\,\\|\\,p_{\mathrm{ref},t}\big)}_{\text{distribution matching (DMD)}}+\underbrace{\frac{w}{\alpha_t}\\,\mathbb{E}_{x,z,\epsilon}\\,\mathcal{B}_t(\tilde{x}_t,x_t)}_{\text{paired discrepancy}}\Big].

Proof.

(i) Decomposition. Adding and subtracting Dref(x~t)D_{\mathrm{ref}}(\tilde{x}_t) gives ΔDFD=ΔDMD+(Dref(x~t)−Dref(xt))\Delta^{\mathrm{DFD}}=\Delta^{\mathrm{DMD}}+\big(D_{\mathrm{ref}}(\tilde{x}_t)-D_{\mathrm{ref}}(x_t)\big). So the left-hand side equals gDMD+w,E[J⊤(Dref(x~t)−Dref(xt))]g_{\mathrm{DMD}}+w\\,\mathbb{E}\big[J^\top\big(D_{\mathrm{ref}}(\tilde{x}_t)-D_{\mathrm{ref}}(x_t)\big)\big].

(ii) First term. By Tweedie's formula, ΔDMD=σt2αt(sfake−sref)(x~t)\Delta^{\mathrm{DMD}}=\frac{\sigma_t^2}{\alpha_t}(s_{\mathrm{fake}}-s_{\mathrm{ref}})(\tilde{x}_t). Since ∇θx~t=αtJ\nabla_\theta\tilde{x}_t=\alpha_tJ, the standard DMD identity ∇θDKL(pθ,t∣pref,t)=E[αtJ⊤(sfake−sref)(x~t)]\nabla_\theta D_{\mathrm{KL}}(p_{\theta,t}\\|p_{\mathrm{ref},t})=\mathbb{E}\big[\alpha_tJ^\top(s_{\mathrm{fake}}-s_{\mathrm{ref}})(\tilde{x}_t)\big] gives gDMD=σt2αt2∇θDKL(pθ,t∣pref,t)g_{\mathrm{DMD}}=\frac{\sigma_t^2}{\alpha_t^2}\nabla_\theta D_{\mathrm{KL}}(p_{\theta,t}\\|p_{\mathrm{ref},t}).

(iii) Second term. By Tweedie's formula, ∇ϕt(y)=(y+σt2∇log⁡pref,t(y))/αt=Dref(y)\nabla\phi_t(y)=\big(y+\sigma_t^2\nabla\log p_{\mathrm{ref},t}(y)\big)/\alpha_t=D_{\mathrm{ref}}(y), and ∇2ϕt(y)=∇Dref(y)=αtσt2Cov[x∣xt=y]⪰0\nabla^2\phi_t(y)=\nabla D_{\mathrm{ref}}(y)=\frac{\alpha_t}{\sigma_t^2}\mathrm{Cov}[x\mid x_t=y]\succeq0. So ϕt\phi_t is convex and Bt≥0\mathcal{B}_t\ge0. Since xtx_t does not depend on θ\theta, ∇θBt(x~t,xt)=αtJ⊤(∇ϕt(x~t)−∇ϕt(xt))=αtJ⊤(Dref(x~t)−Dref(xt))\nabla_\theta\mathcal{B}_t(\tilde{x}_t,x_t)=\alpha_tJ^\top\big(\nabla\phi_t(\tilde{x}_t)-\nabla\phi_t(x_t)\big)=\alpha_tJ^\top\big(D_{\mathrm{ref}}(\tilde{x}_t)-D_{\mathrm{ref}}(x_t)\big). The distribution of (x,z,ϵ)(x,z,\epsilon) does not depend on θ\theta, so gradient and expectation commute, which gives the second term. □\square

The proof is pointwise in (x,z,ϵ)(x,z,\epsilon). It holds for the paired (x,z)(x,z) of our pipeline, and also for independent (x,z)(x,z).

What the two terms mean.

  • Distribution matching (standard DMD). The first term is the reverse KL of DMD, kept in full for every ww. It is an expectation over the student's own samples x~t∼pθ,t\tilde{x}_t\sim p_{\theta,t}, and it is mode-seeking: it moves each sample toward a high-density region of the teacher.
  • Paired discrepancy. The second term is an expectation over the joint distribution x∼pref(⋅∣c)x\sim p_{\mathrm{ref}}(\cdot\mid c), z∼qs(⋅∣x)z\sim q_s(\cdot\mid x). It compares each generation with the reference video its own input was re-noised from, and it vanishes only when the generation reproduces that reference. By the mean-value theorem, Dref(x~t)−Dref(xt)=Mˉt,(Gθ(z,c)−x)D_{\mathrm{ref}}(\tilde{x}_t)-D_{\mathrm{ref}}(x_t)=\bar{M}_t\\,(G_\theta(z,c)-x) with Mˉt=αt2σt2∫01Cov[x∣xt+τ(x~t−xt)],dτ⪰0\bar{M}_t=\frac{\alpha_t^2}{\sigma_t^2}\int_0^1\mathrm{Cov}\big[x\mid x_t+\tau(\tilde{x}_t-x_t)\big]\\,d\tau\succeq0. So it is a regression of Gθ(z,c)G_\theta(z,c) onto xx in the teacher's metric, and it is mean-seeking: its minimizer is E[x∣z]\mathbb{E}[x\mid z].

DFD therefore adds a mean-seeking term to the mode-seeking DMD objective, and ww sets the balance between them; w=0w=0 is DMD.

The reviewer's Gaussian example, for the method we run. We refer to this part as the Gaussian example below; it is Remark 1(iii) of Appendix D.3. Take reference data N(μr,sx2)\mathcal{N}(\mu_r,s_x^2), a teacher N(μT,sx2)\mathcal{N}(\mu_T,s_x^2), and a student G=az^+bG=a\hat{z}+b, where z^\hat{z} is the student input standardized to unit variance. Define

ρ=Corr(z^,x)=αssxαs2sx2+σs2∈[0,1],\rho=\mathrm{Corr}(\hat{z},x)=\frac{\alpha_s s_x}{\sqrt{\alpha_s^2s_x^2+\sigma_s^2}}\in[0,1],

the strength of the pairing: ρ=0\rho=0 when zz is independent of xx, and ρ→1\rho\to1 when zz determines xx. With vt=αt2sx2+σt2v_t=\alpha_t^2s_x^2+\sigma_t^2, the paired term of Proposition 2 is exactly w,αt2sx2vt⋅12E(G−x)2w\\,\frac{\alpha_t^2s_x^2}{v_t}\cdot\tfrac12\mathbb{E}(G-x)^2.

Mean. The KL term pulls the mean toward the teacher mean μT\mu_T, and the paired term pulls it toward the data mean μr\mu_r. Setting the gradient to zero gives

mG∗=σt2μT+w,αt2sx2,μrσt2+w,αt2sx2.m_G^\ast=\frac{\sigma_t^2\mu_T+w\\,\alpha_t^2s_x^2\\,\mu_r}{\sigma_t^2+w\\,\alpha_t^2s_x^2}.

In the reviewer's example μT=μr=0\mu_T=\mu_r=0, both terms point to 0, so mG∗=0m_G^\ast=0 and the mean is correct. When the teacher is shifted (for example by CFG), DFD moves the mean from the teacher toward the data, which DMD cannot do.

Scale. Take sx=1s_x=1, μT=μr=0\mu_T=\mu_r=0, b=0b=0. The combined update is

g(a)=αt2vt[σt2,a,(a2−1)αt2a2+σt2⏟DMD, zero at a=1+w,(a−ρ)⏟paired, zero at a=ρ],g(1)=w,αt2(1−ρ)vt>0,g(a)=\frac{\alpha_t^2}{v_t}\Big[\underbrace{\frac{\sigma_t^2\\,a\\,(a^2-1)}{\alpha_t^2a^2+\sigma_t^2}}_{\text{DMD, zero at }a=1}+\underbrace{w\\,(a-\rho)}_{\text{paired, zero at }a=\rho}\Big],\qquad g(1)=\frac{w\\,\alpha_t^2(1-\rho)}{v_t} > 0,

which we checked by Monte Carlo. The DMD term alone is stationary at the data scale a=1a=1. The paired term alone is stationary at a=ρa=\rho, the scale of E[x∣z]\mathbb{E}[x\mid z]. The combined stationary point lies in (ρ,1)(\rho,1), and ww sets where it lies: w→0w\to0 gives DMD. The balance also depends on ρ\rho. Averaged over tt with w=12w=\tfrac12, the stationary variance is 0.54 of the data variance at our last student step (s=0.624s=0.624, ρ=0.52\rho=0.52) and 0.11 at the first (s=0.999s=0.999, ρ≈0\rho\approx0). This is why we use DFD as a short post-training stage from DMD2 and not as a standalone objective.

Where diversity comes from. The reverse-KL term is an expectation over the student's own samples, so a mode the student has dropped receives no gradient from it. The paired term is an expectation over x∼prefx\sim p_{\mathrm{ref}}: a reference mode with mass πk\pi_k contributes with weight πk\pi_k whether or not the student covers it. For student inputs that still carry information about the mode of xx, this term pulls the corresponding generations toward that mode. Diversity therefore enters through the sampling of xx from the diverse reference set.

Reviewer_trxJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Data-Forcing Distillation (DFD), a post-training method for DMD-based distilled video generators. The core idea is to explicitly integrate real data into the distribution matching target. During training, the teacher is evaluated on a real reference video rather than the student-generated video, while the fake-score model remains evaluated on the generated sample. DFD mixes these updates with ordinary DMD updates.

Critical strengths
  • The writing and ideas of this paper are easy to understand.

  • The proposed method is simple to integrate into existing DMD pipelines. And it mitigates the diversity reduction issues caused by the Reverse KL mode in DMD, by introducing a teacher score difference term on real samples into the original DMD gradient.

Critical weaknesses
  • The robustness of the short post-training protocol is unclear, since the optimal 100–300 fine-tuning steps appear sensitive to the base model, batch size, and other hyperparameters, likely requiring re-selection when these change and undermining the claimed simplicity and reproducibility.

  • DFD treats all noise levels uniformly despite their different properties. In particular, introducing a different reference video at low noise may cause noisy or conflicting gradients and destabilize training.

  • The loss appears heuristic and is not sufficiently justified by the DMD formulation. Moreover, improving diversity requires references covering different modes, yet large reference–generated differences may make the updates unreliable.

Questions for authors
  • Since Causal Forcing is included as a baseline, would the authors consider also applying DFD to it to assess whether the gains extend beyond the Self Forcing framework?

  • Since DFD’s main claim is restoring mode coverage, why does the I2V evaluation omit diversity metrics. A fixed input frame does not determine future camera motion, so the camera diversity metric already used for T2V may also be applicable to I2V.

5 comments

We thank the reviewer for the careful reading, and for recognizing the clarity of the paper and the simplicity and effectiveness of DFD in mitigating the diversity reduction. Below we address the main concerns. All changes are marked in red in the revised paper, and the new videos are in the updated supplementary material.

W1: Re-selection of the fine-tuning steps for DFD

We thank the reviewer for this comment and agree that the best number of DFD steps depends on the base model, the batch size, and other hyperparameters. Our new analysis (W3) shows why we select it instead of training to convergence. In a 1-D Gaussian model of DFD, the stationary point of the combined update has a smaller variance than the data: it lies between the distribution-matching solution of the DMD term and the mean-seeking solution of the paired term. Starting from DMD2, the first DFD steps bring the reference data into the update: the mean moves from the teacher toward the data, and reference modes receive gradient. Training much longer moves the student toward the contracted stationary point. So we do not wait for convergence; we pick the checkpoint within 100–300 steps that balances the two.

We would also like to respectfully note that this is not specific to DFD. Distribution-matching methods such as DMD and DMD2 also degrade after long training, even though their idealized objective has the teacher distribution as a fixed point. In high dimensions the fake score lags behind the generator and the optimization is imperfect, so mode collapse and over-saturation build up as training continues. For DMD too, the number of distillation steps that best balances mode collapse and video quality depends on the base model, the batch size, and the task, and it is chosen by early stopping in the same way. To make this selection easy to reproduce, we will open-source all our code, data, checkpoints, and the detailed checkpoint-selection workflow.

W2: DFD treats all noise levels uniformly, and a different reference video at low noise may cause instability

We thank the reviewer for raising this important point. We realize that the submission did not make two aspects of our implementation clear; with them in place, we believe the concern does not arise in DFD as we run it.

1. The reference is not a different video. In our settings we do not use backward simulation. Each update re-noises a reference video x∼pref(⋅∣c)x\sim p_{\mathrm{ref}}(\cdot\mid c) into the student input, z=αsx+σsηz=\alpha_sx+\sigma_s\eta at a student timestep ss, and the teacher is evaluated on the same xx (Eq. 4 in the revision). So the reference that the teacher sees is always the video that the student's own input came from, and the two stay correlated at every student step. For unit data variance, Corr(z^,x)=αs/αs2+σs2\mathrm{Corr}(\hat z,x)=\alpha_s/\sqrt{\alpha_s^2+\sigma_s^2} is 0.52 at s=0.624s=0.624 and smallest at the first step, s=0.999s=0.999 (0.001).

2. The x0x_0 form turns the low-noise term into a smooth regression. At low tt the teacher's denoiser nearly returns its input, Dref(y)≈y/αtD_{\mathrm{ref}}(y)\approx y/\alpha_t, so the DFD term is Dref(x~t)−Dref(xt)≈Gθ(z,c)−xD_{\mathrm{ref}}(\tilde{x}_t)-D_{\mathrm{ref}}(x_t)\approx G_\theta(z,c)-x, the gradient of 12∣Gθ(z,c)−x∣2\tfrac12\\|G_\theta(z,c)-x\\|^2. This gradient is neither noisy nor conflicting:

  • Not noisy. Its expectation over the joint (x,z)(x,z) is Gθ(z,c)−E[x∣z]G_\theta(z,c)-\mathbb{E}[x\mid z], a pull toward the conditional mean of the references that are consistent with the input; the variation from pair to pair averages out, as in any regression. Even at s=0.999s=0.999, where the correlation is smallest, the target is the prompt-level average, not one unrelated video.
  • Not conflicting. By Proposition 2 (W3), it is part of the gradient of a single objective together with the reverse-KL term, so the two terms trade off, with ww setting the balance.

DFD is also not uniform across noise levels. For Gaussian data with unit variance, the reverse-KL part has weight σt2/αt2\sigma_t^2/\alpha_t^2 and the paired part has weight w,αt2/(αt2+σt2)w\\,\alpha_t^2/(\alpha_t^2+\sigma_t^2). The paired part dominates at low tt (0.94w0.94w vs. 0.060.06 at t=0.2t=0.2), and the reverse KL dominates at high tt (0.06w0.06w vs. 1616 at t=0.8t=0.8). In addition, our code divides each sample's update by ν=mean,∣Gθ−Dref(⋅)∣\nu=\mathrm{mean}\\,|G_\theta-D_{\mathrm{ref}}(\cdot)|, as DMD2 does. At low tt, ν\nu for a DFD step is 4–6 times ν\nu for a DMD step, so the DFD step is damped where the paired part dominates; in the Gaussian model its relative weight at t=0.2t=0.2 drops from 0.5 to 0.15–0.20. This is an implicit noise-level reweighting, which the revision now describes. Consistent with this, we have not seen conflicting or unstable losses at any noise level during training.

We nevertheless agree with the reviewer that an explicit noise-level reweighting could further improve performance, for example weighting the DFD term by αs2/(αs2+σs2)\alpha_s^2/(\alpha_s^2+\sigma_s^2). Routed Forcing [1] uses a related idea, noise-stage routing during distillation, and improves training stability and quality in audio-driven streaming avatar generation. We will add this discussion and [1] as a direction for future work.

[1] Where and When to Force: Routed Forcing for Streaming Avatars.

W3: DFD is not sufficiently justified by the DMD formulation, and large reference–generated differences may make the updates unreliable

Justification. DFD keeps the DMD update and changes only the teacher's input. Our implementation uses the x0x_0-predictions D(⋅)D(\cdot) of the denoisers, with x~t=αtGθ(z,c)+σtϵ\tilde{x}_t=\alpha_tG_\theta(z,c)+\sigma_t\epsilon and xt=αtx+σtϵx_t=\alpha_tx+\sigma_t\epsilon. With exact denoisers, for every w∈[0,1]w\in[0,1] (new Proposition 2, proved in Appendix D.3):

(1−w),gDMD+w,gDFD=∇θ[σt2αt2DKL(pθ,t,∣,pref,t)⏟distribution matching (DMD)+wαt,Ex,z,ϵ,Bt(x~t,xt)⏟paired discrepancy],(1-w)\\,g_{\mathrm{DMD}}+w\\,g_{\mathrm{DFD}}=\nabla_\theta\Big[\underbrace{\frac{\sigma_t^2}{\alpha_t^2}D_{\mathrm{KL}}\big(p_{\theta,t}\\,\\|\\,p_{\mathrm{ref},t}\big)}_{\text{distribution matching (DMD)}}+\underbrace{\frac{w}{\alpha_t}\\,\mathbb{E}_{x,z,\epsilon}\\,\mathcal{B}_t(\tilde{x}_t,x_t)}_{\text{paired discrepancy}}\Big],

where Bt≥0\mathcal{B}_t\ge0 is a Bregman divergence defined by the teacher's denoiser. We hope this shows that DFD is not a heuristic change of the DMD gradient. It has two terms:

  • Distribution matching: the full reverse KL of DMD, an expectation over the student's own samples, which is mode-seeking.
  • Paired discrepancy: an expectation over the joint (x,z)(x,z) that compares each generation with the reference its input came from. It is a regression onto real data in the teacher's metric (for Gaussian data, w,αt2αt2+σt2⋅12E∣Gθ(z,c)−x∣2w\\,\frac{\alpha_t^2}{\alpha_t^2+\sigma_t^2}\cdot\tfrac12\mathbb{E}\\|G_\theta(z,c)-x\\|^2), which is mean-seeking.

DFD balances the two, and ww sets the balance (w=0w=0 is DMD). This is the same structure as the original DMD, which adds a paired regression loss to the reverse KL. The difference is that our pairs come from real videos and re-noising instead of teacher ODE solutions.

What the paired term adds. In a 1-D Gaussian model with unit data variance, teacher mean μT\mu_T and data mean μr\mu_r, the stationary mean is σt2μT+wαt2μrσt2+wαt2\frac{\sigma_t^2\mu_T+w\alpha_t^2\mu_r}{\sigma_t^2+w\alpha_t^2}, between the teacher and the data, so a teacher shifted toward saturated colors is corrected toward the data. For modes: the reverse-KL term is an expectation over the student's own samples, so a mode the student has dropped receives no gradient from it. The paired term is an expectation over x∼prefx\sim p_{\mathrm{ref}}: a reference mode with mass πk\pi_k contributes with weight πk\pi_k whether or not the student covers it. For student inputs that still carry information about the mode of xx, this term pulls the corresponding generations toward that mode. This is why the reference data must cover different modes, as the reviewer rightly notes.

Reviewer_5wnfOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Data-Forcing Distillation, a simple, straightforward, and effective improvement over DMD. The method feeds a reference video sample to the teacher instead of the student's own generation when computing the distribution matching objective. This approach effectively mitigates the reduced diversity and oversaturation issues observed in DMD and DMD2.

Critical strengths
  1. The proposed method is simple and effective, requiring only a modification to the teacher score computation to achieve notable improvements in mitigating diversity reduction and oversaturation.
  2. The paper provides rigorous experimental analysis by training DFD and the compared methods on the same dataset, demonstrating that the performance improvements stem from the proposed method rather than differences in training data.
  3. The paper demonstrates that, under the stated assumptions, the deviation between the DFD and original DMD updates can be bounded.
Critical weaknesses
  1. The paper removes the GAN loss and adopts DFD for teacher score computation without disentangling the individual contributions of these two changes.
  2. Although the paper demonstrates that the deviation between the DFD and original DMD updates can be bounded under the stated assumptions, it lacks a sufficient theoretical explanation of why DFD mitigates mode-seeking behavior.
Questions for authors
  1. Please explain why the proposed method performs less effectively with two or fewer sampling steps, and why it sometimes collapses to nearly static videos.
  2. Please provide additional ablation studies to disentangle the effects of removing the GAN loss and introducing DFD.
  3. Please provide a more detailed theoretical explanation of why DFD mitigates mode-seeking behavior.
5 comments

Thank you for the positive review and the helpful questions. Below we answer each question and report the new experiments (the GAN ablation, the 2-step DMD2 comparison, and two new controls on the DFD term). All changes are marked in red in the revised paper, and the new videos are in the updated supplementary material.

(Reposted with corrected math formatting; the earlier deleted comments had the same content.)


W1 / Q2: Separating the effect of removing the GAN from the effect of DFD.

We agree this is important. Two results in the paper already help here:

  1. Autoregressive setting (Tables 3 and 4). Self Forcing (DMD) does not use a GAN. Self Forcing (DFD) starts from its 1.1k-step checkpoint and trains 70 more steps, also without a GAN (Table 9). So the only difference is what we feed to the teacher. DFD even uses fewer total steps (1.17k vs. 1.2k), but it still gets better diversity (DINO 0.2502 → 0.2658, camera endpoint spread 4.85 → 6.17) and better quality (average VBench 0.9219 → 0.9435). So in this setting, the gain comes from DFD, not from removing the GAN.
  2. Adding the GAN back to DFD (Table 6). This does not clearly improve quality, and it lowers the dynamic degree (0.500 → 0.375). That is why we removed it.

One case was still missing for T2V: DMD without the GAN, with the same training budget. We ran it:

ModelSubject ConsistencyBackground ConsistencyTemporal FlickeringMotion SmoothnessDynamic DegreeAesthetic QualityImaging Quality
w/o GAN w DFD0.96900.96200.97850.98990.50000.71940.7452
w/o GAN w/o DFD0.97720.96700.97630.98860.35710.69680.7258

These results show that the gain comes mainly from adding the DFD loss, not from simply removing the GAN. We also added the qualitative results to the updated supplementary material. Without the GAN and without DFD, the over-saturation problem is still there, and the videos sometimes contain unreasonable artifacts that our method (with DFD, without GAN) does not produce.

Two new controls on the DFD term (Appendix C and supplementary videos). These separate the effect of the DFD term itself. Both use the same DMD2-pretrained student and the same settings as DFD.

  • λ-DMD, a tempered (entropy-regularized) DMD. This is what DFD would be if the reference data did not matter: sfake−12srefs_{\mathrm{fake}}-\tfrac12s_{\mathrm{ref}}, with no reference data. The videos are washed out by iteration 30 and covered by noise by iterations 40–50. So the gain of DFD comes from the reference data.
  • Plain regression. We replace the DFD term by a plain MSE between the generator output and the paired reference trained for the same 100 iterations. The videos become faint, with low contrast and little detail. So DFD is not a simple regression onto the data either (see "The controls and the balance" in Part 3).

Q1: Why DFD works less well with 2 or fewer steps, and why videos are sometimes almost static.

The main reasons are:

  1. The DMD2 starting point. DFD is a short post-training stage on top of a DMD2 model, so it starts from whatever DMD2 gives it. With one or two steps, the generator has to create large motion in one or two forward passes. Fast-moving parts get blurry, and the model tends to fall back to videos with little motion, an "easy", high-density mode that the mode-seeking reverse KL prefers. The 2-step DMD2 model already has these problems.
  2. The paired term at high student noise. As we show in Part 3, DFD adds a mean-seeking paired term to DMD. When the student input carries little information about its reference video, this term pulls the generation toward an average of the prompt's reference videos, and an average of videos with different motion has little motion. With 2 steps the student timesteps are 0.999 and 0.833, and both inputs carry little information about the reference (correlation 0.001 and 0.20). With 4 steps the last student timestep is 0.624 (correlation 0.52). So with 2 steps the balance leans further toward the mean-seeking side, which adds to the static tendency of DMD2.

DFD still improves over DMD here, just not enough to solve the problem. Right now the limitation figure (Fig. 9 in the submission, Fig. 11 in the revision) only shows DFD, so these failures may look like a DFD problem. In the revision, we have added 2-step DMD2 results (numbers and side-by-side videos) next to 2-step DFD in the updated supplementary material. The 2-step DMD2 videos contain unreasonable colorful artifacts. DFD gives much better results and suppresses most of them, but some small artifacts remain in the background and reduce the overall video quality.

W2 / Q3: Why does DFD reduce mode-seeking?

We thank the reviewer for this question, which helped us strengthen the theoretical part of the paper. Proposition 1 only says how far a DFD step can be from a DMD step. Here we show what DFD adds to DMD for the method we run, which explains why it reduces mode-seeking. The analysis is in Sec. 3.3 and Appendix D.3 of the revision, and it continues in Part 3.

The DFD pipeline. We do not use backward simulation. Each update takes a reference video x∼pref(⋅∣c)x\sim p_{\mathrm{ref}}(\cdot\mid c), re-noises it to a student timestep ss, z=αsx+σsηz=\alpha_sx+\sigma_s\eta, and runs the student once on zz. With x~t=αtGθ(z,c)+σtϵ\tilde{x}_t=\alpha_tG_\theta(z,c)+\sigma_t\epsilon and xt=αtx+σtϵx_t=\alpha_tx+\sigma_t\epsilon (shared ϵ\epsilon), a DMD step uses the x0x_0-predictions Dfake(x~t)−Dref(x~t)D_{\mathrm{fake}}(\tilde{x}_t)-D_{\mathrm{ref}}(\tilde{x}_t) and a DFD step uses Dfake(x~t)−Dref(xt)D_{\mathrm{fake}}(\tilde{x}_t)-D_{\mathrm{ref}}(x_t). So xx and zz are paired, z∼qs(⋅∣x)z\sim q_s(\cdot\mid x).

Proposition 2 (new). With exact denoisers, for every w∈[0,1]w\in[0,1],

(1−w),gDMD+w,gDFD=∇θ[σt2αt2DKL(pθ,t,∣,pref,t)⏟distribution matching (DMD)+wαt,Ex,z,ϵ,Bt(x~t,xt)⏟paired discrepancy],(1-w)\\,g_{\mathrm{DMD}}+w\\,g_{\mathrm{DFD}}=\nabla_\theta\Big[\underbrace{\frac{\sigma_t^2}{\alpha_t^2}D_{\mathrm{KL}}\big(p_{\theta,t}\\,\\|\\,p_{\mathrm{ref},t}\big)}_{\text{distribution matching (DMD)}}+\underbrace{\frac{w}{\alpha_t}\\,\mathbb{E}_{x,z,\epsilon}\\,\mathcal{B}_t(\tilde{x}_t,x_t)}_{\text{paired discrepancy}}\Big],

where Bt≥0\mathcal{B}_t\ge0 is the Bregman divergence of the convex potential ϕt(y)=(12∣y∣2+σt2log⁡pref,t(y))/αt\phi_t(y)=\big(\tfrac12\\|y\\|^2+\sigma_t^2\log p_{\mathrm{ref},t}(y)\big)/\alpha_t. Proof idea. A DFD step is a DMD step plus Dref(x~t)−Dref(xt)D_{\mathrm{ref}}(\tilde{x}_t)-D_{\mathrm{ref}}(x_t). The DMD step gives the reverse-KL gradient. By Tweedie's formula ∇ϕt=Dref\nabla\phi_t=D_{\mathrm{ref}} and ∇2ϕt=(αt/σt2),Cov[x∣xt]⪰0\nabla^2\phi_t=(\alpha_t/\sigma_t^2)\\,\mathrm{Cov}[x\mid x_t]\succeq0, so the added part is the gradient of Bt/αt\mathcal{B}_t/\alpha_t with respect to the generator output. The full proof is in Appendix D.3.

The two terms.

  • Distribution matching (standard DMD). The reverse KL is an expectation over the student's own samples. It is mode-seeking: it moves each sample toward a high-density region of the teacher. With CFG the teacher is itself sharpened and shifted toward saturated colors, and DMD follows it.
  • Paired discrepancy. This term is an expectation over the joint distribution of (x,z)(x,z). It compares each generation with the reference video its own input was re-noised from. Since Dref(x~t)−Dref(xt)=Mˉt,(Gθ(z,c)−x)D_{\mathrm{ref}}(\tilde{x}_t)-D_{\mathrm{ref}}(x_t)=\bar{M}_t\\,(G_\theta(z,c)-x) with Mˉt⪰0\bar{M}_t\succeq0 an average of the teacher's posterior covariance, it is a regression onto real data in the teacher's metric. It is mean-seeking: its minimizer is E[x∣z]\mathbb{E}[x\mid z].

Why this reduces mode-seeking. DFD balances the mode-seeking reverse KL with a mean-seeking paired term, and ww sets the balance (w=0w=0 is DMD). A 1-D Gaussian model makes this exact: data N(μr,sx2)\mathcal{N}(\mu_r,s_x^2), teacher N(μT,sx2)\mathcal{N}(\mu_T,s_x^2), a linear student in the standardized input z^\hat z, and ρ=Corr(z^,x)\rho=\mathrm{Corr}(\hat z,x).

  • Mean. The KL term pulls the mean toward μT\mu_T and the paired term toward μr\mu_r. The stationary mean is mG∗=σt2μT+w,αt2sx2,μrσt2+w,αt2sx2m_G^\ast=\frac{\sigma_t^2\mu_T+w\\,\alpha_t^2s_x^2\\,\mu_r}{\sigma_t^2+w\\,\alpha_t^2s_x^2}, between the two. For sx=1s_x=1, μT=1\mu_T=1, μr=0\mu_r=0, w=12w=\tfrac12 it is 0.11, 0.67, 0.97 at t=0.2,0.5,0.8t=0.2,0.5,0.8. A teacher shifted toward saturated colors is corrected toward the data. The reverse KL alone cannot do this, because it never sees the data.
  • Scale. The DMD term alone is stationary at the data scale sxs_x; the paired term alone at ρsx\rho s_x, the scale of E[x∣z]\mathbb{E}[x\mid z]. The combined stationary point lies between the two, and ww sets where.
  • Modes. The reverse-KL term is an expectation over the student's own samples, so a mode the student has dropped receives no gradient from it. The paired term is an expectation over x∼prefx\sim p_{\mathrm{ref}}: a reference mode with mass πk\pi_k contributes with weight πk\pi_k whether or not the student covers it. For student inputs that still carry information about the mode of xx, this term pulls the corresponding generations toward that mode. Diversity therefore enters through the sampling of xx from the diverse reference set.

DFD is not a tempered (entropy-regularized) DMD. A wider distribution is more diverse, so one might think DFD just tempers the teacher. λ-DMD, sfake−λsrefs_{\mathrm{fake}}-\lambda s_{\mathrm{ref}}, targets prefλp_{\mathrm{ref}}^{\lambda}, which is the same as λ\lambda times the DMD objective plus an entropy bonus (1−λ)H(pθ)(1-\lambda)\mathcal{H}(p_\theta). In the Gaussian model it widens the distribution (pλp^{\lambda} of N(μT,sx2)\mathcal{N}(\mu_T,s_x^2) is N(μT,sx2/λ)\mathcal{N}(\mu_T,s_x^2/\lambda)) and keeps the teacher's mean. DFD does neither: its variance stays between ρ2sx2\rho^2s_x^2 and sx2s_x^2, and its mean moves toward the data. In practice λ-DMD with λ=12\lambda=\tfrac12 is covered by noise within 50 iterations, while DFD improves over DMD2 within 100. We also show the video results from λ-DMD in the updated supplementary material

The controls and the balance. DMD2 sits at the mode-seeking end (over-saturated, less diverse) and the plain regression at the mean-seeking end (faint, low contrast). DFD sits between them and improves over DMD2. At high student noise (ρ\rho small) the balance leans toward the mean-seeking side, so we use DFD as a short post-training stage from DMD2 (100–300 steps), which also fits Fig. 6(c).

Revision. (i) Proposition 2 and its proof are in Appendix D.3, with a short statement in Sec. 3.3. (ii) The Gaussian example (mean, scale, the role of ww and ρ\rho) is in Remark 1. (iii) The λ-DMD and plain-regression controls are in Appendix C.

We hope these answers address the reviewer's questions, and we would be glad to provide any further details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.