acceptodds
Under review as a conference paper at ICLR 2027

Drift Forcing: Sample-Based Drift Fields as Initialization for Autoregressive Video Diffusion Distillation

Abstract

Autoregressive video diffusion distillation makes a bidirectional teacher causal, initializes a few-step causal student and refines it with distribution matching distillation (DMD), in which the bidirectional teacher scores the student's own rollouts. The released initializations imitate the causal teacher one trajectory at a time under teacher forcing: the student is trained on ground-truth prefixes, then handed to a stage that conditions it on its own samples. We ask whether the initialization can instead be trained on those samples directly, with no teacher trajectories, no score targets and no trained critic. does this with a drift field in the feature space of a frozen causal teacher: the student's own rollouts are attracted toward positive clips, one bidirectional-teacher clip per training prompt, and repelled from one another, in the self-rollout regime that DMD and inference share. The main recipe runs a coarse field over clip-level features and then a finer field over per-chunk features, for about 400 H800-hours on the public Causal-rCM synthetic training set. On Wan2.1-1.3B, with the DMD trainer, data and budget held fixed, the student that DMD makes from the drift initialization reaches the highest VBench Total among all products compared (84.63), above DMD from the released consistency-distilled initializations and from the causal diffusion model with no initialization stage. The lead comes from VBench's dynamic-degree dimension, the share of clips that move. Code is available at https://anonymous.4open.science/r/DF0E51.

Primary area:
generative models
Keywords:
Autoregressive Video Diffusion Distillation, Drift Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_oVTsOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Drift Forcing replaces the initialization stage of a few-step autoregressive video distillation pipeline. It generates student rollouts and trains them with attraction toward positive teacher-generated clips and repulsion among sibling rollouts in the feature space of a frozen causal teacher. The main recipe first matches clip-pooled features and then chunk-pooled features, before handing the student to an unchanged asymmetric DMD stage. The experiments hold the downstream DMD recipe fixed and compare several initializations on VBench, distributional metrics, and blinded human ratings. The main outcome is a motion-versus-stability trade-off: the proposed initialization gives the highest VBench Total, while consistency-distilled alternatives retain better teacher coverage and human-rated stability.

Critical strengths
  1. Training the initialization on its own causal rollouts is a sensible and technically concrete alternative to per-trajectory imitation. The downstream comparison is well controlled: the DMD trainer, data, batch, and iteration budget are held fixed. The two-stage schedule and repulsion ablations also test actual components of the proposed recipe. At iteration 1000, removing repulsion reduces the final Total from 84.63 to 83.84 despite nearly equal VBench scores before DMD. This supports evaluating an initialization by what the next stage makes of it.

  2. The paper does not hide the limitations of the aggregate metric. It includes teacher-reference precision/coverage, per-dimension results, learned preference measures, and a blinded human comparison. Section 4.5 shows that higher dynamic degree is not simply synonymous with better perceived video: raters distinguish the models on stability even when motion compliance is similar. That analysis is a useful contribution in its own right.

Critical weaknesses
  1. The main proposed recipe is not the better product under the paper's own human evaluation. Section 4.5 reports clear instability on roughly one in five clips for the two-stage product versus one in twenty for TF-dCM. Motion-compliance rates are essentially unchanged, 0.77 versus 0.78. Table 1's VBench improvement over TF-dCM is 0.51, and the paper explains that dynamic degree contributes more than the entire gain. Coverage also falls from 0.223 for TF-dCM's DMD product to 0.137 for the proposed product. The authors acknowledge these trade-offs, which I appreciate, but the evidence does not yet make a compelling case for choosing the main recipe when perceived stability and fidelity matter. A favorable aggregate score is too narrow a basis for that choice.

  2. The practical value of paying for the initialization is not established at equal total cost. DMD from the stage-1 model reaches 84.37, only 0.26 below the proposed 84.63. The proposed initialization adds about 400 H800-hours before the roughly 200-H800-hour DMD stage. The comparison with 2,900 A800-hours is from a different reported consistency-distillation setup; Table 3 lists the actual TF-dCM initialization cost as unpublished. Those numbers cannot establish a hardware-matched speedup. The main comparison does not show what happens when the full initialization budget is instead allocated to the no-initialization baseline. Evaluation seeds and prompt bootstrap intervals also do not quantify variation between independently trained models, which matters for the small aggregate lead.

The authors could also consider discussing BAgger [Po et al., 2026] and History-Guided Video Diffusion [Song et al., 2025]. BAgger offers a related perspective on learning from self-generated trajectories, while History Guidance is relevant to the tradeoff between motion and temporal consistency. Both would help position the contribution, although neither is a direct substitute for the initialization stage studied here. These are optional references to consider, rather than requests for additional experiments.

Optional references: [Po et al., 2026] Ryan Po, Eric Ryan Chan, Changan Chen, and Gordon Wetzstein. "BAgger: Backwards Aggregation for Mitigating Drift in Autoregressive Video Diffusion Models." CVPR 2026.

[Song et al., 2025] Kiwhan Song, Boyuan Chen, Max Simchowitz, Yilun Du, Russ Tedrake, and Vincent Sitzmann. "History-Guided Video Diffusion." ICML 2025.

Questions for authors
  1. Assessment-changing comparison: At the same total H800-hour budget, how does the proposed pipeline compare with spending that budget on DMD from the stage-1 model, including human preference and stability as well as VBench?

  2. Clarification: Is there a checkpoint or configuration that retains the motion gain while matching TF-dCM's human-rated stability? If not, which concrete use case makes the measured trade-off preferable?

  3. Clarification: How variable are the matched-budget Total, dynamic-degree, and stability differences across independent training runs? The reported generation-seed probes do not answer this question.

Reviewer_1SZgOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies few-step initialization before DMD in autoregressive video distillation.

It applies an existing attraction–repulsion drifting objective to the student’s own video rollouts in a frozen causal teacher’s feature space, using a clip-level followed by chunk-level pooling schedule.

On Wan2.1-1.3B under a fixed downstream DMD recipe, the resulting student obtains a higher VBench Total than students trained from consistency-distilled initializations or without a separate initialization stage. The improvement is primarily associated with dynamic degree and comes with stability and coverage trade-offs.

The main contribution is the initialization recipe and its empirical analysis.

Critical strengths
  1. A novel and useful adaptation of drifting to pre-DMD initialization. The paper develops a sample-based initialization for few-step autoregressive video students by applying drifting to their own rollouts in a frozen causal teacher’s feature space, with a clip-level followed by chunk-level pooling schedule. The novelty lies primarily in this initialization role and training recipe. It provides an interesting alternative to trajectory-based consistency or ODE initialization.

  2. Controlled experiments and informative ablations. Table 1 holds the downstream DMD trainer, data, and budget fixed while changing the initialization, providing a clear test of initialization utility. Table 13 is particularly informative: the full and attraction-only variants have nearly identical pre-DMD Totals but substantially different performance after the same DMD training. The paper also reports unfavorable stability and human-preference results, making the trade-offs more transparent.

Critical weaknesses
  1. Generalization across downstream DMD recipes and student scales remains unclear. The proposed method is evaluated only with the Causal Forcing++ DMD trainer and a Wan2.1-1.3B student. Since the main advantage comes from dynamic degree, it remains unclear whether this initialization complements DMD recipes that already improve motion (e.g. Phased DMD, CVPR'26), and whether its benefit on larger students (e.g., Wan2.2-5B, Wan2.1-14B) justifies the additional training cost. The paper appropriately acknowledges its scope, but the restricted validation limits the broader significance of the contribution.

  2. The mechanistic justification for the initialization and key design choices remains incomplete. Section 5.1 and Appendix L provide useful fixed-point and coverage interpretations, and the ablations support several empirical choices. However, these analyses do not sufficiently explain which property of the drift-trained initialization produces its downstream motion advantage under DMD.

In addtion, the asserted necessity of a causal feature extractor in Sec.3.2 also needs justification: applying the same bidirectional encoder to positive and generated videos after a complete self-rollout would not itself violate causal generation.

Questions for authors
  1. Could you provide focused comparisons under another DMD recipe compatible with the causal student and with a larger student model? Repeating the full ablation suite is unnecessary, but it would be valuable to establish whether the benefit and motion–stability trade-off persist.

  2. Which measurable property of the drift-trained initialization explains its downstream DMD advantage? Also, why must the feature extractor be causal when the loss is evaluated after a complete rollout? Have you compared against encoding both positive and generated videos with the same bidirectional teacher?

  3. Could you provide, or clearly point to, synchronized full-length video comparisons for Figure 8, using the same prompts, initial noise, and playback frame rate, and including representative failure cases? The six sampled frames per clip reveal coarse displacement but are insufficient to assess motion smoothness, acceleration, flicker, and transient deformations. These temporal properties are central to evaluating the claimed motion benefit and its stability cost.

  4. Could the A800-hour and H800-hour measurements in Figure 1(b) be compared?

Reviewer_kquSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Within the three-stage pipeline for autoregressive video diffusion distillation (stage 1: make the teacher causal → stage 2: few-step initialization → stage 3: DMD), the paper changes only stage 2. Existing stage-2 recipes imitate trajectories one at a time under teacher forcing (ODE regression / consistency distillation); this paper replaces that with a drift field (the double-softmax attraction–repulsion construction of Deng et al., 2026) in the feature space of a frozen causal teacher, acting on the student's own four-step rollouts — no teacher trajectories, no score targets, no critic. Main recipe: a coarse clip-level mean-pooled field for 1,000 iterations, then a finer chunk-level pooled field for 1,000 more, at roughly 400 H800-hours.

Setting: Wan2.1-1.3B / 480p / the Causal-rCM stack, with the 14B causal model as feature extractor φ and the bidirectional 14B as the DMD real score.

Critical strengths

1.The question is posed correctly. The paper does not ask "is this initialization good on its own?" but "what does an initialization contribute to the final product under a fixed downstream DMD recipe?" That is the right ablation axis, and the controls are clean — trainer, data, batch size, and budget are all frozen while only the initialization varies. 2.Experimental hygiene far above the norm for a submission of this kind. A frozen VBench protocol (944 prompts × 5 seeds, 0–4); local re-measurement of seven released checkpoints (within 0.06 of published values, ordering preserved); noise floors reported for precision/coverage/Vendi; an adjacent-checkpoint noise probe for per-dimension VBench differences; a cross-encoder measurement control over six feature families (Appendix H); a blind study with 6 raters and 397 ratings; an explicit discussion of the metric hierarchy; and a voluntary report that FVD and VBench invert rankings. 3.Negative results are reported candidly and are genuinely useful. The structural failure of the teacher-forced placement (within-prompt spread collapses from 0.35–0.59 to 0.08–0.37); the motion field raising dynamic degree without raising Total; and the finding that coverage is inherited rather than produced by DMD (Appendix L gives a mechanistic account via Eq. 4). These are more likely to be cited than the main claim. 4.The cost story is attractive. 400 H800-h versus 2,900 A800-h for CD and 11,600 A800-h plus 1,900 GiB of cached trajectories for ODE regression; the positive library reuses the existing Causal-rCM synthetic training set at zero construction cost.

Critical weaknesses

1.Three independent human/preference instruments say the method loses; only VBench says it wins 2. The entire advantage comes from dynamic degree — and even that gain is not significant under the VLM judge. Appendix Q states it outright: excluding dynamic degree, the Total change relative to the paper's own initialization is −0.02. Against TF-dCM, +14.2 dynamic degree is worth about 0.85 Total at the protocol's 0.06-per-point weighting, while the Total lead is only 0.5 — i.e. outside dynamic degree, this paper's product is worse. VBench's dynamic degree is essentially the fraction of clips whose optical-flow magnitude exceeds a threshold; it rewards any motion, including jitter, and is widely regarded as the most gameable dimension.

  1. The two-stage ablation is not clean. Stage 2b changes three things at once (mean → chunk pooling, adding the floor r_soft = 0.10, and lr 4e-6 → 2e-6). The conclusion "two-stage beats either field alone" therefore confounds schedule, floor, and learning rate. 4.A margin of 0.26–0.5 is unsupported by any run-to-run variance estimate. Each configuration is trained once, and VBench uses a fixed set of five seeds. The only noise estimate offered is "two checkpoints 50 iterations apart on the same run" (Table 18b reports a 0.00 effect on Total) — that measures evaluation noise only and says nothing about training noise, which is the dominant source of uncertainty at the 0.26 scale. Reviewers will reject the SOTA claim on this basis alone.
Questions for authors

Please see the weakness

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised Manuscript Uploaded

We have uploaded the revised manuscript incorporating the updates and additional results discussed in our initial rebuttal (highlighted in blue), including the multi-run statistics and best-in-1k operating point (Abstract, §4.3), the additional blind human evaluation (§4.5, Appendix M), expanded related work (§2), and clarifications on feature-space design, hardware comparisons, and experimental scope (§1, §3.2, §4.2, §6).

Thank you again for your valuable feedback, and please let us know if you have any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.