acceptodds
Under review as a conference paper at ICLR 2027

ViDA: Exploiting Temporal Dependencies for Autoregressive Video Distillation

10 views

Abstract

Autoregressive video distillation supervises predictions that also serve as context for subsequent generation. Causal self-rollout exposes the student to its own his- tory, but a single re-noising level in Distribution Matching Distillation (DMD) couples the preservation of that context with the correction of generated content. We present ViDA (Video Distillation for Autoregressive generation), which in- corporates temporal dependencies into DMD supervision through ordered score queries. Earlier chunks receive lower, nonzero corruption to retain context while remaining subject to correction. With both score estimators bidirectional and inde- pendent of the generator, the ordered query improves VBench Total by 0.75 over marginal-matched scalar queries, and by 0.64 and 1.40 over shuffled and reversed noise assignments. ViDA also replaces the separate fake-score network with a role-conditioned shared backbone, improving quality at a fixed update ratio. Com- bined with fewer fake updates, it outperforms the independent causal estimator at r= 5 at half the student post-training GPU-hours and 13.6% lower peak allocated memory. On five-second generation, the final 1.3B generator distilled from a 14B teacher reaches 85.62 VBench Total: 1.12 points above matched scalar querying within the same recipe and 1.38 above the locally evaluated official Self Forcing checkpoint. Evaluations on Cosmos and 60-second videos extend these compar- isons across backbones and generation horizons. The autoregressive inference procedure remains unchanged.

Primary area:
generative models
Keywords:
autoregressive video generation, diffusion models, distribution matching distillation, efficient training, long-horizon generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_F93YOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper changes how the DMD query is formed in Self-Forcing-style AR video distillation. Instead of re-noising the whole rollout at one timestep, it samples a per-chunk timestep vector and sorts it along generation order, so earlier chunks get less noise than later ones; the teacher’s timestep conditioning is fine-tuned to accept per-chunk timesteps. Separately, the fake score shares the generator’s backbone, with a small set of role-conditioning parameters telling the network which job it is doing. Empirically, the ordered query gives small but consistent VBench gains over several matched-noise controls, the final system edges out the locally evaluated Self Forcing checkpoint, and the shared fake beats an independent one at a lower fake-update ratio with less memory.

Critical strengths
  1. The experiments are thorough and the ablations are carefully controlled. Reversed landing below scalar is a particularly useful result, it rules out heterogeneous noise by itself as the explanation.
  2. The claims are stated carefully. The paper is candid about what it does not establish, distinguishes literature-reported from locally evaluated numbers.
  3. The paper is well writen.
Critical weaknesses
  1. The sharing gain is confounded. The fake loss also updates the generator, and the trunk gets 600 extra fake-objective updates, so the +1.3 at r=2 could be tracking, regularization, or just more updates. No shared r=5 row, and the independent causal r=5 it is compared to is a single run.
  2. The headline comparisons are thin: one checkpoint per policy in Table 3, one video per prompt in Table 5, Causal Forcing is cited but not compared, and there are no external 60-second baselines.
Questions for authors
  1. Can you add the shared r=5 row?
  2. Teacher adaptation details (batch size, lr, compute), and is the adapted teacher’s denoising loss similar across the four profiles?
  3. For the independent fakes, is the fake update done on a fresh post-step rollout like the shared model? And how was the AR Cosmos initialization obtained?
3 comments
AuthorsOpenReview ↗

Response to Reviewer F93Y

Thank you for highlighting the reversed-query control and for suggesting the shared r = 5 comparison.

(W1/Q1) Shared training at r = 5

As you point out, at r = 2 the shared generator backbone receives 600 updates from the fake loss in addition to its 300 generator updates. We therefore interpret the 1.30-point difference at this ratio as a comparison of the two training designs as a whole. Following your suggestion, we trained the shared model at r = 5 three times and added two runs of the independent causal baseline at the same ratio under the Table 2 protocol.

We will add these results to Table 2. Each entry below is the mean VBench Total ± sample SD over three training runs.

Causal fake estimatorr = 2r = 5
Independent83.75 ± 0.2584.52 ± 0.23
Role-shared85.05 ± 0.2385.07 ± 0.19

Raising r from 2 to 5 increased mean VBench Total by 0.77 points for the independent model and 0.02 for the shared model, narrowing the shared model's lead to 0.55 points. In this setting, the shared model reached nearly the same mean score with 600 fake updates as with 1,500.


(W2) Five-second and 60-second comparisons

Direct five-second comparison with Causal Forcing. Table 3 originally quoted the published Causal Forcing score of 84.04 VBench Total. We have now evaluated its released chunk-wise checkpoint using the same five-second VBench protocol as ViDA. Causal Forcing scored 84.37 and ViDA scored 85.62, a 1.25-point gain for ViDA. We will report this local evaluation alongside the published result in Table 3.

How many training runs support the ordering gain? In Table 3, we trained one model with ordered queries and one with matched scalar queries, both using the 14B teacher. The ordered model scored 1.12 VBench Total points higher in that pair. In a separate experiment with a 1.3B teacher, we kept the same shared causal design and trained each policy with three matched seeds. Ordered queries scored higher in all three pairs, by 0.99 points on average (Appendix Tables 7–8). We will state the run count alongside the Table 3 comparison.

What did we compare at 60 seconds? Table 5 compares the ordered and matched-scalar checkpoints from Table 3 to test whether the ordering gain persists in longer videos. It includes no external 60-second baseline. We generated one 60-second video per checkpoint for each of 128 MovieGen prompts, matching the rendering seed and extension settings within each pair. Across these 128 pairs, the ordered checkpoint scored higher on average on all six reported dimensions, including aesthetic quality (+1.61 points) and imaging quality (+1.35 points).


(Q2) Teacher adaptation cost and denoising loss across profiles

Adaptation settings and cost. For each Wan teacher, we trained only the timestep-conditioning parameters for 2,000 steps on 64 H100 GPUs, using a global batch size of 64 and AdamW with a learning rate of 2 × 10⁻⁶. We used video–text pairs from OpenVidHD-0.4M (Nan et al., 2025) and sampled the four noise profiles uniformly. Adapting each teacher once cost approximately 105 H100 GPU-hours for the 1.3B model and 545 for the 14B model. We then froze each checkpoint and reused it across student runs. Table 2 reports student post-training costs separately, with the same frozen 1.3B teacher for all configurations.

Denoising loss. During adaptation of the Wan 1.3B teacher used in Tables 1 and 2, the four profiles had similar mean flow-matching losses. The highest mean was 2.7% above the lowest over the full 2,000-step run and 3.4% above the lowest over the final 500 steps.

Adaptation steps averagedNon-decreasingShuffledReversedConstant
All 2,0000.11580.11780.11550.1147
Final 5000.11060.11440.11380.1124

We will report the adaptation settings, costs, and per-profile losses in Appendix A.12.


(Q3) Independent fake updates and Cosmos initialization

Do the independent fake estimators also use fresh rollouts? Yes. When both the generator and the fake estimator are updated in the same iteration, we update the generator first. We then generate a fresh rollout with its updated weights, detach it, and use it for the fake update. On fake-only iterations, we also draw a fresh rollout from the current generator. We will clarify this order in Appendix A.9.

How was the Cosmos AR checkpoint obtained? We built it ourselves from the pretrained bidirectional Cosmos-Predict2.5-2B model. We converted its attention to chunk-causal attention and used ODE initialization to obtain a few-step autoregressive generator. Both the matched-scalar and ordered-query models in Table 4 started DMD post-training from this same checkpoint. We will add this detail to Appendix A.8.

I thank the author for their response. The extra experiments and added details have basically addressed my concerns. Although an external 60s baseline comparison was not added, this does not affect the roel of ordered queries within the ablation. Therefore, I'm inclined to maintain my current rating.

Thank you for following up. We’re glad the additional experiments and clarifications helped address your concerns.

Reviewer_vaNDOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ViDA, which structures DMD queries so that earlier predictions retain contextual signals and the entire rollout remains subject to correction. It replaces the fake-score network with a role-conditioned shared backbone, improving quality at a fixed update ratio. Several experiments demonstrate the effectiveness of the method.

Critical strengths
  1. Role-conditioned sharing is efficient, improving quality while reducing memory usage.
  2. The experiments validate the method’s effectiveness from multiple perspectives, including Cosmos transfer, additional 60-second evaluations, and human preference evaluation.
Critical weaknesses
  1. L85 claims that “ViDA also reduces DMD training resources through role-conditioned sharing.” However, in Table 2, Independent causal (r = 2) and Role-shared causal (r = 2) have similar GPU-hour costs. The training resource savings come from reducing r from 5 to 2 rather than from ViDA itself. The paper should clarify that these savings are measured relative to comparable performance.
  2. Why is r = 2 used as the default setting in L259? Has r = 5 been tested?
  3. Why are some results in Table 1 (L274) and Table 2 (L329) based on a single run, while others report averages over three runs? Are these comparisons fair? The experimental settings should be consistent throughout the paper, and results from three runs could be reported as mean ± standard deviation.
  4. Has the method been evaluated on other benchmarks?
Questions for authors

Please see weaknesses, especially W1,W3.

6 comments
AuthorsOpenReview ↗

Response to Reviewer vaND

Thank you for noting the quality and memory benefits of sharing and for asking us to clarify the GPU-hour savings and training run counts.

(W1) What accounts for the training savings?

At the same update ratio, sharing improved quality and reduced peak memory. With two fake-estimator updates per generator update (r = 2), sharing raised mean VBench Total from 83.75 to 85.05 and reduced peak allocated memory from 77.7 to 67.1 GiB (13.6%). As you observed, their training costs were nearly identical (111 H100 GPU-hours for the independent causal model and 110 for the shared model).

The reported 50% GPU-hour saving comes from using fewer fake updates. For the same 300 generator updates, the independent causal model at r = 5 used 1,500 fake updates and 220 H100 GPU-hours. The shared model at r = 2 used 600 fake updates and 110 H100 GPU-hours. At that lower cost, the shared model reached a mean VBench Total of 85.05 versus 84.52 for the independent model across three runs per setting. We will revise L85 to distinguish the two savings and specify that these costs cover student post-training with the frozen 1.3B teacher in Table 2.


(W2) Why use r = 2? What happens at r = 5?

With sharing, raising r from 2 to 5 nearly doubled post-training cost (110 to 215 H100 GPU-hours) for a gain of only 0.02 points in mean VBench Total. We chose r = 2 for its lower cost and tested the shared model at r = 5 in three new training runs for this response. With an independent causal fake estimator, the same increase in r raised mean VBench Total by 0.77 points.

The table gives the mean ± sample SD across three training runs per setting.

Causal fake designVBench Total at r = 2VBench Total at r = 5
Independent83.75 ± 0.2584.52 ± 0.23
Role-shared85.05 ± 0.2385.07 ± 0.19

Because each generator update also changes the shared backbone, the fake estimator may be able to track the generator's changing output distribution with fewer fake updates. We will add the new results to Table 2 and explain the choice of r = 2 at L259.


(W3) Why do run counts differ, and does the half-cost comparison hold?

The shared model still scores higher at half the reported GPU-hours, with three training runs on each side. In the submission, the independent causal baseline at r = 5 had one VBench Total score of 84.57. Two additional runs scored 84.73 and 84.27, bringing its mean to 84.52 ± 0.23. The shared model at r = 2 averaged 85.05 ± 0.23. Its lead was +0.48 points against the original run and is +0.53 against the three-run mean.

Why the run counts differ. We prioritized repeated runs for the main ablations of query ordering and parameter sharing. The remaining single-run rows are reported as individual observations.

  • Table 1 (query ordering). Matched scalar, shuffled, reversed and ViDA each had three training runs with the same adapted teacher, 300 generator updates and the full VBench evaluation. The two reference-scalar rows, which compare the original and adapted teachers, had one run each.
  • Table 2 (parameter sharing). The original comparison at r = 2 already had three runs each for the independent and shared causal designs. With the new r = 5 results, all four causal settings now have three runs each. The independent bidirectional setting at r = 2 remains a single-run result.

We will label every row in Tables 1 and 2 with its run count, report mean ± sample SD for each three-run result, and list the new individual scores in Appendix A.6.


(W4) Does the gain extend beyond VBench?

We evaluated the final ViDA checkpoint from Table 3 with VideoReward (Liu et al., NeurIPS 2025), a scorer outside the VBench suite. Across 128 consecutive MovieGen Video Bench prompts (Polyak et al., 2024), ViDA's mean overall score was 0.54 points higher than Self Forcing's and 0.71 points higher than Causal Forcing's.

We used prompts 129–256 in the published order, immediately following those used in Appendix A.3. For each prompt, we generated five-second videos using the same three rendering seeds across methods. We averaged the scores over seeds and computed paired differences between methods on the same prompts.

ComparisonVisual qualityMotion qualityText alignmentVideoReward Overall (95% CI)
ViDA − Self Forcing+0.33+0.09+0.12+0.54 [+0.40, +0.68]
ViDA − Causal Forcing+0.36+0.29+0.06+0.71 [+0.59, +0.82]

ViDA also scored higher on visual quality, motion quality and text alignment against both baselines. The overall intervals use a paired bootstrap over prompts. We will report the per-dimension intervals and full protocol in the appendix.

Thank you for your response and the additional experiments.

Regarding W2, could you explain why increasing r from 2 to 5 brings almost no improvement for the role-shared design, while the independent causal design gains about 0.8 in VBench score? As a result, the performance gap between the two designs becomes much smaller at r=5.

For W3, I understand that repeating all experiments three times can be costly and may not be necessary. However, I still think that results in the same table should use consistent experimental settings for a fair comparison.

AuthorsOpenReview ↗

Follow-up on W2 and W3

W2. Thank you for highlighting this difference. As you observed, increasing rr from 2 to 5 raises mean VBench Total by 0.77 points for the independent model and by only 0.02 for the shared model, narrowing the gap.

A possible explanation is how the fake estimator keeps up with the generator. An independent estimator relies on its own updates to track the generator’s changing output distribution, which motivates the extra fake updates used in DMD2 (Yin et al., 2024a, Sec. 4.2). In our independent model, the three extra steps may therefore improve the score estimate that guides the next generator update. With sharing, each generator step also updates the backbone used by the fake estimator. If this helps the estimator keep up with just two updates of its own, the extra three would have less effect on that training signal.

Our motivation for sharing the backbone was to make distillation more efficient. In our Wan experiments, even when the independent model received five fake updates per generator step, the shared model with two achieved a higher mean VBench Total (85.05 vs. 84.52) at half the student post-training cost (110 vs. 220 H100 GPU-hours).


W3. We have revised Tables 1 and 2 so that each addresses one question. Every row in both tables now reports the mean VBench Total and sample standard deviation over three training runs, each with the same number of generator updates and the same full VBench evaluation.

  • Table 1 (p. 6) compares four query policies (matched scalar, shuffled, reversed, and ViDA). All four use the same adapted teacher and an independent bidirectional fake estimator at r=5r=5. We moved the two single-run reference-scalar controls to Appendix A.6 (Table 9, p. 17), where they compare the original and adapted teachers under the same scalar sampler.
  • Table 2 (p. 7) compares the independent and role-shared causal estimators side by side at r=2r=2 and again at r=5r=5. At each ratio, the two designs use the same AR initialization, adapted teacher, and query ordering. We moved the independent bidirectional controls at both ratios to Appendix A.6 because their architecture and initialization differ from those of the causal designs. Table 9 also lists the original independent causal r=5r=5 run alongside the two additional runs that now contribute to its three-run mean in Table 2.
Reviewer_mq1ZOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ViDA, a method for improving the distillation of autoregressive models for video generation through two main contributions: a strategy that assigns increasing levels of noise to chunks during training while preserving temporal context, and the sharing of parameters between the generator and the fake-score estimator to reduce computational costs. Experiments show improvements in video quality compared to Self Forcing and greater training efficiency, without altering the inference process.

Critical strengths

The paper shows that using ordered noise levels instead of fixed ones makes the 1.3B generator able to follow the prompt correctly while giving it more freedom to generate the other chunks. The authors also show that they improve memory efficiency, using 13.7% less memory. They also show improved metrics in the VBench comparison, and a substantial preference for their model in the human study.

Critical weaknesses

Unfortunately, this paper is far from my domain of expertise, as I already communicated to the Area Chair. To clarify my doubts, I will read the comments of the other reviewers, and the corresponding answers from the authors. Below I report the list of weaknesses I identified: if the authors manage to bring enough evidence in support of their results, I will consider raising my score.

  1. ViDA assigns noise levels based only on the temporal position of each chunk, assuming that earlier chunks provide useful context for future ones. However, this assumption may not hold for videos containing rapid scene transitions, where the earlier context becomes less relevant. In these cases, the first chunk of a new scene still receives pretty high corruption during distillation, potentially limiting the preservation of scene-specific information. It would be valuable to investigate whether a scene-aware noise schedule could improve performance on videos with frequent scene changes.
  2. The human study does not seem solid enough, as only three raters were employed. I believe this could make the human study noisy. It would be valuable to have more raters.
Questions for authors

Please see W1 in the Weaknesses section: have the authors investigated how the proposed noise schedule behaves in such scenarios? Could a scene-aware noise schedule, which assigns lower noise to the first chunks following a scene transition, improve generation quality?

Also: would it be possible for the authors to make the human study more robust by employing more raters?

5 comments
AuthorsOpenReview ↗

Response to Reviewer mq1Z

Thank you for recognizing ViDA's prompt adherence and for suggesting a scene-aware noise schedule and a larger human study.

(W1/Q1) Does ViDA's ordering make scene cuts harder?

Abrupt cuts were difficult even with a single noise level. Without cut-specific fine-tuning, we compared a student using ViDA's ordering with a matched student using one noise level across chunks. Given only a text prompt for the new shot, neither switched within the first 12 frames in any of the 48 cases. The Table 3 ViDA checkpoint and the official Self Forcing and Causal Forcing models also usually continued the old shot.

Lowering noise at the cut helped when both the location and the subject changed. To test your suggestion, we fine-tuned ViDA under both schedules using the same starting checkpoint, shot-change data and training budget. For cuts to a new location and subject, we compared 288 pairs of clips from 24 held-out scenes, one clip per schedule in each pair. The default ordering produced 196 successful cuts. Reducing query noise by 15% on just the first chunk of the new shot raised that count to 223 (+9.4 percentage points, 95% CI [+1.0, +18.4]). The same reduction made little difference for a new view of the same subject.

We chose the reduction on validation scenes. It applies only to the score models' training queries. The generator uses its own denoised history. A cut counted as successful if at least five of nine raters saw the requested shot for at least eight consecutive frames within the first 12, without ghosting, split frames or collapse. We computed the interval by resampling scenes with their matched clips.

In a separate test with two cuts per clip (A→B→A), none of the tested variants completed both switches. We will report the full cut evaluation in the appendix. Handling repeated scene changes reliably remains future work.


(W2/Q2) Human evaluation with more raters

We expanded the human study from three raters to nine. Among the six new raters alone, overall net preference favored ViDA by 19.14 percentage points over matched scalar (one noise level shared across chunks) and 12.89 points over official Self Forcing.

The six new raters had no role in the project. Each independently judged the same five-second video pairs for the original 128 prompts. As in Appendix A.3, method names were hidden, left–right placement and presentation order were randomized, and ties were allowed.

The full panel also favored ViDA on both questions. The table reports net preference in percentage points, with 95% confidence intervals.

Comparison (all nine raters)Overall preferenceTemporal coherence
ViDA vs. matched scalar+18.49 [+10.24, +26.65]+11.37 [+4.34, +18.40]
ViDA vs. official Self Forcing+11.46 [+7.08, +15.84]+15.54 [+7.99, +22.92]

Net preference is the percentage of ViDA wins minus the percentage of losses, with ties included in the denominator. Each table entry uses 1,152 ratings (128 prompts × 9 raters). The intervals resample prompts for these fixed checkpoints and raters. We will add the nine-rater results alongside Table 6 and summarize them in Sec. 4.5.

Dear Reviewer mq1Z,

Thank you again for your helpful suggestions. We hope the additional scene-cut experiments and expanded human study with nine raters in our response help address your concerns. Please let us know if you have any remaining questions or concerns. We would be happy to provide further clarification.

Thank you for your time and consideration.

Dear authors, I took some time waiting for the other reviewers to follow-up on their review. As I already stated previously, my expertise is very far from this domain. I reviewed the rebuttal from the authors, to both my questions and the questions from the other reviewers. These rebuttals added some clarifications and some new evidence to the original document. The other reviewers responded positively to the paper and even raised the score after the rebuttal. I trust their judgment, and I recognize the effort form the authors. Overall, I decided to raise my score to 3 -- Weak Accept.

However, I would like to repeat that this paper is very far from my expertise. I kindly invite the AC and the other reviewers to take this into account for their final decisions.

Best regards

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Discussion and Manuscript Revisions

Dear AC and Reviewers,

We sincerely thank the AC and all reviewers for the constructive discussion. As the author-reviewer discussion period draws to a close, we summarize the discussion outcomes, the evidence supporting ViDA's contributions, and the revisions incorporated into the manuscript.

1. Discussion outcomes and supporting evidence

  • Reviewer mq1Z stated a decision to raise the initial rating from 2 to 3: "Overall, I decided to raise my score to 3 -- Weak Accept." The reviewer also noted their limited expertise in the area.
  • Reviewer vaND expressed an inclination to increase the initial rating of 2, writing, "since most of my other concerns have been addressed, I am inclined to increase my score." The reviewer retained a minor concern about the narrower sharing advantage at higher update ratios and generalization beyond the evaluated setting.
  • Reviewer F93Y was inclined to maintain the rating of 3, writing, "The extra experiments and added details have basically addressed my concerns." The reviewer noted that the missing external 60-second baseline does not affect the role of ordered queries within the ablation.

ViDA introduces temporally ordered DMD queries to preserve generated history as useful context while keeping it subject to correction, together with role-conditioned sharing for efficient training. The autoregressive inference procedure remains unchanged.

  • Evidence for temporal ordering. With independent bidirectional score estimators and the same adapted teacher, ordered queries improve three-run mean VBench Total by 0.75 over matched scalar, 0.64 over shuffled, and 1.40 over reversed queries (Table 1). These controls demonstrate the ordering benefit independently of parameter sharing. Query gains persist in role-shared causal training (Tables 7-8), with the 14B teacher (Table 3), and on Cosmos (Table 4).
  • Measured training efficiency. The shared design at r = 2 achieves 85.05 mean VBench Total versus 84.52 for the independent causal design at r = 5, at half the student post-training cost (110 versus 220 H100 GPU-hours; Table 2). We clarify that the GPU-hour saving comes from fewer fake updates; sharing reduces peak allocated memory by 13.6% at fixed r = 2. The added shared r = 5 results support the lower-cost choice of r = 2.
  • Expanded validation. The expanded nine-rater study favors ViDA over matched scalar and Self Forcing in both overall preference and temporal coherence (Table 6). Local VBench (85.62 versus 84.37 for Causal Forcing and 84.24 for Self Forcing; Table 3) and VideoReward comparisons (Appendix A.14) also favor ViDA over both baselines. We tested the suggested scene-aware noise adjustment and documented the remaining limitation on repeated scene cuts.

2. Manuscript revisions

  • Controlled comparisons, run counts, and savings attribution: Sections 4.2-4.3 and Tables 1-2 (pp. 6-7), with individual-run results in Appendix A.6 (Table 9 and accompanying text, p. 17).
  • Local Causal Forcing and expanded human evaluation: Sections 4.4-4.5 and Table 3 (p. 8), and Appendix A.3 and Table 6 (pp. 14-15).
  • Cosmos initialization, update order, and teacher-adaptation settings, costs, and losses: Appendices A.8, A.9, and A.12 (pp. 18-20).
  • Scene-cut and VideoReward evaluations: Appendices A.13-A.14 (pp. 20-21).

We thank the AC and reviewers again for their time and careful consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.