- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces History Dropout (HiDrop), a history regularization method for long-horizon autoregressive video generation. During training, it applies step-dependent dropout to historical attention values while retaining all keys for softmax normalization and scaling surviving historical values by the inverse retention probability. During inference, it instead independently samples historical key-value tokens per attention head and attention call, while retaining sink and current-chunk tokens. The method leaves the underlying rollout objective unchanged. Experiments on Self Forcing, Causal Forcing, and LongLive report improvements in aesthetic quality, imaging quality, and dynamic degree at 60 and 240 seconds.
Critical strengths
-
A small, practical intervention. There are no extra learned parameters or auxiliary losses, making the method relatively easy to integrate with existing rollout training.
-
Improvements across multiple baselines. Table 1 reports improvements in aesthetic quality, imaging quality, and dynamic degree across all three evaluated baselines.
Critical weaknesses
-
The training-inference mismatch requires stronger justification and controls. Training uses historical value dropout with full-key normalization and inverse-retention scaling, whereas inference removes both keys and values, renormalizes over the selected keys, and uses independently sampled per-head masks. These operations differ in both attention normalization and mask sharing. The authors do not compare against using the same per-head KV dropout during both training and inference under matched settings. Thus it remains unclear whether the asymmetric design is preferable to a simpler design that uses the same selection rule at both stages. The authors need to provide stronger justification for this design.
-
There is no direct quantitative evaluation of drift over time. The paper's motivation concerns error propagation through generated history, but the quantitative experiments primarily report holistic VBench scores for long videos. These results do not directly establish that quality or identity degrades more slowly within a rollout, or that historical errors propagate less strongly. The drifting metric used in Longlive and Rolling Forcing can be used.
Questions for authors
-
Could the authors compare the proposed value-dropout training/KV-selection inference design against the same per-head KV selection used during both training and inference, with matched retention rates, inference settings, and training budgets? Does the asymmetric design offer an advantage, and what explains that advantage?
-
Could the authors provide quantitative drift measurements on the same prompt rollouts, for example by tracking quality, identity, color, and motion across consecutive temporal windows? Alternatively, could controlled perturbations of generated history demonstrate that HiDrop reduces the error accumulation?
4 comments
Q2: Can quantitative drift measurements on matched rollouts, or controlled perturbations of generated history, demonstrate that HiDrop reduces error accumulation?
HiDrop reduces mean appearance and color drift and better preserves motion dynamics across all three models at 60s and 240s. It also improves the model's ability to continue generation from imperfect history. Please refer to Appendix C.5 (Figures 6–7) and Appendix C.7 (Table 9) in the revised manuscript for the temporal-drift and shared-history continuation experiments.
-
Temporal drift. (1) Settings: We evaluate Self Forcing, Causal Forcing, and LongLive using 326 prompts for 60s videos (VBench-Long) and 128 prompts for 240s videos (MovieGen). Each rollout is divided into non-overlapping 5s windows. AQ and IQ drops are measured relative to the first window; subject and background drift use DINO and CLIP similarity to the first frame; color drift compares normalized hue histograms with the first window. DD change tracks the proportion of dynamic clips relative to the initial window. The curves include pointwise 95% confidence intervals from 2,000 bootstrap resamples of matched baseline–HiDrop video pairs. (2) Results: Averaged over subsequent windows of the 240s videos, HiDrop reduces subject drift by 26–40% and color drift by 26–29%, with smaller AQ drops across all three models. It also better preserves dynamics: the mean change in dynamic proportion from the initial window improves from −19.78 to −8.49 percentage points for Causal Forcing and from −12.27 to +1.30 percentage points for LongLive. Please refer to Figures 6 and 7 in the revised manuscript for the full temporal curves.
-
Continuation from imperfect history. To assess robustness to history perturbations, we use a 10s baseline-generated prefix as a proxy for perturbed ground-truth history. The baseline and HiDrop then each continue generation to 60s from the same historical KV cache and subsequent noise state. This compares their continuation quality under identical imperfect conditioning. VBench evaluates the complete videos, including the shared prefix. HiDrop improves AQ, IQ, and DD across all three backbones (Appendix C.7, Table 9).
Table 9. 60s generation from a shared baseline-generated prefix.
| Backbone | Continuation | AQ ↑ | BC ↑ | DD ↑ | IQ ↑ | MS ↑ | SC ↑ | TF ↑ |
|---|---|---|---|---|---|---|---|---|
| Self Forcing | Baseline | 0.582 | 0.953 | 0.693 | 0.703 | 0.979 | 0.965 | 0.981 |
| Self Forcing | Baseline + Ours | 0.598 | 0.954 | 0.765 | 0.706 | 0.980 | 0.966 | 0.982 |
| Causal Forcing | Baseline | 0.562 | 0.958 | 0.451 | 0.702 | 0.986 | 0.964 | 0.982 |
| Causal Forcing | Baseline + Ours | 0.602 | 0.960 | 0.553 | 0.709 | 0.986 | 0.970 | 0.983 |
| LongLive | Baseline | 0.573 | 0.951 | 0.709 | 0.694 | 0.975 | 0.960 | 0.982 |
| LongLive | Baseline + Ours | 0.595 | 0.958 | 0.773 | 0.698 | 0.978 | 0.968 | 0.982 |
Changes in the paper: Appendix C.5 and Figures 6–7 report temporal drift. Appendix C.7 and Table 9 report continuation from a shared baseline-generated history.
Thank the authors for the detailed rebuttal and the additional experiments. The new ablation studies provide a clearer justification for the asymmetric training-inference design, showing the trade-offs between generation quality and computational efficiency. The temporal drift analysis and shared-history continuation experiments provide more evidence supporting the effectiveness of HiDrop in mitigating error accumulation. I believe my major concerns have been addressed, and I will raise my score accordingly.
Thank you for your constructive feedback
Dear Reviewer M6T6,
Thank you for carefully considering our rebuttal and additional experiments, and for updating your assessment. We are glad that these additions have addressed your major concerns. Your constructive suggestions helped us clarify the training–inference design and strengthen our evaluation of temporal drift and error accumulation.
We sincerely appreciate your time, thoughtful feedback, and support.
Best regards, The Authors
We appreciate the reviewer’s insightful questions and provide our responses below.
We conduct an ablation on different configurations, showcasing that: (1) Using per-head KV dropout during both training and inference (Config. A) yields lower imaging quality and dynamics than HiDrop at both 60s and 240s; (2) Our asymmetric design balances generation quality, training time, and inference efficiency. Please refer to Appendix C.6 (Table 8 and Figure 8) in the revised manuscript for the quantitative and qualitative comparisons.
Table 8. Training configurations and results on Self Forcing. Mask sampling and value dropout refer to training; all configurations use per-head KV selection at inference. Value dropout includes inverse-retention scaling. Gen. step denotes time per generator training step.
The following comparisons explain our training and inference choices.
Training operator: A vs. B and Ours. With per-head training masks, B improves visual quality and dynamics relative to A, and achieves scores comparable to Ours. This combining Table 2 supports value dropout as the training operator.
Training efficiency: B vs. Ours. Compared with B, Per-step masks reduce generator training-step time from 132s to 121s, an 8.3% reduction, with comparable generation quality. This is because independent per-head masks increase the complexity of the Self Forcing training graph. We therefore use per-step masks during training.
Inference efficiency. KV selection shortens attention inputs, reducing steady-state chunk latency from 0.72s to 0.64s (Table 1). We observe no clear improvement in generation quality from value dropout at inference, while chunk latency increases from 0.64s to 0.72s. We therefore retain per-head KV selection at inference while using value dropout during training.
Changes in the paper: Appendix C.6 and Table 8 compare the three configurations and report generator training-step time. Figure 8 presents the corresponding qualitative results.