acceptodds
Under review as a conference paper at ICLR 2027

Mask Forcing: Improving Autoregressive Video Diffusion Distillation via Dual-Noise Masking Rollout

Abstract

Autoregressive (AR) video diffusion models have shown great potential in real-time video generation. Recent methods distill pretrained bidirectional video diffusion models into causal students through Distribution Matching Distillation (DMD), but the generated videos often suffer from over-saturation and over-smoothing issues, resulting in limited visual quality and realism. The key contributing factor is the mode-seeking behavior of the reverse KL objective in DMD, which can cause the student distribution to collapse onto only a few modes of the teacher distribution. To address this, we propose Mask Forcing, a Dual-Noise Masking Rollout strategy that perturbs the causal student self-rollout to mitigate mode collapse induced by reverse-KL mode seeking. The core idea is to inject cleaner signals into noisy rollout inputs via random masks along spatial and temporal axes during the self-rollout process of AR diffusion distillation. Such perturbations encourage the student rollouts to explore broader regions of the teacher distribution, allowing DMD to provide learning signals beyond the modes already covered by the student. Moreover, the cleaner tokens act as denoising guidance for other noisier tokens, improving the intermediate rollout predictions and reducing error accumulation. Extensive experiments demonstrate that our method improves multiple AR video diffusion distillation methods with higher visual quality efficiently, without incorporating real video data or additional post-training stages.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
diffusion models, video generation, autoregressive generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_FXjAOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Mask Forcing on the autoregressive DMD distillation framework of Self Forming, which applies a Dual-Noise Masking Rollout strategy to the input token at each step of the student model's self-rollout. Randomly select a certain proportion of tokens and inject low-noise signals into the noisy rollout inputs to construct inputs with inconsistent local noise levels. The authors motivate this design as a way to improve teacher-mode coverage and reduce error accumulation, and evaluate it on Self Forcing, Causal Forcing, and LongLive, including a 30-second LongLive setting.

Critical strengths
  • This paper is well written and easy to follow.

  • The experimental evaluation is comprehensive, involving multiple metrics, and the results support the effectiveness of the method.

  • The method is simple and direct, requiring no additional training stages or auxiliary components and incurring low extra cost.

Critical weaknesses
  • I think the overall idea of the novel is relatively low, similar to diffusion forcing, and the core is to add noise disturbance to the history. I think this is more like training regularization that improves model robustness and generation quality.

  • Local injection of lower noise tokens may cause a mismatch between the input and global timestep conditions, and the paper has not fully discussed its side effects.

Questions for authors
  • Can the author report the complete VBench Long results for each dimension, not just the overall score? This will clarify the specific benefits of indicators such as consistency and dynamism, and reveal any trade-offs across dimensions.

  • Are the samples used to train the fake score model also drawn from the same dual-noise masked rollout?

1 comment

Thank you for acknowledging Mask Forcing as a simple and direct method with comprehensive experimental evaluation, and giving positive feedback. We address your concerns as follows:

 

W1: Comparison with Diffusion Forcing and history noise augmentation

We thank the reviewer for this comment. We would like to clarify that Mask Forcing perturbs the intermediate denoising inputs of the current chunk, while previously generated history remains uncorrupted. Specifically, the "previous prediction" in Eqs. (8)–(10) refers to the prediction from the preceding denoising step within the same chunk, rather than a previously generated chunk. The completed clean prediction is written to the history KV cache (Algorithm 1).

While both methods involve heterogeneous noise levels, Diffusion Forcing assigns each frame an independently sampled noise level during diffusion training on ground-truth sequences. Mask Forcing introduces masked dual-noise perturbations during autoregressive self-rollout distillation with the DMD objective. We agree that this can have a regularization effect. Our contribution lies in this specific design for autoregressive video distillation and its demonstrated effectiveness.

We additionally evaluate adding noise disturbance to the history as a baseline during autoregressive diffusion distillation in the revision (Appendix J Table 11). Mask Forcing achieves higher HPSv3 (9.84 vs. 9.77), Dynamic Degree (82 vs. 45), and vision reward score (11.37 vs. 9.99). These results support the advantages of Mask Forcing over adding noise disturbance to the history.

 

W2: Discussion of mismatch between inputs and global timestep conditions

We thank the reviewer for raising this point. The mismatch serves as a perturbation to the student’s self-rollout trajectory. Perturbing the rollout trajectory can expose DMD to a broader range of student samples, providing learning signals beyond the modes the student already covers.

Tables 3 and 4 in the paper analyze the effects of this perturbation by varying the mask ratio α\alpha and timestep window Δ\Delta. These parameters control the proportion of tokens at lower noise levels and the maximum timestep difference, respectively. The results reveal trade-offs between visual quality and motion dynamics, which show that increasing the extent of the mismatch does not consistently improve performance. Excessive guidance from cleaner tokens may constrain motion in subsequent denoising, while insufficient perturbation can limit the benefits. We therefore adopt α=0.2\alpha = 0.2 and Δ=250\Delta = 250, achieving HPSv3 of 9.84 and Dynamic Degree of 82 to balance visual quality and motion dynamics.

 

Q1: Complete VBench-Long results

We thank the reviewer for this suggestion. We report the complete per-dimension VBench-Long results below. Mask Forcing improves 12 of the 16 dimensions, matches LongLive on human action, and shows decreases in color, dynamic degree, and appearance style. The largest gains are in spatial relationship (+4.18) and imaging quality (+2.19), alongside improvements in subject and background consistency, temporal flickering, and motion smoothness. The main trade-offs are lower color and dynamic-degree scores, while appearance style remains nearly unchanged. We include the complete results in the revision (Appendix K).

MethodSubject Cons.↑Background Cons.↑Temporal Flickering↑Motion Smoothness↑Dynamic Degree↑Aesthetic Quality↑Imaging Quality↑Object Class↑Multiple Objects↑
LongLive97.3495.8398.8997.4971.8564.6469.6294.2382.92
+ Mask Forcing97.7595.9498.9797.7670.0065.2671.8194.9284.92
MethodHuman Action↑Color↑Spatial Relationship↑Scene↑Appearance Style↑Temporal Style↑Overall Cons.↑Quality↑Semantic↑Total↑
LongLive10087.1281.7951.3221.4725.2126.7484.7180.7083.91
+ Mask Forcing10084.0385.9752.4621.4425.4927.0685.2881.4484.51

 

Q2: Training samples for the fake score model

Yes. The samples used to train the fake score model are drawn from the same dual-noise masked student rollout. The resulting clean samples are then re-noised using the standard diffusion noising process for fake score training. Dual-noise masking is applied only during student rollout, not during this subsequent re-noising step.

Reviewer_iLNWOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Mask Forcing, a simple perturbation strategy for few-step autoregressive video diffusion distillation. Specifically, during (self-forcing-based) student self-rollout, it keeps the original timestep conditioning and DMD objective, but mixes re-noised clean tokens at two different noise levels to essentially augment the input of the next denoising step. This is intended to encourages exploration of more teacher modes and provides cleaner context for subsequent denoising. Inference follows the original schedule. The paper evaluates the method with Self Forcing, Causal Forcing and LongLive, including short and long videos (demos provided), additional distillation baselines, and a preliminary camera-controlled generation application.

Critical strengths
  1. The idea is simple but seems effective, and the paper is clearly written and easy to follow. Method, figures and experiment results are thoroughly presented, and the supplementary video comparisons are very helpful. The improvements in oversaturation and fine details are visible as claimed.

  2. The experiments are quite comprehensive and convincing. It evaluates Self Forcing, Causal Forcing and LongLive in both chunk-wise and frame-wise settings, using VBench, long-video evaluation and human preferences. Appendix B also includes very recent and relevant approaches such as DistillAlign and Causal-rCM. The numerical and qualitative results provide good support for the practical quality gains.

  3. I appreciate the analysis of convergence, which gives useful information about training behavior, and Tables 3–5 study the important hyperparameters and mask configurations, providing insightful information on the design choices and their influence.

Critical weaknesses
  1. I think the experiments do not directly establish that the student covers more teacher modes. The KL decomposition explains a property of mixtures, but does not show that this particular masking strategy reaches previously missed teacher modes. A more direct analysis would be helpful.

    • Following this point, this paper also lacks comparisons with simpler rollout perturbations, such as varying the noise level uniformly within a chunk or adding noise to the generated history, or any other relevant methods for such an essential augmentation purpose.
  2. The long-video results show better quality throughout the rollout, but I am less sure they establish reduced error accumulation. In Table 8, Mask Forcing starts and stays higher, but the HPSv3 dropping trend from the first to the last interval is similar for both methods. It is also not very visible in the Single-Prompt Long Video demos (I think both LongLive and Mask-Forcing kind of equally perform well in terms of degradation over time).

  3. Most of the examples being tested and shown have little dynamics, such as small camera motion or subject motion, so whether the gains remain reliable in more complicated cases during a long rollout is unclear.

Questions for authors

I think my questions are covered in the weaknesses detailed above, I hope the authors can respond to them.

2 comments

Thank you for acknowledging Mask Forcing as a simple and effective method with comprehensive analysis and convincing experimental results. We address your concerns as follows:

 

W1.1: Analysis of broader mode coverage

We thank the reviewer for this suggestion. To evaluate teacher-distribution coverage more directly, we additionally measure precision and recall following [1], and coverage following [2], using V-JEPA2 features from videos generated by students and the teacher on the 100-prompt set. The table below compares Self Forcing and Mask Forcing for k=3,5,10, where k determines the local neighborhood radius as the distance to the k-th nearest neighbor.

kMethodPrecision (%) ↑Recall (%) ↑Coverage (%) ↑
3Self Forcing826366
3Mask Forcing937277
5Self Forcing927275
5Mask Forcing948486
10Self Forcing999084
10Mask Forcing999498

As observed, Mask Forcing consistently improves recall and coverage across k=3,5,10, while maintaining or improving precision. These results support broader coverage of the teacher reference distribution in the evaluated feature space. We further include these results and the evaluation details in the revision (Appendix H).

[1] Kynkäänniemi et al., “Improved Precision and Recall Metric for Assessing Generative Models,” NeurIPS 2019.

[2] Naeem et al., “Reliable Fidelity and Diversity Metrics for Generative Models,” ICML 2020.

 

W1.2: Comparison with simpler rollout perturbations

We thank the reviewer for this suggestion. We additionally compare Mask Forcing with two simpler rollout perturbation baselines: (1) uniform chunk perturbation, which applies one sampled lower noise level uniformly within a chunk using the same sampling window, and (2) history noise augmentation, which perturbs generated chunks before updating the history KV cache, with noise strength uniformly sampled between 0 and 0.05. Both are applied only during training, and inference follows the original schedule.

MethodHPSv3 ↑Vision. ↑Instruct. ↑MQ ↑Dynamic. ↑
Self Forcing9.5510.1038.5015.8870
+ Uniform chunk perturbation9.4711.0044.8618.3489
+ History noise augmentation9.779.9937.0619.2545
+ Mask Forcing9.8411.3745.0320.4982

As observed, uniform chunk perturbation achieves the highest Dynamic Degree but lower HPSv3 than Self Forcing. History augmentation improves HPSv3 and motion quality, but reduces dynamics and the overall vision reward score. Mask Forcing achieves the highest HPSv3 and vision reward scores while maintaining strong dynamics, supporting its benefit over these simpler perturbations. We include these comparisons and implementation details in the revision (Appendix J).

 

W2 & W3: Error accumulation & motion dynamics in long videos

We thank the reviewer for these observations. Table 8 mainly demonstrates that Mask Forcing maintains consistently higher visual quality than LongLive throughout the rollout. Mask Forcing also exhibits a smaller relative HPSv3 decrease of 10.05% from the first to the last interval, compared with 11.58% for LongLive.

To further illustrate error accumulation during long rollouts and compare performance under strong camera or subject motion, we added Figure 13 in the Appendix, presenting additional single-prompt long-video comparisons on VBench-Long. Cases 1–3 feature strong camera or subject motion, where LongLive shows progressive degradation in visual quality during long rollouts, while Mask Forcing better maintains visual quality and fine-grained details over time. In cases 4–5 with relatively subtle camera or subject motion, LongLive gradually loses texture fidelity and object consistency, whereas Mask Forcing better preserves fine-grained textures and consistent object appearance throughout the generated videos. More qualitative results are shown in Figure 14.

Reviewer_iLNWOpenReview ↗

Final Rating

I would like to thank the authors for their rebuttal, which address most of my concerns. After carefully reviewing the comments from all reviewers and the revised manuscript, I am happy to keep my rating.

Reviewer_UuGfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Mask Forcing, a training strategy for autoregressive video diffusion distillation. At training stage, during student self-rollout, random masks mix inputs at the scheduled noise level with lower-noise versions of the student’s intermediate predictions, while retaining the original timestep conditioning and DMD objective. At inference, the original procedure remain unchanged. This method aims to improve mode coverage and reduce error accumulation. Experiments on Self Forcing, Causal Forcing, and LongLive report improvements in generation quality and diversity.

Critical strengths
  • Interesting way of reducing mode-seeking in DMD training: Applying dual-noise masking to the student’s own intermediate predictions is an interesting way to modify the rollout distribution used by DMD. Its integration into autoregressive distillation provides a useful contribution.

  • Consistent empirical improvements: Table 1 reports improvements across three baselines in both chunk-wise and frame-wise settings. Human preference, multi-seed diversity evaluation, and comparisons with Causal-rCM and DistillAlign provide additional support. The long-video experiments further demonstrate its applicability.

  • Practical integration: The method requires no additional model forward passes or real-video supervision during DMD training. It also retains the original inference procedure, making it straightforward to incorporate into existing pipelines.

Critical weaknesses
  • Unclear mechanism for broader mode coverage: Eq. (12) describes a mixture of masked rollout distributions, but does not explain why different masks lead to distinct teacher modes or broader coverage than standard self-rollout. The diversity gains in Table 7 are encouraging, but do not isolate this mechanism. More analyzes are needed.

  • Unexplained training–inference mismatch: The additional cleaner context is provided only during training, whereas inference uses the original noise schedule. Although inference quality improves, the paper does not explain how the cleaner-context mechanism transfers across this mismatch or directly verify the claimed reduction in error accumulation at inference.

Questions for authors
  • What evidence shows that different masking trajectories lead student rollouts to additional teacher modes? Eq. (12) describes the resulting mixture, but does not establish why lower-noise masking produces this broader coverage.

  • Why should the denoising benefit of cleaner context during training carry over to standard inference, where this additional context is absent? What evidence supports the claimed reduction in error accumulation under this training–inference mismatch?

2 comments

Thank you for acknowledging Mask Forcing as an interesting way to reduce mode-seeking in DMD and an effective approach with practical integration into existing pipelines. We address your concerns as follows:

 

W1 & Q1: Mode coverage analysis

We thank the reviewer for this comment. Eq. (12) provides a theoretical analysis of the relationship between rollout-distribution diversity and student–teacher alignment. Its second term quantifies rollout-distribution diversity and reduces the mixture’s marginal reverse KL relative to the average conditional reverse KL, explaining how diversity across mask-conditioned rollout distributions can improve student–teacher alignment. This motivates dual-noise masking as a way to diversify the samples evaluated by DMD and explore teacher-supported regions underexplored by standard self-rollout.

Table 7 reports output diversity, while Figs. 1 and 5 measure student–teacher distributional alignment using CMMD and VMMD against a teacher reference set throughout training. Mask Forcing achieves lower CMMD and VMMD than the corresponding baselines, indicating improved alignment with the teacher distribution. Together, these results show that increased diversity accompanies improved distributional alignment.

To evaluate teacher-distribution coverage more directly, we additionally measure precision and recall following [1], and coverage following [2], using V-JEPA2 features from videos generated by students and the teacher on the 100-prompt set. The table below compares Self Forcing and Mask Forcing for k=3,5,10, where k determines the local neighborhood radius as the distance to the k-th nearest neighbor.

kMethodPrecision (%) ↑Recall (%) ↑Coverage (%) ↑
3Self Forcing826366
3Mask Forcing937277
5Self Forcing927275
5Mask Forcing948486
10Self Forcing999084
10Mask Forcing999498

As observed, Mask Forcing consistently improves recall and coverage across k=3,5,10, while maintaining or improving precision. These results support broader coverage of the teacher reference distribution in the evaluated feature space. We further include these results and the evaluation details in the revision (Appendix H).

[1] Kynkäänniemi et al., “Improved Precision and Recall Metric for Assessing Generative Models,” NeurIPS 2019.

[2] Naeem et al., “Reliable Fidelity and Diversity Metrics for Generative Models,” ICML 2020.

 

W2 & Q2: Training-inference mismatch and error accumulation

The cleaner context is constructed from the student’s own previous prediction, without introducing additional ground-truth or teacher-generated context. Dual-noise masking improves training self-rollouts and the samples used for DMD updates. These DMD updates improve the generator, enabling it to produce higher-quality outputs during standard inference without requiring additional cleaner context.

To evaluate how errors in autoregressive context affect subsequent generation, we additionally conduct a controlled context-perturbation experiment on the 100-prompt set using Self Forcing and Mask Forcing. Under the original inference noise schedule, we perturb the second chunk’s predicted latent with Gaussian noise scaled to 50% of its standard deviation before updating the KV cache. Clean and perturbed runs share the same sampling randomness. We report HPSv3 for subsequent chunks and define d=HPSv3clean−HPSv3perturbedd = \mathrm{HPSv3}_{\mathrm{clean}} - \mathrm{HPSv3}_{\mathrm{perturbed}} as the score decrease under context perturbation.

ChunkSelf Forcing CleanSelf Forcing Perturbedd ↓Mask Forcing CleanMask Forcing Perturbedd ↓
39.6689.5100.1589.8579.8360.022
49.5689.615−0.0479.7969.949−0.153
59.5769.4820.0939.8789.8640.013
69.5079.4440.0639.7149.6980.016
79.4089.2420.1669.6949.716−0.021
Average9.5459.4590.0879.7889.812−0.025

As observed, Mask Forcing achieves higher HPSv3 scores across all evaluated chunks under clean inference, demonstrating that its training-time benefits transfer to standard inference. Moreover, it exhibits smaller degradation across subsequent chunks under context perturbation. The average HPSv3 of Self Forcing drops by 0.087 while Mask Forcing maintains its average quality. These results demonstrate improved robustness to errors in autoregressive context and support reduced error accumulation during inference. We further include these results and the evaluation details in the revision (Appendix I).

Thanks for the detailed rebuttal. Both of my concerns are addressed, and I will retrain my score as weak acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.