acceptodds
Under review as a conference paper at ICLR 2027

RaMP: Rollout-Aware Mode Preconditioning for Video Distillation

Abstract

Few-step autoregressive video generators enable efficient inference, but prolonged self-forcing distribution matching distillation (SF-DMD) can lead to oversaturated colors and abnormal contrast. We attribute this failure to SF-DMD favoring narrow, undesirable regions that are easier for an imperfect student to fit than broader high-quality regions, with fake-score errors further amplifying this bias. These observations motivate reducing undesirable mode before SF-DMD. We therefore propose RaMP, a rollout-aware mode-preconditioning framework that gradually shifts consistency distillation from teacher-forced states to states visited by the student's own autoregressive rollouts before switching to pure SF-DMD. Autoregressive rollout can move states out of the basin of narrow undesirable modes and into transition regions where the broader high-quality mode dominates the teacher posterior. Consistency supervision can then correct these rollout-induced deviations, while the resulting improved prefixes provide better contexts for subsequent autoregressive blocks, progressively reducing undesirable visitation before SF-DMD. Compared with existing autoregressive distillation methods, RaMP achieves the best overall VBench score and lowest HSV saturation, while maintaining strong CFM-7B performance, effectively reducing oversaturation without compromising generation quality.

Primary area:
generative models
Keywords:
Autoregressive Video Generation, Diffusion Distillation, Distribution Matching Distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8WShOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces RaMP, a rollout-aware mode-preconditioning framework designed to reduce oversaturation and bad-mode collapse in few-step autoregressive video distillation. The authors examine how imperfect student initialization and inaccurate fake-score estimation can amplify undesirable modes during SF-DMD. To counter this, RaMP gradually shifts from teacher-forced to self-forcing consistency distillation and leverages student rollout states to improve initialization before standard SF-DMD. Experiments on Wan2.1-T2V-1.3B show gains in VBench scores and color fidelity, and controlled Gaussian-mixture experiments together with ablation studies further support the proposed mechanism.

Critical strengths
  • Insightful analysis of bad-mode collapse. The paper offers a insightful examination of two possible causes of bad-mode collapse in SF-DMD, namely imperfect student initialization and inaccurate fake-score estimation, and supports these explanations with both theoretical derivations and controlled experiments.

  • Comprehensive experiments. The experimental evaluation is fairly comprehensive, covering quantitative and qualitative comparisons, controlled Gaussian-mixture experiments, and ablation studies, all of which support the effectiveness of the proposed method.

  • Clear presentation. The paper is well written and clearly organized. The motivation, theoretical analysis, and proposed method are easy to follow, and the figures and visualizations are helpful.

Critical weaknesses
  • Insufficient training cost comparison. Although the authors highlight the use of only 8 GPUs and a comparable number of training steps, the actual training time and computational cost are not clearly reported. Since different training stages may have substantially different per-step costs, a more detailed comparison of GPU-hours would strengthen the efficiency claims.

  • Limited generalization across models and settings. The experiments are primarily conducted on Wan2.1-T2V-1.3B. Evaluations on larger models or longer video generation would help demonstrate the generalizability of the proposed method.

Questions for authors
  • How sensitive is RaMP to the choice of backbone and video length? Is there any evidence that the improvements extend to larger models or longer autoregressive rollouts?

  • Beyond the Gaussian-mixture experiments, is there any direct evidence from real video models showing that oversaturation stems from bad-mode amplification, particularly when it is driven by inaccurate fake-score estimation?

3 comments

Thank you for the positive assessment of the mechanism analysis and for requesting clearer evidence on cost, longer rollouts, and fake-score estimation. We address these separately so that the scope of the evidence is explicit.

  • W1: Training time and computational cost comparison.

Thank you for the suggestion. We report the per-100-step cost and total GPU-hours for both training stages below. All timings are measured on NVIDIA H200 GPUs, using 8 GPUs for RaMP and the official 64-GPU recipe for Causal Forcing++ [1].

Method / Stage# GPUsStepsTime / 100 StepsTotal GPU-Hours
Causal Forcing++ (Init, official)643,00018.5 min592.0
RaMP (Init)85,50030.0 min220.0
Causal Forcing++ (SF-DMD, official)641,00023.0 min245.3
RaMP (SF-DMD)81,00023.0 min30.7
Total (Causal Forcing++, official)644,000—837.3
Total (RaMP)86,500—250.7

Although RaMP's mixed TF-CD/SF-CD initialization has a higher per-step cost (30.0 vs. 18.5 min / 100 steps) and more steps (5,500 vs. 3,000), its 8-GPU training reduces total compute from 837.3 to 250.7 GPU-hours (3.3× reduction).

  • Q1&W2: Generalization across backbones and longer autoregressive rollouts.

Following all compared baselines (CausVid[2], Self-Forcing[3], Causal Forcing[4], Causal Forcing++, and DistillAlign[5]), we use Wan2.1-T2V-1.3B as the student backbone because official baseline code and checkpoints are only available on this model, ensuring a fair comparison.

To assess generalization to longer horizons, we additionally evaluate our 81-frame-trained models on 161-frame autoregressive rollouts, approximately twice the training length. As shown in the table below, on the VBench-Long[6] benchmark, RaMP consistently outperforms Causal Forcing++ across Total, Quality, and Semantic scores, while reducing HSV saturation and improving CFM-7B. These results support RaMP’s generalizability to longer videos, with consistent improvements in generation quality and color fidelity.

Method (161 frames)Quality ↑\uparrowSemantic ↑\uparrowVBench-Long Total ↑\uparrowHSV ↓\downarrowCFM-7B ↑\uparrow
Causal Forcing++0.83460.66040.79980.478810.61
RaMP0.84080.68740.81010.414011.21
  • Q2: Direct evidence from real video models

Thank you for raising this question.

We conduct a real-video ablation using 1, 5, or 10 fake-score updates per student update, while keeping the total number of student updates fixed. We use this ratio as a practical proxy for fake-score estimation accuracy: more updates allow the fake-score model to better track the evolving student distribution. As shown in the table below, increasing the ratio from 1 to 5 and 10 raises VBench Total from 0.7674 to 0.7998 and 0.8040, and Quality from 0.7984 to 0.8333 and 0.8474. These results show that fake-score training materially affects real-video distillation, consistent with our hypothesis that insufficient fake-score estimation can impair optimization.

Fake Updates / Gen StepTotal Fake UpdatesQuality ↑\uparrowSemantic ↑\uparrowVBench Total ↑\uparrow
12000.79840.64350.7674
51,0000.83330.66580.7998
102,0000.84740.63040.8040

Should you have any additional questions, please do not hesitate to follow up. We greatly look forward to your thoughtful comments.

Reference

[1] Zhao M, Zhu H, Zheng K, et al. Causal forcing++: Scalable few-step autoregressive diffusion distillation for real-time interactive video generation[J]. arXiv preprint arXiv:2605.15141, 2026.

[2] Yin T, Zhang Q, Zhang R, et al. From slow bidirectional to fast autoregressive video diffusion models[C]//2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025: 22963-22974.

[3] Huang X, Li Z, He G, et al. Self forcing: Bridging the train-test gap in autoregressive video diffusion[J]. Advances in Neural Information Processing Systems, 2026, 38: 167283-167308.

[4] Zhu H, Zhao M, He G, et al. Causal forcing: Autoregressive diffusion distillation done right for high-quality real-time interactive video generation[J]. arXiv preprint arXiv:2602.02214, 2026.

[5] Li J, Zou K, Zhou C, et al. Distillalign: Coordinating mode covering and mode seeking in autoregressive video distillation[J]. arXiv preprint arXiv:2607.26811, 2026.

[6] Huang Z, Zhang F, Xu X, et al. Vbench++: Comprehensive and versatile benchmark suite for video generative models[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.

Thank you to the authors for the response. These answers have addressed my questions. Since I have already provided positive feedback, I will keep my score unchanged.

Thank you for keeping your positive score!

Reviewer_bzcROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why SF-DMD can amplify artifacts such as oversaturation, using a simple two-mode analysis and toy experiments. It proposes RaMP, which gradually mixes teacher-forced and student-rollout consistency distillation before SF-DMD. Experiments show improvements in video quality and color fidelity over Causal Forcing++.

Critical strengths
  1. The two-mode analysis useful. It gives an intuitive explanation of how artifacts can be amplified during SF-DMD and why preconditioning might help. The toy experiments make these arguments easier to follow.

  2. The method also improves actual video-generation results over Causal Forcing++. Figure 5 and the LoRA swapping experiment provide useful evidence that changing the initialization affects the subsequent DMD training.

Critical weaknesses
  1. I am still not fully convinced by the explanation of why SF-CD should reduce bad modes more effectively than TF-CD. The paper explains where the teacher can correct a state toward the good mode, but why should student rollout visit these states more often? Rollout errors could also move good trajectories toward bad regions, or simply remain within the bad-mode basin. Figure 6(d) shows that correction is possible from various initial means, but it does not show how frequently these cases occur under the actual TF-CD and SF-CD sampling distributions. This connection is important to the proposed explanation and needs more theoretical support.

  2. It is unclear whether the method preserves diversity. The toy example shows attraction toward one dominant good mode, but real videos have many valid modes. Could the method improve average quality by concentrating on a smaller set of good outputs? The current color and quality metrics do not really answer this, and the paper would benefit from checking whether valid modes are lost during preconditioning.

Questions for authors
  1. Could you explain more directly why student rollout should provide more opportunities for correction than teacher-forced sampling? Even in the two-mode setting, can you derive conditions or show experimentally that SF-CD reduces the actual bad-mode probability more than TF-CD? The mean-shift plot alone does not fully answer this.

  2. Could you report VBench or additional quality metrics for the TF-CD and RaMP checkpoints before SF-DMD? Figure 5 already compares CFM, but a broader comparison would help show what preconditioning itself improves. A direct comparison of artifact frequency would also be useful.

  3. Have you checked diversity before and after preconditioning? Multiple samples per prompt, or a toy experiment with several good modes, could help test whether the method removes bad modes while retaining the valid ones.

6 comments

Thank you for recognizing the usefulness of the two-mode analysis and the initialization ablations. Your questions identify an important distinction between the direction of correction at a queried state and the probability of encountering such states. We address this distinction below and report additional quality and diversity measurements before SF-DMD.

  • W1&Q1: Why student rollout provides more bad-to-good corrective opportunities than TF-CD and reduces bad-mode probability.

We have added a formal derivation in Appendix A.2 (Eqs. 36–45, highlighted in blue) to directly quantify (1) why B→G\mathrm{B}\to\mathrm{G} crossings dominate G→B\mathrm{G}\to\mathrm{B}, and (2) why SF-CD visits these recovery states more often than TF-CD.

(1) Considering only the amplification of the block-0 error for simplicity, the boundary-crossing probabilities at block i of the autoregressive rollout satisfy

\begin{equation} \frac{p_{\mathrm{G}\to\mathrm{B}}^{\mathrm{SF}}(i)}{p_{\mathrm{B}\to\mathrm{G}}^{\mathrm{SF}}(i)}\approx \frac{\delta_{\mathrm{B},t}}{\delta_{\mathrm{G},t}}\exp{\left(-\frac{\delta_{\mathrm{G},t}^2-\delta_{\mathrm{B},t}^2}{2\rho^{2i}\sigma_0^2}\right)}<1, \end{equation}

where pB→GSF(i)p_{\mathrm{B}\to\mathrm{G}}^{\mathrm{SF}}(i) and pG→BSF(i)p_{\mathrm{G}\to\mathrm{B}}^{\mathrm{SF}}(i) denote the bad-to-good and good-to-bad crossing probabilities across the equal-responsibility boundary zt†z_t^\dagger, δB,t=ut⊤μB,t−zt†\delta_{\mathrm{B},t} = u_t^\top\mu_{\mathrm{B},t} - z_t^\dagger and δG,t=zt†−ut⊤μG,t\delta_{\mathrm{G},t} = z_t^\dagger - u_t^\top\mu_{\mathrm{G},t} are the distances from the bad and good mode centers to zt†z_t^\dagger (satisfying δB,t<δG,t\delta_{\mathrm{B},t} < \delta_{\mathrm{G},t} since πG>πB\pi_{\mathrm{G}} > \pi_{\mathrm{B}}), ρ\rho is the effective per-block error amplification factor(usually >1), and σ0\sigma_0 is the initial block-0 perturbation’s standard deviation, so rollout errors move bad trajectories into the good region much more often than they move good trajectories into the bad region.

(2) TF-CD resets to the clean history at every block, so error never accumulates and the perturbation scale stays at σ0\sigma_0. Under SF-CD, error amplification (ρ>1\rho > 1) increases this scale to ρiσ0\rho^i\sigma_0, yielding a strictly higher recovery probability for all i≥1i \ge 1 (Eqs. 44–45): \begin{equation} \quad \frac{p_{\mathrm{B}\to\mathrm{G}}^{\mathrm{SF}}(i)}{p_{\mathrm{B}\to\mathrm{G}}^{\mathrm{TF}}} \approx \rho^i \exp \left[\frac{\delta_{\mathrm{B},t}^2}{2\sigma_0^2}\left(1 - \frac{1}{\rho^{2i}}\right)\right] > 1. \end{equation} Thus, SF-CD's advantage comes from error accumulation across blocks, which pushes bad trajectories into the good-directed region far more often than under TF-CD.

  • Q2: Pre-DMD quality metrics.

Thank you for this helpful suggestion. We evaluate the pre-DMD checkpoints of TF-CD and RaMP on the same VBench benchmark. The overall scores and key sub-metrics are summarized below:

Pre-DMD CkptQuality ↑\uparrowSemantic ↑\uparrowTotal ↑\uparrowImaging Quality ↑\uparrowAesthetic Quality ↑\uparrow
Causal Forcing++ (init)0.81990.59260.77450.65180.5917
RaMP (init)0.81970.60420.77660.70920.6252

Before SF-DMD, RaMP already improves VBench Semantic and Total scores, with particularly clear gains on Imaging Quality (+5.74%) and Aesthetic Quality (+3.35%). Since these two metrics directly penalize low-level artifacts (e.g., blur, noise, overexposure), their gains—alongside higher CFM-7B (Fig. 5a)—confirm that RaMP reduces oversaturation by genuinely suppressing rollout artifacts rather than introducing blur or washed-out textures.

  • W2&Q3: Generation diversity before and after preconditioning & potential loss of valid modes.

We evaluate generation diversity before and after SF-DMD on the full Diversity dimension of VBench 2.0 [1] (following its standard protocol of 20 random seeds per prompt).

Prior to SF-DMD, RaMP preconditioning increases diversity from 0.4068 (TF-CD) to 0.4925, showing that rollout-aware preconditioning better preserves and surfaces diverse modes before DMD begins. After the subsequent SF-DMD stage, RaMP maintains higher diversity (0.5402) than Causal Forcing++ (0.5360), remaining closer to the Wan2.1-T2V-1.3B teacher (0.5873). These results indicate that RaMP preserves sample diversity across both preconditioning and SF-DMD refinement, rather than trading diversity for visual quality. We have added both the quantitative comparison (Table 4) and multi-seed qualitative visualizations (Fig. 8) to Appendix C.1 (highlighted in blue) of the revised paper.

If you have any further questions, please feel free to ask. We look forward to your very thoughtful feedback.

Reference

[1] Zheng D, Huang Z, Liu H, et al. Vbench-2.0: Advancing video generation benchmark suite for intrinsic faithfulness[J]. arXiv preprint arXiv:2503.21755, 2025.

Thank you for the additional experiments and clarifications. I have a few follow-up questions:

  • The diversity results show a substantial improvement before DMD. Could you briefly explain the intuition behind this improvement?

  • I still have some concern about W1 below:

    • Is the total probability mass transferred also depend on the student’s mode weights, wBpB→Gw_Bp_{B\to G} vs. wGpG→Bw_Gp_{G\to B}? Since the student is also initialized from a pretrained model, wGw_G might already be larger than wBw_B. Could you clarify whether the conclusion still holds when these weights are considered?

    • For the SF–TF comparison in part (2), wouldn’t the same argument also imply pG→BSF>pG→BTFp^{SF}_{G\to B} > p^{TF}_{G\to B}? Combining with the mode weights above, could you clarify when the SF-CD reduces bad-mode mass more than TF-CD?

Thank you very much for taking the time to read our response and for raising these thoughtful follow-up questions! We address your first question below, and due to the character limit of a single comment, we respond to Q2 in the immediately following comment.

  • Q1: Intuition behind the pre-DMD diversity improvement.

Thank you for the insightful question.

This pre-DMD diversity gain can be understood from the complementary behaviors of TF-CD and SF-CD. TF-CD samples xtx_t from ground-truth data and thus behaves in a mode-covering manner: trying to cover all GT modes under a regression loss causes the student to predict the average across modes, which blurs different modes together and lowers diversity. In contrast, SF-CD samples xtx_t from the student and behaves in a mode-seeking manner: teacher supervision pulls samples toward their nearest mode peaks, keeping different modes separated rather than averaging them. Therefore, RaMP's mixed TF-CD and SF-CD initialization combines broad mode coverage with clear mode separation, allowing different random seeds to land in distinct modes and increasing pre-DMD Diversity from 0.4068 to 0.4925.

Reviewer_AxULOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies oversaturation in few-step autoregressive video models trained with SF-DMD. It proposes RaMP, which gradually adds student rollout states to consistency distillation before switching to standard SF-DMD. The paper explains the problem using narrow bad modes, fitting errors, and fake-score errors. It tests these ideas and reports better VBench scores and lower saturation on a Wan-based video model.

Critical strengths

The paper studies saturation problem in the commonly adopted SF-DMD method for causal autoregressive models. Improving the student before DMD is a simple approach and does not add extra work at inference time.

Critical weaknesses

The main explanation (in introduction and Sec 3.1 motivation) is not directly tested. The theory uses strong assumptions, however, the paper does not show that oversaturated videos form narrow, easier-to-fit modes. It also didn't provide enough evidence that RaMP reduces visits to those modes.

There are no video results and no supplementary. Sampled frames are not enough to judge motion quality or temporal errors. The metric results are useful, but without videos it is hard to assess the overall quality of this video generation method.

The main idea is understandable, but the method and figures need a clarity pass. In particular, the SF-CD prediction, target, and use of generated history should be explained more directly.

Questions for authors

Overall the method is useful and the reported scores are promising. However, the main explanation needs stronger evidence on real video models, and the lack of video examples makes the quality claims hard to fully assess.

2 comments

Thank you for your efforts to improve our manuscript!

  • W1: Empirical evidence for the proposed mechanism on real video models

Quantifying the CFM-7B distributions in Fig. 4(a) over the exact same set of generated videos used for our VBench evaluation (414 videos per method) directly shows that RaMP reduces visitation to poor color-fidelity modes. Using a fixed low-score tail cutoff of CFM < 5.0, the fraction of degraded videos drops from 6.52% (27/414) for the Causal Forcing++ initialization to 2.17% (9/414) after RaMP preconditioning, and from 1.45% (6/414) to 0.72% (3/414) in the final models.

Fig. 4(b) shows the corresponding training dynamics over 3,000 SF-DMD steps: the mean CFM-7B score of Causal Forcing++ declines steadily from ∼10.35\sim 10.35 to 8.958.95, whereas RaMP stabilizes near 11.011.0 after an initial drop. This indicates that prolonged SF-DMD tends to amplify certain bad modes already present in the initialization, progressively degrading overall color fidelity; by suppressing these poor modes beforehand, RaMP prevents such amplification and stabilizes at a higher level during subsequent optimization.

In addition, we detail the effect of inaccurate fake-score estimation on real videos in our response to Reviewer 8WSh, and we have also expanded Appendix A.2 (Eqs. 36–45) to formally derive why student rollouts yield more bad-to-good than good-to-bad transitions and why SF-CD visits these recovery states more often than TF-CD.

  • W2: Video results

We have newly uploaded the video results of RaMP to the updated supplementary material.

  • W3: Experimental details

Thank you for the suggestion. Sec. 3.2 explicitly defines the SF-CD prediction, target, and generated history. Eq. (7) uses the online student’s clean-endpoint prediction and a stop-gradient EMA target evaluated after the frozen teacher’s ODE step in Eq. (6). The generated history hiSFh_i^{\mathrm{SF}} is constructed from previously generated blocks and conditions the current block’s denoising trajectory, from which the query state is sampled, as described before Eq. (9). Appendix B provides more detailed experimental settings for SF-CD preconditioning.

If you have any further questions, please feel free to ask. We look forward to your very thoughtful feedback.

Thank you for your review. We have responded to all of your comments. We would appreciate it if you could take a look and let us know if any concerns remain.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.