acceptodds
Under review as a conference paper at ICLR 2027

Conditional Residual Prediction: Improving Autoregressive Video Diffusion without a Bidirectional Teacher

Abstract

Causal video diffusion models generate video autoregressively, which suits streaming, interactive, and long-video generation. Under standard training, however, they often yield lower generation quality than bidirectional models of the same size. Many existing approaches address this gap by initializing from or distilling a pretrained bidirectional teacher. We instead train a causal model from an image-model initialization, with no bidirectional video model at any stage. Because this path requires neither a large bidirectional teacher nor a complex distillation pipeline, it is simpler and more scalable. On this path, we find that a causal model trained on ground-truth history becomes strongly dependent on it, so that at inference errors in its own generated history propagate forward. We hypothesize that much of this dependence is unnecessary, because the current input already determines much of what the history provides. We propose Conditional Residual Prediction (CRP), a simple recipe for reducing a model's reliance on a condition: the model first predicts the target without the condition, and the condition may only add a residual on top of this prediction. Applied to history, CRP makes the model predict each chunk from the present as far as it can and use the past only for what the present cannot supply. In controlled experiments, CRP nearly closes the 6.14-point gap to a bidirectional model trained under the same setup. Scaling this recipe, we train Optica, a 2B-parameter causal video model that autoregressively generates 5-second 480p videos and reaches 82.78 on VBench with only about 15M training videos.

Primary area:
generative models
Keywords:
Video Generation, Diffusion Models, Pretraining, Causal Video Generation, Autoregressive Video Generation, Teacher-Free Training, History Dependence

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_utt5OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies why causally-trained autoregressive video diffusion models underperform bidirectional models of the same size, and proposes training a causal model from an image-model initialization with no bidirectional video teacher at any stage. The authors hypothesize that teacher forcing makes the model rely on history even for information the current input already determines ("history-free" information, in their mutual-information decomposition), so errors in self-generated history propagate at inference. Their method, Conditional Residual Prediction (CRP), evaluates the same network twice per step: a History-Free branch predicts the target with history masked out, and a History-Conditioned branch may only add a residual on top of the (stop-gradiented) free prediction, so history supplies only what the present cannot. A second change replaces the video VAE with an image VAE (plus a lightweight temporal patch layer) to remove the latent coupling introduced by causal temporal convolutions. In controlled experiments (4M videos, 256p, 2B model), CRP closes most of the 6.14-point VBench Total gap to a matched bidirectional reference (causal baseline 71.14 → +x₀-prediction 72.41 → +CRP 77.09 → +image VAE 77.48 vs. 77.28 bidirectional), and two diagnostics (teacher-forcing vs. rollout per-frame quality gap; sensitivity to corrupted history) improve consistently. Scaled up, the resulting 2B model "Optica" reaches 82.78 VBench Total at 480p trained on ~15M videos — the best among causal models that use no bidirectional video model.

Critical strengths
  1. Simple and effective approach. CRP addresses exposure bias without requiring a pretrained bidirectional video teacher or a distillation pipeline. The method is straightforward and applicable to causal video diffusion models.
  2. Strong controlled experiments. Table 1 clearly separates the effects of x0-prediction, CRP, and the image VAE. CRP alone improves VBench Total by 4.68 points and substantially reduces quality degradation under generated and corrupted histories (Figure 2).
  3. Useful analysis of video VAE limitations. The authors identify temporal coupling in causal video VAEs as another source of error propagation. The latent perturbation experiments in Appendix E provide convincing evidence of this effect.
Critical weaknesses
  1. decomposition in Section 3.1 motivates CRP, but it does not establish that the model actually separates history-free and history-exclusive information. The residual branch still has access to the full history, and the training objective does not explicitly constrain its dependence on history. The improved robustness supports the method's effectiveness but does not fully validate the proposed explanation. 2.Additional computational cost is not quantified. CRP requires two forward passes through the shared DiT, while the image VAE removes temporal compression. Although the authors acknowledge these costs, inference latency, memory usage, and throughput are not reported. This is important given the paper's emphasis on scalable autoregressive generation.
  2. Evaluation is limited to short videos. All experiments use approximately 5-second videos. Since reducing accumulated errors is the primary motivation, it is unclear whether CRP remains effective over substantially longer autoregressive rollouts. A longer-horizon evaluation would strengthen the practical claims.
Questions for authors
  1. Can the authors provide direct evidence that CRP reduces reliance on redundant history information, rather than simply improving robustness through the auxiliary history-free prediction objective? For example, how does CRP compare with a two-branch model trained without residual coupling?
  2. What are the inference latency, peak GPU memory, and throughput of CRP compared with the causal baseline? How much additional overhead comes from the image VAE?
  3. Does CRP maintain its advantage for longer autoregressive rollouts, such as 10–20 seconds? Even a limited evaluation would help clarify whether its benefits persist as history errors accumulate.
  4. How sensitive is CRP to the quality of the history-free prediction? In scenarios requiring strong temporal dependencies, could the history-free branch reduce motion or identity consistency?
5 comments

We thank the reviewer for the detailed review, and for recognizing the simplicity and effectiveness of CRP, the controlled experiments, and the analysis of video VAE limitations. We address each point below.

Ablation of CRP components and extra history-free training

reviewer (q1+w1): Can the authors provide direct evidence that CRP reduces reliance on redundant history information, rather than simply improving robustness through the auxiliary history-free prediction objective? For example, how does CRP compare with a two-branch model trained without residual coupling?

Thank you very much for the suggestion. This ablation is also added in the revised paper, Table 2. First, please allow us to introduce the components of full CRP again.

  1. At the final output layer, we add the history-free prediction and the history residual prediction. We call this part CRP because it is the core component and directly matches our formula.
  2. We also perform the same operation at intermediate layers, so that the model can fuse information in latent space to further improve performance. We call this latent fusion.
  3. We also add stop-gradient to prevent conditional information from contaminating the history-free branch. We call this stop-gradient.
  4. CRP also needs an extra history-free prediction objective, which we call extra history-free training.

Please see Table 2 and Section 4.5 for the component ablations (full discussion in Appendix H), and Fig. 4 for the training dynamics.

ConfigurationExtra HF trainingLatent fusionCRPStop-gradientTotal ↑Quality ↑Semantic ↑
Teacher Forcing (x0x_0)×××–72.4174.9662.22
Full CRP✓✓✓✓77.0978.0973.10
− Latent fusion✓×✓✓76.3177.0673.32
− CRP✓✓×✓72.6375.1162.74
− Both✓××–71.1474.0859.37
− Stop-gradient✓✓✓×75.8777.3969.79

The two variants the reviewer asks about correspond directly to rows of this table. The − Both row is the two-branch model without residual coupling: it keeps only the auxiliary history-free objective and reaches 71.14, even below Teacher Forcing (x0x_0, 72.41). The − CRP row additionally keeps latent fusion, yet without the output residual composition it reaches only 72.63, close to Teacher Forcing (x0x_0) and far below full CRP (77.09). Thus, the extra history-free objective alone does not explain the gain (Table 2).

The training dynamics show the same pattern: the variants without output residual composition have lower history-conditioned training loss but worse VBench results (Fig. 4). These results support the history over-reliance explanation.

Fig. 7 (Appendix J) of the revised paper additionally visualizes the two branches along the sampling trajectory of one rollout, at steps from σ=0.99\sigma = 0.99 to 0.120.12: the noised input at that step, the history-free prediction, the history residual, and the full CRP prediction. Table 7 quantifies their contributions: on 1,000 held-out clips, the residual removes 25.8% of the history-free error at σ=0.1\sigma = 0.1 and 81.7% at σ=0.9\sigma = 0.9.

This also relates to the reviewer's point that the decomposition in Section 3.1 does not by itself show that the model separates history-free and history-exclusive information. We agree that the residual branch sees the full history; what matters is what the model learns to take from it. With history hh and text yy, a standard causal model can lower its training loss by copying history-free information from the ground-truth history, which is the over-reliance we target. In CRP, the history-free branch must recover this information from the current input alone under stop-gradient, so the residual only needs what the history adds, E[x0∣xσ,h,y]−E[x0∣xσ,y]\mathbb{E}[x_0 \mid x_\sigma, h, y] - \mathbb{E}[x_0 \mid x_\sigma, y] (Appendix J). This is an effect of the model structure, and the results above show it: the variants without residual composition reach lower history-conditioned training loss but worse generation (Table 2, Fig. 4), and the residual corrects a small share of the error at low noise, where the current chunk determines most of the content, and most of it at high noise (Table 7).

Memory, Cost and Image VAE Overhead

reviewer (q2+w2): What are the inference latency, peak GPU memory, and throughput of CRP compared with the causal baseline? How much additional overhead comes from the image VAE?

We report the training and inference cost of CRP and teacher forcing under the same configuration (also added in the revised paper, Table 8).

SettingModelTimeRelativePeak memory (GiB)
Training stepTeacher Forcing3.13 s1.00×42.7
CRP3.99 s1.27×71.0
Inference (per video)Teacher Forcing55.8 s1.00×18.6
CRP103.2 s1.85×18.6

In inference, CRP evaluates both branches at every step, so a video takes 1.85× as long, while peak memory is unchanged. The speed loss from the extra branch could be reduced by distilling CRP into a single-stream student.

The image VAE adds no overhead; it is faster and uses less memory than the video VAE under a correct implementation (also added in the revised paper, Table 5; same Wan 2.1 checkpoint, per clip at 192×336):

VAE modePSNR (dB) ↑Encode (ms) ↓Decode (ms) ↓Peak encode (GiB) ↓Peak decode (GiB) ↓
Video mode (causal temporal)27.93 ± 0.1374612721.181.47
Image mode (per-frame)29.04 ± 0.154227200.930.80

Without temporal dependencies, the image VAE processes frames in parallel with 2D convolutions instead of causal 3D convolutions over the sequence, while inside the DiT the temporal patch layer (about 2% of the parameters; Appendix B.2) keeps the same 20 temporal positions as with the video VAE.

Longer rollouts

reviewer (q3+w3): Does CRP maintain its advantage for longer autoregressive rollouts, such as 10–20 seconds? Even a limited evaluation would help clarify whether its benefits persist as history errors accumulate.

We evaluate the final Optica model on 30-second rollouts using VBench-Long (also added in the revised paper, Table 4), with qualitative examples in Appendix L. We use the same protocol as the official repositories of Self Forcing and Deep Forcing, generating the samples with the first 128 MovieGen prompts and evaluating them with the VBench-Long suite. We also report the performance of several methods that do not use a bidirectional teacher, including MAGI-1, Pyramid Flow, NOVA and SkyReels-V2, following their official repositories. When evaluating our method, we use a standard sliding-window extended generation, without specific techniques for long-video rollout like sinks or RoPE adjustment. The values of the methods distilled from a bidirectional teacher are adopted from Deep Forcing and Relax Forcing (marked †).

MethodSubjectBackgroundMotionDynamicAestheticImagingAverage
Causal, distilled from a bidirectional teacher
Self Forcing†97.3496.4798.6336.6259.4468.5876.18
Rolling Forcing†98.0796.8498.7432.7160.7570.7376.31
LongLive†97.9796.8398.7645.5561.5169.0778.28
Deep Forcing†97.3496.4898.2757.5660.6869.3179.94
Relax Forcing†96.9996.1297.8065.6760.1368.5080.87
Causal, bidirectional initialization only
SkyReels-V297.6997.2299.0139.0660.2863.4376.12
Causal, no bidirectional video model
NOVA93.2795.6999.0532.3439.7829.7764.98
Pyramid Flow93.4495.8498.4833.5942.6441.0767.51
MAGI-197.4296.8299.2945.5258.9258.9676.15
Optica (ours)95.1495.6596.8458.4952.7860.7376.61

Our method shows the highest average score among methods that do not rely on a bidirectional teacher, and is close to those bidirectional-dependent methods without specific tuning for long video. Considering our method is not trained on long videos, this is a promising result. This indicates our method can perform well when facing such a strong OOD setting, and there is room for further improvement by training on longer data or using specific techniques like attention sinks.

Effect of History-Free Branch on Motion and Identity Consistency

reviewer (q4): How sensitive is CRP to the quality of the history-free prediction? In scenarios requiring strong temporal dependencies, could the history-free branch reduce motion or identity consistency?

The history-free prediction only needs to be good where the current chunk is informative. Wherever the current chunk alone is not enough, which is exactly where temporal dependencies are strong, the history residual carries the history-dependent information, and its share grows with the noise level (Table 7 and Fig. 7 in the revised paper). Comparing Teacher Forcing (x0x_0) and CRP, which differ only in CRP (the bidirectional baseline is shown for reference), the fine-grained VBench dimensions show that CRP does not reduce identity or motion consistency:

ModelSubject Consistency ↑Background Consistency ↑Motion Smoothness ↑Dynamic DegreeImaging Quality ↑Aesthetic Quality ↑
Bidirectional baseline89.2996.0896.8883.6163.0049.08
Teacher Forcing (x0x_0)77.0890.3396.4390.0060.4642.26
CRP83.0793.3197.6582.2260.4249.14

CRP improves subject consistency (identity) from 77.08 to 83.07 and motion smoothness from 96.43 to 97.65 over Teacher Forcing (x0x_0). CRP's Dynamic Degree (82.22) is lower than that of Teacher Forcing (x0x_0) (90.00), together with clearly higher consistency, reflecting the known trade-off between motion and consistency on VBench. The history-free branch does not suppress motion itself: over 30-second rollouts, Optica has the highest Dynamic Degree (58.49) among methods without a bidirectional video model (Table 4).

Summary of revisions

We thank you for your constructive comments, which have helped us improve the paper. Following your suggestions, we have added to the revised paper:

  • an ablation of the CRP components, including a two-branch variant without residual coupling, with the training dynamics (Sec. 4.5, Table 2, Fig. 4, Appendix H);
  • an analysis of the two branches across noise levels (Appendix J, Table 7, Fig. 7);
  • the training and inference cost of CRP (Table 8, Appendix K) and the latency and memory of the image VAE (Table 5);
  • 30-second rollouts on VBench-Long with qualitative comparisons (Table 4, Appendices I and L).

The new content is marked in blue in the revised paper, and we would appreciate it if you could take a look.

Reviewer_eF77OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a method for autoregressive video generation, trained just from a scratch (or more specifically, an image generation model), without having to distill from a pretrained bidirectional video generation model. The paper builds on a central insight -- most of the information for denoising a chunk is in the noisy chunk, but models typically overrely on the history, which causes exposure bias during rollout. The paper proposes a residual approach -- denoise hte current chunk just from the noisy current chunk, and then predict a residual from the whole history (with a stop gradient on the first branch). This adds an inductive bias to use as much as possible from the current frame, before building in the history. The work validates on a controlled ablation and on the VBench metric, outperforming other methods that were trained without the benefit of distilling from a bidirectional model.

Critical strengths

The paper makes an interesting fundamental insight that most information on where a video is immediately going is in the current chunk, and to build this insight into the inductive bias of the architecture. The results are convincing overall. In Table 2, the paper shows that the proposed method achieves higher VBench Totals compared to other methods that train a causal model directly.

Critical weaknesses

Though it is impressive that the method can perform well training a causal model directly from an image model, doing so does not currently beat distillation pipelines. But fortunately, the method is not limited to this setting and could be applied to distillation pipelines. I am wondering why the paper did not try this experiment. The method would be significantly more convincing if it could beat other methods in this setting as well. Even if it does not, it would be informative to the reader to know what setting the method works well at, and why. Not applying this to a distillation pipeline is a missed opportunity.

The limitations section mentions there the paper did not look at >5 second rollouts, so it's an open question whether CRP suppresses error accumulation over longer rollouts. Previous work such as LongLive, Self-Forcing++, Resampling Forcing (all cited in the paper already) have now shown up to 240 seconds, minutes long, 15 seconds rollouts, respectively. Why is this experiment currently missing as well?

The paper makes an interesting fundamental insight that most of the information is present in the current chunk and realizes this insight in a specific architectural change (two branches, with a residual). However, there are other possibilities for realizing this, such as random masking on the history as in History Guidance (Song 2025) and history noise augmentation (Diffusion Forcing, Chen 2024). Furthermore, what is the runtime implication of having two branches? The costs relative to the baseline in terms of runtime inference is unclear.

Questions for authors

Could we add baselines for taking into account different lengths of history, as mentioned in the weaknesses section? Please also discuss why distillation experiments were not performed with the method, if it's a fundamental limitation that the method cannot be applied in those settings, or if the method does not work, or if we simply do not know.

Please report the runtime cost of CRP, also w.r.t baselines.

Why aren't rollouts beyond 5 seconds, when previous works have shown much longer?

6 comments

We thank the reviewer for the careful review, and for finding the insight behind CRP interesting and the results convincing. We address each point below.

Longer rollouts

reviewer (q3+w2): Why aren't rollouts beyond 5 seconds, when previous works have shown much longer?

Long rollouts are indeed an important test. We extend the Optica model to 30-second videos and evaluate them on VBench-Long (added in the revised paper, Table 4). We use the same protocol as the official repositories of Self Forcing and Deep Forcing, generating the samples with the first 128 MovieGen prompts and evaluating them with the VBench-Long suite. We also report the performance of several methods that do not use a bidirectional teacher, including MAGI-1, Pyramid Flow, NOVA and SkyReels-V2, following their official repositories. When evaluating our method, we use a standard sliding-window extended generation, without specific techniques for long-video rollout like sinks or RoPE adjustment. The values of the methods distilled from a bidirectional teacher are adopted from Deep Forcing and Relax Forcing (marked †). Qualitative comparisons are provided in Appendix L.

MethodSubjectBackgroundMotionDynamicAestheticImagingAverage
Causal, distilled from a bidirectional teacher
Self Forcing†97.3496.4798.6336.6259.4468.5876.18
Rolling Forcing†98.0796.8498.7432.7160.7570.7376.31
LongLive†97.9796.8398.7645.5561.5169.0778.28
Deep Forcing†97.3496.4898.2757.5660.6869.3179.94
Relax Forcing†96.9996.1297.8065.6760.1368.5080.87
Causal, bidirectional initialization only
SkyReels-V297.6997.2299.0139.0660.2863.4376.12
Causal, no bidirectional video model
NOVA93.2795.6999.0532.3439.7829.7764.98
Pyramid Flow93.4495.8498.4833.5942.6441.0767.51
MAGI-197.4296.8299.2945.5258.9258.9676.15
Optica (ours)95.1495.6596.8458.4952.7860.7376.61

Our method shows the highest average score among methods that do not rely on a bidirectional teacher, and is close to those bidirectional-dependent methods without specific tuning for long video. Considering our method is not trained on long videos, this is a promising result. This indicates our method can perform well when facing such a strong OOD setting, and there is room for further improvement by training on longer data or using specific techniques like attention sinks.

We also added a comparison between TF and CRP on 30-second rollouts in our ablation study. CRP improved the average score from 0.6778 to 0.7321, with the largest gains in dynamic degree, aesthetic quality, and imaging quality.

MethodSubject ↑Background ↑Motion ↑Dynamic ↑Aesthetic ↑Imaging ↑Average ↑
TF(x₀)0.90720.95660.96480.56350.31110.36390.6778
CRP0.92200.95340.97500.62990.41850.49380.7321

Distillation

reviewer (q1+w1): Please also discuss why distillation experiments were not performed with the method, if it's a fundamental limitation that the method cannot be applied in those settings, or if the method does not work, or if we simply do not know.

We are glad to clarify this point. In theory, nothing prevents CRP from being used for distillation. We would like to distinguish two cases: (1) using CRP as the student model and a bidirectional model as the teacher for distillation, and (2) using a causal model trained with CRP as the teacher for distillation.

Since our goal is not to rely on a bidirectional teacher network, the first kind of distillation is beyond the scope of this paper. For the second kind, in fact, when CRP is used as the teacher, the student can be a single-stream, non-CRP network that directly fits the sum of the CRP history-free prediction and history residual. In this way, distillation not only reduces the number of inference steps but also removes the cost of the two CRP forward passes.

The reason we did not further explore distillation in this paper is very simple: Optica has had limited training compared with bidirectional teachers such as Wan2.1, and we hope to perform distillation after scaling has produced a sufficiently strong teacher model. In addition, a very practical factor is that we do not have enough resources to explore and compare different distillation methods.

We believe distillation does not affect the evidence and conclusions presented in this paper: CRP effectively mitigates the history over-reliance problem and significantly improves causal performance in the controlled experiments, and in the scaling experiments it also achieves the best VBench Total among causal models that do not rely on a bidirectional video model.

Diffusion Forcing and History Guidance

reviewer (q1+w3): Could we add baselines for taking into account different lengths of history, as mentioned in the weaknesses section?

We agree that these are natural alternatives for realizing the same insight, so we added Diffusion Forcing and History Guidance to the controlled experiments under the same experimental protocol as the paper (also added in the revised paper, Table 1). In line with observations from previous work, Diffusion Forcing performs worse than teacher forcing. This is consistent with our observation. For History Guidance, we follow the formulation and configuration of MAGI-1, since the original History Guidance formulation considers only history conditioning and does not specify how to combine it with text CFG. History Guidance brings a positive but limited improvement. Unlike Diffusion Forcing, CRP keeps the history clean during both training and inference; unlike History Guidance, it changes what the history-conditioned branch learns rather than only how predictions are combined at sampling time. Overall, in our experiments, CRP produces the most significant improvement, further supporting the effectiveness of our method.

MethodTotal ↑Quality ↑Semantic ↑
Bidirectional77.2879.6467.82
Diffusion Forcing69.1371.7858.50
Teacher Forcing (x0x_0)72.4174.9662.22
History Guidance73.0377.1456.58
CRP77.0978.0973.10

Cost

reviewer (q2+w4): Please report the runtime cost of CRP, also w.r.t baselines.

We show the training and inference speed and memory of CRP and teacher forcing under the same configuration (also added in the revised paper, Table 8).

SettingModelTimeRelativePeak memory (GiB)
Training stepTeacher Forcing3.13 s1.00×42.7
CRP3.99 s1.27×71.0
Inference (per video)Teacher Forcing55.8 s1.00×18.6
CRP103.2 s1.85×18.6

CRP adds only a 1.27× slowdown rather than 2× during training, because the history-free branch does not attend to the history. Memory increases by about 1.66×, coming from the internal activations. In inference, CRP brings about a 1.85× slowdown without increasing memory, mainly due to the extra history-free branch. As mentioned above, the speed loss from the extra branch could be reduced by distilling CRP into a single-stream student.

Summary of revisions

We thank you for your constructive comments, which have helped us improve the paper. Following your suggestions, we have added to the revised paper:

  • 30-second rollouts on VBench-Long with qualitative comparisons (Table 4, Appendices I and L);
  • Diffusion Forcing and History Guidance baselines under the controlled setting (Table 1, Appendix G);
  • the training and inference cost of CRP (Table 8, Appendix K).

All revisions are highlighted in blue in the revised paper; we would be grateful if you could take a look.

Reviewer_HsvKOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies why causal video diffusion models can underperform bidirectional models. It attributes part of the gap to over-reliance on ground-truth history during training, which makes models vulnerable to errors in generated history at inference. The authors propose Conditional Residual Prediction (CRP)​, where a history-free branch predicts the target and a history-conditioned branch adds only a residual. They also use an image VAE to reduce temporal dependence between latents. In controlled experiments, CRP reduces the VBench gap to a bidirectional baseline from 6.14 to 0.19 points; the scaled 2B-parameter Optica model scores 82.78 on VBench.

Critical strengths
  1. The paper tackles an important challenge: training high-quality causal video models without a bidirectional video teacher.
  2. CRP is intuitive, and the controlled experiments show that it substantially narrows the gap to a bidirectional baseline.
Critical weaknesses
  1. The paper does not fully establish whether error accumulation is the main cause of the causal–bidirectional quality gap.
  2. It lacks matched comparisons with related teacher-free methods, such as Resampling Forcing and Diffusion Forcing.
  3. CRP reduces, but does not appear to eliminate, sensitivity to history errors. Optica also scores below several self-forcing methods, though the cross-paper results may not be directly comparable.
Questions for authors
  1. What is the primary cause of the quality gap between causal and bidirectional models, and how much of it is due to error accumulation?
  2. Can the authors quantify the contributions of the history-free and history-conditioned branches? How do they prevent the history-free branch from drifting from the preceding video?
  3. Could CRP be compared with Resampling Forcing, Diffusion Forcing, and Self-Forcing under matched settings?
  4. Could the author detail the inference process and the image-VAE adoption?
8 comments

We thank the reviewer for the thoughtful review, and for recognizing that the paper tackles an important challenge and that CRP is intuitive and substantially narrows the gap to the bidirectional baseline. We address each point below.

The gap between causal and bidirectional models and the role of error accumulation

reviewer (q1+w1): What is the primary cause of the quality gap between causal and bidirectional models, and how much of it is due to error accumulation?

This is a very good question. We will discuss it from two angles: (1) where the gap between bidirectional and causal models comes from, and (2) how much we can gain by addressing error accumulation.

For the first question, we believe error accumulation is a very important cause, but not the only one. For example, based on our experience and observations, bidirectional models have more freedom to model redundant signals. That is, they can use more compute to refine the same content. Another important point is, even if we assume temporal causality, a large part of the information in a video may not be subject to this constraint. For example, a complete room does not strictly require a particular causal order. Proving these points is beyond the scope of this paper. Here, we are simply sharing our views and observations with the reviewer and the community.

The second question is the main issue studied and addressed in this paper. The gains from addressing error accumulation are clear: on VBench, the two changes, x0x_0-prediction and CRP, bring the causal model almost to the level of the bidirectional model (Table 1, Sec. 4.3). The per-frame quality analysis (Fig. 2) shows reduced degradation, and the sailboat example (Fig. 3) shows a clear visual improvement over the causal baseline.

Another important evidence is the training loss v.s. the VBench Score (Fig. 4 and Sec. 4.5 in the revised paper). In our hypothesis, we predict the phenomenon that when history over-reliance happens, the model may have lower training loss but worse generation results, since this over-reliance will also be affected by the error accumulation during generation, and CRP may lead to higher loss since it asks the model to model the history-free information, but helps the generation. Our experimental results closely fit this prediction.

To sum up, is error accumulation the only cause? We do not think so. Is it an important cause? We believe it is, and this view is also supported by our experiments and many previous works.

The contributions of the history-free and history-conditioned branches

reviewer (q2): Can the authors quantify the contributions of the history-free and history-conditioned branches? How do they prevent the history-free branch from drifting from the preceding video?

Thank you for this insightful question. We discuss it from three angles: (1) empirical consistency, (2) the optimal solutions of the two branches, and (3) a new quantitative analysis of what each branch contributes at different noise levels (Appendix J in the revised paper).

1. Compared with Teacher Forcing (x0x_0), CRP does not introduce additional inconsistency. Comparing Teacher Forcing (x0x_0) and CRP, which differ only in CRP (same data, schedule and VBench sampling; the bidirectional baseline is shown for reference), CRP's subject and background consistency are higher:

ModelSubject Consistency ↑Background Consistency ↑Motion Smoothness ↑Dynamic DegreeImaging Quality ↑Aesthetic Quality ↑
Bidirectional baseline89.2996.0896.8883.6163.0049.08
Teacher Forcing (x0x_0)77.0890.3396.4390.0060.4642.26
CRP83.0793.3197.6582.2260.4249.14

2. The optimal solutions of HF and HC. Let xσ=(1−σ),x0+σ,ϵx_\sigma = (1-\sigma)\\,x_0 + \sigma\\,\epsilon, with history hh and text yy. CRP predicts x^0=fHF(xσ,y)+rHC(xσ,h,y)\hat{x}_0 = f_{\mathrm{HF}}(x_\sigma, y) + r_{\mathrm{HC}}(x_\sigma, h, y), where HF is trained with the history-free objective and the sum with the history-conditioned one. At the optimum, fHF∗=E[x0∣xσ,y]f_{\mathrm{HF}}^{\ast} = \mathbb{E}[x_0 \mid x_\sigma, y], and by Tweedie's formula and Bayes' rule,

rHC∗=E[x0∣xσ,h,y]−E[x0∣xσ,y]=σ21−σ ∇xσlog⁡pσ(h∣xσ,y).r_{\mathrm{HC}}^{*} = \mathbb{E}[x_0 \mid x_\sigma, h, y] - \mathbb{E}[x_0 \mid x_\sigma, y] = \frac{\sigma^2}{1-\sigma}\, \nabla_{x_\sigma} \log p_\sigma(h \mid x_\sigma, y).

HF thus points toward the correct (history-free) data distribution, and HC adds the gradient of the history likelihood, a correction that pulls predictions inconsistent with the history back toward history-consistent ones.

The roles also depend on noise: as σ→0\sigma \to 0, rHC∗→0r_{\mathrm{HC}}^{*} \to 0 and the current chunk alone suffices; as σ→1\sigma \to 1, HF tends to the blurry text-conditional mean E[x0∣y]\mathbb{E}[x_0 \mid y] and HC supplies almost all chunk-specific information from the history.

3. Quantitative analysis: the two branches specialize across noise levels. Following the reviewer's suggestion, we measure what each branch contributes (also added in the revised paper, Table 7). On 1,000 held-out real clips (not seen in training), for the same clean history we noise the current chunk at different σ\sigma and compute the latent MSE of the x0x_0 prediction against the ground truth, using the HF branch alone and the full HF+HC prediction (chunks with history).

σ\sigmaHF-Only MSEHF+HC (Full) MSEMSE reduction by HCRelative reduction by HC
0.10.00810.00600.002125.8%
0.30.03340.01620.017251.4%
0.50.06750.02420.043364.1%
0.70.12630.03350.092873.5%
0.90.28590.05240.233581.7%

The results match the analysis above. At low noise, HF alone is already close to the full prediction and HC removes only 25.8% of its error. As σ\sigma grows, HF becomes blurrier, and HC removes an increasing share of the error, up to 81.7%. The added visualization in the revised paper (Fig. 7) shows the same pattern: HF is blurry at high noise and HC restores the history-consistent structure.

Summary. CRP does not exhibit worse consistency than Teacher Forcing (x0x_0). HF provides the history-free direction toward the data distribution, HC adds a history-likelihood correction on top of it, and the two specialize across noise levels.

Comparison with other methods

reviewer (q3+w2): Could CRP be compared with Resampling Forcing, Diffusion Forcing, and Self-Forcing under matched settings?

It is a natural idea to compare with these methods. However, Resampling Forcing does not provide complete VBench scores and uses Wan2.1 for initialization. Its training code has not been released so far, so we are currently unable to make a strict comparison. Self Forcing is a distillation method based on Wan2.1, while Optica, the model trained with CRP in this paper, starts from an image model. CRP can indeed also be used to fine-tune an existing bidirectional model. However, this is beyond the scope of this paper, which focuses on training without a bidirectional video teacher. But we agree this is a good direction for future work.

To provide more comparison, we conducted additional Diffusion Forcing and History Guidance baselines (also added in the revised paper, Table 1; details in Appendix G). In line with observations from previous work, Diffusion Forcing performs worse than teacher forcing. This is consistent with our observation. For History Guidance, we follow the formulation and configuration of MAGI-1, since the original History Guidance formulation considers only history conditioning and does not specify how to combine it with text CFG. History Guidance brings a positive but limited improvement. Unlike Diffusion Forcing, CRP keeps the history clean during both training and inference; unlike History Guidance, it changes what the history-conditioned branch learns rather than only how predictions are combined at sampling time. Overall, in our experiments, CRP produces the most significant improvement, further supporting the effectiveness of our method.

MethodTotal ↑Quality ↑Semantic ↑
Bidirectional77.2879.6467.82
Diffusion Forcing69.1371.7858.50
Teacher Forcing (x0x_0)72.4174.9662.22
History Guidance73.0377.1456.58
CRP77.0978.0973.10

Detail of Image VAE

reviewer (q4): Could the author detail the inference process and the image-VAE adoption?

We are happy to explain the inference process. We use the same Wan 2.1 VAE checkpoint in image mode, so each of the 80 frames is encoded independently into its own latent (Sec. 3.3, Appendix F). Inside the DiT, a lightweight temporal patch layer merges every 4 adjacent latents into one temporal position after the patch embedding and splits them back before the output layer (Fig. 1(c), Appendix B.2), so the transformer operates on 20 temporal positions, as with the video VAE. We perform autoregressive generation over these positions, chunk by chunk. At each denoising step, we first predict x^hf\hat{x}_{hf} directly without looking at history. We then add the residual x^hc\hat{x}_{hc} predicted with access to history to x^hf\hat{x}_{hf} to obtain the full prediction x^\hat{x}. The first chunk has no history, so it does not have this residual part. We then apply CFG using this full prediction. After all chunks are generated, each of the 80 latents is decoded independently by the image VAE into a video frame.

Comments on the paper

1 comment

We sincerely thank the PCs, ACs, and all reviewers for their time and their thoughtful, constructive feedback. We are glad that the reviewers found the problem important (HsvK), the insight interesting (eF77), CRP simple and effective (HsvK, utt5), and the experiments convincing (eF77, utt5).

Following the reviews, we have revised the paper. All changes are marked in blue. The main additions are:

  • More baselines under the controlled setting. Diffusion Forcing and History Guidance are trained and evaluated under the same setup as CRP (Table 1, Appendix G).
  • Ablation of CRP components. We remove each component of CRP in turn, including a two-branch variant without residual coupling, and show the training dynamics of each variant (Sec. 4.5, Table 2, Fig. 4, Appendix H).
  • 30-second rollouts. We evaluate Optica, trained only on 5-second clips, on 30-second videos with VBench-Long, and provide qualitative comparisons (Table 4, Appendices I and L).
  • Roles of the two branches. We quantify how much the history-conditioned residual contributes at different noise levels and visualize both branches along a sampling trajectory (Appendix J, Table 7, Fig. 7).
  • Computational cost. We report the training and inference time and memory of CRP (Appendix K, Table 8), and the latency and memory of the image VAE versus the video VAE (Appendix F, Table 5).

Thank you again for your commitment to the review process, and we would be happy to hear your feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.