- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies why causally-trained autoregressive video diffusion models underperform bidirectional models of the same size, and proposes training a causal model from an image-model initialization with no bidirectional video teacher at any stage. The authors hypothesize that teacher forcing makes the model rely on history even for information the current input already determines ("history-free" information, in their mutual-information decomposition), so errors in self-generated history propagate at inference. Their method, Conditional Residual Prediction (CRP), evaluates the same network twice per step: a History-Free branch predicts the target with history masked out, and a History-Conditioned branch may only add a residual on top of the (stop-gradiented) free prediction, so history supplies only what the present cannot. A second change replaces the video VAE with an image VAE (plus a lightweight temporal patch layer) to remove the latent coupling introduced by causal temporal convolutions. In controlled experiments (4M videos, 256p, 2B model), CRP closes most of the 6.14-point VBench Total gap to a matched bidirectional reference (causal baseline 71.14 → +x₀-prediction 72.41 → +CRP 77.09 → +image VAE 77.48 vs. 77.28 bidirectional), and two diagnostics (teacher-forcing vs. rollout per-frame quality gap; sensitivity to corrupted history) improve consistently. Scaled up, the resulting 2B model "Optica" reaches 82.78 VBench Total at 480p trained on ~15M videos — the best among causal models that use no bidirectional video model.
Critical strengths
- Simple and effective approach. CRP addresses exposure bias without requiring a pretrained bidirectional video teacher or a distillation pipeline. The method is straightforward and applicable to causal video diffusion models.
- Strong controlled experiments. Table 1 clearly separates the effects of x0-prediction, CRP, and the image VAE. CRP alone improves VBench Total by 4.68 points and substantially reduces quality degradation under generated and corrupted histories (Figure 2).
- Useful analysis of video VAE limitations. The authors identify temporal coupling in causal video VAEs as another source of error propagation. The latent perturbation experiments in Appendix E provide convincing evidence of this effect.
Critical weaknesses
- decomposition in Section 3.1 motivates CRP, but it does not establish that the model actually separates history-free and history-exclusive information. The residual branch still has access to the full history, and the training objective does not explicitly constrain its dependence on history. The improved robustness supports the method's effectiveness but does not fully validate the proposed explanation. 2.Additional computational cost is not quantified. CRP requires two forward passes through the shared DiT, while the image VAE removes temporal compression. Although the authors acknowledge these costs, inference latency, memory usage, and throughput are not reported. This is important given the paper's emphasis on scalable autoregressive generation.
- Evaluation is limited to short videos. All experiments use approximately 5-second videos. Since reducing accumulated errors is the primary motivation, it is unclear whether CRP remains effective over substantially longer autoregressive rollouts. A longer-horizon evaluation would strengthen the practical claims.
Questions for authors
- Can the authors provide direct evidence that CRP reduces reliance on redundant history information, rather than simply improving robustness through the auxiliary history-free prediction objective? For example, how does CRP compare with a two-branch model trained without residual coupling?
- What are the inference latency, peak GPU memory, and throughput of CRP compared with the causal baseline? How much additional overhead comes from the image VAE?
- Does CRP maintain its advantage for longer autoregressive rollouts, such as 10–20 seconds? Even a limited evaluation would help clarify whether its benefits persist as history errors accumulate.
- How sensitive is CRP to the quality of the history-free prediction? In scenarios requiring strong temporal dependencies, could the history-free branch reduce motion or identity consistency?
5 comments
Memory, Cost and Image VAE Overhead
reviewer (q2+w2): What are the inference latency, peak GPU memory, and throughput of CRP compared with the causal baseline? How much additional overhead comes from the image VAE?
We report the training and inference cost of CRP and teacher forcing under the same configuration (also added in the revised paper, Table 8).
| Setting | Model | Time | Relative | Peak memory (GiB) |
|---|---|---|---|---|
| Training step | Teacher Forcing | 3.13 s | 1.00× | 42.7 |
| CRP | 3.99 s | 1.27× | 71.0 | |
| Inference (per video) | Teacher Forcing | 55.8 s | 1.00× | 18.6 |
| CRP | 103.2 s | 1.85× | 18.6 |
In inference, CRP evaluates both branches at every step, so a video takes 1.85× as long, while peak memory is unchanged. The speed loss from the extra branch could be reduced by distilling CRP into a single-stream student.
The image VAE adds no overhead; it is faster and uses less memory than the video VAE under a correct implementation (also added in the revised paper, Table 5; same Wan 2.1 checkpoint, per clip at 192×336):
| VAE mode | PSNR (dB) ↑ | Encode (ms) ↓ | Decode (ms) ↓ | Peak encode (GiB) ↓ | Peak decode (GiB) ↓ |
|---|---|---|---|---|---|
| Video mode (causal temporal) | 27.93 ± 0.13 | 746 | 1272 | 1.18 | 1.47 |
| Image mode (per-frame) | 29.04 ± 0.15 | 422 | 720 | 0.93 | 0.80 |
Without temporal dependencies, the image VAE processes frames in parallel with 2D convolutions instead of causal 3D convolutions over the sequence, while inside the DiT the temporal patch layer (about 2% of the parameters; Appendix B.2) keeps the same 20 temporal positions as with the video VAE.
Longer rollouts
reviewer (q3+w3): Does CRP maintain its advantage for longer autoregressive rollouts, such as 10–20 seconds? Even a limited evaluation would help clarify whether its benefits persist as history errors accumulate.
We evaluate the final Optica model on 30-second rollouts using VBench-Long (also added in the revised paper, Table 4), with qualitative examples in Appendix L. We use the same protocol as the official repositories of Self Forcing and Deep Forcing, generating the samples with the first 128 MovieGen prompts and evaluating them with the VBench-Long suite. We also report the performance of several methods that do not use a bidirectional teacher, including MAGI-1, Pyramid Flow, NOVA and SkyReels-V2, following their official repositories. When evaluating our method, we use a standard sliding-window extended generation, without specific techniques for long-video rollout like sinks or RoPE adjustment. The values of the methods distilled from a bidirectional teacher are adopted from Deep Forcing and Relax Forcing (marked †).
| Method | Subject | Background | Motion | Dynamic | Aesthetic | Imaging | Average |
|---|---|---|---|---|---|---|---|
| Causal, distilled from a bidirectional teacher | |||||||
| Self Forcing† | 97.34 | 96.47 | 98.63 | 36.62 | 59.44 | 68.58 | 76.18 |
| Rolling Forcing† | 98.07 | 96.84 | 98.74 | 32.71 | 60.75 | 70.73 | 76.31 |
| LongLive† | 97.97 | 96.83 | 98.76 | 45.55 | 61.51 | 69.07 | 78.28 |
| Deep Forcing† | 97.34 | 96.48 | 98.27 | 57.56 | 60.68 | 69.31 | 79.94 |
| Relax Forcing† | 96.99 | 96.12 | 97.80 | 65.67 | 60.13 | 68.50 | 80.87 |
| Causal, bidirectional initialization only | |||||||
| SkyReels-V2 | 97.69 | 97.22 | 99.01 | 39.06 | 60.28 | 63.43 | 76.12 |
| Causal, no bidirectional video model | |||||||
| NOVA | 93.27 | 95.69 | 99.05 | 32.34 | 39.78 | 29.77 | 64.98 |
| Pyramid Flow | 93.44 | 95.84 | 98.48 | 33.59 | 42.64 | 41.07 | 67.51 |
| MAGI-1 | 97.42 | 96.82 | 99.29 | 45.52 | 58.92 | 58.96 | 76.15 |
| Optica (ours) | 95.14 | 95.65 | 96.84 | 58.49 | 52.78 | 60.73 | 76.61 |
Our method shows the highest average score among methods that do not rely on a bidirectional teacher, and is close to those bidirectional-dependent methods without specific tuning for long video. Considering our method is not trained on long videos, this is a promising result. This indicates our method can perform well when facing such a strong OOD setting, and there is room for further improvement by training on longer data or using specific techniques like attention sinks.
Effect of History-Free Branch on Motion and Identity Consistency
reviewer (q4): How sensitive is CRP to the quality of the history-free prediction? In scenarios requiring strong temporal dependencies, could the history-free branch reduce motion or identity consistency?
The history-free prediction only needs to be good where the current chunk is informative. Wherever the current chunk alone is not enough, which is exactly where temporal dependencies are strong, the history residual carries the history-dependent information, and its share grows with the noise level (Table 7 and Fig. 7 in the revised paper). Comparing Teacher Forcing () and CRP, which differ only in CRP (the bidirectional baseline is shown for reference), the fine-grained VBench dimensions show that CRP does not reduce identity or motion consistency:
| Model | Subject Consistency ↑ | Background Consistency ↑ | Motion Smoothness ↑ | Dynamic Degree | Imaging Quality ↑ | Aesthetic Quality ↑ |
|---|---|---|---|---|---|---|
| Bidirectional baseline | 89.29 | 96.08 | 96.88 | 83.61 | 63.00 | 49.08 |
| Teacher Forcing () | 77.08 | 90.33 | 96.43 | 90.00 | 60.46 | 42.26 |
| CRP | 83.07 | 93.31 | 97.65 | 82.22 | 60.42 | 49.14 |
CRP improves subject consistency (identity) from 77.08 to 83.07 and motion smoothness from 96.43 to 97.65 over Teacher Forcing (). CRP's Dynamic Degree (82.22) is lower than that of Teacher Forcing () (90.00), together with clearly higher consistency, reflecting the known trade-off between motion and consistency on VBench. The history-free branch does not suppress motion itself: over 30-second rollouts, Optica has the highest Dynamic Degree (58.49) among methods without a bidirectional video model (Table 4).
Summary of revisions
We thank you for your constructive comments, which have helped us improve the paper. Following your suggestions, we have added to the revised paper:
- an ablation of the CRP components, including a two-branch variant without residual coupling, with the training dynamics (Sec. 4.5, Table 2, Fig. 4, Appendix H);
- an analysis of the two branches across noise levels (Appendix J, Table 7, Fig. 7);
- the training and inference cost of CRP (Table 8, Appendix K) and the latency and memory of the image VAE (Table 5);
- 30-second rollouts on VBench-Long with qualitative comparisons (Table 4, Appendices I and L).
The new content is marked in blue in the revised paper, and we would appreciate it if you could take a look.
We thank the reviewer for the detailed review, and for recognizing the simplicity and effectiveness of CRP, the controlled experiments, and the analysis of video VAE limitations. We address each point below.
Ablation of CRP components and extra history-free training
Thank you very much for the suggestion. This ablation is also added in the revised paper, Table 2. First, please allow us to introduce the components of full CRP again.
Please see Table 2 and Section 4.5 for the component ablations (full discussion in Appendix H), and Fig. 4 for the training dynamics.
The two variants the reviewer asks about correspond directly to rows of this table. The − Both row is the two-branch model without residual coupling: it keeps only the auxiliary history-free objective and reaches 71.14, even below Teacher Forcing (x0, 72.41). The − CRP row additionally keeps latent fusion, yet without the output residual composition it reaches only 72.63, close to Teacher Forcing (x0) and far below full CRP (77.09). Thus, the extra history-free objective alone does not explain the gain (Table 2).
The training dynamics show the same pattern: the variants without output residual composition have lower history-conditioned training loss but worse VBench results (Fig. 4). These results support the history over-reliance explanation.
Fig. 7 (Appendix J) of the revised paper additionally visualizes the two branches along the sampling trajectory of one rollout, at steps from σ=0.99 to 0.12: the noised input at that step, the history-free prediction, the history residual, and the full CRP prediction. Table 7 quantifies their contributions: on 1,000 held-out clips, the residual removes 25.8% of the history-free error at σ=0.1 and 81.7% at σ=0.9.
This also relates to the reviewer's point that the decomposition in Section 3.1 does not by itself show that the model separates history-free and history-exclusive information. We agree that the residual branch sees the full history; what matters is what the model learns to take from it. With history h and text y, a standard causal model can lower its training loss by copying history-free information from the ground-truth history, which is the over-reliance we target. In CRP, the history-free branch must recover this information from the current input alone under stop-gradient, so the residual only needs what the history adds, E[x0∣xσ,h,y]−E[x0∣xσ,y] (Appendix J). This is an effect of the model structure, and the results above show it: the variants without residual composition reach lower history-conditioned training loss but worse generation (Table 2, Fig. 4), and the residual corrects a small share of the error at low noise, where the current chunk determines most of the content, and most of it at high noise (Table 7).