acceptodds
Under review as a conference paper at ICLR 2027

Mix Forcing: Causal Pre-Training via Mixture Attention for Interactive World Models

Abstract

While interactive video world models require autoregressive generation, directly adapting pretrained bidirectional video diffusion models into autoregressive ones with teacher forcing can disrupt their visual priors and degrade autoregressive generation quality. In this paper, we introduce Mix Forcing, a causal pre-training paradigm for interactive video world models, implemented with the Mixture-of-Bidirectional-and-Autoregressive (MoBA) attention. Mix Forcing jointly trains a video diffusion model with both teacher forcing and bidirectional training, yielding a strong causal initialization for few-step post-training. In self-attention, Mix Forcing leverages teacher forcing to learn causal next-chunk prediction for autoregressive rollout, while using bidirectional attention to preserve pretrained visual priors. To support online text interaction, Mix Forcing further adopts a chunk-wise cross-attention design that conditions each video chunk on a shared environment prompt and its own local prompt, without exposing future instructions to the current chunk. Following causal pre-training on a 14B image-to-video model, we apply consistency and distribution matching distillation to obtain a four-step autoregressive generator. Experiments on our internal benchmark and public benchmarks, including WBench and SANA-WM-Bench, show that our model achieves state-of-the-art visual quality and timely event following with precise camera control. Notably, our model supports high-fidelity long-horizon generation without quality degradation even over one-hour rollouts.

Primary area:
generative models
Keywords:
video world models, Video generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_NkwhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this work, the authors turn a pretrained bidirectional video diffusion model into a causal world model that follows camera controls and prompts that change mid-rollout. The authors argue that standard teacher forcing (TF) degrades visual quality because its attention pattern and noise setup differ from pretraining. Mix Forcing appends a second noisy copy of each clip to the TF sequence and trains it with the original bidirectional objective, using an attention mask so that both losses come from one forward pass. In cross-attention, each chunk sees an environment prompt plus its own local prompt, so a prompt update only affects later chunks and the KV cache is reused. The full system (Wan2.1-I2V-14B, then consistency distillation and DMD to four steps) reports the best visual quality and event following on an internal benchmark and on SANA-WM-Bench. The ablations indicate that Mix Forcing is a better starting point for distillation than TF.

Critical strengths
  • This work presents a simple fix for a real problem in recent causal pre-training pipelines, with no inference cost.
  • Table 5 is the most useful result. With distillation held fixed, starting from Mix Forcing instead of TF gives a much better four-step model. The pre-training objective therefore still matters after distillation.
  • The chunk-wise prompt design is practical, and the on-time metric captures responsiveness that single-prompt benchmarks ignore
Critical weaknesses
  • The main comparisons may not attribute directly the gains to Mix Forcing. The final model uses LingBot-World-Base as its DMD teacher, whereas InBench uses the method's own prompt format.
  • Novelty is modest. Training one model on several objectives by switching attention masks is a known recipe (e.g., UniLM), and LongLive already supports prompt switching.
  • Several details are missing: the long-horizon inference setup, how the per-chunk captions were produced, which VLM judges EA/OT, how baselines were prompted, and inference latency.
Questions for authors
  1. Can you separate attention pattern from noise level, for example with a causal variant trained at a shared noise level?
  2. Does the TF-only arm in Table 5 use chunk-wise or global prompts, and how does the 50-step TF-only model score?
  3. How is LingBot-World-Base conditioned on the environment and local prompts (Eq. 9), and how much of the final quality comes from it?
  4. What cache window and positional scheme do you use for one-hour rollouts, and how does "without quality degradation" square with Table 2?
  5. How were baselines prompted on InBench? Which VLM judges EA/OT, and what is the inference latency of the four-step model?
4 comments
AuthorsOpenReview ↗

Response (Part 1/3)

We sincerely appreciate your careful review and constructive comments. We address each concern below.

1.1 Isolate attention patterns from noise levels (Q1).

Following your suggestion, we train a causal variant at a shared noise level. Starting from TF+BID in Table 3 (one latent frame per chunk, global prompts), we keep all attention masks unchanged and share a single noise level across all TF chunks, so that the causal branch matches the noise configuration of pretraining but not its attention pattern. BID uses a shared noise level in both variants.

VariantTF attentionTF noise levelQuality ↑Rot ↓Trans ↓CMC ↓
TF + BID (Ours)CausalIndependent per chunk70.80.4490.2310.647
TF + BIDCausalShared across chunks68.30.9810.3321.383

Sharing the noise level under causal attention lowers Quality from 70.8 to 68.3, below even TF-only (68.9 in Table 3), and more than doubles the Rot and CMC errors (0.449 → 0.981 and 0.647 → 1.383). Matching the pretraining noise level alone therefore does not reproduce the benefit of BID; this benefit relies on bidirectional attention.

This result is consistent with our motivation for independent noise sampling, following Diffusion Forcing [1]. Since TF predicts each noisy chunk from clean historical context, independent sampling lets different chunks of the same video provide supervision at different denoising timesteps within a single update, whereas a shared noise level reduces this within-video timestep diversity. We have added this ablation as Table S5 and clarified the noise-level configurations in the revised supplementary material.

[1] Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion. Advances in Neural Information Processing Systems, 37:24081–24125, 2024.

2.1 TF-only model in Table 5 (Q2)

In Table 5, the TF-only baseline uses the same chunk-wise prompting scheme and CD/DMD post-training pipeline as Mix Forcing, ensuring consistent prompting across the comparison. The 50-step TF-only model performance is as follows:

ModelDenoising stepsQuality ↑EA ↑[email protected] ↑Rot ↓Trans ↓CMC ↓
TF-only5066.831.218.70.9170.3591.310
TF-only464.133.719.81.0770.4461.331

Before few-step post-training, Mix Forcing achieves Quality of 71.1 versus 66.8 for TF-only; after the same CD/DMD pipeline, the corresponding scores are 70.9 and 64.1. The benefit is already present before DMD and remains in the four-step models. We have added the 50-step TF-only results to Table 5 and clarified that all models in the table use chunk-wise prompt conditioning.

AuthorsOpenReview ↗

Response (Part 2/3)

3.1 Contribution of Mix Forcing and LingBot-World-Base (W1, Q3).

In Eq. (9), LingBot-World-Base serves as the bidirectional DMD teacher. Both the real and fake score models are conditioned on the camera poses and the global prompt for the same video. Our autoregressive generator receives the camera poses, the shared environment prompt, and the corresponding chunk-wise local prompts. We have made these conditioning sets explicit in the revised equation and accompanying text.

Fully isolating the teacher's contribution would require post-training with a different teacher. Instead, the following two comparisons hold the teacher fixed.

In our system-level comparisons, LingBot-World-Fast also uses LingBot-World-Base as its DMD teacher. Our model achieves higher overall Quality on InBench (75.1 vs. 72.4), both splits of SANA-WM-Bench (74.8 vs. 70.1 on Simple and 74.7 vs. 69.9 on Hard), and WBench (78.5 vs. 75.3). These comparisons therefore include a baseline that also benefits from LingBot-World-Base supervision.

To evaluate the contribution of causal pre-training, Table 5 compares the four-step TF-only and Mix Forcing models under the same chunk-wise prompts, CD/DMD post-training pipeline, and LingBot-World-Base DMD teacher. Under this shared setup, Mix Forcing improves Quality from 64.1 to 70.9, EA from 33.7 to 55.1, and [email protected] from 19.8 to 64.3, while reducing CMC error from 1.331 to 0.593. These results support Mix Forcing as a stronger causal initialization whose benefits persist through the same post-training process.

Regarding the InBench prompt format, we evaluate each baseline under two prompting settings and report its best result in Table 1 (see Response 5.1).

4.1 One-hour rollout settings and quality claims (W3, Q4).

Each inference chunk contains four latent frames. During one-hour generation, we use a fixed-size KV cache window: the first latent chunk is retained as an attention sink together with the three most recent latent chunks. To stay consistent with the 561-frame clips used in training, we additionally reset the KV cache every 561 generated frames, keeping only the attention-sink chunk and clearing all other entries. We extend the precomputed RoPE positional range to support long-horizon rollout.

The term “quality” in this claim refers to overall Quality, which aggregates AQ, IQ, TFlk, DD, and HPSv3, whereas IQ in the original Table 2 is only one component of this aggregate. The temporal IQ results are now reported in supplementary Table S3. To clarify the temporal behavior of overall Quality, we further report the overall Quality of all models over the first and last ten seconds, denoted by Quality₀–₁₀ and Quality₅₀–₆₀, respectively, in the table below.

  • Simple trajectory splits
MethodQuality₀–₁₀ ↑Quality₅₀–₆₀ ↑SQ ↑
HY-WorldPlay60.8460.3560.59
JoyAI-Echo-WM70.2364.4567.22
Matrix-Game 3.076.5574.8175.67
SANA-WM-Streaming74.2966.2770.05
LingBot-World-Fast74.6370.1172.30
Mix Forcing (Ours)78.3775.3976.85
  • Hard trajectory splits
MethodQuality₀–₁₀ ↑Quality₅₀–₆₀ ↑SQ ↑
HY-WorldPlay62.3565.4863.88
JoyAI-Echo-WM72.9466.3869.51
Matrix-Game 3.076.3475.0175.67
SANA-WM-Streaming74.8967.1770.82
LingBot-World-Fast75.8172.8974.32
Mix Forcing (Ours)78.3675.1176.70

We additionally report SQ, defined as the harmonic mean of the split-level mean Quality scores in the two windows, where SQ = 2 × Quality₀–₁₀ × Quality₅₀–₆₀ / (Quality₀–₁₀ + Quality₅₀–₆₀). Our model achieves the highest Quality in both windows on both splits, while its Quality declines by about 3 points from the first to the last ten seconds (78.37 → 75.39 on Simple and 78.36 → 75.11 on Hard). Over one-hour rollouts, mean Quality is 71.0 in the first five-minute interval and 69.3 in the last (Figure S12). We agree that "without quality degradation" overstates these results and have revised it to "without significant quality degradation" in the abstract. We have added the Quality₀–₁₀ and Quality₅₀–₆₀ scores and SQ to the revised Table 2, with the SQ definition in Appendix B.1.

AuthorsOpenReview ↗

Response (Part 3/3)

5.1 Baseline prompting, caption construction, and EA/OT evaluation (W1, W3, Q5).

  • Baseline prompting. All variants in Table 3 use global prompts to isolate the effects of the self-attention design, while both TF-only and Mix Forcing in Table 5 use chunk-wise prompts. For the baseline world models on InBench, we test two prompting settings: (i) using a global prompt throughout generation; and (ii) starting with the environment prompt and concatenating the environment prompt with the corresponding local prompt when each event occurs. We report the best results obtained for each world model across these two settings in Table 1. All methods use the same event definitions and evaluation videos. Table 1 evaluates the complete systems, while Table 4 provides the controlled comparison of global and chunk-wise conditioning within our model. We have clarified the baseline prompting protocol in Appendix B.2.

  • Caption construction. We use Qwen3-VL to generate a local caption for each one-second segment of a 16 FPS video. Each segment contains 16 video frames, corresponding to four latent frames and matching our inference chunk size. The resulting caption serves as the local prompt for that chunk. We have added this to Appendix A.1.

  • EA/OT evaluation. We use GPT-6 Astra as the vision-language model judge for EA and OT. We have added this to Appendix B.2.

5.2 Four-step inference latency (W3, Q5).

The four-step model has an end-to-end latency of 0.559 s per chunk at 832×480 resolution, measured with batch size 1 and BF16 precision on 8 H800 GPUs. These measurements exclude the first four chunks of the one-hour rollout. Each inference step generates one chunk of four latent frames, corresponding to 16 video frames (1 s of video at 16 FPS). The end-to-end latency is therefore 0.035 s per frame, or about 28.6 FPS. We use Ulysses sequence parallelism with a degree of 8 for the four-step generator and patch-parallel VAE decoding [1].

Latency4-step generatorVAE decodingEnd-to-end
s/chunk0.5250.0920.559
s/frame0.0330.0060.035

End-to-end latency measures the time from starting the generation of each latent chunk to receiving its decoded video frames. We have added the measurement settings and latency breakdown to supplementary Table S6 and the accompanying text.

[1] Fang et al. xDiT: an Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelism. arXiv:2411.01738, 2024.

6.1 Distinction from UniLM and LongLive (W2).

  • UniLM. UniLM unifies natural language understanding and generation in a shared model, where each attention mask corresponds to a capability used at inference: bidirectional masks for understanding and unidirectional masks for generation. In Mix Forcing, inference is purely causal. The BID objective is never used at inference; instead, it anchors the model to its pretraining configuration, i.e., bidirectional attention with a shared noise level, so that pretrained visual priors are preserved while TF learns causal next-chunk prediction. Moreover, in the diffusion setting, each attention mask must be paired with a matching noise configuration, which has no counterpart in UniLM. Our ablations show that this pairing, rather than mask switching itself, drives the gains: adding a DF objective lowers Quality from 68.9 to 65.6, whereas adding BID raises it to 70.8 (Table 3); with identical attention masks, changing only the TF noise configuration lowers Quality from 70.8 to 68.3 (Response 1.1).
  • LongLive. LongLive conditions the retained history on the current prompt. When the prompt switches, it recomputes the KV cache of the retained frames under the new prompt, so past frames are re-encoded with a prompt that did not generate them, and each switch incurs an extra recomputation. Mix Forcing instead introduces chunk-wise conditioning during causal pre-training: each chunk is conditioned on the shared environment prompt and its own local prompt, so cached chunks remain consistent with the prompts under which they were generated. A new local prompt affects only subsequent chunks, and the KV cache is reused without recomputation. Within our model, this design raises EA from 23.4 to 50.0 and [email protected] from 18.3 to 62.1 compared with global prompting (Table 4).

Dear Reviewer,

Thank you again for your time and thoughtful feedback. We have posted our responses to the points raised in your review. Due to the limited time remaining in the discussion period, we would sincerely appreciate it if you could review them and let us know whether they address your concerns. We welcome any further discussion if needed.

Best regards

Reviewer_HuPAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper changes pre-trained bidirectional video model (Wan2.1-I2V-14B) to autoregressive interactive world model by causal pre-training method, Mix Forcing. It puts teacher forcing branch and bidirectional branch to one sequence in the MOBA attention block. And it uses mask to separate two branch. So the model can learn next-chunk prediction and also keep the pre-training setting. Each causal chunk only sees the shared environment prompt and its own local prompt. So when new prompt comes, it doesn't need to compute the past KV cache again. After that it uses CD and DMD to make 4-step generator. It achieved positive score on quality and EA in InBench. Also Quality is more higher than other methods in both splits of SANA-WM-Bench.

Critical strengths
  1. Table 3 shows effect of adding BID to TF. In Table 3, Quality increases from 68.9 when only TF to 70.8 when TF+BID. Also TF's imaging quality goes down inside the clip (51.5 to 48.3), but TF+BID goes up (55.9 to 61.0). In Table 5, Mix Forcing achieved better score on Quality after same CD and DMD (70.9 vs 64.1). So I think the main idea is checked directly.
  2. Each causal chunk only sees the shared environment prompt and its own local prompt. So when new prompt comes, it doesn't need to compute the past KV cache again. In Table 4, EA increases from 23.4 when global prompt to 50.0 when chunk-wise prompt. Also all three camera error is lower. So the model can follow online prompt well.
  3. Supplementary explains each metric clearly. Figure S12 uses one hour rollout, and Quality is checked every 5 minutes. The mean is between 67.3 and 71.6, and last 5 minutes is 69.3. So it will be clear that the quality doesn't drop a lot in one hour.
Critical weaknesses
  1. Table 3 shows effect of adding BID to TF. However, BID latent is appended to TF sequence and they are trained in one forward pass. So same step is not same compute, and I couldn't find the compute of each run. Also Sec. 4.1 says all three stages use 256 H20 GPUs, but Appendix A.2 says 512 H200 for pre-training and 512 H20 for CD and DMD. It might be typo, but it's unclear that which one is right.
  2. Figure S12 uses one hour rollout, and Quality is checked every 5 minutes. However, Quality is average of AQ, IQ, TFlk, DD and HPSv3. So it doesn't check that the place or object is still same when camera comes back, like memory example in Figure 1. Also I couldn't find how many rollouts were used for standard deviation, and how much history is kept in KV cache in one hour.
  3. In Table 5, 4-step model achieved similar score on Quality with 50-step model (70.9 vs 71.1), and EA is more higher. However, camera error is bigger (Rot 0.319 to 0.435, CMC 0.431 to 0.593). The paper only mentions quality and event following. Also it's unclear that TF-only model in Table 5 uses chunk-wise prompt. Because Table 4 shows prompt design changes event following and camera error.
Questions for authors
  1. Is TF and TF+BID ablation matched with compute, or only with step? If only step is same, is there TF run that has similar compute? Also could you report which GPU setting is right please?
  2. I couldn't find how many rollouts were used in Figure S12. Could you report it please? Also is there result that evaluates the revisit example in Figure 1 separately, not with Quality score?
  3. I'm curious about how KV cache is handled in one hour rollout. Is the window fixed? Also I can't find wall-clock time, so I'm curious about how fast the 4-step model really generates.
  4. I couldn't find which vision-language model is used as judge for EA and OT. Could you report it please?
4 comments
AuthorsOpenReview ↗

Response (Part 1/2)

We sincerely appreciate your assessment of our work and your detailed, constructive feedback. Your suggestions are valuable for improving the clarity and completeness of our paper, and we address each concern below.

1.1 Computational fairness (W1, Q1).

The comparison between TF and TF+BID is matched in both training steps and compute. Adding BID does not increase the number of tokens: in both settings, each training step processes the same total number of tokens across all GPUs. TF+BID assigns half of these tokens to BID training, whereas TF-only assigns all of them to TF training. As a result, the two settings are trained on the same number of tokens with comparable FLOPs. We have clarified this in the revised ablation section.

1.2 GPU configuration (W1, Q1).

All three training stages use 256 H20 GPUs. We have corrected Appendix A.2 to match Section 4.1.

2.1 One-hour rollout statistics (W2, Q2).

Figure S12 evaluates 10 one-hour rollouts, each divided into twelve consecutive five-minute intervals. For each interval, we report the mean and standard deviation of Quality across the 10 rollouts. These details are now included in Appendix C.3 and the Figure S12 caption.

2.2 Revisit consistency (W2, Q2).

To complement the memory example in Figure 1, we additionally evaluate revisit consistency following the protocol in SANA-WM (Zhu et al., 2026), using PSNR, SSIM, and LPIPS. This evaluation covers 60-second videos on the Simple and Hard splits of SANA-WM-Bench.

  • Simple trajectory splits
MethodPSNR ↑SSIM ↑LPIPS ↓
HY-WorldPlay14.700.3690.491
JoyAI-Echo-WM13.480.2630.480
Matrix-Game 3.011.950.2850.504
SANA-WM-Streaming15.020.2750.527
LingBot-World-Fast14.840.3360.506
Mix Forcing (Ours)15.250.4010.503
  • Hard trajectory splits
MethodPSNR ↑SSIM ↑LPIPS ↓
HY-WorldPlay13.640.2760.484
JoyAI-Echo-WM13.600.2620.484
Matrix-Game 3.011.770.2780.504
SANA-WM-Streaming14.420.3030.491
LingBot-World-Fast14.430.3680.512
Mix Forcing (Ours)15.850.3860.471

Mix Forcing achieves the highest PSNR and SSIM on both splits and the lowest LPIPS on the Hard split. These results provide evidence of revisit consistency on the evaluated 60-second trajectories and are now included in Table S4, together with the evaluation protocol.

2.3 KV cache and four-step inference latency (W2, Q3).

  • KV cache of one-hour rollout

Each inference chunk contains four latent frames. During one-hour generation, we use a fixed-size KV cache window: the first latent chunk is retained as an attention sink together with the three most recent latent chunks. To stay consistent with the 561-frame clips used in training, we additionally reset the KV cache every 561 generated frames, keeping only the attention-sink chunk and clearing all other entries. We extend the precomputed RoPE positional range to support long-horizon rollout.

  • Four-step inference latency

The four-step model has an end-to-end latency of 0.559 s per chunk at 832×480 resolution, measured with batch size 1 and BF16 precision on 8 H800 GPUs. These measurements exclude the first four chunks of the one-hour rollout. Each inference step generates one chunk of four latent frames, corresponding to 16 video frames (1 s of video at 16 FPS). The end-to-end latency is therefore 0.035 s per frame, or about 28.6 FPS. We use Ulysses sequence parallelism with a degree of 8 for the four-step generator and patch-parallel VAE decoding [1].

Latency4-step generatorVAE decodingEnd-to-end
s/chunk0.5250.0920.559
s/frame0.0330.0060.035

End-to-end latency measures the time from starting the generation of each latent chunk to receiving its decoded video frames. We have added the measurement settings and latency breakdown to supplementary Table S6 and the accompanying text.

[1] Fang et al. xDiT: an Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelism. arXiv:2411.01738, 2024.

AuthorsOpenReview ↗

Response (Part 2/2)

3.1 Camera control after distillation (W3).

Table 5 shows that, compared with the 50-step model, the four-step model maintains comparable visual quality (71.1 to 70.9) and improves event following (EA: 50.0 to 55.1), but has higher camera-control error (CMC: 0.431 to 0.593). This is because the four-step model uses only four denoising steps and thus performs fewer pose-conditioned refinement updates, which contributes to its lower camera-control accuracy. Moreover, DMD matches the conditional video distribution without an explicit camera-pose error loss, so preserving visual quality does not necessarily mean preserving the same camera-control accuracy.

We have added this discussion to the revised manuscript. Despite this trade-off, the four-step Mix Forcing model still has much lower camera error than the four-step TF-only model (CMC: 0.593 vs. 1.331, Table 5), and our final model achieves the lowest camera errors among all world models in Table 1 (CMC: 0.539 vs. 0.694 for the next best).

3.2 Prompting settings in Tables 3 and 5 (W3).

In Table 3, all self-attention variants use global prompts to isolate the effects of the self-attention design. In Table 5, the TF-only baseline uses the same chunk-wise prompting scheme and CD/DMD post-training pipeline as Mix Forcing, ensuring consistent prompting across the comparison. We have clarified these prompting settings in the captions of Tables 3 and 5.

4.1 Evaluation model (Q4).

We use GPT-6 Astra as the vision-language model judge for EA and OT. We have added this to the evaluation details in the Appendix.

Thank you for the detail response.

  1. GPU setting, rollout number, latency and VLM judge are answered. Also revisit result on SANA-WM-Bench is helpful.
  2. For Q1, the response says total token is same in TF and TF+BID. However, the paper says BID latent is appended to TF sequence. So I'm curious about how the token is matched. Is the TF clip more shorter, or batch size is smaller?

Thanks for your follow-up comments on our response. Specifically, TF and TF+BID use the same training steps. To construct the TF-only baseline, we replace each BID instance in TF+BID with another TF instance. This ensures that the model is trained with matched computation, but different attention patterns. Therefore, both settings use the same number of training tokens and comparable FLOPs during each training step.

Reviewer_tYAxOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

Mix Forcing turns a 14B Wan 2.1 I2V model into a causal video model by adding a pre-training stage with joint teacher-forcing (TF) and bidirectional (Bidi) training. Inside a same model, while sharing the parameter, a block attention mask lets a model learn both distributions and authors also propose chunk-wise text conditioning alongside an global environment description. Following the recent paradigm, they perform Consistency Distillation, followed by DMD-based few-step distillation. Authors provide benchmark results on SANA WM Bench / InBench and the User Study.

Critical strengths
  • Training recipe sounds reasonable, and the paper includes nice ablation studies (Table 5) supporting that MOBA training is helpful.

  • I can see much of engineering efforts involved in this work from data curation, prompt generation, and larger model training. While I don’t see any video samples, result images in the supplementary looks nice.

Critical weaknesses
  • My biggest concern is the relation of this paper to “Infinite Worlds with Versatile Interactions” which is listed as the reference. In my opinion, this paper’s major contribution is in the finding that MOBA pre-training is helpful for causal distillation. But I feel like this is already known question, from “Infinite Worlds with Versatile Interactions” paper and an ICML paper this year (“Flex Forcing”). Considering that most other components are from typical causal distillation pipeline, I would expect more distinctive insights regarding MOBA but I don’t see any new contributions here. In fact, Figure 3 almost 1-1 maps to Figure 4 in “Infinite Worlds with Versatile Interactions” and attention masks look exactly the same.

  • This paper is a bit ambiguous on positioning too. I’m not sure if it's a research paper claiming technically new ideas or a technical report focusing more on the performance itself. When we look at Table 1 and Table 2, authors blindly compare with diverse set of models including the HY-WorldPlay / Sana WM / LingBot series and so on but without a matched model parameter budget / data control, this is less meaningful. I would’ve appreciated more reasons on why Mix Forcing performs better on Camera Error / Event Following and so on compared to different baselines but it’s hard to figure that out given that they have 1) different base model, 2) dataset and 3) training budget. I find Table 4/5 to be helpful but for other components authors would need to elaborate more on where these performance improvements came from.

  • Many details are omitted. What is the inference context? When authors say DMD, does it mean Self Forcing with DMD (with rollouts?) or just DMD? How does it work over one hour setup?

Questions for authors

Please see my weakness section for questions.

5 comments
AuthorsOpenReview ↗

Response (Part 1/2)

Thank you for recognizing the value of our ablation studies. Video results, including one-hour rollouts, are available on our anonymous project page (https://mixforcing.github.io), which is also linked in the submitted manuscript. We address your comments below.

1.1 Novelty and relation to prior work (W1, Q1)

Flex-Forcing. Flex-Forcing and Mix Forcing address different problems at different training stages. Flex-Forcing is a post-training method: starting from ODE initialization, it randomly samples chunk partitions within each self-generated rollout during DMD distillation, so a single sequence mixes causal and bidirectional attention, with the goal of flexible autoregressive, bidirectional, or hybrid inference. Mix Forcing instead studies causal pre-training, where each training instance follows a single mode: teacher forcing learns causal next-chunk prediction, while bidirectional training keeps the attention pattern and noise level identical to pretraining, thereby preserving pretrained visual priors. Inference is always autoregressive.

Since Flex-Forcing operates only in post-training, it does not examine whether the pre-training objective benefits causal distillation. Table 5 isolates this question: under the same CD/DMD pipeline, DMD teacher, and chunk-wise prompts, Mix Forcing improves four-step Quality from 64.1 to 70.9 and EA from 33.7 to 55.1 over TF-only pre-training, and reduces CMC error from 1.331 to 0.593. The two methods are thus orthogonal, as Flex-Forcing's flexible chunking could be applied on top of a Mix Forcing initialization. We have added this discussion of Flex-Forcing (ICML 2026) to Related Work in the revised manuscript.

"Infinite Worlds with Versatile Interactions." We respectfully note that this report is available only on arXiv. Per the ICLR 2027 Reviewer Guidelines, "arXiv is not considered a peer-reviewed venue," authors "are not required to compare to papers solely on arXiv," and "the lack of such comparisons cannot be a basis for rejection." We have also provided a confidential clarification regarding this report to the Program Chairs. We therefore kindly ask that the novelty of this submission be assessed on its own content, in particular the controlled analyses in Tables 3–5 and the additional ablations in the revised manuscript.

AuthorsOpenReview ↗

Response (Part 2/2)

2.1 Research contribution and evaluation scope (W2, Q2)

We position this work as a research paper with three contributions: (1) MoBA self-attention, which combines teacher forcing and bidirectional training during causal pre-training to preserve pretrained visual priors; (2) MoBA cross-attention, which conditions each chunk on a shared environment prompt and its own local prompt to support online prompt interaction; and (3) Mix Forcing as a stronger causal initialization for few-step post-training. Tables 1–2 are system-level comparisons intended to show the overall strength of the full system. Since the baselines differ in base model, data, and training budget, and their training data and full recipes are mostly not publicly available, a matched comparison with them is not feasible. We therefore validate each contribution through controlled ablations (Tables 3–5), where all variants share the same base model and training setup:

TableDesign evaluatedAblation
Table 3Self-Attention Design of MoBA for Causal Pre-TrainingUse TF as the baseline to evaluate adding DF and BID training, and study different training chunk sizes.
Table 4Cross-Attention Design of MoBA for Causal Pre-TrainingCompare global and chunk-wise prompt conditioning, both built on the best self-attention setting (TF+BID) in Table 3.
Table 5Mix Forcing for Few-Step Post-TrainingCompare four-step models obtained from TF-only and Mix Forcing pre-training under the same CD/DMD post-training pipeline.

2.2 Camera control and event following (W2)

Camera control. MoBA improves camera control through both attention designs. In self-attention, bidirectional training preserves the pretrained prior and avoids the within-clip quality degradation of TF (IQ 51.5 → 48.3 under TF vs. 55.9 → 61.0 under TF+BID), providing a stable scene for the pose condition; CMC drops from 1.425 to 0.647 (Table 3). In cross-attention, local prompts indicate what should appear in each chunk, which helps the pose condition control viewpoints more precisely; CMC further drops to 0.431 (Table 4).

Prompt following. MoBA's chunk-wise cross-attention conditions each chunk on its own local prompt, so a new instruction takes effect in the following chunk. This raises EA from 23.4 to 50.0 and [email protected] from 18.3 to 62.1 (Table 4). Existing world models are trained with a single global prompt, which corresponds to the global-prompt setting in Table 4.

3.1 More Details (W3, Q3)

  • Inference context: Each inference chunk contains four latent frames. In standard comparisons and ablation studies, our model uses full KV caching and attends to all preceding chunks.

  • DMD: We use the former, i.e., DMD with autoregressive self-rollout following Self Forcing. The generator rolls out chunks of four latent frames, each conditioned on its previously generated chunks through the KV cache, and the DMD objective in Eq. (9) is applied to the rollout. We have clarified this in the revised manuscript.

  • One-hour generation: We use a fixed-size KV cache window: the first latent chunk is retained as an attention sink together with the three most recent latent chunks. To stay consistent with the 561-frame clips used in training, we additionally reset the KV cache every 561 generated frames, keeping only the attention-sink chunk and clearing all other entries. We extend the precomputed RoPE positional range to support long-horizon rollout.

Dear Reviewer,

Thank you again for your time and thoughtful feedback. We have posted our responses to the points raised in your review. Due to the limited time remaining in the discussion period, we would greatly appreciate it if you could review them and let us know whether they address your concerns. We welcome any further discussion if needed.

Best regards

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.