acceptodds
Under review as a conference paper at ICLR 2027

Training expressive diffusion world models from scratch in under 50 GPU hours

Abstract

Autoregressive video diffusion has emerged as a promising approach for training expressive world models which can simulate complex interactions in dynamic environments. However, existing video diffusion models are typically highly expensive to train, which can often limit open-source research to fine-tuning of existing models or inference-time approaches with limited flexibility. In this work, we introduce WDDQ, a computationally efficient training recipe for a world model for Minecraft. In particular, we develop an autoencoder with a highly compressed latent space, a high-fidelity causal diffusion decoder, and a world model architecture combining an efficient masked predictor with a diffusion model that generates latent frames conditional on predictor outputs. Using this recipe, we train a world model that outperforms existing open-source models while training from scratch using less than 50 GPU hours. Unlike prior work, we accomplish this without distilling pretrained visual encoders, overfitting on specific scenes, or leveraging privileged information about 3D positions or geometry.

Primary area:
generative models
Keywords:
video diffusion, world models

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_6JEKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents WDDQ, a low-cost training recipe for action-conditioned Minecraft video world models. It combines a highly compressed video autoencoder with a pixel diffusion decoder and a separable architecture consisting of a masked next-latent predictor and a prediction-conditioned diffuser. The authors report training both stages from scratch in under 40 GH200 GPU-hours, achieving better generation quality and action controllability than the evaluated open-source baselines.

Critical strengths
  1. The paper combines highly compressed representations, independent predictor supervision, and masked training into a low-cost recipe with practical value for lowering the barrier to world-model research.
  2. The model achieves better generation quality and action controllability than the evaluated open-source baselines while maintaining competitive inference speed.
  3. The paper is clearly presented, with helpful architecture diagrams and well-organized experiments.
Critical weaknesses
  1. Section 3.1 states: “One way to prevent additional latency is by resampling the latent each time a new action is received.” The authors subsequently propose sharing the initial noise to improve within-chunk consistency. However, Section 4.2 does not clearly specify whether the main evaluation receives actions strictly one at a time or how unavailable actions within a chunk are handled. As a reference, Dreamer 4 [1] evaluates closed-loop interaction through human players performing tasks such as digging pits and building walls inside the model. I did not find a comparable evaluation in this paper.

  2. Section 4.2 describes the FVD computation as “dividing each sampled rollout into eight chunks of 16 frames each.” This short-clip evaluation does not directly assess consistency across clips, such as whether a placed block remains present after the player looks away and returns. Long-Context State-Space Video World Models [2] quantitatively evaluates reconstruction accuracy when revisiting previously observed locations in Memory Maze and analyzes how accuracy changes with the temporal retrieval distance. Although this paper provides qualitative examples of visual consistency, I did not find a comparable systematic evaluation of scene memory or state persistence.

  3. Section 4.3 states: “All models are trained with the same number of steps, with the dataset and optimizer held constant.” However, the variants have different computational costs per step, so equal training steps do not correspond to equal wall-clock budgets. The reported FLOPs comparisons support computational savings but do not establish performance advantages under a fixed training-time budget. SCD [3], for example, reports generation quality as a function of actual training time. Given that low-cost training is a central contribution, a similar comparison would strengthen the evidence.

  4. Section 3.2 explicitly states that “potentially noisy gradients from the diffusion model are not backpropagated to the causal predictor.” Consequently, the predictor is trained only through the next-frame prediction objective. However, the paper lacks an ablation that isolates this choice by comparing enabled and disabled diffusion-gradient backpropagation while keeping the separable architecture and other settings unchanged.

  5. The paper follows an established IDM-based evaluation protocol but does not further examine whether its scores agree with human judgments on the videos generated by this model. MineWorld [4] evaluates the correlation between automatic controllability metrics and human ratings. I did not find a comparable validation here.

  6. The experiments use a fixed configuration of 10 world-model sampling steps and 8 diffusion-decoder steps, without analyzing generation quality, action controllability, and throughput under alternative step counts. Since both sampling stages contribute to end-to-end inference cost, varying their step counts separately would clarify the quality–speed trade-off and provide a rationale for the chosen configuration.

[1] Danijar Hafner, Wilson Yan, and Timothy Lillicrap. Training Agents Inside of Scalable World Models. arXiv:2509.24527, 2025. https://arxiv.org/abs/2509.24527

[2] Ryan Po et al. Long-Context State-Space Video World Models. ICCV, 2025. https://arxiv.org/abs/2505.20171

[3] Xingjian Bai et al. Causality in Video Diffusers is Separable from Denoising. arXiv:2602.10095, 2026. https://arxiv.org/abs/2602.10095

[4] Junliang Guo et al. MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft. arXiv:2504.08388, 2025. https://arxiv.org/abs/2504.08388

Questions for authors

Please see the issues raised in the Weaknesses.

5 comments

We thank the reviewer for their constructive feedback. We are encouraged that they find our proposed approach to be practical for lowering the barrier to world-model research, while achieving strong performance in terms of generation quality and action controllability and maintaining competitive inference performance.

W1. “Section 3.1 states: ‘One way to prevent additional latency is by resampling the latent each time a new action is received’ [...] However, Section 4.2 does not clearly specify whether the main evaluation receives actions strictly one at a time or how unavailable actions within a chunk are handled. As a reference, Dreamer 4 [1] evaluates closed-loop interaction through human players performing tasks [...] I did not find a comparable evaluation in this paper.”

We have revised the paper to clarify that in all evaluations, when generating frame tt, only action inputs up to and including action t−1t-1 are provided to WDDQ (and the input chunk of actions is right-padded with zeros, if a full chunk is not yet available), and the latent is re-sampled for each new action and subsequent frame. As such, our evaluations match the same input structure and observation-action causality of interactive, human-in-the-loop inference. Using fixed starting observations and identical action sequences as input to each model in our evaluations enables more closely controlled comparisons between models and avoids introducing human variability.

W2. “Section 4.2 describes the FVD computation as ‘dividing each sampled rollout into eight chunks of 16 frames each.’ This short-clip evaluation does not directly assess consistency across clips, such as whether a placed block remains present after the player looks away and returns. Long-Context State-Space Video World Models [2] quantitatively evaluates reconstruction accuracy when revisiting previously observed locations [...] I did not find a comparable systematic evaluation of scene memory or state persistence”

We add a quantitative evaluation of long-horizon consistency in Section 4.4 of the paper. To broadly emulate the evaluation in Long-Context State-Space Video World Models for our dataset, we filter for videos with a high proportion of ‘revisit’ frames, with a precise definition and filtering thresholds provided in the paper. For each video, all models receive 32 ground-truth frames as context, and then each model autoregressively generates the following 96 frames conditional on this context. We report average LPIPS of the rollout with respect to the ground-truth continuation, evaluating the model’s ability to accurately reconstruct parts of the scene that are visible in the provided context frames. WDDQ outperforms both Oasis and MineWorld in terms of revisit LPIPS.

ModelRevisit LPIPS
Oasis0.5548
MineWorld0.4849
WDDQ0.4619

W3. “The variants have different computational costs per step, so equal training steps do not correspond to equal wall-clock budgets. The reported FLOPs comparisons support computational savings but do not establish performance advantages under a fixed training-time budget. SCD [3], for example, reports generation quality as a function of actual training time. Given that low-cost training is a central contribution, a similar comparison would strengthen the evidence.”

We observe in our experiments that all models achieve similar MFU on our hardware and training configuration, so FLOP counts are approximately proportional to the elapsed training time. Unlike wall-clock training time, which can vary based on unpredictable factors (like GPU temperature or filesystem pressure) and across software versions, FLOP counts represent a more deterministic and accurate metric for training budget, at least for these experiments, where no method achieves significantly higher tensor core utilization.

W4. “Section 3.2 explicitly states that “potentially noisy gradients from the diffusion model are not backpropagated to the causal predictor.” Consequently, the predictor is trained only through the next-frame prediction objective. However, the paper lacks an ablation that isolates this choice by comparing enabled and disabled diffusion-gradient backpropagation while keeping the separable architecture and other settings unchanged.”

We are currently running this ablation and anticipate that results will be available before the end of the discussion period.

W5. “The paper follows an established IDM-based evaluation protocol but does not further examine whether its scores agree with human judgments on the videos generated by this model. MineWorld [4] evaluates the correlation between automatic controllability metrics and human ratings. I did not find a comparable validation here.”

We provide additional validation of the IDM evaluation protocol in Appendix F by comparing its outputs to those from a VLM judge. We pass controls as a text prompt and pairs of videos to Qwen3-VL-32B-Instruct, a strong open-source model, and ask it to select the video that follows actions more faithfully while ignoring visual quality. In pairwise comparisons between rollouts sampled from Oasis, Mineworld, and WDDQ, the VLM’s preference agrees with the IDM’s relative ranking for discrete-action F1 approximately 85.3% of the time. Both metrics prefer WDDQ over Oasis in terms of action controllability over 96% of the time, and both prefer WDDQ over MineWorld about 56% of the time. Thus, the IDM demonstrates high agreement with a VLM judge when comparing individual video pairs, as well as in terms of aggregated preference rates when comparing samples from two models.

W6. “The experiments use a fixed configuration of 10 world-model sampling steps and 8 diffusion-decoder steps, without analyzing generation quality, action controllability, and throughput under alternative step counts. Since both sampling stages contribute to end-to-end inference cost, varying their step counts separately would clarify the quality–speed trade-off and provide a rationale for the chosen configuration.”

Our choice of 10 world model sampling steps was picked simply to match Oasis’s sampling configuration. To clarify the quality-speed tradeoff, we sweep the number of sampling steps for both the world model and autoencoder and report FVD, action controllability, and throughput under each configuration evaluated. Due to compute limitations, we sample only 500 rollouts of length 64 for each configuration, and due to hardware availability, FPS values are reported on an H200 (rather than GH200, used by the original experiments). FVD improves monotonically when either WM steps or AE steps increase, all else held constant. Action F1 improves as WM steps increase and appears to remain the same or decrease by a relatively small amount as AE steps increase, potentially as a result of the presence of additional, noisy high-frequency features making it slightly more difficult to accurately determine the actions taken. As expected, FPS decreases as either WM or AE steps increase.

WM stepsAE stepsFVD@500, T=64Action F1FPS (H200)
4465.40.81827.8
4861.60.81719.2
41659.50.81711.9
8451.50.82021.7
8848.80.81816.0
81648.10.81610.6
16447.40.82315.0
16845.60.82112.1
161644.60.8208.69

Dear Reviewer 6JEK,

We have completed the requested ablation, in which gradients from the diffuser are backpropagated to the predictor while keeping the separable architecture and other settings unchanged.

ConfigurationFVD ↓Discrete-action F1 ↑Mouse L1 ↓
Diffusion gradients detached63.40.8120.261
Diffusion gradients enabled54.20.8070.272

We find that enabling gradients from the diffuser results in improved visual fidelity, reducing FVD, while giving slightly worse action controllability in terms of both discrete action F1 and mouse L1 error. Backpropagating diffuser gradients introduces a new hyperparameter for the weight of the diffusion loss relative to the prediction loss, potentially increasing the difficulty of hyperparameter tuning. We also note that it is no longer possible to train the predictor and diffuser separately for this variant. For simplicity, we recommend the main configuration of WDDQ with gradients disabled, but the optimal choice likely depends on hyperparameter tuning budget and the importance of controllability relative to visual fidelity.

We have incorporated these results into the revised manuscript in Section 4.3, and provided additional clarifications about the trade-offs of this approach in Section 3.2. We hope our responses and additional experiments have addressed your concerns. If so, we would be grateful if you might reconsider your overall assessment in light of our new results and clarifications. Thank you again for your thoughtful feedback, which has helped us improve the paper.

Best, WDDQ Authors

Reviewer_utC2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents WDDQ, an efficient training recipe for video diffusion world models that can be trained from scratch on a single GH200 GPU within 50 hours. The proposed approach combines an autoencoder with 4× temporal and 32× spatial compression and a pixel-space diffusion decoder. The world model is further divided into a predictor with 50% tube masking and a spatial diffuser, with gradients stopped between the two modules.

Experiments on the Minecraft VPT dataset show that WDDQ achieves better video generation quality (FVD 63.4) and action controllability (mouse L1 error 0.261) than Open Oasis and MineWorld, while reaching 8.81 FPS.

Critical strengths
  • The most compelling contribution is the ability to train a high-resolution video world model from scratch on a single GPU in under 40 hours (16 hours for the autoencoder and 24 hours for the world model). This significantly lowers the computational barrier to developing and experimenting with video world models.
  • The authors make reasonable architectural choices to reduce computational cost. Tube masking reduces the predictor's temporal modeling cost, while the pixel-space diffusion decoder helps recover visual details lost through aggressive latent compression. The overall design achieves a useful trade-off between training efficiency and generation quality.
Critical weaknesses
  • All experiments are conducted in Minecraft, where visual appearances and environmental dynamics are relatively constrained compared to real-world videos. While the results are promising, it remains unclear whether the proposed efficiency techniques would remain effective in more visually diverse and complex environments.
  • Most components, including the diffusion decoder, VideoMAE-style tube masking, and the decoupled predictor-diffuser design, build on existing techniques. The contribution appears to lie primarily in their effective integration rather than a fundamentally new modeling approach.
  • The comparison with existing baselines is not fully controlled in terms of training data and computational budget. Therefore, it is unclear whether the reported performance gains are primarily due to the proposed architecture or differences in the training setup. A comparison under matched training conditions would help better isolate the contribution of the proposed method.
Questions for authors
  • The proposed design relies on aggressive spatiotemporal compression and a relatively small latent representation. Have the authors tested whether these choices remain effective on more visually complex datasets, such as real-world videos? Even a small-scale experiment on another domain would help support the broader applicability of the approach.

  • The autoencoder requires approximately 16 hours of training. Have the authors considered using an existing pretrained video encoder to reduce the total training cost? Although this would change the from-scratch setting, it would be useful to understand the trade-off between training a specialized autoencoder and leveraging pretrained representations.

  • The current experiments do not clearly establish whether WDDQ's performance gains come from the proposed architecture or the use of a curated high-quality data subset. To better isolate the contribution of the proposed method, it would be valuable to conduct comparisons in both directions: scaling down the baseline models to match WDDQ's training conditions and scaling up WDDQ to evaluate its performance with larger datasets and model sizes.

9 comments

We thank the reviewer for their constructive feedback. We are encouraged that they find our approach’s capability to train a high-resolution video world model from scratch in approximately 40 GPU hours to be compelling, and are glad that they find that our method lowers the computational barrier to world-model research via architectural designs the achieve a useful trade-off between computational efficiency and generation quality.

W1/Q1. “All experiments are conducted in Minecraft, where visual appearances and environmental dynamics are relatively constrained compared to real-world videos.”

We agree that experiments in additional domains would provide further evidence of the generalizability of our approach. However, Minecraft is widely used for world modeling research precisely due to its high visual diversity and complex environment interactions. This is especially true for the specific dataset we use, i.e. OpenAI’s VPT contractor dataset. Much of the prior literature on efficient world modeling uses the TECO Minecraft dataset [1, 2, 3, 4], a low-resolution and low-diversity dataset that only includes videos of players navigating swamp scenes. The VPT contractor dataset, in comparison, contains significantly more visual diversity and involves complex interactions with the environment, including breaking and placing blocks, crafting, smelting, and interacting with mobs and entities. Additionally, since players often enter areas and new biomes previously unseen in the episode, accurate world modeling on this dataset requires learning to emulate the game’s underlying and complex world generation logic. As such, a variety of recent, large-scale world modeling works, including Oasis, MineWorld, and DreamerV4, use the VPT Minecraft dataset as their primary benchmark. To our knowledge, WDDQ is the first work that demonstrates training from scratch on the significantly more general and diverse VPT dataset with a computational budget on the order of 50 GPU hours.

Minecraft also serves as a challenging world-modeling task due to its high-dimensional, high-frequency action space, with a complex and state-dependent relationship between actions and their effect on future observations. Some internet-scale video datasets lack low-level actions altogether (e.g. Koala, InternVid). Other real-world video world modeling datasets, such as Sekai2, provide only camera pose annotations, which have a comparatively deterministic and predictable impact on the video observations. Many physical-robot datasets, such as SOAR and BridgeDataV2, are limited to only tabletop manipulation tasks, and are arguably less diverse in terms of visual diversity than the VPT Minecraft dataset. Thus, the VPT Minecraft dataset is one of the few datasets that simultaneously involve high visual diversity, complex dynamics, and expressive low-level control.

W2. “Most components [...] build on existing techniques. The contribution appears to lie primarily in their effective integration rather than a fundamentally new modeling approach.”

Although components which are similar at a high level exist in prior literature, effective approaches for adapting these components for causal world modeling are nontrivial and are meaningful contributions of our work. For example, typical causal video VAEs (such as HunyuanVideo-1.5, a baseline we compare to in the paper) condition the decoder on previous latents, a design decision that enables parallelization across time. However, action-conditional world modeling necessarily implies streaming inference. We introduce a novel, hierarchical diffusion decoder architecture with both downsampling and upsampling streams and additional input channels for the previous reconstruction, improving the decoder’s ability to generate low-level details that are consistent across time. As a result, our autoencoder experimentally demonstrates improved reconstruction fidelity relative to baselines.

In addition, adapting masked training (typically used in training image diffusion or bidirectional video diffusion models) to a causal world modeling setting introduces additional challenges in terms of error accumulation and the train-test gap it induces. The specific instantiation of the separable architecture we propose, in which only latent-space predictions from the causal predictor are passed as input to the decoder, is a novel adaptation designed to address this train-time gap that meaningfully improves computational efficiency while preserving generation quality. Thus, WDDQ involves novel designs for each of the underlying components of our method, crucial for the experimentally demonstrated improvements in generation quality and computational efficiency.

W3/Q3. “The comparison with existing baselines is not fully controlled in terms of training data and computational budget [...] a comparison under matched training conditions would help better isolate the contribution of the proposed method.”

We agree that retraining every baseline under an identical data and compute budget would provide the most direct controlled comparison. Unfortunately, the publicly released Oasis and MineWorld implementations do not provide reproducible training code, making it infeasible to perform a fully-controlled comparison. However, the overall parameter count of our autoencoder and world model are similar to that of Oasis, and we target the same resolution, so we can rule out the hypothesis that parameter count or resolution differences alone explains the improvement of our method.

We therefore perform the closest controlled comparisons available within our framework. In Section 4.3, we ablate variants that replace our proposed components with architectural choices similar to those found in the baseline approaches while holding the training data, optimizer, and number of training steps fixed. In particular, the ‘non-separable architecture’ variant uses a unified diffusion transformer architecture, rather than a separable predictor and diffuser, making it extremely similar to the diffusion model architecture used by Oasis. Under controlled conditions, the non-separable variant achieves a substantially worse FVD of 87.0 compared to 63.4 for WDDQ, while requiring significantly more inference compute, at 2.06×10122.06 \times 10^{12} FLOPs per frame compared to 1.27×10121.27 \times 10^{12} FLOPS per frame for WDDQ.

Thus, the comparisons to Oasis and MineWorld establish that WDDQ achieves stronger end-to-end performance relative to existing open-weight models, while our ablation study demonstrates that the improved performance can be attributed to the proposed method.

Q2: “Have the authors considered using an existing pretrained video encoder to reduce the total training cost? Although this would change the from-scratch setting, it would be useful to understand the trade-off between training a specialized autoencoder and leveraging pretrained representations.”

A pretrained video autoencoder would reduce up-front training cost and provide useful pretrained representations, however, depending on domain, off-the-shelf autoencoders may not be available, or have suboptimal tradeoffs between latency, resolution, and reconstruction fidelity. Efficient end-to-end training of both the autoencoder and diffusion model is thus a central desiderata of this work and increases the broader applicability of our results. Leveraging pretrained representations, in either the autoencoder or the world model, constitutes an interesting and broad research direction complimentary to our work, but our primary focus is on the setting of training from-scratch.

References

[1] Wilson Yan, Danijar Hafner, Stephen James, and Pieter Abbeel. Temporally consistent transformers for video generation. In International Conference on Machine Learning, pp. 39062–39098. PMLR, 2023.

[2] Ryan Po, Yotam Nitzan, Richard Zhang, Berlin Chen, Tri Dao, Eli Shechtman, Gordon Wetzstein, and Xun Huang. Long-context state-space video world models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8733–8744, 2025.

[3] Yuchao Gu, Weijia Mao, Mike Zheng Shou. Long-context autoregressive video modeling with next-frame prediction. arXiv preprint arXiv:2503.19325, 2025.

[4] Kiwhan Song, Boyuan Chen, Max Simchowitz, Yilun Du, Russ Tedrake, Vincent Sitzmann. History-guided video diffusion. arXiv preprint arXiv:2502.06764, 2025.

Dear Reviewer utC2,

Thank you again for your thoughtful review and for recognizing WDDQ’s potential to make world-model research more accessible. We have worked to strengthen both the empirical evaluation and the explanation of our contributions. For convenience, we summarize our responses and additions below:

  • Clarified the methodological contributions: We explained the specific designs needed for causal world modeling, including the hierarchical diffusion decoder conditioned on previous reconstructions and the separable predictor–diffuser architecture.
  • Highlighted controlled architectural comparisons: With training data, optimizer, and training steps held fixed, WDDQ achieves 63.4 FVD versus 87.0 for the non-separable variant, while reducing inference computation from 2.06 to 1.27 TFLOPs per frame.
  • Added a quantitative scene-revisit evaluation: WDDQ achieves 0.4619 LPIPS, compared with 0.4849 for MineWorld and 0.5548 for Oasis, providing additional evidence of consistency when revisiting previously observed scenes.
  • Expanded validation of action controllability: We added an independent VLM-based assessment, which agrees with the inverse-dynamics metric’s pairwise rankings 85.3% of the time.
  • Examined efficiency and design trade-offs: We added a nine-configuration sampling-step study covering generation quality, controllability, and throughput, and completed the gradient-backpropagation ablation to clarify the trade-off between visual fidelity and action control.
  • Clarified the benchmark and training choices: We explained the diversity and interaction complexity of the VPT dataset and the motivation for training both the autoencoder and world model from scratch.

We hope these additions address your concerns and strengthen your confidence in the contribution. As the discussion period approaches its end, we would be very grateful if you could revisit your assessment in light of this work. We would warmly welcome any remaining questions.

Thank you again for your time and consideration.

Best,
WDDQ Authors

Reviewer_1Kn1OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces World Diffusion Done Quick (WDDQ), an efficient recipe for training action-conditioned video world models from scratch. The method combines a highly compressed video autoencoder with a diffusion decoder, a masked temporal predictor, and a spatial-only latent diffuser conditioned on the predictor’s output. To accommodate online interaction despite temporal compression, it proposes resampling the latent as new actions arrive while reusing the initial noise within each frame chunk. On Minecraft, the authors report a combined autoencoder and world-model training time of less than 40 GH200 GPU hours, alongside improvements over Open Oasis and MineWorld in generation quality and action controllability.

Critical strengths
  1. Practical contribution to training efficiency. Training both the autoencoder and the world model within the reported budget is a meaningful result that could make video world-model research more accessible. The paper provides substantial architectural, optimization, and preprocessing details, making the training recipe informative for subsequent work.

  2. Strong reported visual quality and controllability. WDDQ achieves an FVD of 63.4, compared with 104.9 for MineWorld and 365.1 for Open Oasis, while improving inverse-dynamics-based controllability across most action categories. These results are obtained at the native 640 × 360 resolution with an end-to-end throughput of 8.81 FPS, demonstrating a useful balance between quality and inference efficiency.

  3. Useful evidence on latent representation design. The comparison between the 32-channel and 128-channel autoencoders illustrates that stronger reconstruction performance does not necessarily translate into better downstream world modeling under a limited training budget. The more compressed representation achieves better world-model results, providing a useful insight into the trade-off between reconstruction fidelity and modeling difficulty.

Critical weaknesses
  1. Limited evaluation of long-horizon state consistency. The experiments demonstrate strong short-horizon visual quality and action controllability, and Figures 3–5 provide encouraging examples of interactions. However, evaluation is restricted to 128-frame rollouts, with FVD computed on 16-frame segments. The inverse-dynamics metrics measure whether actions can be recovered from generated videos, but do not directly establish whether their consequences persist in the environment. Consequently, the results provide limited quantitative evidence of whether environmental changes and previously observed scene structure remain consistent over longer interactions. Although the conclusion acknowledges this limitation, targeted persistence tests would help establish the scope of the model’s capabilities.
Questions for authors
  1. Section 3.1 describes per-action latent resampling, while Appendix D mentions retaining short runs of null actions for inference. Are unavailable entries in the four-action chunk filled with null actions until the corresponding inputs arrive? Please also clarify whether Tables 2–3 and the reported FPS use this per-action resampling protocol or generate each latent after the complete action chunk is available.

  2. Figures 3–5 show examples of block placement, breaking, and inventory behavior. How consistently are these state changes preserved over longer rollouts, particularly when the camera moves away and revisits an edited location? A quantified persistence test would help complement the current qualitative examples.

  3. Section 3.2 motivates gradient detachment as a way to avoid potentially noisy diffusion gradients. With the separable architecture and masking held fixed, how does allowing the diffusion loss to update the predictor, while retaining its prediction loss, affect FVD, action controllability, and training cost? This would help isolate the contribution of gradient detachment beyond the architectural comparisons in Table 4.

  4. For the reconstruction results in Table 1, does the decoder condition on the ground-truth previous frame or its own preceding reconstruction? If ground-truth frames are used, how do the reconstruction metrics change under recursive decoding with reconstructed previous frames?

3 comments

We thank the reviewer for the constructive feedback. We are encouraged that they recognize WDDQ’s practical contribution to substantially lowering the cost of training video world models, as well as its strong visual quality and action controllability, and we are glad that they find our latent-representation experiments, which study the trade-off between latent modelability and reconstruction fidelity, to be insightful.

W1/Q2. “Limited evaluation of long-horizon state consistency.”

We add an evaluation for long-horizon state consistency in Section 4.4 of the paper, measuring the model’s reconstruction accuracy when ‘revisiting’ previously observed parts of the scene. We sample sequences from out dataset containing a high proportion of frames that revisit previously observed parts of the environment, with a precise definition of ‘revisit’ and filtering thresholds provided in the paper. For each video, each model receives 32 ground-truth frames as context, and then autoregressively generates the following 96 frames conditional on this context. We report average LPIPS of the generated frames with respect to the ground-truth continuation, evaluating the model’s ability to accurately reconstruct parts of the scene that were previously observed. WDDQ outperforms both Oasis and MineWorld in terms of revisit LPIPS.

ModelRevisit LPIPS
Oasis0.5548
MineWorld0.4849
WDDQ0.4619

Q1.i. “Are unavailable entries in the four-action chunk filled with null actions until the corresponding inputs arrive?”

Yes, if a full four-action chunk is not yet available when generating any particular frame, we right-pad the action-conditioning inputs with null actions.

Q1.ii. “Clarify whether Tables 2–3 and the reported FPS use this per-action resampling protocol or generate each latent after the complete action chunk is available.”

Yes, all experimental evaluations with our model use the same input structure that would be used during streaming interactive inference, in which latents are re-sampled at every timestep.

Q3. “How does allowing the diffusion loss to update the predictor, while retaining its prediction loss, affect FVD, action controllability, and training cost?”

We are currently running this ablation and anticipate that results will be available before the end of the discussion period.

Q4. “For the reconstruction results in Table 1, does the decoder condition on the ground-truth previous frame or its own preceding reconstruction?”

During inference, the preceding reconstructed frames are provided as input to the model, ensuring that the autoencoder evaluations use the same input structure that would be used during streaming interactive inference.

Dear Reviewer 1Kn1,

We have completed the requested ablation, in which gradients from the diffuser are backpropagated to the predictor while retaining the prediction loss and keeping the architecture and other settings unchanged.

ConfigurationFVD ↓Discrete-action F1 ↑Mouse L1 ↓
WDDQ, diffusion grads detached63.40.8120.261
WDDQ, diffusion grads enabled54.20.8070.272

We find that enabling gradients from the diffuser improves FVD, while slightly reducing action controllability in terms of macro-averaged discrete action F1 and mouse L1 error. We note that backpropagating diffuser gradients introduces a new hyperparameter for the weight of the diffusion loss relative to the prediction loss, potentially increasing the difficulty of hyperparameter tuning. We also note that it is no longer possible to train the predictor and diffuser separately for this variant. For simplicity, we recommend the main configuration of WDDQ with gradients disabled, but the optimal choice likely depends on hyperparameter tuning budget and the importance of controllability relative to visual fidelity. We have revised Section 4.3 of the paper to incorporate the new results and clarify the trade-offs of this approach.

Thank you for suggesting this ablation. We hope the added results and our responses have addressed your concerns, and we would be happy to discuss any remaining questions. If your concerns have been resolved, we would sincerely appreciate it if you would consider raising your rating. Thank you again for helping us improve the paper.

Best, WDDQ Authors

Thank you for the detailed response and the new experiments. The new Revisit LPIPS evaluation effectively addresses my main concern regarding long-horizon state consistency, and the clarifications on the inference protocol have resolved my minor questions. Overall, the rebuttal has addressed my concerns, and I will maintain my positive rating for this work.

Reviewer_iQvBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces WDDQ (World Diffusion Done Quick), an efficient training recipe for autoregressive video diffusion world models. The approach combines a highly compressed video autoencoder with a generative diffusion decoder and a separable world model architecture consisting of a masked causal predictor and a spatial diffusion model. WDDQ enables training from scratch in under 50 GPU hours on a single GH200 GPU. Experiments on Minecraft demonstrate improved video generation quality, action controllability, and computational efficiency compared to existing open-weight world models.

Critical strengths

The ability to train a competitive video world model from scratch in under 50 GPU hours is impressive and substantially reduces the computational barrier to world model research.

The combination of highly compressed video latents, a diffusion-based decoder, and a separable predictor-diffuser architecture provides a practical approach to reducing training and inference costs.

WDDQ achieves substantially better FVD and action controllability than Open Oasis and MineWorld while operating at higher resolution and competitive inference speed.

Critical weaknesses

Most components, including latent compression, diffusion decoding, masked prediction, and separable architectures, build upon existing techniques. The main contribution is their effective integration and optimization rather than a fundamentally new modeling approach.

Experiments are restricted to Minecraft, making it unclear whether the proposed training recipe generalizes to more diverse visual environments, real-world videos, or tasks requiring complex physical reasoning.

The compared models differ in training datasets, architectures, parameter counts, and resolutions. Without retraining baselines under matched computational budgets, it is difficult to attribute the improvements solely to the proposed method.

Questions for authors

none

2 comments

We thank the reviewer for the constructive feedback and for recognizing the practical significance of WDDQ. We are encouraged that the reviewer finds our approach’s capability to train a competitive video world model from scratch in under 50 GPU hours to be impressive, and that they find the architecture we propose to be practical, with experimentally demonstrated improvements in FVD and action controllability.

W1. “Most components [...] build upon existing techniques. The main contribution is their effective integration and optimization rather than a fundamentally new modeling approach”

Although components which are similar at a high level exist in prior literature, effective approaches for adapting these components for causal world modeling are nontrivial and are meaningful contributions of our work. For example, typical causal video VAEs (such as HunyuanVideo-1.5, a baseline we compare to in the paper) condition the decoder on previous latents, a design decision that enables parallelization across time. However, action-conditional world modeling necessarily implies streaming inference. We introduce a novel, hierarchical diffusion decoder architecture with both downsampling and upsampling streams and additional input channels for the previous reconstruction, improving the decoder’s ability to generate low-level details that are consistent across time. As a result, our autoencoder experimentally demonstrates improved reconstruction fidelity relative to baselines.

In addition, adapting masked training (typically used in training image diffusion or bidirectional video diffusion models) to a causal world modeling setting introduces additional challenges in terms of error accumulation and the train-test gap it induces. The specific instantiation of the separable architecture we propose, in which only latent-space predictions from the causal predictor are passed as input to the decoder, is a novel adaptation designed to address this train-time gap that meaningfully improves computational efficiency while preserving generation quality. Thus, WDDQ involves novel designs for each of the underlying components of our method, crucial for the experimentally demonstrated improvements in generation quality and computational efficiency.

W2. “Experiments are restricted to Minecraft, making it unclear whether the proposed training recipe generalizes to more diverse visual environments, real-world videos, or tasks requiring complex physical reasoning.”

We agree that experiments in additional domains would provide further evidence of generalization. However, Minecraft is widely used for world modeling research precisely due to its high visual diversity and complex environment interactions. This is especially true for the specific dataset we use, i.e. OpenAI’s VPT contractor dataset. Much of the prior literature on efficient world modeling uses the TECO Minecraft dataset [1, 2, 3, 4], a low-resolution and low-diversity dataset that only includes videos of players navigating swamp scenes. The VPT contractor dataset, in comparison, contains significantly more visual diversity and involves complex interactions with the environment, including breaking and placing blocks, crafting, smelting, and interacting with mobs and entities. Additionally, since players often enter areas and new biomes previously unseen in the episode, accurate world modeling on this dataset requires learning to emulate the game’s underlying and complex world generation logic. As such, a variety of recent, large-scale world modeling works, including Oasis, MineWorld, and DreamerV4, use the VPT Minecraft dataset as their primary benchmark. To our knowledge, WDDQ is the first work that demonstrates training from scratch on the significantly more general and diverse VPT dataset with less than 50 GPU hours.

Minecraft also serves as a challenging world-modeling task due to its high-dimensional, high-frequency action space, with a complex and state-dependent relationship between actions and their effect on future observations. Some internet-scale video datasets lack low-level actions altogether (e.g. Koala, InternVid). Other real-world video world modeling datasets, such as Sekai2, provide only camera pose annotations, which have a comparatively deterministic and predictable impact on the video observations. Many physical-robot datasets, such as SOAR and BridgeDataV2, are limited to only tabletop manipulation tasks and are arguably less diverse in terms of visual diversity than the VPT Minecraft dataset.

W3. “Without retraining baselines under matched computational budgets, it is difficult to attribute the improvements solely to the proposed method.”

We agree that retraining every baseline under an identical data and compute budget would provide the most direct controlled comparison. Unfortunately, the publicly released Oasis and MineWorld implementations do not provide reproducible training code, making it infeasible to perform a fully-controlled comparison. However, the overall parameter count of our autoencoder and world model are similar to that of Oasis, and we target the same resolution, so we can rule out the hypothesis that parameter count or resolution differences alone explains the improvement of our method.

We therefore perform the closest controlled comparisons available within our framework. In Section 4.3, we ablate variants that replace our proposed components with architectural choices similar to those found in the baseline approaches while holding the training data, optimizer, and number of training steps fixed. In particular, the ‘non-separable architecture’ variant uses a unified diffusion transformer architecture, rather than a separable predictor and diffuser, making it extremely similar to the diffusion model architecture used by Oasis. Under controlled conditions, the non-separable variant achieves a substantially worse FVD of 87.0 compared to 63.4 for WDDQ, while requiring significantly more inference compute, at 2.06×10122.06 \times 10^{12} FLOPs per frame compared to 1.27×10121.27 \times 10^{12} FLOPS per frame for the full method.

Thus, the comparisons to Oasis and MineWorld establish that WDDQ achieves stronger end-to-end performance relative to existing open-weight models, while our ablation study demonstrates that the improved performance can be attributed to the proposed method.

References

[1] Wilson Yan, Danijar Hafner, Stephen James, and Pieter Abbeel. Temporally consistent transformers for video generation. In International Conference on Machine Learning, pp. 39062–39098. PMLR, 2023.

[2] Ryan Po, Yotam Nitzan, Richard Zhang, Berlin Chen, Tri Dao, Eli Shechtman, Gordon Wetzstein, and Xun Huang. Long-context state-space video world models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8733–8744, 2025.

[3] Yuchao Gu, Weijia Mao, Mike Zheng Shou. Long-context autoregressive video modeling with next-frame prediction. arXiv preprint arXiv:2503.19325, 2025.

[4] Kiwhan Song, Boyuan Chen, Max Simchowitz, Yilun Du, Russ Tedrake, Vincent Sitzmann. History-guided video diffusion. arXiv preprint arXiv:2502.06764, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.