- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents WDDQ, a low-cost training recipe for action-conditioned Minecraft video world models. It combines a highly compressed video autoencoder with a pixel diffusion decoder and a separable architecture consisting of a masked next-latent predictor and a prediction-conditioned diffuser. The authors report training both stages from scratch in under 40 GH200 GPU-hours, achieving better generation quality and action controllability than the evaluated open-source baselines.
Critical strengths
- The paper combines highly compressed representations, independent predictor supervision, and masked training into a low-cost recipe with practical value for lowering the barrier to world-model research.
- The model achieves better generation quality and action controllability than the evaluated open-source baselines while maintaining competitive inference speed.
- The paper is clearly presented, with helpful architecture diagrams and well-organized experiments.
Critical weaknesses
-
Section 3.1 states: “One way to prevent additional latency is by resampling the latent each time a new action is received.” The authors subsequently propose sharing the initial noise to improve within-chunk consistency. However, Section 4.2 does not clearly specify whether the main evaluation receives actions strictly one at a time or how unavailable actions within a chunk are handled. As a reference, Dreamer 4 [1] evaluates closed-loop interaction through human players performing tasks such as digging pits and building walls inside the model. I did not find a comparable evaluation in this paper.
-
Section 4.2 describes the FVD computation as “dividing each sampled rollout into eight chunks of 16 frames each.” This short-clip evaluation does not directly assess consistency across clips, such as whether a placed block remains present after the player looks away and returns. Long-Context State-Space Video World Models [2] quantitatively evaluates reconstruction accuracy when revisiting previously observed locations in Memory Maze and analyzes how accuracy changes with the temporal retrieval distance. Although this paper provides qualitative examples of visual consistency, I did not find a comparable systematic evaluation of scene memory or state persistence.
-
Section 4.3 states: “All models are trained with the same number of steps, with the dataset and optimizer held constant.” However, the variants have different computational costs per step, so equal training steps do not correspond to equal wall-clock budgets. The reported FLOPs comparisons support computational savings but do not establish performance advantages under a fixed training-time budget. SCD [3], for example, reports generation quality as a function of actual training time. Given that low-cost training is a central contribution, a similar comparison would strengthen the evidence.
-
Section 3.2 explicitly states that “potentially noisy gradients from the diffusion model are not backpropagated to the causal predictor.” Consequently, the predictor is trained only through the next-frame prediction objective. However, the paper lacks an ablation that isolates this choice by comparing enabled and disabled diffusion-gradient backpropagation while keeping the separable architecture and other settings unchanged.
-
The paper follows an established IDM-based evaluation protocol but does not further examine whether its scores agree with human judgments on the videos generated by this model. MineWorld [4] evaluates the correlation between automatic controllability metrics and human ratings. I did not find a comparable validation here.
-
The experiments use a fixed configuration of 10 world-model sampling steps and 8 diffusion-decoder steps, without analyzing generation quality, action controllability, and throughput under alternative step counts. Since both sampling stages contribute to end-to-end inference cost, varying their step counts separately would clarify the quality–speed trade-off and provide a rationale for the chosen configuration.
[1] Danijar Hafner, Wilson Yan, and Timothy Lillicrap. Training Agents Inside of Scalable World Models. arXiv:2509.24527, 2025. https://arxiv.org/abs/2509.24527
[2] Ryan Po et al. Long-Context State-Space Video World Models. ICCV, 2025. https://arxiv.org/abs/2505.20171
[3] Xingjian Bai et al. Causality in Video Diffusers is Separable from Denoising. arXiv:2602.10095, 2026. https://arxiv.org/abs/2602.10095
[4] Junliang Guo et al. MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft. arXiv:2504.08388, 2025. https://arxiv.org/abs/2504.08388
Questions for authors
Please see the issues raised in the Weaknesses.
5 comments
W5. “The paper follows an established IDM-based evaluation protocol but does not further examine whether its scores agree with human judgments on the videos generated by this model. MineWorld [4] evaluates the correlation between automatic controllability metrics and human ratings. I did not find a comparable validation here.”
We provide additional validation of the IDM evaluation protocol in Appendix F by comparing its outputs to those from a VLM judge. We pass controls as a text prompt and pairs of videos to Qwen3-VL-32B-Instruct, a strong open-source model, and ask it to select the video that follows actions more faithfully while ignoring visual quality. In pairwise comparisons between rollouts sampled from Oasis, Mineworld, and WDDQ, the VLM’s preference agrees with the IDM’s relative ranking for discrete-action F1 approximately 85.3% of the time. Both metrics prefer WDDQ over Oasis in terms of action controllability over 96% of the time, and both prefer WDDQ over MineWorld about 56% of the time. Thus, the IDM demonstrates high agreement with a VLM judge when comparing individual video pairs, as well as in terms of aggregated preference rates when comparing samples from two models.
W6. “The experiments use a fixed configuration of 10 world-model sampling steps and 8 diffusion-decoder steps, without analyzing generation quality, action controllability, and throughput under alternative step counts. Since both sampling stages contribute to end-to-end inference cost, varying their step counts separately would clarify the quality–speed trade-off and provide a rationale for the chosen configuration.”
Our choice of 10 world model sampling steps was picked simply to match Oasis’s sampling configuration. To clarify the quality-speed tradeoff, we sweep the number of sampling steps for both the world model and autoencoder and report FVD, action controllability, and throughput under each configuration evaluated. Due to compute limitations, we sample only 500 rollouts of length 64 for each configuration, and due to hardware availability, FPS values are reported on an H200 (rather than GH200, used by the original experiments). FVD improves monotonically when either WM steps or AE steps increase, all else held constant. Action F1 improves as WM steps increase and appears to remain the same or decrease by a relatively small amount as AE steps increase, potentially as a result of the presence of additional, noisy high-frequency features making it slightly more difficult to accurately determine the actions taken. As expected, FPS decreases as either WM or AE steps increase.
| WM steps | AE steps | FVD@500, T=64 | Action F1 | FPS (H200) |
|---|---|---|---|---|
| 4 | 4 | 65.4 | 0.818 | 27.8 |
| 4 | 8 | 61.6 | 0.817 | 19.2 |
| 4 | 16 | 59.5 | 0.817 | 11.9 |
| 8 | 4 | 51.5 | 0.820 | 21.7 |
| 8 | 8 | 48.8 | 0.818 | 16.0 |
| 8 | 16 | 48.1 | 0.816 | 10.6 |
| 16 | 4 | 47.4 | 0.823 | 15.0 |
| 16 | 8 | 45.6 | 0.821 | 12.1 |
| 16 | 16 | 44.6 | 0.820 | 8.69 |
Dear Reviewer 6JEK,
We have completed the requested ablation, in which gradients from the diffuser are backpropagated to the predictor while keeping the separable architecture and other settings unchanged.
| Configuration | FVD ↓ | Discrete-action F1 ↑ | Mouse L1 ↓ |
|---|---|---|---|
| Diffusion gradients detached | 63.4 | 0.812 | 0.261 |
| Diffusion gradients enabled | 54.2 | 0.807 | 0.272 |
We find that enabling gradients from the diffuser results in improved visual fidelity, reducing FVD, while giving slightly worse action controllability in terms of both discrete action F1 and mouse L1 error. Backpropagating diffuser gradients introduces a new hyperparameter for the weight of the diffusion loss relative to the prediction loss, potentially increasing the difficulty of hyperparameter tuning. We also note that it is no longer possible to train the predictor and diffuser separately for this variant. For simplicity, we recommend the main configuration of WDDQ with gradients disabled, but the optimal choice likely depends on hyperparameter tuning budget and the importance of controllability relative to visual fidelity.
We have incorporated these results into the revised manuscript in Section 4.3, and provided additional clarifications about the trade-offs of this approach in Section 3.2. We hope our responses and additional experiments have addressed your concerns. If so, we would be grateful if you might reconsider your overall assessment in light of our new results and clarifications. Thank you again for your thoughtful feedback, which has helped us improve the paper.
Best, WDDQ Authors
We thank the reviewer for their constructive feedback. We are encouraged that they find our proposed approach to be practical for lowering the barrier to world-model research, while achieving strong performance in terms of generation quality and action controllability and maintaining competitive inference performance.
W1. “Section 3.1 states: ‘One way to prevent additional latency is by resampling the latent each time a new action is received’ [...] However, Section 4.2 does not clearly specify whether the main evaluation receives actions strictly one at a time or how unavailable actions within a chunk are handled. As a reference, Dreamer 4 [1] evaluates closed-loop interaction through human players performing tasks [...] I did not find a comparable evaluation in this paper.”
We have revised the paper to clarify that in all evaluations, when generating frame t, only action inputs up to and including action t−1 are provided to WDDQ (and the input chunk of actions is right-padded with zeros, if a full chunk is not yet available), and the latent is re-sampled for each new action and subsequent frame. As such, our evaluations match the same input structure and observation-action causality of interactive, human-in-the-loop inference. Using fixed starting observations and identical action sequences as input to each model in our evaluations enables more closely controlled comparisons between models and avoids introducing human variability.
W2. “Section 4.2 describes the FVD computation as ‘dividing each sampled rollout into eight chunks of 16 frames each.’ This short-clip evaluation does not directly assess consistency across clips, such as whether a placed block remains present after the player looks away and returns. Long-Context State-Space Video World Models [2] quantitatively evaluates reconstruction accuracy when revisiting previously observed locations [...] I did not find a comparable systematic evaluation of scene memory or state persistence”
We add a quantitative evaluation of long-horizon consistency in Section 4.4 of the paper. To broadly emulate the evaluation in Long-Context State-Space Video World Models for our dataset, we filter for videos with a high proportion of ‘revisit’ frames, with a precise definition and filtering thresholds provided in the paper. For each video, all models receive 32 ground-truth frames as context, and then each model autoregressively generates the following 96 frames conditional on this context. We report average LPIPS of the rollout with respect to the ground-truth continuation, evaluating the model’s ability to accurately reconstruct parts of the scene that are visible in the provided context frames. WDDQ outperforms both Oasis and MineWorld in terms of revisit LPIPS.
W3. “The variants have different computational costs per step, so equal training steps do not correspond to equal wall-clock budgets. The reported FLOPs comparisons support computational savings but do not establish performance advantages under a fixed training-time budget. SCD [3], for example, reports generation quality as a function of actual training time. Given that low-cost training is a central contribution, a similar comparison would strengthen the evidence.”
We observe in our experiments that all models achieve similar MFU on our hardware and training configuration, so FLOP counts are approximately proportional to the elapsed training time. Unlike wall-clock training time, which can vary based on unpredictable factors (like GPU temperature or filesystem pressure) and across software versions, FLOP counts represent a more deterministic and accurate metric for training budget, at least for these experiments, where no method achieves significantly higher tensor core utilization.
W4. “Section 3.2 explicitly states that “potentially noisy gradients from the diffusion model are not backpropagated to the causal predictor.” Consequently, the predictor is trained only through the next-frame prediction objective. However, the paper lacks an ablation that isolates this choice by comparing enabled and disabled diffusion-gradient backpropagation while keeping the separable architecture and other settings unchanged.”
We are currently running this ablation and anticipate that results will be available before the end of the discussion period.