- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work introduces MoTWorld, which is an interactive world model that can jointly predict video frames, voxels and camera poses in a unified MoT architecture. In a typical cascaded pipeline, geometry is only one condition for video generation. However, MoTWorld denoises 2D and 3D in a shared attention, thus enabling the appearance and 3D structure to inform each other. The author introduced SpRoPE, a rotation position encoding based on the camera geometry and estimated depth, to eliminate the coordinate gap between pixels and voxels. In the Minecraft benchmark, this model outperformed baseline methods such as PERSIST and Oasis, especially in visual video display and single-image voxel reconstruction.
Critical strengths
2D-3D joint prediction is good. The shared attention between images, voxels and cameras enable better information flow across modalities. SpatialRoPE is also natural here, solving 2D image tokens-3D voxel tokens correspondence. Independent noise levels also makes the framework flexible, supporting reconstruction, rendering and rollouts tasks, all trained within a unified model architecture. The long-horizon results are also strong and support the claim of explicit 3D state.
Critical weaknesses
-
The current practice is bound to voxel representation and Minecraft. This model uses exact voxel states in the main experiments and is also highly dependent on privileged geometric information, including depth supervision of the simulator and known camera poses. In addition, the fixed voxel resolution limits the expressive power, resulting in a possible lower upper limit of the method and making it difficult to express more refined geometers. Therefore, it is still unclear whether the same design can be well applied to more general 3D representations and noisier geometries in real-world scenarios.
-
The evidence that joint prediction is superior to cascaded prediction is insufficient. In Table 5, the inference configuration of the same model weight is changed to compare the joint/cascade, but the model weights are trained under the joint condition and not trained as a cascade model. Therefore, this experiment shows that the jointly trained model works better in the joint inference mode, but it cannot fully prove that the joint prediction is fundamentally superior to the properly trained cascade.
Questions for authors
-
Why did you choose the voxel representation instead of the point cloud/set-based representation? Can these designs of yours be extended to other 3D representations?
-
For the cascade comparison in Table 5, have you ever attempted to directly train a cascade model with matching architecture/data/computation? In this case, can the advantage of joint prediction still be maintained? I want to know whether joint training and joint inference actually improve the performance of each individual task, or just providing a unified framework across tasks.
-
Why choose the manually history corruption Diffusion Forcing instead of Self Forcing? Since long-horizon inference is based on the history generated by the model itself, self-forcing seems to be more directly consistent with the inference distribution. Have you compared these two training strategies, especially in terms of long-term stability?
If author can solve these problems with convincing experimental evidence, I would be happy to increase my score.
7 comments
[W1, Q1] Privileged information, resolution, and other 3D representations [Part 2/2]
Motivation for semantic voxels. A semantic voxel grid has three advantages as the persistent state of a world model.
- Effective: it represents the whole space explicitly. Every cell holds an explicit state, a solid block with its class or air, including the cells outside the current view. The model can therefore remember that a region is empty as well as what fills it, and keep what it has generated for regions that are out of view. Point-based representations can also hold this information, but they are organised around surfaces and leave free space implicit. A grid states occupied and free space equally explicitly, which we find a more natural and convenient form for this memory. In our rollouts, the carried grid stays accurate over 600 frames and keeps revisited places consistent (Tables RA1 and RA2 in our reply to Reviewer 2Xei).
- Efficient: it is a regular array. Its memory and compute are set by the resolution and do not grow with scene content. Any location is read or written by direct indexing rather than a nearest-neighbour search; neighbourhood, visibility and collision queries are simple array operations, and a local change of the world, such as a block placed or removed, is a write to one cell.
- Directly usable: it is the representation planners already consume. 3D semantic occupancy is a standard prediction target and world-model state in autonomous driving [A, B]. A world model that carries such a grid next to the video therefore also outputs a state that can feed path planning or collision checking directly.
Using voxels is a design choice based on these advantages for this setting and we do not claim that they are better than other 3D representations in general. Point clouds, meshes, and 3D Gaussians describe surfaces compactly and precisely and suit other downstream tasks, such as fine-grained reconstruction or high-fidelity rendering. For this reason we also show below that our design is not tied to voxels.
The design extends to other 3D representations. Neither the MoT nor SpRoPE depends on voxels. SpRoPE needs, for every 3D token, a metric centre and the distribution of its position: each code of Eq. (4) is the expected rotary factor under that distribution, i.e. its characteristic function, with the phase at the centre and an attenuation equal to the characteristic function's magnitude. A voxel is uniform over a cube and an image token is uniform over its depth interval. A token from a point cloud or point map is a point, or a Gaussian if its position is uncertain (σ), with attenuation exp(-ω²σ²/2); a 3D Gaussian with covariance Σ uses σ² = Σ_aa on axis a. A set-based representation therefore plugs into the same encoding by changing only the attenuation formula, and the projected image address applies unchanged to any token with a 3D center.
[W2, Q2] Joint vs a properly trained cascade, and whether joint training helps the individual tasks [Part 1/2]
Thanks for your insightful comment. In Table 5, S0, S3 and S4 run passes which are jointly trained, while S1 and S2 rely on passes never seen in training (PV→V, and PV→P), so they show the zero-shot capability. As suggested, we added two experiments to separate the architecture from a mismatch between training and inference.
(1) A cascade trained for its own inference, at a matched training cost. The zero-shot settings of Table 5 cannot separate the architecture from a training-inference mismatch in S1 and S2, so we trained the cascade. Starting from the Table 1 checkpoint, we finetune on the same data and objective, but replace joint PV→PV generation in the task mixture by the two passes of S1 and S2. The cascade is thus trained for exactly the inference it is evaluated with. We also finetune the joint model from the same checkpoint, at the same training cost for comparison. Also, we rename S0 and S1 as joint with one-pass and cascade, individually, since S1 also benefits from the input of both pixel and voxel modalities and, thus, we differentiate them between one-pass and cascade. For S2, we rename it as voxel-bottleneck, cascade since everything the video carries from one window to the next passes through the voxel grid.
Table RC2: Joint generation vs a trained cascade (MoT-eval, 200 episodes, 600 frames, window 1+7, recorded poses). Joint, one-pass (S0): the pixels and voxels of a window are denoised together in one pass (PV→PV). Joint, cascade (S1): a voxel pass with the future frames hidden (PV→V), then a pixel pass that renders the frames from those voxels and the pixel history (PV→P). Voxel-bottleneck, cascade (S2): the same voxel pass, then pixels rendered from the voxels alone (V→P over the whole 1+7 window).
| Setting | FVD@200 ↓ | FVD@400 ↓ | FVD@600 ↓ |
|---|---|---|---|
| joint, one-pass (S0): Table 1 | 57.9 | 57.5 | 56.3 |
| joint, cascade (S1): Table 1, zero-shot (Table 5) | 72.2 | 72.8 | 71.4 |
| voxel-bottleneck, cascade (S2): Table 1, zero-shot (Table 5) | 120.6 | 117.7 | 115.1 |
| joint, one-pass (S0): trained as a joint | 44.8 | 42.2 | 40.8 |
| joint, cascade (S1): trained as a cascade | 47.3 | 44.4 | 42.5 |
| voxel-bottleneck, cascade (S2): trained as a cascade | 102.3 | 98.0 | 96.0 |
Trained for its own inference, the joint cascade closes most of its gap to one-pass generation, but one-pass generation still stays ahead at every horizon. The much larger difference is between the two cascades: after training, the voxel-bottleneck cascade stays at more than twice the FVD of the joint cascade, and training improves it less. This demonstrates two key findings: (1) Even when the cascade is explicitly trained for its own sequential inference, joint one-pass denoising retains a consistent advantage at every horizon (40.8 vs. 42.5 FVD@600) while requiring only a single forward pass rather than two. (2) Access to multimodal history is critical: the voxel-bottleneck cascade remains more than 2x worse in FVD (96.0 vs. 42.5), confirming that maintaining joint multimodal representations across time is necessary.
We appreciate your thoughtful questions and valuable suggestions!
[W1, Q1] Privileged information, resolution, and other 3D representations [Part 1/2]
The gains do not rest on privileged depth, poses or initial geometry. We separate the three inputs.
Table RC1: Rollouts without the ground-truth initial grid or future poses (MoT-eval, 600 frames, one MoTWorld rollout checkpoint). FVD (↓). P→V: v0 reconstructed from the frame-0 image, by MoTWorld's P→V model or, for PERSIST, by PERSIST itself (its setting without v0). The FVD improvement of the first row compared with Table 1 comes from a longer training detailed in Appendix D.7 in the updated draft.
With neither privileged input (bold row), MoTWorld receives exactly what PERSIST without v0 receives, the first image, its camera and the actions, and still lowers FVD at 600 frames by roughly two thirds. Inferring v0 and predicting the camera cost a similar amount, so neither input carries the result: with both removed, FVD stays at about half that of PERSIST given the true v0.
Resolution. We use a 48^3 grid for two reasons. It matches what the camera observes in this setting: the pixel frames are rendered with “fog” beginning at the grid's half-width (24 blocks from the camera in default), so the grid covers approximately the part of the scene that is rendered without fog. It is also the resolution of the voxel tokenizer of PERSIST, our main baseline, which we reuse for a controlled comparison. This resolution is fixed by the voxel tokenizer, not by the MoT backbone: apart from the input and output projections of the voxel latents, no learned parameter depends on the number or the spacing of the voxel tokens. SpRoPE is defined in metric units, and the cell size enters only through the attenuation. A finer tokenizer therefore keeps the coordinate frame: every physical offset between two tokens keeps the phase it had in training, and only the attenuation changes. An index-based RoPE would instead double every token index and with it the phase of every offset, so its learned positional relations would not carry over without rescaling. A backbone trained with one tokenizer can therefore initialize training with a finer one instead of starting from scratch, and the tokenizer can be trained for the granularity an application needs. The cost of a finer grid is the number of tokens, which we list as a limitation for large real scenes.