acceptodds
Under review as a conference paper at ICLR 2027

MoTWorld: Geometry-Grounded World Modeling with Joint Video-Voxel Generation

Abstract

Interactive world models predict future observations from actions of an agent, but maintaining spatial consistency over long rollouts remains challenging. Explicit 3D states provide a persistent representation of the scene, yet cascaded approaches typically predict geometry first and then use it to condition video generation, limiting information exchange between the two modalities. We propose MoTWorld, a world model that jointly generates video frames and semantic voxel grids within a Mixture-of-Transformers framework. Specifically, we represent images, voxels, and camera parameters as separate streams with modality-specific weights and shared attention. By changing which streams are observed or generated, we use the same model for image-to-voxel reconstruction, voxel-to-image rendering, and action-conditioned rollout. We further identify that shared attention alone does not explicitly establish correspondence between image and voxel tokens, which occupy different coordinate systems. We propose a novel rotary positional encoding that uses camera geometry and predicted depth to represent both modalities in a common 3D coordinate system. We account for voxel extent and depth uncertainty in the encoding to guide cross-modal attention. Experiments on two Minecraft-style test sets with exact 3D ground truth shows that MoTWorld significantly improves single-image voxel reconstruction and video quality over 600-frame rollouts compared with state-of-the-art methods, while supporting reconstruction, rendering, and generation within a unified architecture.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
world model, geometry-grounded, occupancy, video diffusion model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wKyfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces MoTWorld, which is an interactive world model that can jointly predict video frames, voxels and camera poses in a unified MoT architecture. In a typical cascaded pipeline, geometry is only one condition for video generation. However, MoTWorld denoises 2D and 3D in a shared attention, thus enabling the appearance and 3D structure to inform each other. The author introduced SpRoPE, a rotation position encoding based on the camera geometry and estimated depth, to eliminate the coordinate gap between pixels and voxels. In the Minecraft benchmark, this model outperformed baseline methods such as PERSIST and Oasis, especially in visual video display and single-image voxel reconstruction.

Critical strengths

2D-3D joint prediction is good. The shared attention between images, voxels and cameras enable better information flow across modalities. SpatialRoPE is also natural here, solving 2D image tokens-3D voxel tokens correspondence. Independent noise levels also makes the framework flexible, supporting reconstruction, rendering and rollouts tasks, all trained within a unified model architecture. The long-horizon results are also strong and support the claim of explicit 3D state.

Critical weaknesses
  1. The current practice is bound to voxel representation and Minecraft. This model uses exact voxel states in the main experiments and is also highly dependent on privileged geometric information, including depth supervision of the simulator and known camera poses. In addition, the fixed 48348^3 voxel resolution limits the expressive power, resulting in a possible lower upper limit of the method and making it difficult to express more refined geometers. Therefore, it is still unclear whether the same design can be well applied to more general 3D representations and noisier geometries in real-world scenarios.

  2. The evidence that joint prediction is superior to cascaded prediction is insufficient. In Table 5, the inference configuration of the same model weight is changed to compare the joint/cascade, but the model weights are trained under the joint condition and not trained as a cascade model. Therefore, this experiment shows that the jointly trained model works better in the joint inference mode, but it cannot fully prove that the joint prediction is fundamentally superior to the properly trained cascade.

Questions for authors
  1. Why did you choose the voxel representation instead of the point cloud/set-based representation? Can these designs of yours be extended to other 3D representations?

  2. For the cascade comparison in Table 5, have you ever attempted to directly train a cascade model with matching architecture/data/computation? In this case, can the advantage of joint prediction still be maintained? I want to know whether joint training and joint inference actually improve the performance of each individual task, or just providing a unified framework across tasks.

  3. Why choose the manually history corruption Diffusion Forcing instead of Self Forcing? Since long-horizon inference is based on the history generated by the model itself, self-forcing seems to be more directly consistent with the inference distribution. Have you compared these two training strategies, especially in terms of long-term stability?

If author can solve these problems with convincing experimental evidence, I would be happy to increase my score.

7 comments

We appreciate your thoughtful questions and valuable suggestions!

[W1, Q1] Privileged information, resolution, and other 3D representations [Part 1/2]

The gains do not rest on privileged depth, poses or initial geometry. We separate the three inputs.

  • Depth supervision. R1 of Table 4 has no depth head and never sees simulator depth, yet it already reconstructs far better than PERSIST (Table 2), and the depth head adds almost nothing on top. The reconstruction gain therefore comes from the shared metric frame of SpRoPE, not from privileged depth.
  • Initial grid and camera poses. Table RC1 removes both: v0 is reconstructed from the first image by MoTWorld's P→V model, and every camera after frame 0 is predicted by the camera stream.

Table RC1: Rollouts without the ground-truth initial grid or future poses (MoT-eval, 600 frames, one MoTWorld rollout checkpoint). FVD (↓). P→V: v0 reconstructed from the frame-0 image, by MoTWorld's P→V model or, for PERSIST, by PERSIST itself (its setting without v0). The FVD improvement of the first row compared with Table 1 comes from a longer training detailed in Appendix D.7 in the updated draft.

Methodv0CameraFVD@200 ↓FVD@400 ↓FVD@600 ↓
MoTWorldGTGT39.537.636.1
MoTWorldP→VGT57.359.860.9
MoTWorldGTpredicted55.155.158.3
MoTWorldP→Vpredicted64.262.865.1
PERSISTGTpredicted79.5104.4123.5
PERSISTP→Vpredicted137.1158.4173.6

With neither privileged input (bold row), MoTWorld receives exactly what PERSIST without v0 receives, the first image, its camera and the actions, and still lowers FVD at 600 frames by roughly two thirds. Inferring v0 and predicting the camera cost a similar amount, so neither input carries the result: with both removed, FVD stays at about half that of PERSIST given the true v0.

Resolution. We use a 48^3 grid for two reasons. It matches what the camera observes in this setting: the pixel frames are rendered with “fog” beginning at the grid's half-width (24 blocks from the camera in default), so the grid covers approximately the part of the scene that is rendered without fog. It is also the resolution of the voxel tokenizer of PERSIST, our main baseline, which we reuse for a controlled comparison. This resolution is fixed by the voxel tokenizer, not by the MoT backbone: apart from the input and output projections of the voxel latents, no learned parameter depends on the number or the spacing of the voxel tokens. SpRoPE is defined in metric units, and the cell size enters only through the attenuation. A finer tokenizer therefore keeps the coordinate frame: every physical offset between two tokens keeps the phase it had in training, and only the attenuation changes. An index-based RoPE would instead double every token index and with it the phase of every offset, so its learned positional relations would not carry over without rescaling. A backbone trained with one tokenizer can therefore initialize training with a finer one instead of starting from scratch, and the tokenizer can be trained for the granularity an application needs. The cost of a finer grid is the number of tokens, which we list as a limitation for large real scenes.

[W1, Q1] Privileged information, resolution, and other 3D representations [Part 2/2]

Motivation for semantic voxels. A semantic voxel grid has three advantages as the persistent state of a world model.

  • Effective: it represents the whole space explicitly. Every cell holds an explicit state, a solid block with its class or air, including the cells outside the current view. The model can therefore remember that a region is empty as well as what fills it, and keep what it has generated for regions that are out of view. Point-based representations can also hold this information, but they are organised around surfaces and leave free space implicit. A grid states occupied and free space equally explicitly, which we find a more natural and convenient form for this memory. In our rollouts, the carried grid stays accurate over 600 frames and keeps revisited places consistent (Tables RA1 and RA2 in our reply to Reviewer 2Xei).
  • Efficient: it is a regular array. Its memory and compute are set by the resolution and do not grow with scene content. Any location is read or written by direct indexing rather than a nearest-neighbour search; neighbourhood, visibility and collision queries are simple array operations, and a local change of the world, such as a block placed or removed, is a write to one cell.
  • Directly usable: it is the representation planners already consume. 3D semantic occupancy is a standard prediction target and world-model state in autonomous driving [A, B]. A world model that carries such a grid next to the video therefore also outputs a state that can feed path planning or collision checking directly.

Using voxels is a design choice based on these advantages for this setting and we do not claim that they are better than other 3D representations in general. Point clouds, meshes, and 3D Gaussians describe surfaces compactly and precisely and suit other downstream tasks, such as fine-grained reconstruction or high-fidelity rendering. For this reason we also show below that our design is not tied to voxels.

The design extends to other 3D representations. Neither the MoT nor SpRoPE depends on voxels. SpRoPE needs, for every 3D token, a metric centre and the distribution of its position: each code of Eq. (4) is the expected rotary factor under that distribution, i.e. its characteristic function, with the phase at the centre and an attenuation equal to the characteristic function's magnitude. A voxel is uniform over a cube and an image token is uniform over its depth interval. A token from a point cloud or point map is a point, or a Gaussian if its position is uncertain (σ), with attenuation exp(-ω²σ²/2); a 3D Gaussian with covariance Σ uses σ² = Σ_aa on axis a. A set-based representation therefore plugs into the same encoding by changing only the attenuation formula, and the projected image address applies unchanged to any token with a 3D center.

[W2, Q2] Joint vs a properly trained cascade, and whether joint training helps the individual tasks [Part 1/2]

Thanks for your insightful comment. In Table 5, S0, S3 and S4 run passes which are jointly trained, while S1 and S2 rely on passes never seen in training (PV→V, and PV→P), so they show the zero-shot capability. As suggested, we added two experiments to separate the architecture from a mismatch between training and inference.

(1) A cascade trained for its own inference, at a matched training cost. The zero-shot settings of Table 5 cannot separate the architecture from a training-inference mismatch in S1 and S2, so we trained the cascade. Starting from the Table 1 checkpoint, we finetune on the same data and objective, but replace joint PV→PV generation in the task mixture by the two passes of S1 and S2. The cascade is thus trained for exactly the inference it is evaluated with. We also finetune the joint model from the same checkpoint, at the same training cost for comparison. Also, we rename S0 and S1 as joint with one-pass and cascade, individually, since S1 also benefits from the input of both pixel and voxel modalities and, thus, we differentiate them between one-pass and cascade. For S2, we rename it as voxel-bottleneck, cascade since everything the video carries from one window to the next passes through the voxel grid.

Table RC2: Joint generation vs a trained cascade (MoT-eval, 200 episodes, 600 frames, window 1+7, recorded poses). Joint, one-pass (S0): the pixels and voxels of a window are denoised together in one pass (PV→PV). Joint, cascade (S1): a voxel pass with the future frames hidden (PV→V), then a pixel pass that renders the frames from those voxels and the pixel history (PV→P). Voxel-bottleneck, cascade (S2): the same voxel pass, then pixels rendered from the voxels alone (V→P over the whole 1+7 window).

SettingFVD@200 ↓FVD@400 ↓FVD@600 ↓
joint, one-pass (S0): Table 157.957.556.3
joint, cascade (S1): Table 1, zero-shot (Table 5)72.272.871.4
voxel-bottleneck, cascade (S2): Table 1, zero-shot (Table 5)120.6117.7115.1
joint, one-pass (S0): trained as a joint44.842.240.8
joint, cascade (S1): trained as a cascade47.344.442.5
voxel-bottleneck, cascade (S2): trained as a cascade102.398.096.0

Trained for its own inference, the joint cascade closes most of its gap to one-pass generation, but one-pass generation still stays ahead at every horizon. The much larger difference is between the two cascades: after training, the voxel-bottleneck cascade stays at more than twice the FVD of the joint cascade, and training improves it less. This demonstrates two key findings: (1) Even when the cascade is explicitly trained for its own sequential inference, joint one-pass denoising retains a consistent advantage at every horizon (40.8 vs. 42.5 FVD@600) while requiring only a single forward pass rather than two. (2) Access to multimodal history is critical: the voxel-bottleneck cascade remains more than 2x worse in FVD (96.0 vs. 42.5), confirming that maintaining joint multimodal representations across time is necessary.

Reviewer_DksfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposed MoTWorld which leverages Mixture-of-Transformer architecture to model voxel, video and camera in a unified manner. The proposed method supports various modality transform tasks, and can be used in various application scenarios. The authors uses SpRoPE, a novel positional embedding to guide cross-modal attention, which is the main contribution of the paper.

Critical strengths
  1. The research topic is well motivated.
  2. The proposed solving method, including Mixture of Transformer architecture and SpRoPE positional embedding, models different modality jointly. This makes a significant evolution compared to PERSIST on generation consistency and accuracy.
  3. The evaluation results and the ablation studies show the effectiveness of the method.
Critical weaknesses
  1. The model is bidirectional and roll out is performed using sliding window mechanism. Compared to causal model using KV cache, this manner requires more computational resources.
  2. All the experiments are done on Minecraft which is naturally structured. Implementing this method into more complex games / simulators or real-world scenes still requires validation.
  3. Compared to PERSIST, the consistency of MoTWorld is much better, however, the visual quality seems to drop a little, creating some blurry effects.
  4. The main task PV->PV requires the initial voxel grid, which can be hard to obtain, especially in more complex scenarios.
  5. The SpRoPE relys on the true camera information, making MoTWorld a camera controlled world model rather than an action-controlled one. This resist MoTWorld from generalizing to in-the-wild cases with no ground-truth future camera information, causing clipping issues.
Questions for authors

Please refer to the Weaknesses part.

5 comments

We thank the reviewer for these valuable questions and constructive feedback!

[W1] Computational cost: the windowed rollout is several times cheaper than the KV-cached cascade. Its context can be cached exactly, and a trained causal variant with a KV cache is as fast, with slightly higher FVD but slightly more accurate voxels.

We thank the reviewer for raising efficiency. As shown in Figure 2, our model is only bidirectional across generated frames and observed frames never attend to the frames being generated. An observed token attends only to itself and never to the target frames being generated, so its keys and values do not depend on the generated tokens or on the denoising step. This is exactly the property a KV cache relies on, so the context can be computed once per window and reused for all its denoising steps without approximation, which means rollouts with and without this cache are bit-identical. As suggested, we also finetune a causal version using the same dataset and roll it out with a KV cache. Tables RB1 and RB2 report its throughput and generation quality.

Table RB1: Rollout cost on one H200 (batch 1, 360x640, bf16; 16 sampler steps for MoTWorld). FPS counts generated frames. Ours: our submitted rollout, the context recomputed at every step (as in Figure S2 of the draft); causal + KV cache: the vanilla causal variant, with the keys and values of the history frames computed once per window; Ours + KV cache: ours with the keys and values of the history frames computed once per window.

MethodFPS ↑ OursFPS ↑ Causal + KV CacheFPS ↑ Ours + KV Cache
PERSIST-0.46-
MoTWorld PV→PV 1+32.092.272.42
MoTWorld PV→PV 4+42.112.542.71
MoTWorld PV→PV 1+73.083.703.49

Table RB2: Quality of the causal variant, 600-frame rollouts with the protocol of Table 1 (GT camera, first-frame anchor, 1+7 window: 1 history frame and 7 generated frames per pass). The causal model is finetuned from MoTWorld-ours. The FVD improvement of the first row compared with Table 1 comes from a longer training detailed in Appendix D.7 in the updated draft.

ModelPERSIST-eval FVD 200 / 400 / 600 ↓MoT-eval FVD 200 / 400 / 600 ↓Class Acc@600 PERSIST-eval / MoT-eval ↑
MoTWorld-ours31.8 / 30.5 / 29.739.5 / 37.6 / 36.189.7 / 82.1
MoTWorld-causal36.5 / 35.8 / 34.244.1 / 44.4 / 43.391.0 / 84.0

Regarding throughput, the KV cache consistently accelerates inference across all window sizes, with our model and the causal variant achieving comparable FPS. In terms of generation quality, our model outperforms the causal variant by 4.5 and 7.2 FVD at 600 frames on PERSIST-eval and MoT-eval, respectively, although the causal variant maintains slightly higher voxel accuracy. MoTWorld therefore supports both attention patterns, and we retain the bidirectional window as default for its superior visual fidelity.

[W2] Scope: Minecraft-style worlds give exact 3D ground truth, and the model holds up under shifted data and noisy inputs

We agree that extending multimodal world models to photorealistic simulators and real-world perception is an important direction, as we state in Section 5. We chose the Minecraft-style simulator deliberately because the 2D and 3D ground truth enables rigorous verification of geometric persistence without confounding errors from approximate multi-view depth estimators. Also, the Minecraft-style simulator serves as the standard benchmark established by contemporary baselines (PERSIST, Oasis), ensuring strictly fair and reproducible comparisons.

Within this benchmark, we explicitly evaluate the generalization challenges encountered in real deployments:

  1. Zero-Shot Domain Shift: MoTWorld was trained exclusively on our own dataset, yet it outperforms PERSIST on PERSIST's own evaluation benchmark (PERSIST-eval), which was generated using its own environment setting and action policy.
  2. Autonomous Unprivileged Rollouts: In realistic deployments, privileged initial grids and future camera poses are absent. When tested under inferred initial voxels (P → V) and predicted camera paths, MoTWorld preserves its lead over competitive baselines by roughly 60% in FVD (Table RB5 under the reply of [W4, W5] below).
  3. Robustness to Visual Degradation: To evaluate resilience to sensor corruption, Table RB3 reports 600-frame rollouts where Gaussian noise ({0.1,0.2}) corrupts the conditioning initial frame.

Table RB3: PV→PV rollouts of MoTWorld with noisy inputs, 600 frames, protocol of Table 1 (window 1+7, first-frame anchor, GT camera given for every frame). Gaussian noise of std σ is added on the first frame, which is also the anchor of every window. The FVD improvement of the first row compared with Table 1 comes from a longer training detailed in Appendix D.7 in the updated draft.

Input Image NoisePERSIST-eval FVD 200 / 400 / 600 ↓MoT-eval FVD 200 / 400 / 600 ↓Class Acc@600 PERSIST-eval / MoT-eval ↑
No noise31.8 / 30.5 / 29.739.5 / 37.6 / 36.189.7 / 82.1
σ = 0.134.6 / 33.4 / 31.841.4 / 39.2 / 37.089.8 / 82.1
σ = 0.239.8 / 34.3 / 32.450.5 / 41.5 / 37.889.7 / 82.0
Reference (PERSIST + v0, no noise)95.6 / 133.1 / 148.879.5 / 104.4 / 123.561.5 / 53.6

The results show noise does not accumulate over the rollout. With a noisy first image the FVD gap to the clean rollout shrinks from frame 200 to frame 600. Two parts of the design contribute.

  1. Diffusion forcing with history corruption. In training, the frames of a window are denoised at independent noise levels, and in most windows the history frames the model conditions on are corrupted (noise, blur, contrast changes, ghosting and small shifts on the images; noise on the voxel latents) while still labelled clean. The model therefore learns to use its context as evidence rather than to copy it. The test noises differ from the training corruptions (since we keep the anchor frame clean during training), so this is generalisation rather than a matched augmentation.
  2. The voxel stream carries the scene. In the MoT framework, the knowledge of voxel can help the rollout be robust on a degraded image instead of being propagated.

Finally, the architectural foundation of MoTWorld is not limited to Minecraft. Dense 3D semantic occupancy grids aligned with calibrated multi-camera streams and vehicle trajectories are standard representations in real-world autonomous driving benchmarks (e.g., Occ3D on nuScenes [A]). Recent driving scene generators and world models already provide such occupancy representations with video (e.g., UniScene [B], GenieDrive [C]). Adapting MoTWorld to these domains are targeted for our future extensions.

[A] Tian, Xiaoyu, et al. Occ3d: A large-scale 3d occupancy prediction benchmark for autonomous driving. NeurIPS 2023.
[B] Li, Bohan, et al. Uniscene: Unified occupancy-centric driving scene generation. CVPR 2025.
[C]Yang, Zhenya, et al. Geniedrive: Towards physics-aware driving world model with 4d occupancy guided video generation. CVPR 2026.

[W3] Visual quality: continued training further improves visual quality without architecture changing.

We thank the reviewer for the close look. The blur came largely from under-training: continuing to train the same model, with no architectural change, greatly reduces it. Table RB4 measures this per segment of the same rollouts as Table 1; both models decode with the same frozen pixel autoencoder.

Table RB4: Image quality over the 600-frame rollouts on MoT-eval set. Sharpness = Laplacian variance relative to the ground-truth frame (1.0 = as sharp as GT). Blurred frames = share of frames whose Laplacian variance is below 0.4 of the ground-truth frame's. FID = mean per-frame FID over the segment.

MethodFramesSharpness ↑Blurred Frames ↓LPIPS ↓SSIM ↑PSNR ↑FID ↓
PERSIST-XL + v00-990.606.8%0.4000.62717.6085.8
MoTWorld + v0, submitted0-990.594.0%0.2560.70321.8279.3
MoTWorld + v0, continued training0-990.602.8%0.2210.73222.6971.1
PERSIST-XL + v0100-2990.5812.5%0.5290.57614.72101.9
MoTWorld + v0, submitted100-2990.586.1%0.3060.67920.4987.7
MoTWorld + v0, continued training100-2990.594.2%0.2670.70921.2879.2
PERSIST-XL + v0300-5990.5716.3%0.5850.54713.48108.7
MoTWorld + v0, submitted300-5990.587.8%0.3290.67319.7490.1
MoTWorld + v0, continued training300-5990.595.9%0.2980.69820.3483.4

Continued training. Blurred frames become fewer, and every reference metric improves, as does FVD at 600 frames, on both evaluation sets. Figure S4 in the updated draft shows frames across the rollout for the ground truth, PERSIST-XL + v0, MoTWorld + v0, and MoTWorld + v0 after continued training. Compared with PERSIST, the new version of MoTWorld is more competitive in visual quality and more consistent.

[W4, W5] Neither the GT initial grid nor the GT cameras are required

The initial grid can be inferred by the same model. MoTWorld's P→V task reconstructs the voxel world from a single image far better than PERSIST (Table 2), and even with no voxel state at all the same checkpoint rolls out with a much lower FVD than PERSIST without v0 (Table 1). As suggested, Table RB5 now chains the two: v0 is reconstructed from the first image and the rollout continues from it, with recorded or with predicted cameras.

Table RB5: Rollouts without privileged inputs, MoT-eval, 600 frames. P→V = v0 reconstructed from the frame-0 image, by the Table 2 P→V model for MoTWorld or, for PERSIST, by PERSIST itself (its setting without v0); predicted cameras come from the camera stream (per window the camera cell generates the new poses, then the frames are rendered given them). FVD (↓).

Methodv0CameraFVD@200 ↓FVD@400 ↓FVD@600 ↓
MoTWorldGTGT39.537.636.1
MoTWorldP→VGT57.359.860.9
MoTWorldGTpredicted55.155.158.3
MoTWorldP→Vpredicted64.262.865.1
PERSISTGTpredicted79.5104.4123.5
PERSISTP→Vpredicted137.1158.4173.6

With the same inputs as PERSIST without v0 (the first image and camera, and the actions; bold row vs the last row), MoTWorld lowers FVD at 600 frames by more than half. Inferring v0 and predicting the camera cost a similar amount of FVD, and with both removed MoTWorld still has about half the FVD of PERSIST given the true v0.

Also, the camera is not required to be GT. MoTWorld can be action-controlled: conditioning on the current pixel, voxel, and pose, player actions drive the camera stream to predict subsequent poses (Appendix C, Eq. S7) to avoid clipping, which then parameterizes SpRoPE for the visual and voxel towers. With predicted cameras (rows 3 and 4 of Table RB5), video quality drops slightly compared with using GT poses, but remains well ahead of PERSIST with its dedicated camera model, and, with neither privileged input, well ahead of PERSIST with or without v0.

Reviewer_2XeiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MoTWorld jointly generates video and semantic voxel grids using a flow-matching Mixture-of-Transformers with modality-specific weights and shared attention. One model supports image-to-voxel reconstruction, voxel-to-image rendering, and several rollout configurations. Its main technical contribution, SpRoPE, aligns image and voxel tokens through shared spatial coordinates, predicted depth intervals, and voxel projections.

Experiments on two Minecraft-style evaluation sets show substantial reconstruction and video-quality gains. In-frustum reconstruction IoU improves from 69.5% to 88.2% on PERSIST-eval, and 600-frame FVD decreases from 148.8 to 46.8. However, the main joint-rollout results use ground-truth future camera poses and initial voxels; the predicted-camera extension reports pose errors separately.

Critical strengths
  • A coherent framework for exchanging information between images and explicit geometry. The same model performs reconstruction, rendering, and joint prediction, while SpRoPE supplies correspondence information across otherwise incompatible token coordinates. This is a meaningful architectural contribution beyond simply concatenating modalities.

  • Substantial improvements on complementary evaluation tasks. Reconstruction gains appear on both evaluation sets and across all reported geometry metrics. The video results also show large improvements across rollout horizons. Together, these provide stronger evidence of usefulness than video-quality improvements alone.

Critical weaknesses
  • The evaluation does not directly establish long-horizon geometric consistency under action-driven rollout. FVD is computed over non-overlapping 16-frame clips, so it measures local video statistics across the rollout rather than persistence across distant viewpoints. Quantitative voxel accuracy is reported only for initial reconstruction. Furthermore, the strongest joint-rollout results assume true future poses and initial geometry, while the predicted-camera extension lacks corresponding video and voxel-quality results. These gaps limit the principal world-modeling claim.

  • The ablations do not adequately isolate the proposed contributions. Table 4 starts with a configuration that already uses geometric encoding, leaving no direct comparison against the same model without SpRoPE. Table 5 compares joint denoising against cascade configurations explicitly described as untrained inference settings. Their poorer performance could therefore reflect a training–inference mismatch. These experiments demonstrate useful behavior of the checkpoint, but do not establish the claimed architectural advantage over a comparably trained cascade.

Questions for authors
  • Do PERSIST and Oasis receive the same ground-truth future camera information as MoTWorld in Table 1? The released PERSIST pipeline normally predicts its own camera states. Please specify the conditioning provided to each baseline.

  • What are the video-quality and rollout voxel-accuracy results when MoTWorld reconstructs its initial voxels from the input image and predicts subsequent cameras? Quantitative geometry accuracy over time would directly address the persistence claim.

  • How does performance change when the same architecture is trained without SpRoPE, and when the cascade configurations are explicitly trained under comparable data and compute budgets?

  • Equation (4) is presented as the exact expected rotary encoding over a uniform depth interval, but Equation (5) uses �0��0� as its center rather than the interval’s arithmetic midpoint. Appendix A acknowledges this distinction. Is the implemented encoding an approximation, and how does this affect the stated derivation?

Suggestion: Consider discussing and citing Lift-Splat-Shoot as a related depth-aware lifting approach, clarifying how SpRoPE uses depth uncertainty in positional encoding rather than explicit feature lifting and aggregation. This would help contextualize the contribution.

6 comments

We thank the reviewer for the helpful questions and valuable suggestions!

[W1, Q2] Evaluation on Long-horizon geometric consistency: the carried voxel state, revisits, and the fully free-running setting

FVD is computed on 16-frame clips, but each later clip is generated from the model's own earlier output, so any drift of the world state carries forward and accumulates as the agent moves to distant viewpoints, which reflects on FVD. Considering FVD also takes the visual quality into account and measures persistence indirectly, we follow the reviewer's suggestion and conduct three further evaluations w/ or w/o FVD below to assess long-horizon geometric consistency.

(1) Voxel accuracy over the whole rollout. Our evaluation records the carried grid at every frame, so we now score it against the simulator grid along the full rollout.

Table RA1: Accuracy (%) of the solid voxel state during the 600-frame rollouts. Bold best among models. Both models use the GT camera.

Evaluation DatasetMethodIoU @200 ↑IoU@400 ↑IoU@600 ↑Class Acc @200 ↑Class Acc @400 ↑Class Acc @600 ↑
MoT-evalPERSIST + v089.084.380.964.358.054.9
MoT-evalMoTWorld + v097.796.595.689.985.382.1
PERSIST-evalPERSIST + v091.989.988.173.267.864.7
PERSIST-evalMoTWorld + v098.798.197.693.891.389.7

The voxels are more accurate than PERSIST's throughout: IoU and class accuracy are clearly higher across 200, 400, and 600 frames, indicating better geometric consistency than PERSIST.

(2) Revisit consistency. When an agent moves away and back, the image and voxel are expected to be consistent with the last visit. The image and voxel capturing the revisited scene can be used to evaluate the revisit consistency. We consider a frame as a revisit when the scene was seen earlier but was out of view for at least the previous 50 frames. Both models use the GT camera pose. Please refer to Appendix D.8 in the updated draft for more details.

Table RA2: Revisits scored per pixel at the GT pose (600-frame rollouts). Both models use the GT pose as input. PSNR / SSIM / LPIPS are computed on the revisit pixels, between the generated frame and the GT at that pose. Voxel drift: share of the revisited cells whose class in the model's voxel memory changed since the cell was last seen.

Evaluation datasetMethodPSNR ↑ / SSIM ↑ / LPIPS ↓Voxel Drift (%) ↓
MoT-evalPERSIST + v015.89 / 0.365 / 0.71343.9
MoT-evalMoTWorld + v017.08 / 0.376 / 0.47114.6
PERSIST-evalPERSIST + v015.66 / 0.367 / 0.63333.6
PERSIST-evalMoTWorld + v017.06 / 0.384 / 0.4317.6

Our method outperforms PERSIST. Particularly, while a place is out of view, PERSIST largely changes the voxel class, whereas MoTWorld drifts less. The results show that our model has better long-horizon geometric consistency than the baseline.

(3) Fully free-running rollouts: no GT pose and no GT initial grid. MoTWorld reconstructs v0 from the first image with its own P→V task, and its camera stream predicts the pose of every new frame. It therefore receives exactly the same inputs as PERSIST without v0 and GT camera pose.

Table RA3: Comparison of rollout results. The FVD improvement of the first row compared with Table 1 comes from a longer training detailed in Appendix D.7 in the updated draft.

Evaluation datasetModelv0CameraFVD@200 ↓FVD@400 ↓FVD@600 ↓Rot. Err.@600 (°) ↓Transl. Err.@600 (blk) ↓
PERSIST-evalMoTWorldGTGT31.830.529.700
PERSIST-evalMoTWorldP→Vpredicted60.457.559.26.55.44
PERSIST-evalPERSISTP→Vpredicted176.2194.0199.148.36.47
MoT-evalMoTWorldGTGT39.537.636.100
MoT-evalMoTWorldP→Vpredicted64.262.865.13.39.71
MoT-evalPERSISTP→Vpredicted137.1158.4173.627.811.60

Fully free-running MoTWorld lowers FVD at 600 frames by roughly two thirds relative to PERSIST without v0 on both evaluation datasets, and camera pose is competitive, which further validates the effectiveness of our camera tower and P→V reconstruction performance. To be noted, the class accuracy of voxels is dominated by the imagined voxels which are unseen in the first image during the P→V reconstruction stage. Instead of comparing the imagined unseen voxels with GT voxels which introduce randomness, we report “with v0” consistency and revisit results above to validate the geometric consistency. And we are also open to any additional questions on fully free-running rollout results or others.

[W2, Q3] Isolating SpRoPE and joint denoising: a model without SpRoPE and an explicitly trained cascade

Thanks for your insightful feedback. As suggested, we ran the two additional experiments.

(1) The same model without SpRoPE. We report the model performance of plain joint self-attention without SpRoPE as R0 in Table RA4. Each modality keeps a standard RoPE.

Table RA4: Revised Table 4 in the draft.

SpRoPEDepth HeadProjected (h,w)GramIoU in ↑IoU out ↑Cls Seen ↑Cls Unseen ↑
R077.782.480.765.4
R1✓81.983.783.066.9
R2✓✓82.183.683.166.1
R3✓✓✓82.984.785.067.7
R4✓✓✓✓83.884.585.268.0

The biggest improvement comes from R0 to R1. Its voxels are also less specific to the input, which means the plain model uses the image, but without a shared geometric code, it generates the voxel by referring to image content but does not place image content in the geometrically aligned cells. The qualitative comparison can be observed in Figure S3 in the updated draft, where we can find R0 has clear misalignment problems while the methods with SpRoPE clearly improve the geometric alignment.

(2) An explicitly trained cascade. In Table 5, S0, S3 and S4 run passes which are jointly trained, while S1 and S2 rely on passes never seen in training (PV→V, and PV→P), so they show the zero-shot capability. As suggested, we trained the cascade to separate the architecture from a mismatch between training and inference. Starting from the Table 1 checkpoint, we finetune on the same data and setting for one day, but replace joint PV→PV generation in the task mixture by the two passes of S1 and S2. The cascade is thus trained for exactly the inference it is evaluated with. As a control, we also finetune the joint model from the same checkpoint, at the same training cost for comparison. We also rename the settings to make their difference clearer. We call S0 and S1 joint one-pass and joint cascade, respectively: both condition on pixels and voxels, and they differ only in whether a window is generated in a single pass or in two (voxels first, then pixels). We call S2 the voxel-bottleneck cascade, since its pixel pass sees only the voxels, so everything the video carries from one window to the next must pass through the voxel grid.

Table RA5: Joint generation vs a trained cascade (MoT-eval, 200 episodes, 600 frames, window 1+7, recorded poses). Joint, one-pass (S0): the pixels and voxels of a window are denoised together in one pass (PV→PV). Joint, cascade (S1): a voxel pass with the future frames hidden (PV→V), then a pixel pass that renders the frames from those voxels and the pixel history (PV→P). Voxel-bottleneck, cascade (S2): the same voxel pass, then pixels rendered from the voxels alone (V→P over the whole 1+7 window).

SettingFVD@200 ↓FVD@400 ↓FVD@600 ↓
joint, one-pass (S0): (Table 5)57.957.556.3
joint, cascade (S1): zero-shot (Table 5)72.272.871.4
voxel-bottleneck, cascade (S2): zero-shot (Table 5)120.6117.7115.1
joint, one-pass (S0): trained as a joint44.842.240.8
joint, cascade (S1): trained as a cascade47.344.442.5
voxel-bottleneck, cascade (S2): trained as a cascade102.398.096.0

Trained for its own inference, the joint cascade closes most of its gap to one-pass generation, but one-pass generation still stays ahead at every horizon. The much larger difference is between the two cascades: after training, the voxel-bottleneck cascade stays at more than twice the FVD of the joint cascade, and training improves it less. That shows the benefit of joint modelling therefore comes mostly from getting access to both modalities, and relatively less from doing so in one pass; the video gap of the joint cascade in Table 5 is mostly a training-inference mismatch, and we have updated this in Section 4.4.

[Q1] Baseline conditioning: PERSIST is reported in its best camera configuration in Table 1 and Oasis only supports action as input.

In Table 1, PERSIST predicts its own camera after frame 0, as in its released pipeline. We chose this because PERSIST is better with its own camera than with the GT one as mentioned in the PERSIST paper, so Table 1 reports PERSIST's stronger configuration. Table RA6 lists the inputs of every row and adds PERSIST with the GT pose on every frame, i.e. exactly MoTWorld's inputs.

Table RA6: Inputs given to each method and FVD (↓) at 200/400/600 frames. p0, v0: image, voxel grid at frame 0. All rollouts are driven by the recorded actions. The PERSIST GT-camera row is new.

MethodGiven at startCameraPERSIST-eval FVD 200 / 400 / 600 ↓MoT-eval FVD 200 / 400 / 600 ↓
Oasisp0none519.6 / 642.3 / 788.9509.5 / 706.3 / 859.2
PERSIST w/ v0 (Table 1)p0, v0predicted by PERSIST95.6 / 133.1 / 148.879.5 / 104.4 / 123.5
PERSIST w/ v0, GT camerap0, v0GT pose106.0 / 136.4 / 158.2128.7 / 187.7 / 226.3
MoTWorld w/ v0 (Table 1)p0, v0GT pose49.6 / 48.1 / 46.857.9 / 57.5 / 56.3

With identical inputs the gap widens: MoTWorld's FVD at 600 frames is obviously lower than PERSIST’s, a larger margin than in Table 1. Providing ground-truth poses slightly degrades PERSIST's FVD compared to using its own predicted camera trajectories, consistent with the Camera-GT ablation reported in the PERSIST paper. Thus, reporting PERSIST with its own camera predictor in Table 1 reflects its stronger configuration. Oasis receives only the first image and the actions: no camera pose and no voxel input. Whether cameras are predicted or provided from ground truth, MoTWorld outperforms PERSIST by over 50% in long-horizon FVD. Performance under GT-pose-free or v0-free rollouts are already detailed in part (3) of the [W1, Q2] reply above.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Rebuttal for AC [1/2]

Dear AC,

We provide below a summary including contributions, assessments, and how we addressed concerns in the rebuttal and the revised draft.


Contributions

Joint video-voxel-camera world model. MoTWorld represents images, semantic voxels and camera poses as separate streams in a Mixture-of-Transformers, with modality-specific weights and shared attention. By choosing which streams are observed and which are generated, one model performs single-image voxel reconstruction (P→V), voxel-to-image rendering (V→P) and action-conditioned rollouts that generate video and voxels together.

SpRoPE: one metric frame for image and voxel tokens. Shared attention alone does not tell an image token which voxels it sees. SpRoPE places both modalities in a common 3D coordinate system and clearly improves the alignment between 2D and 3D features.

Experimental results On two Minecraft-style evaluation sets with exact 3D ground truth, MoTWorld improves single-image voxel reconstruction and 600-frame video quality over PERSIST and Oasis. The rebuttal adds that the carried voxel state stays accurate over the whole rollout and that MoTWorld keeps a large lead when it receives no privileged input at all.


Positive Assessments

All three reviewers recognized the architecture, SpRoPE and the strength of the results. They described MoTWorld as "a coherent framework for exchanging information between images and explicit geometry" and "a meaningful architectural contribution beyond simply concatenating modalities" (2Xei), "well motivated" and "a significant evolution compared to PERSIST on generation consistency and accuracy" (Dksf), and found SpRoPE "natural here, solving 2D image tokens-3D voxel tokens correspondence" (wKyf). On the experiments, they noted "substantial improvements on complementary evaluation tasks" (2Xei), that "the evaluation results and the ablation studies show the effectiveness of the method" (Dksf), and that "the long-horizon results are also strong and support the claim of explicit 3D state" (wKyf).

After the discussion, all three reviewers rate the paper as weak acceptance (3):

  • 2Xei: 2 → 3 (raised after the rebuttal)
  • Dksf: 3 (kept)
  • wKyf: 3 (kept)

Addressing reviewer concerns

[Reviewer 2Xei, score: 2 → 3]

Concerns raised:

  • Long-horizon geometric consistency is not measured directly: FVD is local, voxel accuracy is reported only at frame 0, and the main rollouts use ground-truth future poses and initial voxels.

  • Table 4 has no model without SpRoPE, and the cascades of Table 5 are untrained inference settings.

  • What conditioning does each baseline receive?

  • Eq. (5) centres the phase at approximation point rather than at the interval midpoint of Eq. (4).

  • Suggested discussing Lift-Splat-Shoot.

How we addressed them:

  • Measured long-horizon consistency directly. The carried voxel grid stays far more accurate than PERSIST's over the whole rollout (Table RA1), and revisited places change much less in MoTWorld's voxel memory (Table RA2, Appendix D.8). In fully free-running rollouts, with the initial grid reconstructed from the first image and every camera predicted, MoTWorld keeps a large lead over PERSIST given the same inputs (Table RA3).

  • Trained the plain self-attention model without SpRoPE (new row R0 of the revised Table 4). It falls below every SpRoPE rung, and its voxels are visibly misaligned with the image (Figure S3). Trained the cascade for its own inference at matched cost: it closes most of the zero-shot gap but does not overtake joint one-pass generation, and the voxel-bottleneck cascade stays far behind (Table RA5, Table 5).

  • Listed the inputs of every baseline (Table RA6). Table 1 reports PERSIST in its stronger configuration, with its own camera predictor; given exactly MoTWorld's inputs it does worse, so the gap widens. Oasis receives only the first image and the actions.

  • Confirmed that Eq. (5) is a deliberate approximation with a bounded error. Swapping in the exact midpoint at inference leaves reconstruction and rollout geometry essentially unchanged (Tables RA7, RA8).

  • Discussed Lift-Splat-Shoot: it lifts features along each ray in one direction, while SpRoPE moves no features and lets attention run in both directions, which joint generation needs.

AuthorsOpenReview ↗

Summary of Rebuttal for AC [2/2]

[Reviewer Dksf, score: 3]

Concerns raised:

  • The bidirectional sliding window costs more than a causal model with a KV cache.

  • All experiments are in Minecraft; complex and real scenes still need validation.

  • Visual quality is slightly blurrier than PERSIST.

  • PV→PV requires the initial voxel grid, and SpRoPE relies on true cameras, so the model is camera-controlled rather than action-controlled.

How we addressed them:

  • Showed that the history of each window can be KV-cached exactly, and fine-tuned a causal variant with a KV cache. Both run several times faster than PERSIST; the bidirectional window keeps the better video quality, the causal variant slightly more accurate voxels, so MoTWorld supports both (Tables RB1, RB2).

  • Explained the choice of Minecraft-style worlds (exact 2D and 3D ground truth; the benchmark of PERSIST and Oasis) and showed robustness within it: zero-shot transfer to PERSIST's own benchmark, the lead without privileged inputs, and rollouts from a noisy first frame in which the noise does not accumulate (Table RB3). Pointed to driving occupancy benchmarks of real scenes for future works.

  • Continuing to train the same architecture leaves fewer blurred frames and improves every image metric and FVD (with larger advantages against the baseline) (Table RB4, Figure S4, Appendix D.7).

  • Showed that neither input is required: the initial grid is reconstructed by P→V, and the camera stream predicts the poses from the actions (Appendix C). With both inferred, MoTWorld stays well ahead of PERSIST without the initial grid, and even ahead of PERSIST given the true one (Table RB5).


[Reviewer wKyf, score: 3]

Concerns raised:

  • The method is bound to voxels and Minecraft, depends on privileged depth supervision, known poses and exact voxels, and the fixed 48 x 48 x 48 resolution may limit it.

  • Table 5 does not show that joint prediction beats a properly trained cascade; does joint training help each task or only unify them?

  • Why Diffusion Forcing with history corruption rather than Self Forcing?

How we addressed them:

  • Separated the privileged inputs. Without depth supervision, ours already reconstructs better than PERSIST, so the gain comes from SpRoPE's shared metric frame rather than from privileged depth; the initial grid and the cameras can both be inferred (Table RC1). Explained the 48 x 48 x 48 grid setting, motivated semantic voxels, and showed the potential of SpRoPE to extend to point and Gaussian tokens.

  • Trained the cascade for its own inference at matched cost (as for 2Xei, Table RC2), and trained single-task models of the same architecture at the compute the joint model spends on each task. The joint model wins on every task (Tables RC3 to RC5), so joint training improves the individual tasks through positive transfer, not only unifies them.

  • Compared Self Forcing at equal compute. Diffusion Forcing with history corruption gives better rollouts on both evaluation sets as in prior works (Table RC6).


Summary and final remarks

The three reviewers agree on the value of jointly generating video and explicit geometry and on SpRoPE as the mechanism that aligns them, and all three now rate the paper as weak acceptance. For every concern, we answered one by one with a new experiment or a clarification.

The points that remain after the discussion are that training uses 3D ground truth (2Xei) and that real-world environments are still to be shown (Dksf). We agree these are the future works as discussed in Section 5 for the extension to real-world scenes.

We hope this summary provides a useful reference for the AC. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.