- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces 3D World Simulator, which is a multimodal 3D world model that can predict scene geometry, visual appearance, and spatially aligned tactile signals. 3D World Simulator is a 2 stage architecture. The first stage is a hybrid particle-voxel autoencoder that compresses the input point cloud into compact 3D voxel latents. The second stage is an action-conditioned 3D latent dynamics model which runs spatiotemporal attention on the history latents and current latents. The predicted latents can be decoded to point cloud and produce 3DGS to be rendered at any viewpoint. Experiments show that the model is competitive on real-world manipulation tasks while also enabling data generation for policy training & evaluation, as well as model-based planning.
Critical strengths
- The model has a shared representation for multiple modalities, including geometry, visual appearance and spatially aligned tactile signals.
- The policies trained on generated data seem to transfer well to real world tasks requiring tactlile, novel viewpoints, etc.
- The simulated and real world policy success rates show strong correlations across tasks.
- The model does not depend on point tracks, which allows it to be trained on a broader range of dataset.
Critical weaknesses
- Despite the mostly reasnoable design choices, it seems that there were no ablations that fully justify the proposed design and why the architecture is better than existing works such as: https://gaussian-world-model.github.io/.
- Was the data generation experiment generating replays of recorded real-world actions or unseen teleop trajectories during world model training? If the model performs well even for unseen teleop trajectories, then it enables scalable new demonstration collection, otherwise, the experiment primarily demonstrates observation generation along recorded action trajectories, and the ability to collect new demonstrations remains unclear.
- Was the model evaluated on held out objects excluded from the autoencoder and dynamics model training? It was unclear to me what held-out object means precisely mean in Table 2. Please clarify the object-level splits for both training stages and the number of held-out objects used for evaluation.
- The planning experiments seem a bit limited, where the zero shot planning result only shows a few examples with final state chamfer distance. Could the authors report success rate (with clearly defined success criteria) for these experiments as well?
Questions for authors
Please see weaknesses, thanks.