acceptodds
Under review as a conference paper at ICLR 2027

Hybrid Particle-Voxel Dynamics for Multimodal 3D World Simulation

11 views

Abstract

Predicting how the world evolves in response to robot actions is central to policy learning, evaluation, and planning. However, many action-conditioned world models predict future observations in 2D image or video space without explicitly modeling the underlying 3D scene, making spatial consistency across viewpoints and over long-horizon interactions challenging. We introduce **3D World Simulator**, an action-conditioned multimodal world modeling framework that jointly predicts appearance, geometry, and tactile observations from a unified 3D representation. Our approach combines a hybrid particle-voxel autoencoder that compresses point clouds into compact 3D voxel latents with an action-conditioned latent dynamics model that predicts their evolution. The shared 3D representation supports geometrically consistent rendering from arbitrary viewpoints and flexible integration of spatially aligned tactile signals. Our experiments demonstrate interactive, long-horizon, multimodal prediction at 25 Hz. We then show how our model enables scalable data generation for policy training and evaluation using tactile and multi-view visual observations, including wrist-mounted and novel camera views. Furthermore, we scale our method to large datasets containing diverse objects and demonstrate model-based planning with unseen objects and robot configurations. Together, these results highlight the potential of hybrid particle-voxel dynamics as a foundation for multimodal 3D world simulation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot world model, action-conditioned 3D video prediction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5CZzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces 3D World Simulator, which is a multimodal 3D world model that can predict scene geometry, visual appearance, and spatially aligned tactile signals. 3D World Simulator is a 2 stage architecture. The first stage is a hybrid particle-voxel autoencoder that compresses the input point cloud into compact 3D voxel latents. The second stage is an action-conditioned 3D latent dynamics model which runs spatiotemporal attention on the history latents and current latents. The predicted latents can be decoded to point cloud and produce 3DGS to be rendered at any viewpoint. Experiments show that the model is competitive on real-world manipulation tasks while also enabling data generation for policy training & evaluation, as well as model-based planning.

Critical strengths
  1. The model has a shared representation for multiple modalities, including geometry, visual appearance and spatially aligned tactile signals.
  2. The policies trained on generated data seem to transfer well to real world tasks requiring tactlile, novel viewpoints, etc.
  3. The simulated and real world policy success rates show strong correlations across tasks.
  4. The model does not depend on point tracks, which allows it to be trained on a broader range of dataset.
Critical weaknesses
  1. Despite the mostly reasnoable design choices, it seems that there were no ablations that fully justify the proposed design and why the architecture is better than existing works such as: https://gaussian-world-model.github.io/.
  2. Was the data generation experiment generating replays of recorded real-world actions or unseen teleop trajectories during world model training? If the model performs well even for unseen teleop trajectories, then it enables scalable new demonstration collection, otherwise, the experiment primarily demonstrates observation generation along recorded action trajectories, and the ability to collect new demonstrations remains unclear.
  3. Was the model evaluated on held out objects excluded from the autoencoder and dynamics model training? It was unclear to me what held-out object means precisely mean in Table 2. Please clarify the object-level splits for both training stages and the number of held-out objects used for evaluation.
  4. The planning experiments seem a bit limited, where the zero shot planning result only shows a few examples with final state chamfer distance. Could the authors report success rate (with clearly defined success criteria) for these experiments as well?
Questions for authors

Please see weaknesses, thanks.

Reviewer_isY9OpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents an action-conditioned 3D world model that combines a hybrid particle-voxel autoencoder with latent dynamics to jointly predict geometry, multi-view RGB-D observations, and tactile signals. The paper evaluates the model on diverse scenarios and demonstrate applications in policy training, evaluation, and planning with unseen objects.

Critical strengths

The shared 3D representation provides a natural way to maintain consistency across camera views and spatially align tactile predictions, without requiring explicit point correspondences. The real-world policy training results and simulation-to-real performance correlations show valuable information beyond observation reconstruction.

Critical weaknesses

The unseen-object and unseen-robot planning results cover only four objects and one robot setup, which limits the support for the broader generalization claim.

Questions for authors

Which types of interactions are most challenging for the model, and what failure modes do the authors typically observe during long rollouts? It would also be helpful to clarify whether errors in geometry and tactile predictions tend to occur together or independently.

Reviewer_h3hrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a hybrid particle-voxel auto-encoder (a sparse 3D conv encoder to a voxel latent, decoded coarse-to-fine to occupancy, 3D Gaussian particles and tactile particles) and an action-conditioned 3D U-Net latent dynamics model trained with multi-step rollouts, a curriculum and history-noise injection. On six real ALOHA bimanual tasks it beats 2D video baselines (Cosmos, Robot4DGen, IWS) and particle baselines (PGND, PointWorld, ParticleFormer) on image, depth, Chamfer and tactile metrics over open-loop rollouts. Policies trained only on simulator data reach success close to real-data training on three tasks, and simulator success correlates with real success (r = 0.86-0.97). On Deform360 it beats particle baselines and is used for qualitative zero-shot planning on unseen objects.

Critical strengths
  • A single 3D latent serves RGB, depth, geometry and tactile signals, with tactile represented as spatially aligned particles. This is a principled multimodal design with view consistency that 2D video world models lack.
  • The paper tests downstream usefulness directly, through policy training from simulator data, a sim-to-real correlation of policy performance across four policy families, and planning, instead of relying only on reconstruction metrics.
Critical weaknesses
  • The downstream claims are weakly supported. The policy-training protocol is unclear: it does not say how the 100 simulator demonstrations were produced, or whether the world model's roughly 200 training episodes overlap with the real demonstrations or evaluation initial states. There is no comparison against data from a baseline world model (for example IWS), so 'comparable to real data' is hard to interpret. Planning is qualitative only: four objects, single Chamfer numbers, no trial counts, success rates or baseline models, and the 'unseen robot arm' generalization is weak because the model represents only object particles with gripper-pose actions.
  • The proposed approach needs robot-specific multi-view, depth and tactile data, which is hard to acquire at scale, and the paper trains on about 200 episodes per task, so scaling is unclear.
  • There are no ablations and the baseline comparison is thin. Nothing isolates the consistency loss, history noise, multi-step curriculum, render loss, voxel-only vs particle decoding, the tactile channel or latent size, so the 'hybrid particle-voxel' claim is not isolated. Particle baselines are trained on CoTracker tracks, and the 2D baselines get depth tiled as RGB, so large margins may reflect adaptation choices. Geometry metrics for the video baselines are not explained, and no variance or per-task results are given. The closest no-tracking 3D particle model (Visual Particle Dynamics) is cited but not compared, and no 3D Gaussian world model (for example GWM) is included.
Questions for authors
  1. How exactly were the 100 simulator demonstrations collected, and are the world-model training episodes and policy-evaluation initial states disjoint from them? This determines whether Fig. 6 shows genuine data generation or memorized trajectories.
  2. Can you provide ablations of the key components (hybrid decoder vs voxel-only and particle-only, consistency loss, history noise, multi-step loss and curriculum)? This is needed to attribute the gains to the proposed design.
  3. How are the geometry and tactile metrics computed for the 2D video baselines, and how were those baselines tuned? Can you give per-task results with variance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.