acceptodds
Under review as a conference paper at ICLR 2027

Does Latent Planning Survive Point Clouds? Action-Conditioned JEPA World Models for Geometric Observations and Goals

Abstract

Latent action world models let agents plan new behaviors at test time by predicting how actions change the environment, and joint-embedding predictive architectures (JEPAs) do so by forecasting future latent states rather than pixels. Yet nearly all such models see the world through a camera, even though robotic manipulation is fundamentally geometric: in robotics goals for manipulation are traditionally specified by target object poses, not by images of the object once placed. We ask whether latent planning survives a shift from appearance to geometry, on the observation side as well as on the goal specifications side. To answer this, we extend the stable-worldmodel evaluation platform with simulated LiDAR-style raycast point clouds as a new sensor modality, and adapt three JEPA designs to point clouds: a frozen-encoder model built on Utonia features, a distribution-prior model based on LeWM, and an action-sensitive model based on Delta-JEPA. We further introduce a goal-encoding mechanism that constructs the goal latent from the current latent and a 3D target pose, removing the need for goal images or goal point clouds. A comparative evaluation of the different anti-collapse mechanisms shows that point-cloud world models can match their image-based counterparts, demonstrating that the modality shift from appearance to geometry is achievable. All models are released as open weights with open-source training and inference code, to make world-model planning accessible for LiDAR-driven and pose-directed robotic tasks.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world models, latent representation, robotics, control, point clouds

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_shTEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper extends the stable-worldmodel suite of tasks with a LiDAR point clouds modality, and provides evaluation of several JEPA-style models adapted to point clouds. In addition, the paper proposes a mechanism for converting goals from latents and 3D poses. The main takeaway is that the existing approaches for image-based setups also work with point clouds.

Critical strengths
  • Paper is easy to follow and simple.
  • The contribution is overall clear, and complimenting the benchmark with 3D is a nice addition, though not extremely novel.
  • Analysis of different architectures and objectives, trained and pre-trained.
  • The noise ablations indicate an interesting avenue of future work.
  • Comprehensive appendix.
  • Open-source.
Critical weaknesses
  • Generating goals is not trivial and requires access to privileged information, such as object positions. It is unclear how this would transfer to real-world setups. In addition, it requires training in a separate module.
  • Missing related work: Locate 3D, Point-JEPA
  • Writing: while the paper is overall clear, some sentences read like an output of an LLM, e.g., Line 360: “Table 2 answers the title question.”.

Minor:

  • Line 286: “equation 1” -> “Equation 1”. Line 293: “equation 4” -> “Equation 4”. Line 312. Line 344. Please fix all those references.
Questions for authors
  • Can the authors please report the complexity of each architecture/objective? Namely, inference and planning times.
  • What would it take to use the proposed training and inference setups in the real-world, especially for planning?
  • Overall, I’m positive regarding this submission. Though not extremely novel, it does provide a contribution that might be useful for future research.
3 comments
AuthorsOpenReview ↗

First Rebuttal Comment

Thank you for reading our paper so carefully and for highlighting its strengths.

shTE-Q1: "Generating goals is not trivial and requires access to privileged information, such as object positions. It is unclear how this would transfer to real-world setups. In addition, it requires training in a separate module." 3D targets are indeed needed, but only as training labels. At test time, the goal is simply a desired pose, and no goal scan is required. In real setups, these labels (e.g., object poses) can be obtained offline from off-the-shelf 3D pose estimators run on the recorded scans, or from the robot's proprioception (e.g., the gripper pose). Since no real-time requirements must be met, we can use more accurate but slower methods and also post-processing techniques for that. The target-to-latent module itself is lightweight, trained on frozen latents, and leaves the world model unchanged. It also improves robustness when the sensor is degraded (Table 13).

shTE-Q2: "Missing related work: Locate 3D, Point-JEPA" Both Papers are now discussed in the "World models on point clouds" paragraph in Related Work and set in relation to our work. Thanks!

shTE-Q3: "Writing: while the paper is overall clear, some sentences read like an output of an LLM, e.g., Line 360: “Table 2 answers the title question.”" Thank you for letting us know. We rewrote that sentence. Please let us know if you find any other formulations that one might find distracting.

shTE-Q4: "Line 286: “equation 1” -> “Equation 1”. Line 293: “equation 4” -> “Equation 4”. Line 312. Line 344. Please fix all those references." Thank you for pointing those out. We went through the Paper and hopefully fixed all of them.

shTE-Q5: "Can the authors please report the complexity of each architecture/objective? Namely, inference and planning times." Thank you for this suggestion. We added a Compute & Runtime section (Appendix A.6) with training, inference, and planning times for all architectures.

shTE-Q6: "What would it take to use the proposed training and inference setups in the real-world, especially for planning?"" This is a very interesting question. For the example of OGB-Cube, the JEPA training itself would not change much, since it only needs recorded sequences of point clouds and actions, without URDF data, camera calibration, or dense 3D tracks. Most of the work lies in data collection, since that requires a range sensor observing the robot, recording states and actions for real episodes (OGB-Cube contains 10k) using teleoperation or a scripted policy. For planning, the robot has to react to actions the same way as during recording, which usually holds for position-controlled arms. Besides the sensor setup and robot software, the recorded action and sensor data have to be synced, and safety concepts for real-life planning have to be implemented. Simulation-to-real approaches can help here, but require very accurate simulation of both the robotic behaviour and the scanned observations. As this is a high-effort project, the goal of our work is to deliver ideas and empirical evidence for the functionality of this approach to provide a foundation for future work in this research direction.

Reviewer_shTEOpenReview ↗

Thank you for your effort

I thank the authors for their effort during the rebuttal period, I appreciate the clarifications and the additional details. Overall, my impression is that there is a solid contribution in the form of adding another modaility to a widely-used benchmark, while the novelty on the application side is modest. Given the ratings scale of this year's ICLR, I believe my current score reflects my impression of the paper well and I'm happy to keep it. I acknowledge that I have read the other reviews and responses as well.

Thank you for your time and for engaging with our rebuttal. We are glad the clarifications were helpful.

Reviewer_SYfKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper mainly discuss if the latent world model based on JEPA can be applied to point cloud. The authors proposed Point-LeWM and Utonia-WM, which use 3D point cloud backbone to build connections between current coordinates and latent state. The the results show that encoding targeting state can help to enhance planning ability in most cases.

Critical strengths

The idea is straightforward and easy to understand: using point cloud to improve latent-space planning. The paper have some interesting ablations and useful analysis, including attention visualization, state probing, frozen-encoder. These experiments have shown some interesting points.

Critical weaknesses
  1. The experiments are conducted in simluation environment, where point cloud are easy to get and all date obtained are clean. However, in real scene, point cloud are usually abtained by Lidar or depth sensor, which often contain noise and are easily influenced by environment. The authors are encouraged to include more experiments in real scenes.
  2. Utonia-WM uses fixed voxel pooling. Ablations on voxel resolution, feature hierarchy, or projection design would help, as these factors often influence the performance of point cloud encoder.
Questions for authors

This paper offers useful implementations and a systematic initial evaluation of point-cloud latent world models. However, the methodological novelty is more or less incremental. please see the weakness part to include more evidence of the effectiveness of the method.

4 comments
AuthorsOpenReview ↗

First Rebuttal Comment

Thank you for your positive feedback on our analyses and for your constructive comments.

SYfK-Q1: "The experiments are conducted in simluation environment,...The authors are encouraged to include more experiments in real scenes." We agree and also think that this strengthens our position in motivating real-world applications. Hence, in Section 4.1, we added experiments on the real-world RH20T dataset, where we train Point-LeWM and Point-Delta-JEPA without adjustments and show that both can encode the gripper position well, and that Point-Delta-JEPA can even encode the full gripper pose.

Since real-world planning experiments require building the corresponding platforms, which is resource-intensive and out of scope (see shTE-Q6), we instead refer to our planning experiments under noise and dropout, in which we simulate noisy real-world data (Section 4.1 with Table 12 in the Appendix, and Section 4.2 with Table 13). We leave planning on real-world point cloud data as future work that our research enables.

SYfK-Q2: "Utonia-WM uses fixed voxel pooling. Ablations on voxel resolution,..."" That is a good point, and we have now provided ablations for voxel resolution, feature hierarchy, and projection (PCA vs. seeded random orthogonal) in Appendix A.2. The higher voxel resolution evaluations are still running, and we will add them once finished. For every configuration, precomputing the Utonia features takes roughly three days on two A100 GPUs and 300-400 GB of storage, followed by two to three days of training on the same hardware and five hours of evaluation on ten A40 GPUs, but we are there soon.

AuthorsOpenReview ↗

More experiments in real scenes & Utonia fine grid insights

On RH20T, we previously showed that the latents encode the gripper state. Taking another step towards planning on real data, a new surprise experiment (Section 4.1, Appendix D.1) now shows that the predictor works on real data too, as it reacts when the target object is teleported or removed, but not when points are just missing. The other Utonia-WM ablations are in Appendix A.2 as requested. The higher voxel resolution makes the predictor collapse, which we explain there.

With this, the paper now evaluates both the encoder and the predictor on real data and ablates Utonia-WM, which we hope addresses your main concern about the simulation-only setting and the point cloud encoder's effectiveness.

Please take a moment to respond to the rebuttal. Thank you!

AuthorsOpenReview ↗

Utonia-WM ablation

Out of interest, we also retrained the finer-grid configuration with query-key normalization. This removes the collapse. The attention logits stay small, and the validation loss decreases throughout training. We refer to Appendix D.1 for more information. The longer sequences and the normalization slow training considerably. However, for the OGB-Cube we can see that a finer voxel grid does not increase performance (69.8 ± 6.8 success rate). The same holds for Push-T, which even decreases in performance (28.0 ± 5.9), showing that the selected voxel resolution was already meaningfully chosen.

As stated at the beginning, we did these experiments out of curiosity and do not include them in the paper since they follow a different architecture (qk-normalization) and training regime, but we thought you might be interested in them too.

Reviewer_rLZjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether JEPA-based latent world models can be extended from image observations to point-cloud observations for robotic planning. Several existing world models are adapted to point-cloud inputs, including LeWorldModel, Delta-JEPA, and DINO-WM. The authors extend the established SWM evaluation platform with simulated Lidar-style raycast point clouds and evaluate the methods on four simulated environments, comparing point-cloud-based and image-based latent planning. The results show that point-cloud-based world models can achieve comparable performance to image-based world models.

Critical strengths

The paper is well written and clear.

The experiments are comprehensive and support the claim. The authors evaluate three different types of latent world models and extend them to point cloud input.

Critical weaknesses

Even though it is interesting that the authors demonstrate that point clouds can be used as inputs for latent world models, I do not think this is very surprising, since prior world-model work such as PointWorld has already shown that explicit point-cloud representations can work even in the real world. In addition, many world-action models use RGB-only video models as backbones, and prior work has shown that adding 3D/4D modalities can be helpful. So I am curious and do not fully understand:

  • Why does using point clouds perform worse than RGB-only inputs in some settings?
  • Would combining point clouds and RGB in a latent world model perform better than using RGB alone?
  • Beyond showing that existing latent world models can be extended to point clouds, what other takeaways can we draw? Why is this important and helpful, and what insights can we gain from it?

Another concern is that, since the proposed methods are extensions of existing latent world models, the novelty might be limited. The evaluation would be stronger if the methods could be applied to more complicated scenarios, demonstrating the advantages of point-cloud inputs, especially for geometry-related or precision-demanding tasks where RGB-only approaches might struggle.

Questions for authors

Please see the weaknesses above.

3 comments
AuthorsOpenReview ↗

First Rebuttal Comment

Thank you for your detailed review and for giving us the chance to clarify our work.

rLZj-Q1: "..., I do not think this is very surprising, since prior world-model work such as PointWorld has already shown that explicit point-cloud representations can work even in the real world." We agree that PointWorld shows explicit point-cloud dynamics can be learned for real-world tasks. However, it predicts per-point displacements (12K points) supervised by dense 3D tracks, an objective that cannot collapse. Additionally, it relies on an URDF and calibrated cameras, among other requirements. We, on the other hand, only encode a raw, unordered point cloud (x,y,z) into a 192-dim vector (Point-LeWM & -Delta-JEPA) and learn dynamics from sparse actions in a JEPA-framework that can collapse. Moreover, with this latent encoding, the risk of the slow-feature shortcut and the geometric shortcut (Lines 141-159) is much more present.

rLZj-Q2: "In addition, many world-action models use RGB-only video models as backbones, and prior work has shown that adding 3D/4D modalities can be helpful." It is true that geometry can help RGB video backbones of WAMs. However, these findings do not directly transfer to our setting. These works combine geometry with RGB, are trained using reconstruction, and use a learned policy to directly predict actions. Our model, on the other hand, uses geometry only, is trained without reconstruction, and plans in latent space with MPC over predicted rollouts. To the best of our knowledge, it was unexplored how geometry-only inputs behave for action-conditioned JEPA world models, and our work is a first step in this direction. Related work (Lines 134-136) now also briefly distinguishes WAMs from latent-planning world models.

rLZj-Q3: "Would combining point clouds and RGB in a latent world model perform better than using RGB alone?" In this JEPA context, we see detailed analysis of the fusion of point cloud & image in one model as future work. Nevertheless, thanks to your inputs and questions, we ran experiments where image and point cloud latents have been combined during planning, which led to the conclusion that this naive combination already improves results compared to single-modality planning. Experiment description and results have been added to Section 4.3.

rLZj-Q4: "Why does using point clouds perform worse than RGB-only inputs in some settings?"" Other than adjusting the optimization settings (batch size, learning rate & schedule) to match the hardware used in this project, we used the (hyper-)parameters that were optimized for the RGB models. This, and the fact that we are dealing with sparse instead of dense inputs, could influence performance. That being said, the aim of this work is not to outperform image-based JEPAs, but to show the problems (geometric shortcut and slow feature) that arise when working with point clouds, and to propose solutions to them. Whether a point-cloud model outperforms an image-based JEPA depends heavily on many factors such as sensor placement and sensor resolution. Such benchmarking would require an applied setting and a specific use case the sensors are optimized for.

rLZj-Q5: "Beyond showing that existing latent world models can be extended to point clouds, what other takeaways can we draw? Why is this important and helpful, and what insights can we gain from it?"

  • First, the practical takeaway is that a world model for point clouds can be trained from recorded observations and actions alone, without knowledge of how the robot is built or a complex calibration. We think this is important because it lowers the barrier to using such models on new real-world platforms. Thanks to your and Reviewer SYfK's feedback, we emphasized this with the added experiments (Section 4.1, 4.3).
  • Goals can be given as 3D target poses instead of goal observations. Success matches the goal-cloud reference, and the clean target makes planning more robust to sensor noise.
  • Even without a supervised objective as in PointWorld, latent prediction and planning do not fail, although 80--98% of returns lie on a static ground plane and only 0.3--15% of points move.

rLZj-Q6: "...more complicated scenarios, demonstrating the advantages of point-cloud inputs, especially for geometry-related or precision-demanding tasks where RGB-only approaches might struggle." In Section 4.1, we added an evaluation on a dataset with real-world data demonstrating the capabilities in more complicated, noisy, and sparse point clouds. In comparison to the simulated data, every episode in the real dataset covers a different object or scene.

Because our method operates directly on point clouds, it inherits advantages from LiDAR sensors. Unlike RGB cameras, LiDAR is an active sensor and is therefore largely unaffected by illumination, performing equally well in darkness. It also provides accurate geometric measurements at long range, where image-based depth estimation degrades.

AuthorsOpenReview ↗

More experiments on complex scenarios

This might also interest you regarding more complicated scenarios: in a new surprise experiment on real RH20T data (Section 4.1, Appendix D.1), the predictor reacts when the target object is teleported or removed, but not when points are just missing. So even without dense supervision as in PointWorld and despite working in latent space, the model picks up how real scenes should evolve. We hope this, together with the probing and multi-modal results, makes the contribution and its takeaways clearer, and we are happy to answer any further questions.

Please take a moment to respond to the rebuttal. Thank you!

Comments on the paper

3 comments
AuthorsOpenReview ↗

Thank you and first general comment

First, we thank the reviewers for their feedback. Given what we all hear about the current state of the review process, we consider ourselves lucky to have received such constructive feedback that will definitely improve our work.

In this general comment, we would like to address the novelty concerns that were a common remark in the reviews. While we do agree that our proposed methods are extensions of existing work, we still would like to point out that:

  1. We are the first to show a way that action-conditioned latent world models (JEPA models) for planning also work on raw point clouds, which was, given the failure modes of latent prediction and self-supervised learning in point clouds as discussed in Lines 151-158, not clear. More info is provided in the discussion with Reviewer rLZj.

  2. We address and provide an approach for the important research question regarding the problem of requiring a goal observation at test-time in JEPA models for planning. Here, our target-to-latent module represents a successful method for solving this problem while keeping the planning in latent space. More info is provided in the discussion with Reviewer shTE.

  3. We extend the established stable-worldmodel benchmark suite by point clouds, enabling a unified benchmark for (latent) world model research on point clouds and multi-modal settings.

We consider this new empirical evidence for 1) and 2) to be impactful for JEPA research at a theoretical level, as it shows that point cloud methods can match image-based methods for this type of latent planning. For practitioners, knowing that this approach works supports the development of lidar-based systems for unstructured environments. Finally, paired with 3), we believe this will accelerate research on action-conditioned latent planning for point clouds and image + point clouds, which, to the best of our knowledge, had not yet been explored before our work.

Thanks to the feedback from Reviewers SYfK and rLZj, we added real-data evaluations showing that gripper poses can be encoded in real point clouds (see SYfK-Q1) and implemented a baseline showing that a naive combination of an image and a point cloud model can improve planning success rate (see rLZj-Q2), which strengthens our position.

AuthorsOpenReview ↗

Paper Update (12 Nov 2026)

The latest version of the paper is now available. The following changes are worth looking at:

  • New real-world experiment (Section 4.1, Appendix D.1)
  • New multi-modal planning (Section 4.3, Appendix D.4)
  • Minor Related Work changes (Section 2)
  • New appendix material:
    • Utonia-WM encoder ablation (Appendix A.2)
    • Inference and planning runtimes (Appendix A.6)
AuthorsOpenReview ↗

Paper Update (16 Nov 2026)

Another new version of the Paper is now available. The following changes are worth looking at:

  • New latent surprise experiment (Section 4.1, Appendix D.1): teleporting or removing the target object in real RH20T point clouds makes both models' prediction error spike. Removing points elsewhere or dropping half of them barely does.
  • New figures (Appendix D.1): surprise over time (32 episodes), perturbed point clouds frame by frame, and probing examples for eight more episodes.
  • Finer Utonia-WM grid insights (Appendix A.2): more tokens make the predictor collapse (more tokens -> logit growth -> saturated softmax), and they would unfairly favour Utonia-WM over DINO-WM.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.