- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper extends the stable-worldmodel suite of tasks with a LiDAR point clouds modality, and provides evaluation of several JEPA-style models adapted to point clouds. In addition, the paper proposes a mechanism for converting goals from latents and 3D poses. The main takeaway is that the existing approaches for image-based setups also work with point clouds.
Critical strengths
- Paper is easy to follow and simple.
- The contribution is overall clear, and complimenting the benchmark with 3D is a nice addition, though not extremely novel.
- Analysis of different architectures and objectives, trained and pre-trained.
- The noise ablations indicate an interesting avenue of future work.
- Comprehensive appendix.
- Open-source.
Critical weaknesses
- Generating goals is not trivial and requires access to privileged information, such as object positions. It is unclear how this would transfer to real-world setups. In addition, it requires training in a separate module.
- Missing related work: Locate 3D, Point-JEPA
- Writing: while the paper is overall clear, some sentences read like an output of an LLM, e.g., Line 360: “Table 2 answers the title question.”.
Minor:
- Line 286: “equation 1” -> “Equation 1”. Line 293: “equation 4” -> “Equation 4”. Line 312. Line 344. Please fix all those references.
Questions for authors
- Can the authors please report the complexity of each architecture/objective? Namely, inference and planning times.
- What would it take to use the proposed training and inference setups in the real-world, especially for planning?
- Overall, I’m positive regarding this submission. Though not extremely novel, it does provide a contribution that might be useful for future research.
3 comments
Thank you for your effort
I thank the authors for their effort during the rebuttal period, I appreciate the clarifications and the additional details. Overall, my impression is that there is a solid contribution in the form of adding another modaility to a widely-used benchmark, while the novelty on the application side is modest. Given the ratings scale of this year's ICLR, I believe my current score reflects my impression of the paper well and I'm happy to keep it. I acknowledge that I have read the other reviews and responses as well.
Thank you
Thank you for your time and for engaging with our rebuttal. We are glad the clarifications were helpful.
First Rebuttal Comment
Thank you for reading our paper so carefully and for highlighting its strengths.
shTE-Q1: "Generating goals is not trivial and requires access to privileged information, such as object positions. It is unclear how this would transfer to real-world setups. In addition, it requires training in a separate module." 3D targets are indeed needed, but only as training labels. At test time, the goal is simply a desired pose, and no goal scan is required. In real setups, these labels (e.g., object poses) can be obtained offline from off-the-shelf 3D pose estimators run on the recorded scans, or from the robot's proprioception (e.g., the gripper pose). Since no real-time requirements must be met, we can use more accurate but slower methods and also post-processing techniques for that. The target-to-latent module itself is lightweight, trained on frozen latents, and leaves the world model unchanged. It also improves robustness when the sensor is degraded (Table 13).
shTE-Q2: "Missing related work: Locate 3D, Point-JEPA" Both Papers are now discussed in the "World models on point clouds" paragraph in Related Work and set in relation to our work. Thanks!
shTE-Q3: "Writing: while the paper is overall clear, some sentences read like an output of an LLM, e.g., Line 360: “Table 2 answers the title question.”" Thank you for letting us know. We rewrote that sentence. Please let us know if you find any other formulations that one might find distracting.
shTE-Q4: "Line 286: “equation 1” -> “Equation 1”. Line 293: “equation 4” -> “Equation 4”. Line 312. Line 344. Please fix all those references." Thank you for pointing those out. We went through the Paper and hopefully fixed all of them.
shTE-Q5: "Can the authors please report the complexity of each architecture/objective? Namely, inference and planning times." Thank you for this suggestion. We added a Compute & Runtime section (Appendix A.6) with training, inference, and planning times for all architectures.
shTE-Q6: "What would it take to use the proposed training and inference setups in the real-world, especially for planning?"" This is a very interesting question. For the example of OGB-Cube, the JEPA training itself would not change much, since it only needs recorded sequences of point clouds and actions, without URDF data, camera calibration, or dense 3D tracks. Most of the work lies in data collection, since that requires a range sensor observing the robot, recording states and actions for real episodes (OGB-Cube contains 10k) using teleoperation or a scripted policy. For planning, the robot has to react to actions the same way as during recording, which usually holds for position-controlled arms. Besides the sensor setup and robot software, the recorded action and sensor data have to be synced, and safety concepts for real-life planning have to be implemented. Simulation-to-real approaches can help here, but require very accurate simulation of both the robotic behaviour and the scanned observations. As this is a high-effort project, the goal of our work is to deliver ideas and empirical evidence for the functionality of this approach to provide a foundation for future work in this research direction.