- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a Grounded World Model (GWM) framework that learns a latent world model. Different from previous works that enable planning by the image goals, the proposed GWM allows the world model planning by the language goals. GWM adopts the pretrained video-language embedding space as the shared interface for aligning both visual and textual modalities. The experiments are conducted on simulation benchmarks and the real-world Franka Panda robot.
Critical strengths
- Latent world models have potential to allow zero-shot planning, which is crucial for out-of-distribution (OOD) scenarios in real-world applications. It could also alleviate efforts to collect data for every task, reducing the teleoperation burden.
- The proposed method adopts the language goals rather than image goals, which is also practical and reasonable for real-world inference and deployment.
- The overall paper is easy to follow and well-organized.
Critical weaknesses
- As described in lines 369-371, the overall pipeline highly relies on external modules to produce the action candidates. However, this way would highly restrict the flexibility and generalibility. If the correct action sequence can not be properly produced by this pipeline, GWM cannot successfully address the task.
- The experiments are mainly conducted on the pick-and-place tasks. The effectiveness of more contact-rich and complex manipulation tasks are not sufficiently verified. It would potentially be a limitation of using VLM’s embedding space as the latent world model space. More clarification and analysis on it would be beneficial.
- The diversity of the unseen scenarios are also insufficient. One of the benefits from world models is the capability to perform long-horizon tasks. The current results mainly perform on short-horizon tabletop manipulation without demonstrating the ability to handle long-horizon or multi-stage manipulation tasks.
Questions for authors
- Is the proposed method model-agnostic? Whether the proposed approach can be applied to different VLMs or different model sizes?
Thanks to the authors for the detailed response! The new results with the diffusion policy are helpful, since I was concerned about GWM choosing from ready made motions while other methods had to generate their own. The camera calibration and candidate quality experiments also answer most of my other questions. I still think the lack of multiple training runs and error bars is a limitation. But overall, I think the authors have made a good effort to address the concerns and the new experiments make the paper more convincing. So I am updating my rating from 2 to 3 (Weak Acceptance).