acceptodds
Under review as a conference paper at ICLR 2027

Grounded World Model: Latent Planning with Language Goals

221 views

Abstract

World models such as DINO-WM and LeWM specify the goal with an image, which is difficult to obtain in advance for novel tasks. We present the Grounded World Model (GWM), a latent world model that enables zero-shot planning in the real world from language goals alone. Given a candidate action sequence and the current observation, GWM predicts the future in the visual space of a pretrained video-language embedding model. The frozen readout of this embedding model maps this imagined future and the task description into the same embedding space, where their negative cosine similarity serves as the planning cost. Training GWM requires only offline and task-agnostic video-action pairs and no language labels. In simulated experiments on WISER, planning with GWM, which executes the candidate action of lowest cost, solves 87% of 288 tasks with unseen instructions and visual signals, while ten fine-tuned VLAs average 22%. We then scale GWM up with real robot data, and use it for zero-shot planning in realistic simulation and real scenes. In the IsaacSim evaluation, planning with GWM completes all 70 trials across 14 tasks that require reasoning over referring expressions, matching a modular planner grounded by a frontier VLM, while π0.5 reaches 37/70. Deployed on a real Franka, the same stack completes 55/60 separately evaluated pick-and- place sub-tasks, comparable to the modular planner’s 52/60, with the full system running locally on a single consumer GPU

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Models, Planning, Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ocPDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a Grounded World Model (GWM) framework that learns a latent world model. Different from previous works that enable planning by the image goals, the proposed GWM allows the world model planning by the language goals. GWM adopts the pretrained video-language embedding space as the shared interface for aligning both visual and textual modalities. The experiments are conducted on simulation benchmarks and the real-world Franka Panda robot.

Critical strengths
  1. Latent world models have potential to allow zero-shot planning, which is crucial for out-of-distribution (OOD) scenarios in real-world applications. It could also alleviate efforts to collect data for every task, reducing the teleoperation burden.
  2. The proposed method adopts the language goals rather than image goals, which is also practical and reasonable for real-world inference and deployment.
  3. The overall paper is easy to follow and well-organized.
Critical weaknesses
  1. As described in lines 369-371, the overall pipeline highly relies on external modules to produce the action candidates. However, this way would highly restrict the flexibility and generalibility. If the correct action sequence can not be properly produced by this pipeline, GWM cannot successfully address the task.
  2. The experiments are mainly conducted on the pick-and-place tasks. The effectiveness of more contact-rich and complex manipulation tasks are not sufficiently verified. It would potentially be a limitation of using VLM’s embedding space as the latent world model space. More clarification and analysis on it would be beneficial.
  3. The diversity of the unseen scenarios are also insufficient. One of the benefits from world models is the capability to perform long-horizon tasks. The current results mainly perform on short-horizon tabletop manipulation without demonstrating the ability to handle long-horizon or multi-stage manipulation tasks.
Questions for authors
  1. Is the proposed method model-agnostic? Whether the proposed approach can be applied to different VLMs or different model sizes?
Reviewer_NF7mOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a language-driven world model planning framework called Grounded World Model (GWM). Unlike existing world models, GWM does not rely on pre-provided target images and can execute language instructions. To achieve this, GWM learns world dynamics in a visual latent space aligned with pre-trained videos and language models.

Critical strengths
  1. The research question is both important and novel, and represents a promising direction. Natural language, as an interaction interface, is highly suitable for practical needs.
  2. The method is designed to be relatively simple. The authors did not introduce any additional complex language alignment training or generative VLM evaluation modules, and the overall framework has a certain degree of simplicity and scalability.
  3. It demonstrated outstanding zero-shot generalization ability in the Wiser test.
  4. The entire system can run on a single RTX 5090 GPU without needing to invoke a remote large-scale generative VLM for each action selection, thus having practical deployment value.
Critical weaknesses
  1. The paper directly uses the cosine similarity between predicted future actions and linguistic instructions to evaluate the quality of actions, but semantic similarity is not necessarily equivalent to the degree of task completion. The authors need to further discuss this issue.
  2. Some comparative experiments raise fairness issues. For example, in WISER, GWM can retrieve actions from existing demonstration trajectories, while VLA needs to directly generate robot actions; the difficulty of action generation faced by the two is not entirely the same.
  3. Planning performance is highly dependent on the external candidate trajectory generation module. GWM is essentially responsible for predicting and scoring existing candidate actions, rather than generating actions independently.
  4. The research question is very interesting, but the solution method lacks novelty.
Questions for authors
  1. Why can language-video cosine similarity reliably evaluate the success of an action? Can it be compared with specially trained task success discriminators or visual reward models?
  2. How much performance improvement does the world model itself actually contribute? Can we further eliminate the influence of pre-trained multimodal models and external planners through ablation experiments?
  3. Please provide a fair comparison of GWM and VLA.
Reviewer_5BJNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Grounded World Model, a latent world model for robot planning from language goals without language-labeled world-model training data. GWM predicts future video-language token latents from the current observation and candidate action trajectory. A frozen video-language embedding model then maps the predicted future and the instruction into a shared space, and negative cosine similarity ranks candidate trajectories. Training needs only robot videos with actions, no text labels.

Critical strengths

The idea is simple, it reuses a model that already links video and text instead of training a new language connection. The ablations clearly show which parts matter. Actions are fed in as drawn images of the robot arm. This makes it easy to move to a new robot (83% on xArm6 with no retraining). The whole system runs on one consumer GPU. It needs no action tokenizer, and the zero-shot xArm6 transfer is good evidence for cross-embodiment use.

Critical weaknesses

Simulation trials aren’t really 70. GWM chose one plan per task and replayed it 5 times. TIPTOP planned fresh every time. So GWM made 14 decisions, not 70. TIPTOP had to drop the block into the bin. GWM seems to pass while still holding the block over the bin. GWM only picks from 12 ready-made motions, and the right one is always included. The VLAs must create their own motions. It’s a much harder job. GWM only ranks options. If none of them is good, it can’t help, and this isn’t tested. Results drop from other camera angles, and camera calibration errors aren’t tested.

Questions for authors
  1. Can you rerun the simulation with fresh planning on every trial for all methods, and judge them all the same way?
  2. How did you choose your final checkpoints? Can you report several training runs with error bars?
  3. How do camera angle and calibration errors affect results?
  4. What happens when the options are fewer, worse, or all wrong?
2 comments

Thanks to the authors for the detailed response! The new results with the diffusion policy are helpful, since I was concerned about GWM choosing from ready made motions while other methods had to generate their own. The camera calibration and candidate quality experiments also answer most of my other questions. I still think the lack of multiple training runs and error bars is a limitation. But overall, I think the authors have made a good effort to address the concerns and the new experiments make the paper more convincing. So I am updating my rating from 2 to 3 (Weak Acceptance).

AuthorsOpenReview ↗

Thank you for the feedback!

Thank you for the positive feedback and for raising your score! We are glad that the diffusion policy results and the camera calibration and candidate quality experiments addressed your concerns : )

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.