- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents WAP, which combines a pose-image-conditioned world model with VLM-guided global optimization and local search for robot planning. Experiments demonstrate gains across several generalization settings, while theoretical analyses examine model-based planning and its potential advantages over imitation learning.
Critical strengths
- Effective integration of world modeling and planning. WAP combines pose-image-conditioned world modeling with VLM-guided global optimization and local search.
- Substantial effort toward theoretical understanding. The paper investigates the theoretical basis of world-model-based planning and provides extensive formal analyses and proofs to complement its empirical findings.
Critical weaknesses
- Limited evidence for broad generalization. While Table 3 shows gains, the evaluation covers only 5 selected LIBERO-Long tasks, 6 modified-layout LIBERO-Object tasks, and 2 novel tasks. Besides, the two real-robot “novel tasks” are green/purple variants of the same can-to-basket operation, rather than distinct manipulation skills. Thus, the results establish effectiveness in the evaluated settings, but provide limited evidence for broad novel-task generalization.
- The VLM baseline removes world-model imagination, global optimization, and local search together. Although Table 16 demonstrates the benefits of individual planning stages, it remains unclear whether comparable gains could be achieved by global optimization and local search without world-model imagination under a similar inference budget.
Questions for authors
- How does WAP perform when transferred to tasks with more diverse action patterns?
- How does WAP compare with a VLM baseline using global optimization and local search without world-model imagination, under comparable candidate and reasoning budgets?
3 comments
Response to Reviewer oADL (2/2)
Response to Weakness 2 (W2) & Question 2 (Q2): Isolating World-Model Imagination from Search
Both global optimization and local search depend fundamentally on visual world-model imagination:
- Outcome-Based Candidate Evaluation: Selecting the optimal action requires evaluating candidates based on their downstream physical consequences (e.g., rim collisions, grasp stability, or off-center releases), which only become apparent in the world-model-imagined rollouts.
- Limitations of VLM-Only Reasoning: A standalone VLM cannot directly foresee physical contact outcomes from the initial observation alone. As noted in Appendix D.2 (Page 45, Lines 2386–2396), even when we explicitly prompt the VLM with known failure modes (such as maintaining clearance over rims) and grant additional reasoning budget, it still fails to avoid collisions without visual rollouts.
To strictly isolate the contribution of our visual world model under an identical search structure and candidate budget, we conducted a new ablation experiment (Section 5.4, Page 10, Table 6 of the revised manuscript). In this baseline, the VLM performs the exact same global optimization and local candidate search under the same candidate budget, but instead of using our visual world model, the VLM writes and executes code to numerically simulate the end-effector trajectory in text and reasons over these textual outcomes to refine and rank the candidates.
Rebuttal Table 2 (Table 6 in Revision): Search with VLM Code Simulation vs. World-Model Visual Imagination (Success Rate %)
| Method | Compositional Task | New Layout | Novel Task |
|---|---|---|---|
| VLM Planner Baseline (No Search) | 42.0 | 37.7 | 40.0 |
| WAP w/ VLM Textual/Code Imagination (Same Search & Budget) | 52.0 | 48.0 | 44.0 |
| WAP (Ours, w/ World-Model Visual Imagination) | 72.0 | 82.0 | 78.0 |
As shown in Rebuttal Table 2, while performing global optimization and local search via VLM code simulation improves over the direct VLM planner by 4.0% to 10.3%, WAP with visual world-model imagination achieves a further 20.0% to 34.0% improvement across all settings. This confirms that the search structure alone is insufficient: simulating end-effector trajectories in code and text cannot capture fine-grained visual geometry and physical contacts as effectively as our action-conditioned world model.
Response to Question 1 (Q1): Performance on Diverse Action Patterns
Beyond standard pick-and-place motions, WAP handles diverse manipulation skills, including 6-DoF orientation alignment (e.g., rotating the gripper to grasp the long edge of the cream cheese box in Fig. 21), side-grasping mugs by their rims/handles, and articulated object manipulation (Put the Black Bowl in the Bottom Drawer and Close It).
To further evaluate WAP on more diverse action patterns, we tested an additional multi-stage task from LIBERO-Long: Turn on the Stove and Put the Moka Pot on It. Unlike top-down pick-and-place, this task requires non-standard trajectory control, including rotational/pushing manipulation to turn on a stove knob followed by grasping a moka pot by its angled handle and placing it onto the burner. As shown in Rebuttal Table 3, WAP achieves substantial gains over the VLM planner baseline across both articulated and multi-skill tasks (and strongly outperforms , which achieves only 6% on the stove task as shown in Table 12 of the revision).
Rebuttal Table 3: Performance on Diverse Manipulation Skills and Action Patterns (Success Rate %)
| Task | Manipulation Skills Involved | (VLA) | VLM Planner | WAP (Ours) |
|---|---|---|---|---|
| Turn on Stove & Put Moka Pot on It | Knob rotation/pushing + complex grasp | 6.0 | 24.0 | 58.0 |
| Put Bowl in Bottom Drawer & Close It | Precision placement + articulated drawer pushing | 44.0 | 48.0 | 72.0 |
Dear Reviewer oADL,
As the discussion period draws to a close, we wanted to briefly follow up to see if our response and revised manuscript have addressed your questions, including our new zero-shot real-robot tasks, evaluations on diverse manipulation skills, and the ablation comparing world-model visual imagination against VLM search with text simulation.
Please let us know if you have any remaining questions or concerns we can clarify before the deadline. Thank you again for your time and helpful feedback!
Best regards,
The Authors
Response to Reviewer oADL (1/2)
We thank the reviewer for their thoughtful feedback and constructive comments. Below, we address each point regarding broad generalization, diverse manipulation skills, and isolating world-model imagination from the search procedure.
Response to Weakness 1 (W1): Evidence for Broad Novel-Task Generalization
Our framework demonstrates strong generalization beyond the training distribution across both simulation and real-robot evaluations, substantially outperforming state-of-the-art policy-based baselines.
Crucially, we directly reused the world model checkpoint previously adapted on Put the Can in the Basket as-is, collecting zero new trajectories and performing zero parameter updates, even though the camera placement, lighting, manipulated objects, and task geometry all differ entirely from the adaptation data.
Rebuttal Table 1 (Table 4 in Revision): Zero-Shot Real-Robot Transfer to Unseen Tasks (20 Trials per Task)
As shown in Rebuttal Table 1, a world model adapted from failure data on one task can be deployed zero-shot to solve completely novel tasks with unseen objects, outperforming the VLM planner baseline by 45% to 50%. Furthermore, both global optimization (to prevent knocking over the target cup) and local search (to center the release over the narrow cup opening) are necessary to achieve high success rates in fine-grained manipulation.