acceptodds
Under review as a conference paper at ICLR 2027

World Action Planner: Generalizable Robot Decision-Making with Action-Conditioned World Models

Abstract

Building generalizable robot agents for diverse applications remains a fundamental challenge. While imitation learning-based policies can perform well in familiar training environments, they often struggle to generalize to novel scenes, layouts, and task compositions. To this end, we present **World Action Planner**, an agentic robot planning system in which the agent searches for and composes executable action plans through imagination with an action-conditioned world model. The search proceeds in a coarse-to-fine manner. First, the agent performs global action optimization by reasoning over imagined world-model rollouts to identify potential failures and refine the proposed action plan. It then performs local action search, comparing the imagined future outcomes of neighboring candidates to select the best action for execution. Across compositional long-horizon tasks, novel object layouts, and real-robot planning on novel tasks without expert demonstrations, World Action Planner consistently outperforms state-of-the-art end-to-end generalist policy models and VLM planners, demonstrating the effectiveness of world-model-based action search for generalizable robot decision making.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Model, Generalization, Planning, Agent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_oADLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents WAP, which combines a pose-image-conditioned world model with VLM-guided global optimization and local search for robot planning. Experiments demonstrate gains across several generalization settings, while theoretical analyses examine model-based planning and its potential advantages over imitation learning.

Critical strengths
  • Effective integration of world modeling and planning. WAP combines pose-image-conditioned world modeling with VLM-guided global optimization and local search.
  • Substantial effort toward theoretical understanding. The paper investigates the theoretical basis of world-model-based planning and provides extensive formal analyses and proofs to complement its empirical findings.
Critical weaknesses
  • Limited evidence for broad generalization. While Table 3 shows gains, the evaluation covers only 5 selected LIBERO-Long tasks, 6 modified-layout LIBERO-Object tasks, and 2 novel tasks. Besides, the two real-robot “novel tasks” are green/purple variants of the same can-to-basket operation, rather than distinct manipulation skills. Thus, the results establish effectiveness in the evaluated settings, but provide limited evidence for broad novel-task generalization.
  • The VLM baseline removes world-model imagination, global optimization, and local search together. Although Table 16 demonstrates the benefits of individual planning stages, it remains unclear whether comparable gains could be achieved by global optimization and local search without world-model imagination under a similar inference budget.
Questions for authors
  • How does WAP perform when transferred to tasks with more diverse action patterns?
  • How does WAP compare with a VLM baseline using global optimization and local search without world-model imagination, under comparable candidate and reasoning budgets?
3 comments
AuthorsOpenReview ↗

Response to Reviewer oADL (1/2)

We thank the reviewer for their thoughtful feedback and constructive comments. Below, we address each point regarding broad generalization, diverse manipulation skills, and isolating world-model imagination from the search procedure.


Response to Weakness 1 (W1): Evidence for Broad Novel-Task Generalization

Our framework demonstrates strong generalization beyond the training distribution across both simulation and real-robot evaluations, substantially outperforming state-of-the-art policy-based baselines.

  • Principled Benchmark Selection in Simulation: In our Compositional Task Generalization setting (Appendix D.2.1, Table 10, Page 46), the 5 LIBERO-Long tasks were not cherry-picked; rather, they represent all tasks in LIBERO-Long that are direct compositions of two atomic LIBERO-90 subtasks within the same scene, allowing a strict evaluation of compositional transfer when trained exclusively on LIBERO-90. We also expanded our evaluation to additional complex tasks requiring non-standard skills (such as Turn on the Stove and Put the Moka Pot on It in Q1, Rebuttal Table 3 below).
  • Zero-Shot Real-Robot Generalization to Unseen Tasks and Objects: To provide stronger evidence of broad real-robot generalization beyond the green/purple can variants, we conducted new, fully zero-shot real-robot experiments on the Franka Panda setup across two distinct manipulation tasks (Section 5.2, Page 9, Table 4, Fig. 5, and Appendix D.2.4, Figs. 18–19 of the revised manuscript):
    1. Cube-in-Cup: Placing a black cube into a narrow green cup.
    2. Cup Covering: Placing an upside-down large dark-green cup over an upside-down small light-green cup to cover it.

Crucially, we directly reused the world model checkpoint previously adapted on Put the Can in the Basket as-is, collecting zero new trajectories and performing zero parameter updates, even though the camera placement, lighting, manipulated objects, and task geometry all differ entirely from the adaptation data.

Rebuttal Table 1 (Table 4 in Revision): Zero-Shot Real-Robot Transfer to Unseen Tasks (20 Trials per Task)

TaskVLM Planner BaselineWAP w/ Global Opt. OnlyWAP (Ours, Full)
Cup Covering10%45%55%
Cube-in-Cup15%40%65%

As shown in Rebuttal Table 1, a world model adapted from failure data on one task can be deployed zero-shot to solve completely novel tasks with unseen objects, outperforming the VLM planner baseline by 45% to 50%. Furthermore, both global optimization (to prevent knocking over the target cup) and local search (to center the release over the narrow cup opening) are necessary to achieve high success rates in fine-grained manipulation.

AuthorsOpenReview ↗

Response to Reviewer oADL (2/2)

Response to Weakness 2 (W2) & Question 2 (Q2): Isolating World-Model Imagination from Search

Both global optimization and local search depend fundamentally on visual world-model imagination:

  1. Outcome-Based Candidate Evaluation: Selecting the optimal action requires evaluating candidates based on their downstream physical consequences (e.g., rim collisions, grasp stability, or off-center releases), which only become apparent in the world-model-imagined rollouts.
  2. Limitations of VLM-Only Reasoning: A standalone VLM cannot directly foresee physical contact outcomes from the initial observation alone. As noted in Appendix D.2 (Page 45, Lines 2386–2396), even when we explicitly prompt the VLM with known failure modes (such as maintaining clearance over rims) and grant additional reasoning budget, it still fails to avoid collisions without visual rollouts.

To strictly isolate the contribution of our visual world model under an identical search structure and candidate budget, we conducted a new ablation experiment (Section 5.4, Page 10, Table 6 of the revised manuscript). In this baseline, the VLM performs the exact same global optimization and local candidate search under the same candidate budget, but instead of using our visual world model, the VLM writes and executes code to numerically simulate the end-effector trajectory in text and reasons over these textual outcomes to refine and rank the candidates.

Rebuttal Table 2 (Table 6 in Revision): Search with VLM Code Simulation vs. World-Model Visual Imagination (Success Rate %)

MethodCompositional TaskNew LayoutNovel Task
VLM Planner Baseline (No Search)42.037.740.0
WAP w/ VLM Textual/Code Imagination (Same Search & Budget)52.048.044.0
WAP (Ours, w/ World-Model Visual Imagination)72.082.078.0

As shown in Rebuttal Table 2, while performing global optimization and local search via VLM code simulation improves over the direct VLM planner by 4.0% to 10.3%, WAP with visual world-model imagination achieves a further 20.0% to 34.0% improvement across all settings. This confirms that the search structure alone is insufficient: simulating end-effector trajectories in code and text cannot capture fine-grained visual geometry and physical contacts as effectively as our action-conditioned world model.


Response to Question 1 (Q1): Performance on Diverse Action Patterns

Beyond standard pick-and-place motions, WAP handles diverse manipulation skills, including 6-DoF orientation alignment (e.g., rotating the gripper to grasp the long edge of the cream cheese box in Fig. 21), side-grasping mugs by their rims/handles, and articulated object manipulation (Put the Black Bowl in the Bottom Drawer and Close It).

To further evaluate WAP on more diverse action patterns, we tested an additional multi-stage task from LIBERO-Long: Turn on the Stove and Put the Moka Pot on It. Unlike top-down pick-and-place, this task requires non-standard trajectory control, including rotational/pushing manipulation to turn on a stove knob followed by grasping a moka pot by its angled handle and placing it onto the burner. As shown in Rebuttal Table 3, WAP achieves substantial gains over the VLM planner baseline across both articulated and multi-skill tasks (and strongly outperforms π0.5\pi_{0.5}, which achieves only 6% on the stove task as shown in Table 12 of the revision).

Rebuttal Table 3: Performance on Diverse Manipulation Skills and Action Patterns (Success Rate %)

TaskManipulation Skills Involvedπ0.5\pi_{0.5} (VLA)VLM PlannerWAP (Ours)
Turn on Stove & Put Moka Pot on ItKnob rotation/pushing + complex grasp6.024.058.0
Put Bowl in Bottom Drawer & Close ItPrecision placement + articulated drawer pushing44.048.072.0

Dear Reviewer oADL,

As the discussion period draws to a close, we wanted to briefly follow up to see if our response and revised manuscript have addressed your questions, including our new zero-shot real-robot tasks, evaluations on diverse manipulation skills, and the ablation comparing world-model visual imagination against VLM search with text simulation.

Please let us know if you have any remaining questions or concerns we can clarify before the deadline. Thank you again for your time and helpful feedback!

Best regards,

The Authors

Reviewer_tFVROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduce World Action Planner (WAP), an agentic planning framework that plan through action-conditioned world model. The authors proposed coarse-to-fine search, which first let agent reason imagined world-model rollout and then perform local action search by comparing the neighboring candidates. The author also includes the theoretical analysis of the generalization of model-based planning and imitation learning.

Critical strengths
  • S1: WAP shows a strong performance across generalization setting, including novel object and compositional tasks. The method can compose previously learned skills to solve longer-horizon tasks and adapt to new object positions without task-specific demonstrations.
Critical weaknesses
  • W1: WAP requires prior knowledge of object dimensions to determine step sizes. In the implementation, the local step size requires object’s size information. Also, the global step size requires the object’s height. Both parameters require manually tuning for each task. The grasping and releasing skill rely on hard-coded top-down motions, limiting the method’s generality to different manipulation requirements.

  • W2: WAP relies on the VLM’s visual grounding ability to identify corresponding pixel across camera views and triangulate the 3D target position. However, this requires the target to be visible in at least two camera views with accurate pixel correspondences. If the object is only visible from only one camera, triangulation may fail or produce inaccurate 3D positions. This raises concerns about generalizability in cluttered environments.

Questions for authors
  • Why were only 10 real-world trajectories collected, and why were all of them failed episodes? How does the amount of collected data affect performance?

  • Have the authors tried WAP without fine-tuning on the 10 real-world trajectories? How much of WAP’s success depends on this adaptation rather than the planning method itself as the 10 real-world trajectories are all failed episodes.

  • How robust is the triangulation when the VLM identifies inconsistent pixel locations across camera views?

4 comments
AuthorsOpenReview ↗

Response to Reviewer tFVR (1/3)

We thank the reviewer for their constructive feedback and insightful comments. Below, we address each concern regarding search scale hyperparameters, VLM triangulation robustness, real-robot sample efficiency, and zero-shot generalization.


Response to Weakness 1 (W1): Prior Knowledge Requirements and Action Primitives

  • No Exact Prior Object Dimensions Required: WAP does not require exact object dimensions or manual per-task tuning. The search step sizes only require rough, order-of-magnitude estimates, which the VLM can estimate directly from camera observations (Appendix C.2.3).
  • Insensitivity to Search Scale Hyperparameters: As shown in Appendix D.4.3 (Table 20, Page 59 of the revised manuscript), WAP is highly robust to variations in both local and global search scales. Varying step sizes by 20% to 40% yields consistent success rates across all tasks:

Rebuttal Table 1 (Table 20 in Revision): Robustness to Local and Global Search Scales (Success Rate %)

Local Search Scale Ablation (Compositional Tasks)

Local Search ScalePnP soup & tomato saucePnP white mug & yellow-white mugPnP mug & puddingPnP soup & cheese
0.0466667472
0.05 (Default)72687870
0.0770727466

Global Search Scale Ablation (New-Layout Tasks)

Global Search ScalePnP soupPnP cheesePnP salad dressingPnP ketchupPnP milkPnP pudding
0.04788086628076
0.06 (Default)888690668478
0.08868284709080
  • Generality of Manipulation Skills: In our Compositional and New-Layout settings, WAP does not use hard-coded primitives; instead, it flexibly composes learned diffusion policies for complex, non-top-down skills such as side-grasping mugs or closing drawers (Section 3.2.1). Simple GRASP and RELEASE primitives are used only in the Novel Task settings where zero expert demonstrations exist (Appendix D.2.3). Even there, these primitives are defined purely in the gripper frame without object- or scene-specific assumptions, while all spatial decisions (approach position, 3D ROTATE orientation, and placement pose) are synthesized by the planner and reused unchanged across simulation and real-robot tasks.

Response to Weakness 2 (W2): Partial Occlusions and Imperfect Multi-View Triangulation

WAP does not require accurate VLM grounding or triangulation, as the triangulated 3D point serves only as a coarse initial proposal. Even when an object is only clearly visible in a single camera view and occluded in others, we prompt the VLM to leverage its 3D spatial reasoning to approximately point to the inferred object location in the occluded views. While this naturally introduces larger initial triangulation errors, correcting such grounding inaccuracies is a core motivation for our world-model-based search framework.

Rather than executing the triangulated coordinate directly, WAP simulates the candidate trajectory in visual space using our multi-view action-conditioned world model. This allows the agent to directly observe spatial misalignments in unoccluded views and correct them via global optimization and local search (consistent with Section 4.2, where initial proposal errors ϵAgent.Proposal\epsilon_{\text{Agent.Proposal}} within the search reach hglobal+hlocalh^{\text{global}} + h^{\text{local}} are absorbed). For instance, in StackCube, where the top-down view is occluded by the robot arm and held cube, the VLM's initial multi-view triangulation has a noticeable offset, but inspecting the imagined rollout in the front and side views enables the agent to detect and correct the misalignment before execution (Fig. 13 in the revision). We provide quantitative verification of this error recovery in Q3 (Rebuttal Table 4) below.

AuthorsOpenReview ↗

Response to Reviewer tFVR (2/3)

Response to Question 1 (Q1): Real-Robot Trajectory Count, Failure Episodes, and Sample Efficiency

  • Why only 10 trajectories were collected: Real-world execution is costly, making sample efficiency a primary design goal. We collected just 10 autonomous rollouts to demonstrate that our pose-image-conditioned world model can rapidly adapt to a new real-world domain with minimal physical interaction.
  • Why all 10 trajectories were failures: To avoid expensive human teleoperation, we collected these trajectories autonomously using zero-shot VLM proposals. Because standalone VLMs reason semantically rather than physically, they fail to account for 3D obstacle clearance (colliding with the basket rim) and suffer from uncorrected localization errors (Appendix D.2.4, Fig. 14). This highlights a key practical advantage of WAP: unlike imitation learning policies that require expert demonstrations, our world model only needs to learn local environment dynamics and visual appearance. It can learn effectively from cheap, autonomous failure trajectories and still accurately imagine out-of-distribution successful maneuvers at test time (Fig. 17).
  • Impact of data volume on performance: Below, we evaluate WAP across varying numbers of autonomous adaptation trajectories from 0 to 40:

Rebuttal Table 2: Real-Robot Performance Across Different Numbers of Adaptation Trajectories

Number of Adaptation TrajectoriesSuccess Rate (%)
0 (Zero-shot sim-to-real)20
545
10 (Reported in paper)65
2075
4080

WAP's performance scales smoothly with additional interaction data, reaching 75% to 80% with 20 to 40 trajectories. While prior end-to-end policies typically require 50+ human-teleoperated successful demonstrations for real-world fine-tuning, WAP achieves 65% success using only 10 autonomous failure rollouts.


Response to Question 2 (Q2): Zero-Shot Real-Robot Generalization Without Task-Specific Adaptation

First, on Put the Can in the Basket (Rebuttal Table 2), evaluating the simulation-pretrained world model directly on the real robot without fine-tuning (0 trajectories) achieves 20% success (vs. 10% for the VLM baseline). Because the base world model was pretrained purely in simulation (LIBERO-90), the 10 failure trajectories primarily bridge the visual sim-to-real domain gap (camera viewpoints, lighting, and appearance) rather than teaching task-specific solutions.

To decouple visual domain adaptation from task-specific planning, we conducted new, fully zero-shot real-robot experiments on two unseen tasks (Section 5.2, Page 9, Table 4, Fig. 5, and Appendix D.2.4, Figs. 18–19 of the revised manuscript):

  1. Cube-in-Cup: Placing a black cube into a narrow green cup.
  2. Cup Covering: Placing an upside-down large dark-green cup over an upside-down small light-green cup to cover it.

We directly reused the world model checkpoint from can-to-basket as-is, collecting zero new trajectories and performing zero parameter updates, even though the camera placement, lighting, objects, and task objectives differ entirely from the adaptation data.

Rebuttal Table 3 (Table 4 in Revision): Zero-Shot Real-Robot Transfer to Unseen Tasks (20 Trials per Task)

TaskVLM Planner BaselineWAP w/ Global Opt. OnlyWAP (Ours, Full)
Cup Covering10%45%55%
Cube-in-Cup15%40%65%

As shown in Rebuttal Table 3, WAP generalizes zero-shot to these unseen tasks and objects, outperforming the VLM baseline by 45% to 50%:

  • Global Optimization: The VLM baseline frequently lowers the object onto the target cup's rim, knocking it over (10% to 15% success; Fig. 5b). Global optimization foresees the rim collision and raises the placement pose, boosting success to 40% to 45%.
  • Local Search: In Cube-in-Cup, raising the pose alone can leave the release point slightly off-center, causing the cube to bounce off the rim (Fig. 5c). Local search evaluates imagined lateral offsets and selects the best-aligned candidate, increasing success to 65% (Fig. 5f).

These results confirm that the 10 failure episodes serve only as a one-time visual calibration to the real workspace, while the gains come from WAP's coarse-to-fine world-model planning.

AuthorsOpenReview ↗

Response to Reviewer tFVR (3/3)

Response to Question 3 (Q3): Quantitative Robustness to Inconsistent VLM Triangulation

Building on W2, we quantified how WAP recovers from inconsistent VLM pixel grounding on the precision-intensive StackCube task, where top-view occlusion degrades multi-view triangulation.

As shown in Rebuttal Table 4, initial VLM triangulation suffers from a large average 3D position error of 0.110 (over twice the target cube width of 0.05), causing frequent misses and a 22.0% success rate. By refining the proposal through global and local world-model search, WAP reduces the position error by 68% (to 0.035), well within the cube's support region, and increases success to 76.0%. This confirms that world-model search effectively compensates for noisy VLM triangulation.

Rebuttal Table 4: Localization Error and Success Rate on StackCube (Target Cube Size: 0.05)

MethodTarget Position ErrorSuccess Rate (%)
VLM Planner Baseline (Initial Triangulation)0.11022.0
WAP (After World-Model Search)0.03576.0

Dear Reviewer tFVR,

As the discussion period draws to a close, we wanted to briefly follow up to see if our response and revised manuscript have addressed your questions, including our experiments on search-scale robustness, VLM triangulation error recovery, real-robot data scaling, and zero-shot real-robot transfer.

Please let us know if you have any remaining questions or concerns that we can clarify before the deadline. Thank you again for your time and helpful feedback!

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.