acceptodds
Withdrawn from ICLR 2027

World Action Agent: Harnessing VLM Plays Robots via World Action Rehearsal

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

General-purpose vision-language models (VLMs) bring broad knowledge and spatial reasoning to robot manipulation, yet existing systems either use them indirectly, to predict constraints or write programs, or give them a view of the scene rather than a world in which to act. We present World Action Agent (WAA), a multi-agent harness through which VLMs pilot robots with basic tools, making every decision within a visual action workspace. The workspace has three properties. Contact views, selected automatically from the scene geometry, present the scene around the current interaction. Action rehearsal turns each action into an editable proposal that the agent, alone or through an Imagination Agent, previews and revises against planning feedback before execution. In-view correction closes the loop between observation, rehearsal, and low-level execution, letting the agent remove residual offsets in the view where it observes them. Through the same workspace, WAA acquires embodied procedural knowledge in two ways: it evolves multimodal skills from expert videos and human teaching under evidence-based review and consults them through a Skill Agent, and its interaction traces train smaller VLMs to pilot the same harness. On LIBERO-Pro, WAA with skills evolved only from LIBERO-90 reaches a state-of-the-art 75.6% average success, outperforming end-to-end VLAs, code-as-policy agents, and a visual-harness baseline with the same backbone; the same skills remain effective on robosuite without further learning. Fine-tuning Qwen3.5-9B on harness traces raises its out-of-domain success from 1.7% to 43.3%.

Primary area:
applications to robotics, autonomy, planning
Keywords:
VLM Agent, Embodied Agent

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_r17pOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents World Action Agent (WAA), a VLM-based harness for robot manipulation combining interaction-centered views, action previews with planning feedback, and closed-loop visual corrections. It also supports multimodal skill learning and smaller-model training from interaction traces. Experiments report 75.6% average success on LIBERO-Pro using skills learned from LIBERO-90, transfer of frozen skills to robosuite, and improved performance after fine-tuning a Qwen3.5-9B main agent.

Critical strengths
  1. A comprehensive and coherent system

    WAA integrates multi-view observation, action rehearsal, in-view correction, and skill guidance into a unified robot-control framework. The system addresses multiple practical challenges across the manipulation workflow, giving the work meaningful engineering value.

  2. Substantial gains on spatial tasks

    WAA reports 80.0% and 73.3% success on the two LIBERO-Pro Spatial splits, compared with ASPIRE’s reported 51.0% and 60.0%. These improvements provide concrete evidence of the system’s value for tasks requiring precise spatial alignment.

Critical weaknesses
  1. Insufficient ablation of the core designs

    The paper lacks ablations of Contact views, action rehearsal, and in-view correction. Consequently, it is difficult to identify each component’s contribution to the reported gains or assess whether all three are necessary.

  2. Limited difficulty and coverage of the robosuite evaluation

    The evaluation includes only cube lifting, stacking, and restacking. Lifting and stacking already achieve 100% success without learned skills. This relatively simple and narrow task set provides limited evidence for skill transfer to more diverse and challenging manipulation tasks.

Questions for authors
  1. Additional evidence—core component contributions:

    Can the authors provide controlled ablations of Contact views, action rehearsal, and in-view correction, keeping the backbone and skill library fixed? These results would help establish which proposed mechanisms account for the performance gains.

  2. Clarification—robosuite evaluation:

    How many trials were conducted, and how varied were the initial states for each task? Given that lifting and stacking already achieve 100% success without learned skills, how broadly should the reported skill-transfer results be interpreted?

Reviewer_HjD8OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents WAA, a system that enables VLMs to control robots through local views, action previews with motion-planning feedback, and image-space corrections. It also constructs a multimodal skill library from demonstrations and fine-tunes a smaller VLM on interaction traces. Evaluation is conducted in LIBERO-Pro and robosuite.

Critical strengths
  • The system integrates local observation, action preview, and correction into a common interface. The examples make the intended workflow understandable.

  • The skill library improves performance over the no-skill and seed-skill settings. It is frozen before evaluation, and its reuse is tested on three robosuite tasks.

Critical weaknesses
  • The contribution of the main components is not established. There are no separate ablations of Contact views, action rehearsal, or image-space correction. The skill experiments also change both textual procedures and visual references. These comparisons therefore do not establish which components account for the improvement or whether separate Imagination and Skill Agents are necessary. The comparison with Show-Harness changes several aspects of the system simultaneously and cannot resolve this issue.

  • The practical applicability and manipulation limits are insufficiently evaluated. All experiments are conducted in simulation, including the RGB-D and VGGT variants. Since the method relies on reconstructed geometry and calibrated image-to-motion mappings, real-robot experiments are needed to assess robustness to sensing errors, calibration errors, and imperfect contact. The evaluated tasks also provide limited evidence about dexterous manipulation. The paper does not establish whether the interface can support coordinated finger motions, in-hand manipulation, or force-sensitive contact. These limitations should be discussed explicitly. Furthermore, although execution cost is reported, the average episode time of approximately 150 seconds leaves responsiveness in changing environments unresolved.

Questions for authors
  • Can you provide separate ablations of Contact views, action rehearsal, and image-space correction under the same backbone, skills, and interaction budget? How much of the skill improvement comes from evolved text versus visual references?

  • Have you tested the system on a real robot? If not, what evidence supports its robustness to depth errors, imperfect calibration, and contact disturbances?

  • What are the current limitations for dexterous manipulation and tasks requiring frequent contact adjustments? Please clarify whether the available action primitives and model latency can support such tasks, and report decision latency separately from total episode time.

Reviewer_EjMuOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The World Action Agent (WAA) is a system for using a vision-language model (VLM) to control a robot using tool calls. The WAA has local views so you can see how the object relates to the gripper; it can preview possible actions with motion planning feedback; and it allows for image-based corrections as the robot executes an action. Also, the WAA creates a multimodal skill library from demonstrations provided by experts which can be used to direct the agent. In simulation, the WAA achieved 75.6% average success on LIBERO-Pro and transferred the learned skills to three robosuite tasks. The authors also fine-tuned Qwen3.5-9B based on interaction traces collected to act as the primary agent.

Critical strengths
  1. The paper shows promising simulation results in general. On LIBERO-Pro, it has an average success rate of 75.6%, and it performs well in spatial tasks. Also, the use of a frozen skill library improves performance on robosuite restacking, indicating that there is some transfer between environments.
  2. The paper shows a successful skill learning component. This allows the average success rate to be increased on LIBERO-Pro from 28.9% to 75.6%.
Critical weaknesses
  1. The paper includes several different components (contact views, action rehearsal, in-view correction, and skill learning), however, it is unclear what amount of contribution each component makes. Ablation studies should be conducted for each component separately.
  2. The paper appears to be a combination of various methods from other papers; therefore, the authors should compare and contrast the components of their method with those of [1,2,3] and explain how their method's components differ from those of [1,2,3] and other existing methods.
  3. All evaluations were done in simulation (including both RGB-D and VGGT), thus it is unknown how well the proposed method will handle real sensor noise, calibration errors, and contact dynamics.

[1] IVE: Imagine, Verify, Execute.

[2] VoLo: A Physical Orchestrator for Open-Vocabulary Long-Horizon Manipulation.

[3] SAIL: Test-Time Scaling for In-Context Imitation Learning with VLM.

Questions for authors

See weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.