acceptodds
Withdrawn from ICLR 2027

Rethinking World-Action Model for Compositional and In-Context Robotic Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-horizon compositional manipulation has become increasingly important for real-world robot deployment, where a single task involves multiple coordinated subtasks. Existing world-action models (WAMs) jointly predict short-horizon visual futures and actions, but typically lack explicit subtask-level reasoning. We propose **Visual Goal-conditioned Action Reasoning (ViGAR)**, a hierarchical framework that factorizes manipulation into a visual subgoal planner and a subgoal executor. Given the current observation and global instruction, the subgoal planner predicts a visual subgoal for the next subtask. The subgoal executor then jointly generates future visual trajectories and actions conditioned on the predicted subgoal. Both components share a pretrained world-model representation, enabling task-level planning and action generation to benefit from common physical knowledge. Moreover, our framework naturally supports in-context learning: using a global goal image as context can induce different subtask decompositions and behaviors without parameter updates. On the RoboTwin Clean2Random benchmark, ViGAR achieves 82.00% and 67.02% success rates under the *Clean* and *Random* settings, respectively, surpassing the strongest baseline by 12.81% in average success rate. Real-robot experiments on five compositional and two in-context learning tasks further confirm the effectiveness of ViGAR.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotics manipulation, World-action model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iXodOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ViGAR, a hierarchical framework for long-horizon compositional robotic manipulation that separates visual subgoal planning from subgoal-conditioned execution. A planner predicts intermediate goal images from the current observation and task instruction, while a world-action policy jointly generates future video and actions conditioned on these subgoals. Both modules are initialized from Cosmos3-Nano. The method introduces subgoal look-ahead supervision to improve stage transitions and end-effector ROI weighting to emphasize manipulation-relevant regions. An optional global goal image enables different task compositions without parameter updates. Experiments on RoboTwin Clean2Random, five real-world manipulation tasks, and two goal-conditioned composition tasks demonstrate improvements over the reported baselines, with additional evaluations of subgoal prediction quality and key design components.

Critical strengths
  1. Clear and practically motivated hierarchical design. ViGAR separates visual subgoal planning from action generation, providing an explicit connection between task-level planning and execution. The subgoal look-ahead mechanism also addresses a meaningful challenge in long-horizon manipulation: avoiding regression to already completed subtasks.

  2. Strong experimental results and useful validation. ViGAR achieves 67.02% success on RoboTwin Random, compared with 22.46% for the controlled Cosmos3-Nano baseline. The inclusion of this baseline, together with subgoal-quality evaluations, component ablations, and real-world experiments, provides encouraging evidence for the practical value of visual subgoal guidance.

Critical weaknesses
  1. Inconsistent evaluation metrics for in-context learning. The reported task progress scores in Table 5 cannot be reproduced using Equation (5). The stage-wise success rates yield 0.35 for both OOD Fruit Arrangement and Desktop Item Storage, rather than the reported 0.43 and 0.52. Without clarification of the normalization procedure and stage definitions, the reported 47.5% average OOD progress remains difficult to verify, weakening a central experimental claim.

  2. Insufficient training controls and unclear compositional generalization. The planner receives substantial additional training, while the data available to each baseline and the procedure for replacing incorrect predicted subgoals with ground truth are not sufficiently documented. Furthermore, the planner is explicitly trained with global goal images, and the exact held-out task compositions are unclear, making it difficult to distinguish genuine compositional generalization from standard goal-conditioned behavior. The aggregate RoboTwin results also combine 19 multi-stage and 31 single-stage tasks without separating their performance, limiting evidence for the claimed long-horizon benefits.

Questions for authors
  1. Task progress metric (clarification). Could the authors reconcile Equation (5) with Table 5? The reported stage-wise success rates yield 0.35 for both OOD tasks, rather than 0.43 and 0.52. If task lengths vary by goal, how are individual trials normalized, and how is the reported 47.5% average OOD progress calculated?

  2. Long-horizon performance (additional analysis). Can the authors report the existing RoboTwin results separately for the 19 multi-stage and 31 single-stage tasks? This would help determine whether the improvements primarily come from better subgoal-conditioned control or better coordination across subtask transitions.

  3. Training data and controls (clarification). What training data are available to the planner, executor, and controlled baselines? How are incorrect predicted subgoals identified and replaced with ground-truth images during policy training, and what fraction of predictions is replaced?

  4. In-context generalization (clarification). Which exact goal compositions were included in training and which were held out for the two in-context tasks? How does the evaluation distinguish generalization to unseen task compositions from standard goal-conditioned behavior, given that the planner is explicitly trained with terminal goal images?

Reviewer_444zOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses a common limitation of existing world-action models (WAMs): the lack of explicit subtask-level reasoning. The authors propose Visual Goal-conditioned Action Reasoning (ViGAR), which separates manipulation into visual subgoal planning and subgoal execution. An additional planner predicts a visual target for the next subtask to guide the WAMs. The authors further claim that the design of ViGAR supports in-context learning. ViGAR achieves strong results on RoboTwin Clean2Random and reports improvements on real-world manipulation tasks.

Critical strengths
  1. The motivation is clear and relevant: extending WAMs beyond short-horizon action prediction by introducing explicit subgoal planning.
  2. The overall approach is clearly presented, and the method performs well in the RoboTwin C2R setting.
Critical weaknesses
  1. Limited novelty and insufficient analysis. The use of subgoal images offers limited novelty, given existing approaches such as 𝜋0.7[1] and VISTA[2] (both of which are cited in this paper). In addition, (1) there is also no comparison establishing whether visual subgoals offer an advantage over textual subgoals for a WAM. (2) The criteria for consistent subtask segmentation need further clarification. (3) The paper provides limited analysis of the reliability of stage transitions during inference and the planner’s ability to handle OOD states and recover from execution failures, which are important in real-world deployment.

  2. Insufficient evidence for the claimed in-context learning capability. The global goal image appears to act as an additional conditioning input, and the existing experiment does not clearly distinguish this from conventional goal-conditioned generalization. The authors should clarify the relationship to the typical in-context learning methods. The real-world ICL experiments in Figure.5 also lack baseline comparisons and sufficient analysis to establish whether the robot follows the supplied goal rather than relying on cues from the initial configuration.

[1]𝜋0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilities [2]Scaling World Model for Hierarchical Manipulation Policies

Questions for authors
  1. Please explain what evidence shows that the model acquires a new in-context capability, rather than generalizing to new task compositions through ordinary goal conditioning.

  2. Both the subgoal planner and executor are adapted from Cosmos3-Nano, a 16B-total/8B-active-parameter model. Considering the size of these models, please report the end-to-end latency, including both subgoal generation and action prediction. How do these computational costs affect real-world execution, and how are inference delays handled?

  3. ViGAR is mid-trained on approximately 900 hours of real-robot data before task-specific adaptation in real-world experiments. Were π0.5 and Cosmos3-Nano-Policy also mid-trained on the same data before finetuning?

  4. In the visual subgoal prediction comparison, ViGAR substantially outperforms RxBrain and VISTA on all four metrics. Were these baselines adapted using the same training data and comparable training conditions, or were they evaluated zero-shot? Please clarify their training settings and discuss what accounts for the large performance gap.

Reviewer_jNMdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a hierarchical framework, ViGAR. A visual subgoal planner predicts an intermediate target image, and a world-action policy uses this image to generate future video frames and actions. The framework also accepts a final goal image at inference time to guide new task combinations without updating model parameters.

Critical strengths
  1. The simulation gains are substantial, especially on the Random setting.
Critical weaknesses
  1. The in-context learning claim needs stronger comparisons. The planner is trained with final goal images and receives a new goal image at test time. This is a reasonable way to specify tasks. However, changing the goal without updating parameters is also expected behavior for a goal-conditioned policy. The two ICL tasks mainly change object selection, placement counts, and object-to-container assignments. These results support generalization to new goals, but provide limited evidence for broader compositional planning.

  2. Some important implementation and evaluation details are missing. Appendix B states that incorrect or distorted predicted subgoals are replaced with ground-truth subgoals when training the policy. The paper does not explain who makes this decision, what criteria are used, or how often replacement occurs. This matters for reproducibility and for understanding how the policy handles planner errors at deployment. For the real-robot experiment, absolute inference times are also missing. Since the generation of sub-goal images clearly would introduce newly-added time cost.

Questions for authors
  1. How are the OOD progress scores calculated?

  2. How does ViGAR compare with a direct policy that receives the same final goal image but does not generate intermediate subgoals? How are unseen goal combinations excluded from all relevant training datasets?

  3. What ratio of predicted subgoals is replaced by ground truth during training?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.