acceptodds
Under review as a conference paper at ICLR 2027

Agentic World Action Model

32 views

Abstract

Recent world action models provide visual foresight and future prediction for robot manipulation, but lack explicit mechanisms for compositional skill reuse. Agentic robotics combines code-based planning and reasoning with reusable skills, but often lacks predictive grounding in physical dynamics. We introduce AW-0, an agentic world action model framework that integrates code-based planning with joint video and action prediction. We jointly model code, future video, and robot actions as three coupled generative streams within a unified mixture-of-transformers architecture. Within this architecture a code expert generates parameterized skill routines and control logic while video and action experts generate future visual states and low-level motor commands. Code serves two complementary roles: it structures multi-step prediction and control within the world model and represents skills as reusable programs for subsequent agentic execution. We further introduce an automated curation pipeline that labels robot and human video trajectories with aligned programs and consolidates recurring routines into reusable skill libraries. Evaluations on RoboTwin 2.0 and physical bimanual manipulation tasks show improved task performance over VLA and WAM baselines, alongside gains in language following, semantic reasoning and robustness, demonstrating that agentic code enables connection between planning, prediction and execution within WAMs.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world action models, agentic robotics, robotics foundation model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_KLJmOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

he paper proposes AW-0, a world action model (WAM) that adds a “code expert” to a video-plus-action generative policy. A Mixture-of-Transformers backbone holds three experts: a code expert (a diffusion LLM with a frozen Qwen3-VL ViT), a video expert (Cosmos-Predict2.5-2B), and an action DiT. They are denoised jointly: masked discrete diffusion for code, flow matching for video and actions, with blockwise-causal attention ordered code -> video -> action. The code consists of Python “skill programs” written against primitive robot APIs. An agentic pipeline produces these labels: Gemini 3.1 Pro writes them for demonstration slices, and a Claude Opus 4.8 coordinator distills recurring patterns into a frozen library of 50 skills (23 for YAM, 27 for egocentric human data). The model is pretrained on about 2,900 hours of YAM teleoperation and egocentric human video annotated this way. By default the generated code is not executed and serves only as conditioning. An optional “hybrid” mode runs the code Code-as-Policies style when it passes validity gates, and otherwise falls back to the action chunk. On RoboTwin 2.0 (50 tasks), AW-0 reports 80.8% clean and 20.6% randomized success, versus 64.8% and 27.7% for the strongest respective baselines. On 8 real-world tasks across two embodiments (parallel grippers and 22-DoF dexterous hands), it reports a 71.1% average rubric score versus 46.0% for the best baseline. Ablations cover attention ordering, ViT vs. VAE conditioning, label content (comments, skills, language subgoals) and joint vs. autoregressive decoding, the last with latency measurements.

Critical strengths
  1. Putting programmatic skill planning inside a jointly denoised video-action model, rather than wrapping a policy with an external agent, is a fresh idea. Code serves both as conditioning and as an optionally executable artifact.

  2. C->V->A beats V->C->A and V->A. Full code beats code without comments, code without skills, and natural-language subgoals on most tasks. This gives fairly direct evidence that structured code, and not just extra language supervision, helps.

  3. Joint denoising seems to outperform autoregressive code generation on 7 of 8 tasks while being 2.4× faster, and it seems to add only 0.04 s over the code-free WAM.

  4. The tasks span two quite different embodiments, including dexterous 22-DoF hands, and include tasks needing conditional logic and counting (Replace Lightbulb, Flip Scoreboard, Arrange Fruits), where the semantic benefit of code is plausible.

  5. Appendix openly reports that code drives only 5–9% of steps overall, and that hybrid mode costs 17.7 points in the clean setting.

  6. The annotation pipeline seems to covers about 2.07M subtask slices, with a documented coordinator protocol, skill-admission criteria, and cache-based reuse.

Critical weaknesses
  1. By my count, 36 of the 50 references are cited only as arXiv preprints, 12 point to peer-reviewed venues, and 2 are industry model cards. This matters because the arXiv-only set includes the things the paper’s validity rests on: a) every WAM and VLA baseline it reimplements (DreamZero, EgoScale, WLA); b) the evaluation benchmark (RoboTwin 2.0); c) core building blocks (SDAR, Cosmos-Predict2.5, Qwen3-VL, Wan2.2, MoT); d) the explanations for two key results. Both the drop under randomization and the ViT-over-VAE result are justified solely by Yan et al. (arXiv 2608.05903), a preprint posted roughly one month before submission. Several cited preprints (2606., 2607., 2608.*) are essentially concurrent and unvetted. Separately, several works that are peer-reviewed are cited as arXiv only. Positioning against unreviewed work weakens the novelty and SOTA claims, and the bibliography needs a careful pass.

  2. AW-0 is pretrained on about 1,900 hours of YAM data, the same embodiment as half the real-world tasks, while baselines start from their public checkpoints. Training budgets also differ widely: DreamZero gets 1 epoch per task, while π0.5, EgoScale and WLA get 50 epochs. The 25-point real-world gap therefore cannot be attributed to the method. This suggests that there is a confounded comparison with baselines.

  3. The hybrid-mode comparison seems not controlled. Indeed, in RoboTwin, hybrid mode is trained on demonstrations collected by executing CaP programs, which differ from the standard demos used for the default mode and the baselines. The real-world hybrid evaluation covers only one task.

  4. The headline simulation claim depends on averaging. Under randomization, AW-0 is below π0.5. “Highest average” seems to come from combining clean and random scores. The robustness claim in the abstract rests on hybrid mode, whose training data differs (see above).

  5. The code-free variant results to be below the DreamZero baseline, yet adding code gives a gain. This raises the question of whether the code labels carry privileged structure: sensor-derived gripper-event sequences, explicit “STAGE k/N” headers, and loop counts read from frames.

  6. The quality of the produced code seems quite low, and the quality of the annotations seems unmeasured. Indeed, the pretrained code expert has an entity F1 of 0.534 and a primitive F1 of 0.365. No human audit of the Gemini/Claude labels is reported, even though labels contain “[auto-repair]” insertions. The method’s success may come from code acting as a structured conditioning signal rather than as a correct plan, which undercuts the “agentic reuse” framing.

  7. The paper lacks of a proper statistical rigor. Indeed, the paper does not seem to report any seeds, standard errors or significance tests. Real-world metrics are additive partial-credit rubrics, not success rates. Some task names overstate difficulty: “Solve Rubik’s Cube” is a single quarter-turn un-scramble.

  8. The pipeline depends on proprietary frontier models (Gemini 3.1 Pro, Claude Opus 4.8), a private 2,900-hour corpus, and 48-GPU pretraining, and there is no release commitment for data, labels or code. THus, the reproducibility is questionable.

  9. The skill-reuse claim is under-tested. The YAM and human libraries are disjoint. Cross-embodiment or cross-task reuse of skills, and agentic re-execution beyond one real task, are not demonstrated.

Questions for authors

Q0) Please address all the criticisms above if not properly transformed in questions below.

Q1) Could you replace preprint citations with peer-reviewed versions where they exist? For the claims that currently rest only on recent arXiv work (pixel-level OOD degradation, VAE features being poorly suited to semantic tasks), could you provide your own supporting experiments?

Q2) Were the V->A, V->C->A, subgoal and AR ablations all pretrained on the same 2,900-hour corpus with identical compute? If not, what does each variant see during pretraining?

Q3) Can you report baselines fine-tuned (or ideally co-pretrained) on the same YAM pretraining data, and with matched epochs and compute, so that the method contribution can be separated from the data contribution?

Q4) How much of the code gain survives if you strip privileged information from the labels (gripper-event sequences, stage indices, frame-derived loop counts), keeping only skill calls?

Q5) For RoboTwin hybrid mode, what is AW-0’s default-mode performance when trained on the same CaP-collected demonstrations? Conversely, how do baselines perform when trained on them?

Q6) What fraction of automatically generated labels is correct according to a human audit on a random sample? How often does the “[auto-repair]” path trigger?

Q7) Please report standard errors over rollouts or seeds, binary task-success rates alongside rubric scores for the real-world tasks, and the scrambled-cube distribution used for “Solve Rubik’s Cube”.

Q8) The memory cache (K slots, replacement probability p) and the noise-resampling mixture (α = 0.5) are not ablated. What are their individual contributions?

Q9) How many denoising steps and code-canvas tokens underlie the 0.23 s latency, and how does performance trade off against fewer steps?

Q10) Is any skill invoked across embodiments or on unseen downstream tasks? Can the library be extended at fine-tuning time without retraining the code expert?

Q11) Which diffusion LLM initializes the code expert (LLaDA or SDAR)? Which parameters are frozen versus trained during downstream fine-tuning?

Q12) Will the annotated program dataset, skill library, API definitions and checkpoints be released?

Reviewer_xPe1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AW-0 combines code generation, future video prediction, and robot action generation through joint denoising. It also introduces an automated program annotation pipeline and an optional hybrid execution mode.

Critical strengths

1, Using programs to connect task planning and action prediction is an interesting idea.

2, The real-world results across two robot platforms are promising. The ablations on code content, visual conditioning, and generation strategy are useful.

Critical weaknesses

1, The evidence for compositional generalization is limited. The model learns from programs generated by frontier models using a fixed library of 50 skills. It is unclear how well it handles unseen skill combinations or tasks outside this library. More detail on the skill definitions, selection process, and overlap with evaluation tasks would help.

2, Code reliability needs closer evaluation. The paper mentions validity and progress checks, but it is unclear how these handle programs that are syntactically valid but semantically wrong. Also, the fallback action branch is itself conditioned on code, so an incorrect plan could affect both execution paths. Code error rates and recovery results would be useful.

3, The contribution of future video prediction is not fully isolated. A code-to-action baseline with the same visual inputs and program supervision would help establish whether the gains require world modeling, beyond the added semantic planning branch.

4, Robustness remains a concern. Default AW-0 drops from 80.8% to 20.6% under visual randomization. Hybrid execution improves the random setting but substantially reduces clean performance, so its benefits appear dependent on the evaluation setting.

Questions for authors

see above

Reviewer_ptEnOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces AW-0, a framework that combines code-based planning with joint video and action generation. An automated annotation pipeline converts demonstrations into aligned programs and reusable skills, while an optional hybrid execution mode combines program execution with predicted actions. Experiments on RoboTwin 2.0 and real-world bimanual tasks demonstrate strong performance. However, additional controlled comparisons would help establish the contributions of video prediction, hybrid execution, and joint denoising.

Critical strengths
  • Provides an interesting integration of structured program generation, video prediction, and action modeling.

  • Demonstrates strong simulation and real-world results, including dexterous manipulation.

  • Includes useful ablations on attention structure, visual conditioning, and program content.

Critical weaknesses
  • The contribution of video prediction is not sufficiently isolated. Figure 5 compares different attention orderings and a variant without code, but does not establish whether video prediction provides additional benefits once code planning is available. A code-to-action baseline without the video expert would clarify whether video generation improves task performance enough to justify its computational cost.

  • The benefits of hybrid execution require further disentanglement. In Table 1, hybrid execution improves success under randomization from 20.6% to 29.2%, but reduces clean-setting success from 80.8% to 63.1%. The appendix discusses this trade-off and reports control-channel usage, but a code-only execution baseline using the same generated programs and primitive library is missing. Such a comparison would help distinguish the contributions of learned actions, program execution, and the switching mechanism.

  • The autoregressive baseline is insufficiently specified. Table 4 reports substantial performance and latency improvements over AR, but its backbone, initialization, attention structure, and decoding procedure are not clearly documented. Although the paper identifies teacher-forced AR training, the available details are insufficient to determine whether the comparison isolates the effect of joint denoising or also reflects other architectural and training differences.

Questions for authors
  • Could the authors evaluate a code-to-action model without video prediction under comparable training conditions, reporting both task performance and inference cost?

  • How does code-only execution compare with direct-action and hybrid execution in clean and randomized settings, using the same model checkpoint and primitive library?

  • What is the exact AR baseline configuration? Are initialization, training data, output lengths, and video/action sampling budgets matched, and how does generated code condition the video and action outputs?

1 comment

Thank you for the clarifications and additional experiments, which substantially address my main concerns. I will maintain my positive recommendation.

Reviewer KLJm also raised important concerns about the different training datasets used in the original execution-mode comparison and the use of rubric scores in the real-world evaluation. The authors’ controlled comparisons and additional binary success rates with uncertainty estimates provide essential context for interpreting the results. I strongly encourage incorporating both responses into the revised manuscript, together with the updated robustness discussion and clarified task definitions.

Reviewer_EifKOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AW-0, which integrates code-based skill planning, future video prediction, and robot action generation within a unified Mixture-of-Transformers architecture. Through joint denoising, code provides planning guidance for video and action generation. The paper also introduces an automated program annotation and skill discovery pipeline, along with a hybrid execution mechanism combining program execution and action-chunk prediction. Experiments on RoboTwin 2.0 and eight real-world tasks across two robotic platforms demonstrate improved performance.

Critical strengths
  1. The method is coherent and practically relevant. Incorporating structured skill planning into world action models provides a viable approach to connecting high-level task reasoning, future-state prediction, and low-level control.

  2. The real-world experiments are compelling. The evaluation covers both parallel grippers and dexterous hands. AW-0 achieves an average score of 71.1% across eight real-world tasks, substantially exceeding the strongest baseline's 46.0%.

Critical weaknesses
  1. Although all methods use the same downstream demonstrations, AW-0 additionally introduces a code expert, a separate visual encoder, extra program supervision, and large-scale pretraining. The ablations support the effectiveness of these components, but do not sufficiently establish how much of the overall gain comes from code-based planning itself versus model capacity, pretraining, or additional supervision.

  2. The automated annotation pipeline relies on program generation, skill abstraction, and cache reuse based on contact and motion patterns. Errors in these processes may compromise the quality of planning supervision.

  3. In the default mode, generated code is not executed but instead conditions video and action prediction. The resulting improvements therefore primarily demonstrate the value of code as a planning representation, rather than sufficiently establishing the model’s ability to reliably invoke and compose executable skills.

Questions for authors
  1. Could you isolate the contribution of code-based planning by controlling for model capacity, pretraining data, and additional supervision?
  2. How do you assess the correctness and temporal alignment of program annotations, including errors introduced by skill abstraction and cache reuse?
  3. Can you provide direct evaluations of generated programs’ execution success and generalization to unseen skill compositions, beyond their use as conditioning representations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.