acceptodds
Withdrawn from ICLR 2027

Humanoid World Action Model With Joint State–Action Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Humanoid robots are a promising platform for general-purpose manipulation. Recent Vision-Language-Action (VLA) policies learn actions directly from multimodal observations, while World Action Models (WAMs) further incorporate future visual prediction to improve action generation. However, in hierarchical humanoid systems, VLA and WAM policies output reference actions that are subsequently realized through whole-body control, robot dynamics, balance, and contact. This hierarchy creates an action–execution gap: the reference produced by the policy can differ from the motion realized by the robot. Without explicitly modeling the realized body state, future visual prediction must jointly explain scene evolution and discrepancies between reference actions and executed motion, making it difficult to associate an action with its physical outcome. We propose HWAM, a Humanoid World Action Model with joint state–action generation, which makes the robot's post-execution proprioceptive state an explicit prediction target. By jointly generating reference actions and their realized body states, HWAM directly incorporates supervision of executed motion into action learning. HWAM is trained through three complementary conditional paths. The Policy path jointly denoises state–action trajectories conditioned only on current observations, matching deployment conditions. Forward Dynamics Modeling (FDM) predicts future visual observations conditioned on actions and post-execution states, while Inverse Dynamics Modeling (IDM) reconstructs the joint trajectory from visual transitions. Together, these paths connect policy references, realized body motion, and visual outcomes. HWAM achieves the highest success rate among evaluated baselines on three real-robot tasks on the LimX OLI humanoid, without pretraining on additional robot data. On Candy Picking, HWAM achieves a 70.6% success rate, compared with 62.5% for pi_0.5 and 43.3% for Fast-WAM.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Action Model, Vision-Language-Action Model, Embodied Intelligence, Humanoid Robot

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xBq9OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the phenomenon that, in hierarchically controlled humanoid robots, there exists a discrepancy between the high-level reference command a_t and the proprioceptive state s_{t+1} after actual execution by the low-level controller, and proposes HWAM (Humanoid World Action Model). The method concatenates the executed proprioceptive state onto the action stream of the World Action Model, and performs joint denoising generation of state and action; the training follows the three-way FDM / IDM / Policy objectives in the UWM / Cosmos3 style, the architecture interface references Fast-WAM, and the video backbone is based on Wan2.2. The authors first quantify the discrepancy between the reference and the execution on the LimX OLI platform (direct MAE 4.42°, steady-state 2.885°) as the motivation; subsequently, on the three real-robot tasks of OLI (Candy Picking, Object Collection, Plush-Toy Picking) and the ALOHA clothes-folding task, they compare with baselines including π0.5, Fast-WAM, Cosmos3-Edge, and DiT4DiT, report the highest success rates, and provide ablations of the state objective and the training paths, as well as video prediction PSNR diagnostics.

Critical strengths

The research question is genuine and meaningful. In hierarchical humanoid control, the high level outputs reference commands, while the low-level WBC/impedance control rewrites them; "command ≠ actual execution" is a real problem. The authors quantified this phenomenon and attempted to incorporate it into WAM modeling, and this direction is worth studying. The method is simple, easy to reproduce, and has real-robot evaluation. The method only concatenates state onto the action channel, making it easy to plug into existing WAMs. The experiments are conducted on two real platforms and four tasks, with a relatively large number of baselines, and Wilson confidence intervals are provided.

Critical weaknesses

The necessity of predicting state is never argued, and the core causal chain is missing. a_t and s_{t+1} are both ground truth and highly correlated (they differ by only about 4°), s_t is already part of the input, and the predicted s is discarded at deployment—so the state objective is likely just redundant auxiliary supervision. In the teleoperation data, a_t already implicitly absorbs the execution deviation, and the low-level controller stays the same at deployment; the authors never explain why s still needs to be modeled explicitly. Their own results do not support it either: adding the state objective under VGM/Policy drops the success rate from 60.0% to 45.5%, open-loop MAE improves by only about 5%, and the PSNR gain under GT is almost trivial. Beyond that, the deviation sources (control/contact/dynamics) are never decomposed, the per-joint statistics claimed in the main text are missing from the appendix, and there is no latency compensation.

Questions for authors
  1. Decomposition of the deviation sources: can the MAE be reported separately by joint groups (legs/waist, arms, dexterous hands) and by contact phase (pre-contact/contact)? Can the MAE after latency compensation (searching for the optimal time shift) be reported? And please supplement the per-joint statistics claimed in the main text but missing in the appendix.

  2. What is the specific dimensional composition of state? Apart from the 31 shared joints, does it include quantities that are not in action, such as base velocity, IMU, hand currents? (The wording "first 31 shared joints" implies that state is higher-dimensional.) If it does, state provides genuinely additional information, which would significantly change my judgment of the method's motivation.

  3. Please report the number of trials for each task and each configuration (including all ablations), and give the confidence interval or significance test for the 32.8-point interaction effect. Are there any multi-seed results?

  4. The data is collected by teleoperation, and the operator has already corrected the commands in a closed loop based on actual execution, and the low-level controller is also unchanged at deployment. In this case, compared with directly learning a by BC, what extra does explicitly modeling s provide?

Reviewer_XMvUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose creating a world action model for humanoids (HWAM) that jointly generates high-level action and next states, conditioned on current observations. This is done through an state-action expert diffution transformer. The HWAM also generates future video through a video expert diffusion transformer. Both diffusion transformers have mixed cross attention. The generated actions are then executed on a robot through a whole-body tracking controller. The model can be used as a forward dynamics model that takes current observation and predicted future actions and states to generate future video. The model can also be used as an inverse dynamics model that takes current observation and future video to generate future states and actions. Finally the model can be used as a policy that takes current observation and generates future states and actions. At deployment, the model is used as a policy. The paper presents experimental results on a LimX full-size humanoid on a candy picking task demonstrating higher success rate compared to the pi 0.5 VLA or the Fast-WAM. Experimental results are also presented on an ALOHA bimanual manipulator for cloth folding.

Critical strengths
  • Model can potentially be used as a forward dynamics / inverse dynamics / policy modes.

  • Proposed method has higher success rate than other methods like Cosmos, GROOT, pi 0.5, Dit4Dit, Fast-WAM, SmolVLA.

Critical weaknesses
  • Sec. 2 on related work does not provide a critical review. It just presents prior works but does not identify the critical gaps in prior work that this paper fills.

  • It would be nice to know what enabled the higher success rate for the proposed approach. Is it the joint state-action generation or is it the training of the model to be a forward and inverse dynamics model as well as a policy?

  • The quality of the predicted video (in the FDM mode) is not presented. Would be nice to see a graph like in Fig. 5 for this case as well. How good is the model at predicting the future video frames?

Questions for authors
  • Fig. 5 presents the MAE for state and action prediction. Is this when the model is used as a policy or as a finite dynamics model?
Reviewer_CbELOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets hierarchical humanoid control. A high-level policy outputs kinematic reference actions that a whole-body tracking controller executes, so the state the robot actually reaches can differ from the reference; the authors call this the action-execution gap. HWAM concatenates each reference action a_t with the measured next state s_{t+1} into a single token and denoises the joint state-action chunk with flow matching.

A Wan2.2-based video expert and a state-action expert interact through path-dependent masked attention. Training samples uniformly among three paths: forward dynamics (future video from state-action), inverse dynamics (state-action from video), and a deployment-matched policy path (state-action from current observations only). At deployment, predicted states still participate in joint denoising, while only actions are sent to the controller. The evaluation includes real-robot trials on three LimX OLI tasks, a 2×2 ablation on the mobile Plush-Toy task, open-loop prediction diagnostics on Candy Picking, and an ALOHA clothes-folding task in the appendix.

Critical strengths
  1. The paper evaluates a full-size humanoid against recent baselines including π0.5, GR00T N1.5 and Fast-WAM. On Candy Picking, each of the four methods with reported counts was evaluated over about 120 trials. The joint target is a simple extension of existing WAMs and requires no future-video generation at deployment, giving it practical value.

  2. The motivation comes from a measurable property of the data (Section 4.1, Appendix A), and the paper is careful about scope: it says the ablation does not isolate individual paths, that there is no video-prediction advantage, and that ALOHA is not zero-shot transfer.

Critical weaknesses
  1. Incomplete reporting of evaluation and baseline settings limits confidence in the stability of the gains. In Table 2, each change reduces success when introduced alone, while their combination improves success from 60.0% to 73.3%. The authors reasonably interpret this as an interaction between the components. However, complete trial counts and uncertainty estimates are missing for this ablation and for the OLI tasks other than Candy Picking, and evidence from independent training runs is not reported. Baseline adaptation and training settings also need more detail. In particular, Fast-WAM scores 0% on Object Collection because it fails to grasp the basket; stage-level results and adaptation details would help assess the comparability of the experimental setups.

  2. The added value of joint generation over ordinary auxiliary state supervision has not been isolated. Predicted states participate in joint denoising at deployment and may help action generation, but the existing ablation only compares targets with and without states. Without a simpler control such as a separate state-prediction head, it remains unclear how much of the gain comes from joint denoising and how much comes from adding state supervision. This limits the interpretation of the specific design.

Questions for authors
  1. Could you provide the missing success counts, trial counts and confidence intervals for Tables 1 and 2, and explain how initial conditions were set and aborted trials were counted? Since Table 9 uses partial credit, please report its trial counts, scoring rules and score variability separately. For Table 2, please also include any available independent training repeats or other evidence that the observed component interaction is stable.

  2. Could you describe each baseline's initialization, control-interface adaptation, training budget and model selection, and clarify whether Fast-WAM in Table 1 is the same configuration as Action + VGM/Policy in Table 2? Any available stage-level results, such as basket-grasp success on Object Collection, would help explain the failures of Fast-WAM and Cosmos3-Edge.

  3. Could you compare HWAM with an auxiliary state-prediction head that does not participate in joint denoising, using the same FDM/IDM/Policy recipe and original action labels, with training budgets and state-loss settings matched as closely as possible? One representative task would suffice to distinguish the roles of joint generation and ordinary state supervision. If an additional experiment is not feasible, please clarify what conclusions the existing ablation supports.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.