acceptodds
Under review as a conference paper at ICLR 2027

HarmoWAM: Harmonizing Generalizable and Precise Manipulation via Adaptive World Action Models

Abstract

World Action Models (WAMs) have emerged as a promising paradigm for robot control by modeling physical dynamics. Current WAMs generally follow two paradigms: the "Imagine-then-Execute" approach, which uses video prediction to infer actions via inverse dynamics, and the "Joint Modeling" approach, which jointly models actions and video representations. Based on systematic experiments, we observe a fundamental trade-off between these paradigms: the former explicitly leverages world models for generalizable transit but lacks interaction precision, whereas the latter enables fine-grained, temporally coherent action generation but is constrained by the exploration space of the training distribution. Motivated by these findings, we propose HarmoWAM, an end-to-end WAM that fully leverages a world model to unify predictive and reactive control, enabling both generalizable transit and precise manipulation. Specifically, the world model provides spatio-temporal physical priors that condition two complementary action experts: a predictive expert that leverages latent dynamics for iterative action generation, and a reactive expert that directly infers actions from predicted visual evolution. To enable adaptive coordination, a Process-Adaptive Gating Mechanism is proposed to automatically determine the timing and location of switching between them. This allows the world model to drive the reactive expert to expand the exploration space and the predictive expert to perform precise interactions across different task stages. For evaluation, we construct three training-unseen test environments across six real-world robotic tasks, covering variations in background, position, and object semantics. Notably, HarmoWAM achieves strong zero-shot generalization across these scenarios, significantly outperforming prior state-of-the-art VLA models and WAMs by margins of 33% and 29%, respectively.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Action Model, Robotic Manipulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3cRSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents HarmoWAM, motivated by empirical evidence of complementary strengths between Joint Modeling and Imagine-then-Execute approaches. It combines two action experts conditioned on a shared world model and uses stage-aware gating to support generalizable transit and precise manipulation. Experiments on six real-world tasks evaluate its effectiveness under in-domain and out-of-distribution conditions.

Critical strengths
  1. The paper presents a clear motivation grounded in real-world robotic experiments. The proposed framework is conceptually sound and well aligned with the empirical observations.
  2. The experimental evaluation provides convincing evidence of the method’s effectiveness across both in-domain and out-of-distribution settings.
Critical weaknesses
  1. The empirical study supporting the central motivation is limited to two real-world tasks. While the results suggest complementary strengths between the two paradigms, they do not convincingly establish the generality of the claimed trade-off between generalization and manipulation precision. A broader and more diverse set of tasks would be needed to substantiate this claim.
  2. Although the extensive real-world evaluation is valuable, the results may be sensitive to environmental variability and hardware-specific factors. The lack of complementary comparisons on standardized simulation benchmarks leaves concerns about reproducibility and the robustness of the reported gains.
Questions for authors
  1. The rationale behind the design of the two action experts should be explained more clearly, particularly why their respective architectures and conditioning inputs are appropriate for the intende roles. Moreover,additional controlled ablations are needed to validate these design choices and isolate their individual contributions.
  2. (Related to Question 1)What motivates conditioning the predictive expert on the current-step latent features, while providing future frames and corresponding latent features to the reactive expert? Have alternative conditioning schemes been evaluated?
  3. How does the proposed framework compare with a single action expert that receives both real observations and predicted visual information under a comparable parameter budget? This would help clarify the benefit of explicit expert specialization and routing.
14 comments

We sincerely thank you for your careful review and for recognizing HarmoWAM’s motivation, conceptual soundness, and convincing real-world results. Your suggestions on broader evaluation, standardized benchmarks, and expert design are valuable.

W1: Generality of the Observed Complementarity

The original Stack Coke Cans and Put Flowers in Vase tasks involve both spatial target approach and contact-sensitive fine interaction, allowing us to examine the two paradigms’ generalization and manipulation precision across different manipulation structures. Building on this evidence and following your suggestion, we extend the phase-wise evaluation protocol to the remaining four tasks, Pick Fruit to Plate, Pour Coke into Beaker, Write “Yes”, and Put Items to Bag and Zip, further covering pick-and-place, rotational control during pouring, writing trajectory control, deformable-object manipulation, and zipper control. The protocol covers all six real-world tasks and specifies 10 trials per method and task under each ID/OOD setting. We evaluate Joint Modeling interaction with near-target initialization to isolate manipulation precision, following Appendix D.2 (Lines 1085 to 1089).

Table R1. Phase-wise evaluation on six real-world tasks.

The first two rows are from Table 1 of the paper, and the remaining four rows cover the additional tasks. ITE denotes Imagine-then-Execute and JM denotes Joint Modeling. Each cell reports transit / interaction success rates. JM interaction uses near-target initialization, consistent with the evaluation setting in the paper.

TaskID ITEID JMBackground ITEBackground JMPosition ITEPosition JMObjects ITEObjects JM
Stack Coke Cans1.00 / 0.700.90 / 0.901.00 / 0.600.50 / 0.801.00 / 0.500.30 / 1.001.00 / 0.700.00 / 1.00
Put Flowers in Vase1.00 / 0.800.90 / 1.001.00 / 0.600.50 / 0.901.00 / 0.200.00 / 1.001.00 / 0.700.60 / 1.00
Pick Fruit to Plate1.00 / 0.900.90 / 0.900.90 / 0.600.60 / 0.800.90 / 0.800.40 / 0.800.90 / 0.500.70 / 0.80
Pour Coke into Beaker0.90 / 0.700.90 / 0.900.90 / 0.500.80 / 0.800.80 / 0.300.30 / 0.700.90 / 0.600.80 / 0.70
Write “Yes”0.90 / 0.600.90 / 0.800.90 / 0.600.60 / 0.700.90 / 0.400.50 / 0.700.90 / 0.600.60 / 0.60
Put Items to Bag and Zip0.90 / 0.400.90 / 0.700.80 / 0.300.60 / 0.600.80 / 0.400.30 / 0.600.80 / 0.400.50 / 0.50

We first observed complementary strengths in transit generalization and interaction precision through controlled comparisons on two real-world tasks (Table 1 and Section 3.2, Lines 194 to 202). This observation is consistent with the action-generation mechanisms of the two paradigms. Imagine-then-Execute first predicts future visual states with the world model, then converts the predicted motion into actions through an inverse-dynamics mapping. This allows control to follow the motion path planned by the world model, bringing visual prediction generalization to transit in unseen scenes. However, errors in translating predicted visual states into control actions can still affect end-effector positioning and gripper timing, limiting performance during fine interaction.

Joint Modeling instead uses implicit world-model features to condition action generation and jointly denoises an entire action chunk through Action DiT. Under demonstration supervision, it learns temporally coherent actions that support precise local interaction. However, its action policy is still learned from downstream demonstrations and may remain constrained by their spatial coverage. Thus, even when the world model provides target-related information, the policy may not translate it into actions that reach unseen positions.

The basis for broader applicability is that visual-to-action translation errors and the influence of the demonstration distribution arise from how actions are generated, rather than from the particular objects or layouts of the original tasks. The same sources of error also affect other tasks requiring both target approach in unseen scenes and precise interaction, providing a mechanism-based rationale for expecting similar complementarity.

Table R1 reports results for all six tasks. Across the four additional tasks and three OOD settings, Imagine-then-Execute achieves 86.7% average transit success, compared with 55.8% for Joint Modeling. For interaction, Joint Modeling with near-target initialization achieves 69.2%, compared with 50.0% for Imagine-then-Execute. These results extend the observed complementarity across the additional manipulation skills.

Figure 13 in Appendix F.5 visualizes transit and interaction across all six tasks, complementing the phase-wise results in Table R1.

Thank you for asking about generality, which we discuss in Section 3.2 of the revised manuscript (Lines 204 to 211), with the six-task evaluation protocol and results in Appendix F.5 (Lines 1514 to 1551, Table 15).

Q1 and Q2: Action-Generation Mechanisms and Conditioning Design

1. Why Are the Architectures Suited to Their Intended Roles?

The two experts are designed to exploit complementary action-generation mechanisms. The Reactive Action Expert fuses predicted-frame DINOv2 features with corresponding Wan latents and maps them directly to actions through the Orientation Decoder. This allows control to follow the world model's predicted motion, supporting transit in unseen scenes (Section 3.3, Lines 269 to 277). The Predictive Action Expert jointly denoises an entire action chunk through Action DiT, learning temporally coherent actions under demonstration supervision to support precise interaction (Section 3.3, Lines 261 to 267).

This division of roles is supported jointly by the motivation experiments and expert-removal ablations. The phase-wise evaluation in the motivation experiments identifies the two mechanisms' respective strengths in OOD transit and near-target interaction (Table 1 and Section 3.2, Lines 194 to 202). In Figure 6(a), removing the Reactive Action Expert reduces Position OOD performance from 90% to 14%, while removing the Predictive Action Expert reduces Object OOD performance from 90% to 60% (Section 4.4, Lines 457 to 463). Together, these results support the complementary roles of the two experts and show that either mechanism alone falls short of the full framework.

2. Why Do the Experts Use Different Conditioning Inputs?

The conditions match how each expert generates actions. The Reactive Action Expert infers actions from predicted future visual states, so it receives future frames and temporally corresponding future latents. The Predictive Action Expert generates the next action chunk from the current state, so current observations and the current-step latent condition its denoising process.

This rationale does not imply that alternative conditions are unusable or that additional future information cannot help. We use the controlled comparisons below to examine the contribution of the conditions and whether changing the inputs can recover the full framework's performance.

3. Which Ablations Validate These Choices?

Latent-removal ablations establish the additional value of Wan latent conditioning. Figure 6(c) retains each expert's architecture and visual inputs while removing Wan latent features. ID performance drops from 74% to 65% for the Reactive Action Expert and from 95% to 62% for the Predictive Action Expert (Section 4.4, Lines 483 to 495).

Alternative-conditioning comparisons retain the action-generation mechanism and change only the conditions within each expert. We use Put Flowers in Vase with 10 real-robot rollouts per variant under each ID/OOD setting. The world model is frozen, expert architectures and training settings are held fixed, and each expert acts alone. Table R3 reports average cumulative stage success rate, with Avg. denoting the mean across the four settings. Each modified expert is paired with its original conditioning, while HarmoWAM provides a full-framework reference.

Reactive conditioning. We hold the predicted RGB video fixed and vary the Wan latent at each future step. Table 5 lists four-setting means of 37% for None, 39% for repeated Current, and 46% for aligned Future (Lines 511 to 518).

Predictive conditioning. We retain the original conditions and add DINOv2 features from 12 predicted future frames. These features are pooled and projected to the Action DiT conditioning dimension, retain frame-order information, and enter cross-attention at each denoising step. The augmented expert achieves 90.0% ID and 20.0% average OOD performance, compared with HarmoWAM's 77.8% OOD average. Table 5 displays overall means of 27% for None, 38% for Current, and 38% for +Future.

Reviewer_a6TZOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes HarmoWAM, a world action model that aims to balance generalization and precise manipulation. The authors identify the limitations of existing WAM paradigms and introduce a framework that combines predictive and reactive action experts with an adaptive gating mechanism. The method is evaluated on six real-world manipulation tasks with both in-domain and OOD settings, showing improvements over existing VLA and WAM baselines.

Critical strengths
  1. The paper addresses an important problem with a well-motivated design.

The authors clearly analyze the trade-off between generalization and manipulation precision in existing WAMs, and the proposed expert coordination mechanism is closely aligned with this observation.

  1. The paper provides strong real-world experimental validation.

The evaluation covers diverse manipulation tasks and OOD scenarios, with comparisons against strong baselines and comprehensive ablations, providing solid evidence for the effectiveness of the proposed method.

Critical weaknesses
  1. Lack of standardized simulation benchmark evaluation.

The evaluation is mainly conducted on real-world tasks. Including simulation benchmarks could further validate the generalization ability and improve reproducibility.

  1. The training and evaluation protocol is not sufficiently clear.

It is unclear whether the six tasks are jointly trained with a single checkpoint or trained separately with task-specific checkpoints, and whether the OOD evaluations use the same policy.

Questions for authors
  1. Do the authors plan to release the code and model checkpoints?
  2. Could the authors clarify the training and evaluation protocol, including whether a unified checkpoint or task-specific checkpoints are used?
  3. Could the authors provide more details about the computational requirements, such as GPU configuration and training cost?
5 comments

We sincerely thank you for your careful review and strong support for HarmoWAM, particularly your recognition of its motivation, expert coordination, and real-world validation. Your suggestions on reproducibility, evaluation protocols, and computational requirements are valuable. We have addressed these points in the revised manuscript and clarify them below.

Q1: Code and Model Checkpoints

Thank you for emphasizing reproducibility. We agree that access to the complete implementation and execution workflows is important for reproducing the results. The release package is prepared, and we commit to releasing the complete codebase, checkpoints, and detailed reproduction instructions immediately upon acceptance. The code covers data preparation, two-stage training, integrated inference, and real-robot evaluation, accompanied by configuration files, environment setup instructions, and step-by-step execution guidance to support reproduction from training through evaluation.

W1: Standardized Simulation Evaluation

Thank you for recommending standardized simulation benchmarks. We agree that they provide valuable complementary evidence and improve reproducibility. We have evaluated HarmoWAM on all 50 bimanual tasks in RoboTwin 2.0, with results reported in Appendix F.4 and Table 14. The benchmark uses an ALOHA-AgileX robot in SAPIEN. Each method is jointly trained on all 50 tasks using 50 Clean demonstrations per task, for a total of 2,500 demonstrations. The resulting unified checkpoint is evaluated over 50 Clean rollouts per task (Appendix F.4, Lines 1443 to 1450). HarmoWAM achieves the highest average success rate of 67.60%, compared with 56.56% for π₀.₅ and 50.00% for Wan+AnyPos (Appendix F.4, Lines 1451 to 1454). Following your suggestion, we have moved these simulation results into the main text and clarified the evaluation protocol.

Table R1. Average task-completion success on the 50 RoboTwin 2.0 tasks under the Clean setting (%).

MethodClean
VPP19.52
Cosmos-Policy25.32
π₀.₅56.56
QwenVLA-OFT33.76
Wan+AnyPos50.00
HarmoWAM67.60

Detailed scores for each of the 50 tasks and all methods are provided in Appendix F.4, Table 14.

W2 and Q2: Training and Evaluation Protocol

Training. We train HarmoWAM separately on each of the six real-robot tasks, yielding one task-specific checkpoint per task. Each task has 100 teleoperated demonstration trajectories (Section 4.1, Lines 354 to 359). We first fine-tune the Wan2.2-TI2V-5B world model on that task’s demonstration videos. We then freeze the world model and train the two action experts and Process-Adaptive Gating on the same task’s demonstrations (Appendix C.2, Lines 862 to 880).

Checkpoint selection for ID and OOD evaluation. For a given task, we use the same checkpoint in ID and all three OOD settings. Task-specific training uses only ID demonstrations, and the OOD settings are unseen during this training. The OOD settings vary the background, target position, or target object (Appendix C.4, Lines 931 to 949). We evaluate each checkpoint directly, without additional demonstrations, fine-tuning, or online adaptation.

Baseline comparison. Each real-robot baseline is also trained separately for each task, yielding one checkpoint per baseline per task. Each baseline uses that task’s checkpoint for ID and all three OOD settings. For each task, the baselines use the same training demonstrations, task instructions, and test protocol as HarmoWAM (Appendix C.1, Lines 854 to 858, and Appendix C.2, Lines 862 to 863). In the main real-robot evaluation, we run 20 independent episodes per method and task, randomize tabletop object positions, and report the average cumulative stage success rate across the task’s key stages (Section 4.1, Lines 365 to 368).

Q3: Computational Requirements

Thank you for requesting these details. We agree that clear hardware and training cost information is important for assessing practical reproducibility.

For the real-robot experiments, HarmoWAM is trained separately for each task on eight NVIDIA H20 GPUs. Stage 1 takes approximately two hours to fine-tune the task-specific world model. Stage 2 takes approximately four hours to train the two action experts and Process-Adaptive Gating with the world model frozen.

At inference, real-robot execution uses one NVIDIA RTX 4090 GPU. The world model is invoked once per action chunk and uses five denoising steps to generate a 13-frame video and its latent features. Process-Adaptive Gating selects one expert to generate a 12-step action chunk. The complete real-robot control loop runs at approximately 4 Hz, from observation acquisition through world-model prediction, expert selection, and action generation to action execution on the robot. The model inference frequency is 4 Hz, and the action-output throughput is 48 predicted actions/s (Appendices C.2 and C.3).

Thank you again for your positive assessment of our work and your valuable suggestions. We have incorporated your feedback and provided a fuller description of the experimental setup and reproduction details in the revised manuscript.

Thank you for the response. It addresses my concerns. However, I wonder why Motus and Fast-WAM were not included in the RoboTwin 2.0 comparison. Would it be possible to compare them under the same training data setting?

Reviewer_CxhCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HarmoWAM to address the trade-off between generalization and precise manipulation in existing World Action Models (WAMs). The authors first analyze the complementary properties of two representative paradigms, Imagine-then-Execute and Joint Modeling: the former achieves stronger cross-distribution transfer by leveraging the world model, but lacks fine-grained manipulation capability; the latter attains higher operational precision, yet exhibits limited exploration in unseen environments. Motivated by these observations, the authors propose a dual-expert framework that combines a world model, a predictive action expert, and a reactive action expert, and dynamically selects the appropriate expert through a process-adaptive gating mechanism according to the current task stage. The method is evaluated on six real-robot tasks and three out-of-distribution (OOD) scenarios. Experimental results show that HarmoWAM outperforms multiple existing VLA and WAM baselines in both in-domain and out-of-domain settings, and ablation studies confirm the effectiveness of the two action experts as well as the latent features of the world model.

Critical strengths
  1. Well-motivated research. The paper addresses the trade-off between generalization and manipulation precision in World Action Models (WAMs), and supports the motivation with real-robot experiments that reveal the complementary properties of two representative paradigms.
  2. Strong empirical results. The method is evaluated on multiple real-robot tasks and out-of-distribution (OOD) scenarios, achieving clear improvements over existing baselines. Ablation studies also provide preliminary evidence for the effectiveness of the dual-expert design.
Critical weaknesses
  1. Insufficient justification of necessity and novelty. The proposed method mainly combines two existing action modeling strategies and uses a gating network for stage-wise switching. Although the design is reasonable, the paper does not compare it with simpler switching strategies, such as rule-based gating. The current ablations demonstrate the complementarity of the two experts, but they are not sufficient to establish the necessity of the learned gating mechanism. In addition, the respective contributions of the world model’s dynamics modeling and the pretrained visual representations remain unclear.
  2. Lack of computational cost analysis. The framework integrates a 5B world model, a 1B predictive expert, and an additional reactive expert, and adopts a two-stage training procedure, which may introduce substantial training and inference overhead. Although the paper reports an inference frequency of approximately 4 Hz, it does not provide a fair comparison with baselines in terms of training cost, inference latency, and GPU memory usage. As a result, it is difficult to assess whether the performance gain justifies the added model complexity.
Questions for authors
  1. Could the authors add comparisons with rule-based stage switching or oracle gating to further verify the necessity of the learned gating mechanism?
  2. Could the authors further disentangle the contributions of the world model’s dynamics modeling and the pretrained visual features to the observed performance gains?
  3. Could the authors provide comparisons of training cost, inference latency, and GPU memory usage with the main baselines under the same hardware setup? Does the reported 4 Hz reflect the complete end-to-end inference pipeline?
17 comments

We sincerely thank you for your positive comments on the motivation behind HarmoWAM and on its real-world performance. We would like to address your questions regarding Process-Adaptive Gating, the world-model latent features, and computational cost below, and we hope these clarifications are helpful.

W1 and Q1: Necessity and Novelty of Process-Adaptive Gating

1. Necessity and Novelty of Stage-Aware Coordination

The need for stage-aware coordination arises because the two action-generation mechanisms have complementary strengths, and neither alone reliably handles both OOD transit and precise interaction. Table 1 and Section 3.2 (Lines 194 to 202) show that Imagine-then-Execute achieves 100% OOD transit success but 55% interaction success. Joint Modeling achieves approximately 32% OOD transit success, yet reaches 95% interaction success with near-target initialization. Process-Adaptive Gating therefore learns to select the mechanism suited to the current stage from visual observations. The novelty lies in translating this empirically identified complementarity into learned stage-aware coordination of two distinct action-generation mechanisms under a shared world model. The comparisons below with Averaging, Rule-based switching, and Oracle gating further clarify the value of learned gating.

2. Comparisons Evaluating Learned Gating

The paper already demonstrates the advantage of Process-Adaptive Gating over Averaging. Figure 6(b) and Section 4.4 (Lines 464 to 467) show that Averaging causes a 46 percentage-point drop under Position OOD. Through comparisons with Averaging, Rule-based switching, and Oracle gating, we further examine the value of learned gating.

The evaluation protocol follows Section 4.4 (Lines 454 to 456), using Put Flowers in Vase and Pick Fruit to Plate with 10 trials per task under each ID/OOD setting. The world model and both action experts remain fixed. Table R1 reports the mean of the two tasks’ average cumulative stage success rates.

Rule-based Switching

Rule-based switching uses a task state machine and updates object and target-container positions online using Grounding DINO and RGB-D observations. At each routing decision, Grounding DINO localizes the task objects and target containers in the current RGB image. We align depth to RGB, select valid target-surface depth within the detected regions, and deproject the pixels using camera intrinsics to obtain 3D positions. Each gripper position is estimated as the median 3D position of valid depth points in its orange image region, and gripper-to-target distances are computed in a common coordinate frame. Demonstration-calibrated offsets define the interaction reference points for each stage. For grasping, insertion, and placement, the state machine computes the distance from the acting gripper to its interaction reference point. For bimanual handover, it computes the distance between the two grippers in the same coordinate frame. It selects the Reactive Action Expert during approach and switches to the Predictive Action Expert when the calibrated spatial trigger conditions are met, retaining it until the specified gripper event and stage-exit conditions are satisfied. Put Flowers in Vase follows grasp, handover, and insertion, with distance thresholds of 102.3, 173.0, and 27.4 mm, respectively. Pick Fruit to Plate follows banana grasp, banana placement, carrot grasp, and carrot placement, with respective distance thresholds of 208.0, 157.0, 145.0, and 159.0 mm. Switching rules, target offsets, and distance thresholds remain fixed across ID and OOD settings, while object and container positions update from online observations.

In Figure 10 of Appendix D.3 in the revised manuscript, we show the detected targets, gripper positions, and expert-switching conditions for both tasks.

Approximate Oracle Gating (GPT-6)

Because ground-truth stage labels are unavailable during execution, we use GPT-6 as an external estimator of the current stage. The router uses this judgment as a proxy stage label, selecting the Reactive Action Expert for transit and the Predictive Action Expert for interaction. Table R1 compares learned gating with this approximate Oracle under the same closed-loop protocol.

We also compare the learned gate’s predictions offline with annotated ground-truth stage labels on held-out demonstrations to evaluate its stage-recognition performance. These labels are constructed from recorded proprioceptive events. Agreement reaches 96.95% on 1,637 ID frames and 94.77% on 4,207 OOD frames (Appendix F.3.2, Lines 1409 to 1414). This high agreement supports reliable stage recognition under changes in background, position, and objects, while the learned gate requires only current visual features at inference.

Results and Practical Complexity

Table R1. Expert coordination comparison averaged over Put Flowers in Vase and Pick Fruit to Plate (%).

SettingAveragingRule-based switchingOracle (GPT-6)Process-Adaptive Gating (Ours)
ID76909493
Background59858887
Position44829290
Objects57839190

In Table R1, the learned gate leads rule-based switching by 2, 8, and 7 percentage points under Background, Position, and Objects OOD, respectively, and stays within 2 points of the GPT-6 reference across all settings. Rule-based switching requires a separate state machine for each task. For every stage, the designer must define the interaction reference point, calibrate target offsets and distance thresholds from demonstrations, and define stage-transition rules based on gripper feedback. The implementation also requires RGB-depth alignment, online object detection, depth processing, and 3D deprojection. This task-specific rule design and geometric calibration must be repeated for each new task, adding substantial implementation and calibration work. The approximate Oracle calls an external GPT-6 model at every routing decision, incurring additional model-call costs. Process-Adaptive Gating reuses the frozen Wan VAE encoder to extract features from the current observation using the world model’s pretrained visual representations and trains a lightweight MLP on automatically generated labels. HarmoWAM thus selects experts within the policy with little additional computation, without per-task calibration or external model calls. Among these approaches, learned gating therefore provides a simpler, more efficient, and more readily deployable way to coordinate experts.

3. Reliability and Robustness of Learned Gating

In the window-sensitivity experiment, with the world model and experts fixed, changing the interaction-label window from ±20 to ±10 or ±30 frames changes average task performance by at most 3.4 percentage points (Figure 6(d), Section 4.4, Lines 467 to 471, and Appendix F.3.2, Lines 1436 to 1439).

Closed-loop analysis includes three successful OOD rollouts and one failure. In successful rollouts, gate scores remain low during transit and rise during grasping, handover, and insertion, supporting alignment with task progression during execution (Table 13 and Appendix F.3.2, Lines 1416 to 1425).

These results demonstrate the need for stage-aware coordination, highlight the practical advantages of learned gating over rule-based switching and approximate Oracle gating, and support its robustness and effectiveness under OOD conditions.

Thank you for suggesting these controls. The revised manuscript reports the gating comparison in Section 4.4 (Lines 471 to 482, Table 4) and the implementation details in Appendix D.3 (Lines 1118 to 1172).

W1 and Q2: World Model’s Dynamics Modeling and Pretrained Visual Representations

The world model’s dynamics modeling and the pretrained visual representations contribute through both the explicit and implicit paths of HarmoWAM. We explain their roles in each path and provide complementary evidence below.

Explicit Path

In the explicit path, the world model’s dynamics modeling provides predicted RGB observations of how the scene evolves, while the pretrained visual representations provide visual knowledge that supports generalization to unseen scenes. The Reactive Action Expert converts the predicted observations into actions through the Orientation Decoder (Section 3.3, Lines 269 to 277). The world model’s dynamics modeling and the pretrained visual representations serve interdependent functions in the current explicit path. The former provides coherent future state evolution, while the latter provides visual knowledge supporting generalization to unseen scenes. Together, they support action generation from predicted video. Directly removing either changes how this path operates, making isolated attribution difficult. We therefore retain the complete explicit path and use the latent conditioning comparison below to further examine the additional contribution of latent conditions.

Implicit Path

Wan latent conditions carry information related to both the world model’s dynamics modeling and the pretrained visual representations. Below, we examine the roles of these two types of information and their contributions to performance gains in the Reactive Action Expert and the Predictive Action Expert separately.

Reactive Action Expert

For the Reactive Action Expert, we compare three conditioning schemes while retaining the predicted RGB video: no Wan latent features, future Wan latent features aligned with each predicted frame, and the current-step Wan latent repeated across future frames.

We use the closed-loop protocol on Put Flowers in Vase with 10 real-robot rollouts per variant in each of the ID, Background, Position, and Objects settings. We freeze the world model, retrain each Reactive Action Expert under the same settings, and evaluate it alone. Table R2(a) reports the average cumulative stage success rate, with Avg. denoting the mean across the four settings.

Table R2(a). Reactive Action Expert latent conditioning comparison on Put Flowers in Vase.

ConditioningIDBackgroundPositionObjectsAvg.
Predicted future frames without Wan latent features0.430.330.330.370.37
Predicted future frames with corresponding future Wan latent features0.530.470.430.400.46
The same predicted future frames with repeated current-step Wan latent features0.470.330.370.400.39

Comparing no latents with aligned future latents measures the joint contribution of the pretrained visual representations and predicted dynamics information carried by the Wan latents. Adding aligned future latents raises mean cumulative stage success from 37% to 46%, a gain of 9 percentage points over the same predicted RGB input.

Replacing the aligned future latents with the current-step latent repeated across future frames retains Wan latent conditioning but changes both its future-state content and its temporal correspondence with the predicted frames. This comparison probes how temporal information associated with the world model’s dynamics modeling contributes to action prediction. With the predicted RGB video held fixed, Table R2(a) gives a mean cumulative stage success rate of 39% for the repeated current-step latent, 7 percentage points below aligned future latents (46%) and only 2 points above no latents (37%). This pattern suggests that the Reactive Action Expert can use information about how the predicted state changes across frames to generate actions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response(1/2)

We sincerely thank all three reviewers for their careful reading and constructive feedback. We appreciate their recognition of HarmoWAM’s research value, methodological design, and empirical evidence:

  • Clear and practically meaningful motivation. HarmoWAM addresses the central trade-off between generalization and manipulation precision in existing WAMs and reveals the complementary strengths of the two paradigms through real-robot experiments. Reviewer CxhC highlights the “well-motivated research,” a6TZ recognizes an “important problem with a well-motivated design,” and 3cRS notes the “clear motivation grounded in real-world robotic experiments.”
  • An expert-coordination framework closely aligned with empirical findings. HarmoWAM unifies two complementary action-generation mechanisms under a shared world model and coordinates generalizable transit and precise interaction through stage-adaptive routing. Reviewer a6TZ finds the expert coordination mechanism “closely aligned” with the observed trade-off, and 3cRS considers the framework “conceptually sound and well aligned with the empirical observations.”
  • Convincing real-world performance and OOD generalization. HarmoWAM outperforms existing baselines across diverse real-world tasks and three OOD settings, with ablations validating its key design choices. Reviewer CxhC recognizes the “strong empirical results,” a6TZ highlights the “strong real-world experimental validation,” and 3cRS finds “convincing evidence” of effectiveness across ID and OOD settings.

We have addressed the main concerns through additional experiments and clarifications, summarized below. Detailed responses are provided under each review.

Summary of New Results and Clarifications

  1. Broader transit–interaction study (3cRS). We extended the phase-wise evaluation from two to all six real-world tasks. Across the four additional tasks and three OOD settings, Imagine-then-Execute achieves 86.7% transit success versus 55.8% for Joint Modeling. For interaction, Joint Modeling with near-target initialization achieves 69.2% versus 50.0% for Imagine-then-Execute, supporting the observed complementarity across more manipulation skills.

  2. Learned routing controls and robustness (CxhC). With the world model and experts fixed, learned gating outperforms rule-based switching by 2–8 percentage points under OOD and remains within 2 points of the GPT-6 approximate Oracle across all settings. It reuses the frozen Wan VAE encoder’s pretrained visual representations through a lightweight MLP, avoiding per-task geometric calibration and external model calls. Label-window sensitivity and closed-loop analyses further support routing robustness.

  3. World-model features and expert conditioning (CxhC, 3cRS). We analyze how world-model dynamics and pretrained visual representations support action generation through the explicit future-video and implicit latent-conditioning paths, using both experts’ designs and controlled ablations. For Reactive, aligned future latents yield 46% mean success, compared with 37% without latents and 39% with repeated current-step latents, supporting the value of dynamics-related temporal information. For Predictive, latent-removal ablations and Grad-CAM provide complementary evidence for supplementary Wan conditioning and pretrained visual representations. We also clarify why each architecture and its inputs suit its intended role.

  4. Single-expert alternatives (3cRS). Predictive + Future receives current observations and predicted future-frame features without expert selection, achieving 20.0% mean OOD success versus 77.8% for HarmoWAM. Unified All-Condition Policy, a supplementary control that switches input modes within a shared 28-layer Action DiT, achieves 28.9%. These results support preserving and coordinating complementary action-generation mechanisms beyond combining conditioning inputs.

  5. Standardized simulation and additional baselines (a6TZ, 3cRS). On all 50 RoboTwin 2.0 tasks, each method is jointly trained with 2,500 Clean demonstrations, and HarmoWAM achieves 67.60% Clean success versus 56.56% for π₀.₅. Our follow-up comparisons use the same downstream data and evaluation protocol. Fast-WAM achieves 77.80% Clean and 1.90% Randomized success, compared with HarmoWAM’s 67.60% and 50.16%, respectively.

  6. Computational cost and reproducibility (CxhC, a6TZ). We compare training cost, end-to-end inference latency, and peak allocated GPU memory with all five original baselines. HarmoWAM uses 48.0 H20 GPU-hours per real-world task, with 250.0 ms inference latency and 22.3 GB peak memory on an RTX 4090. Its training cost and end-to-end latency are lower than VPP and Cosmos-Policy, while maintaining approximately 4 Hz across the complete real-robot loop on a single RTX 4090, combining stronger OOD performance with practical deployment feasibility.

AuthorsOpenReview ↗

General Response(2/2)

Summary of Revisions

The revised manuscript has been uploaded, with substantive changes marked in blue. References below refer to that version.

  1. Broader motivation study: Section 3.2 and Appendix F.5, Table 15 and Figure 13 (3cRS).
  2. Routing controls and implementation: Section 4.4, Table 4, and Appendix D.3, Figure 10. Robustness analyses appear in Figure 6 and Appendix F.3.2 (CxhC).
  3. World-model features, expert conditioning, and single-expert comparisons: Section 4.4, Figure 6 and Table 5, and Appendix F.3.1 (CxhC, 3cRS).
  4. Standardized simulation: Section 4.5, Table 6, and Appendix F.4, Table 14 (a6TZ, 3cRS). Additional Fast-WAM and Motus results are reported in our follow-up response to a6TZ.
  5. Computational cost: Appendix C.3, Table 7, with inference settings in Appendix C.2 (CxhC, a6TZ).
  6. Training and evaluation protocols: Section 4.1, Appendices C.1–C.4, and Appendix F.4 (a6TZ).

We thank the reviewers again for suggestions that have strengthened the evidence and presentation. We hope these additions clarify the value of coordinating complementary experts for generalizable transit and precise interaction, and we remain happy to address further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.