- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents HarmoWAM, motivated by empirical evidence of complementary strengths between Joint Modeling and Imagine-then-Execute approaches. It combines two action experts conditioned on a shared world model and uses stage-aware gating to support generalizable transit and precise manipulation. Experiments on six real-world tasks evaluate its effectiveness under in-domain and out-of-distribution conditions.
Critical strengths
- The paper presents a clear motivation grounded in real-world robotic experiments. The proposed framework is conceptually sound and well aligned with the empirical observations.
- The experimental evaluation provides convincing evidence of the method’s effectiveness across both in-domain and out-of-distribution settings.
Critical weaknesses
- The empirical study supporting the central motivation is limited to two real-world tasks. While the results suggest complementary strengths between the two paradigms, they do not convincingly establish the generality of the claimed trade-off between generalization and manipulation precision. A broader and more diverse set of tasks would be needed to substantiate this claim.
- Although the extensive real-world evaluation is valuable, the results may be sensitive to environmental variability and hardware-specific factors. The lack of complementary comparisons on standardized simulation benchmarks leaves concerns about reproducibility and the robustness of the reported gains.
Questions for authors
- The rationale behind the design of the two action experts should be explained more clearly, particularly why their respective architectures and conditioning inputs are appropriate for the intende roles. Moreover,additional controlled ablations are needed to validate these design choices and isolate their individual contributions.
- (Related to Question 1)What motivates conditioning the predictive expert on the current-step latent features, while providing future frames and corresponding latent features to the reactive expert? Have alternative conditioning schemes been evaluated?
- How does the proposed framework compare with a single action expert that receives both real observations and predicted visual information under a comparable parameter budget? This would help clarify the benefit of explicit expert specialization and routing.
14 comments
Table R1 reports results for all six tasks. Across the four additional tasks and three OOD settings, Imagine-then-Execute achieves 86.7% average transit success, compared with 55.8% for Joint Modeling. For interaction, Joint Modeling with near-target initialization achieves 69.2%, compared with 50.0% for Imagine-then-Execute. These results extend the observed complementarity across the additional manipulation skills.
Figure 13 in Appendix F.5 visualizes transit and interaction across all six tasks, complementing the phase-wise results in Table R1.
Thank you for asking about generality, which we discuss in Section 3.2 of the revised manuscript (Lines 204 to 211), with the six-task evaluation protocol and results in Appendix F.5 (Lines 1514 to 1551, Table 15).
Q1 and Q2: Action-Generation Mechanisms and Conditioning Design
1. Why Are the Architectures Suited to Their Intended Roles?
The two experts are designed to exploit complementary action-generation mechanisms. The Reactive Action Expert fuses predicted-frame DINOv2 features with corresponding Wan latents and maps them directly to actions through the Orientation Decoder. This allows control to follow the world model's predicted motion, supporting transit in unseen scenes (Section 3.3, Lines 269 to 277). The Predictive Action Expert jointly denoises an entire action chunk through Action DiT, learning temporally coherent actions under demonstration supervision to support precise interaction (Section 3.3, Lines 261 to 267).
This division of roles is supported jointly by the motivation experiments and expert-removal ablations. The phase-wise evaluation in the motivation experiments identifies the two mechanisms' respective strengths in OOD transit and near-target interaction (Table 1 and Section 3.2, Lines 194 to 202). In Figure 6(a), removing the Reactive Action Expert reduces Position OOD performance from 90% to 14%, while removing the Predictive Action Expert reduces Object OOD performance from 90% to 60% (Section 4.4, Lines 457 to 463). Together, these results support the complementary roles of the two experts and show that either mechanism alone falls short of the full framework.
2. Why Do the Experts Use Different Conditioning Inputs?
The conditions match how each expert generates actions. The Reactive Action Expert infers actions from predicted future visual states, so it receives future frames and temporally corresponding future latents. The Predictive Action Expert generates the next action chunk from the current state, so current observations and the current-step latent condition its denoising process.
This rationale does not imply that alternative conditions are unusable or that additional future information cannot help. We use the controlled comparisons below to examine the contribution of the conditions and whether changing the inputs can recover the full framework's performance.
3. Which Ablations Validate These Choices?
Latent-removal ablations establish the additional value of Wan latent conditioning. Figure 6(c) retains each expert's architecture and visual inputs while removing Wan latent features. ID performance drops from 74% to 65% for the Reactive Action Expert and from 95% to 62% for the Predictive Action Expert (Section 4.4, Lines 483 to 495).
Alternative-conditioning comparisons retain the action-generation mechanism and change only the conditions within each expert. We use Put Flowers in Vase with 10 real-robot rollouts per variant under each ID/OOD setting. The world model is frozen, expert architectures and training settings are held fixed, and each expert acts alone. Table R3 reports average cumulative stage success rate, with Avg. denoting the mean across the four settings. Each modified expert is paired with its original conditioning, while HarmoWAM provides a full-framework reference.
Reactive conditioning. We hold the predicted RGB video fixed and vary the Wan latent at each future step. Table 5 lists four-setting means of 37% for None, 39% for repeated Current, and 46% for aligned Future (Lines 511 to 518).
Predictive conditioning. We retain the original conditions and add DINOv2 features from 12 predicted future frames. These features are pooled and projected to the Action DiT conditioning dimension, retain frame-order information, and enter cross-attention at each denoising step. The augmented expert achieves 90.0% ID and 20.0% average OOD performance, compared with HarmoWAM's 77.8% OOD average. Table 5 displays overall means of 27% for None, 38% for Current, and 38% for +Future.
We sincerely thank you for your careful review and for recognizing HarmoWAM’s motivation, conceptual soundness, and convincing real-world results. Your suggestions on broader evaluation, standardized benchmarks, and expert design are valuable.
W1: Generality of the Observed Complementarity
The original Stack Coke Cans and Put Flowers in Vase tasks involve both spatial target approach and contact-sensitive fine interaction, allowing us to examine the two paradigms’ generalization and manipulation precision across different manipulation structures. Building on this evidence and following your suggestion, we extend the phase-wise evaluation protocol to the remaining four tasks, Pick Fruit to Plate, Pour Coke into Beaker, Write “Yes”, and Put Items to Bag and Zip, further covering pick-and-place, rotational control during pouring, writing trajectory control, deformable-object manipulation, and zipper control. The protocol covers all six real-world tasks and specifies 10 trials per method and task under each ID/OOD setting. We evaluate Joint Modeling interaction with near-target initialization to isolate manipulation precision, following Appendix D.2 (Lines 1085 to 1089).
Table R1. Phase-wise evaluation on six real-world tasks.
The first two rows are from Table 1 of the paper, and the remaining four rows cover the additional tasks. ITE denotes Imagine-then-Execute and JM denotes Joint Modeling. Each cell reports transit / interaction success rates. JM interaction uses near-target initialization, consistent with the evaluation setting in the paper.
We first observed complementary strengths in transit generalization and interaction precision through controlled comparisons on two real-world tasks (Table 1 and Section 3.2, Lines 194 to 202). This observation is consistent with the action-generation mechanisms of the two paradigms. Imagine-then-Execute first predicts future visual states with the world model, then converts the predicted motion into actions through an inverse-dynamics mapping. This allows control to follow the motion path planned by the world model, bringing visual prediction generalization to transit in unseen scenes. However, errors in translating predicted visual states into control actions can still affect end-effector positioning and gripper timing, limiting performance during fine interaction.
Joint Modeling instead uses implicit world-model features to condition action generation and jointly denoises an entire action chunk through Action DiT. Under demonstration supervision, it learns temporally coherent actions that support precise local interaction. However, its action policy is still learned from downstream demonstrations and may remain constrained by their spatial coverage. Thus, even when the world model provides target-related information, the policy may not translate it into actions that reach unseen positions.
The basis for broader applicability is that visual-to-action translation errors and the influence of the demonstration distribution arise from how actions are generated, rather than from the particular objects or layouts of the original tasks. The same sources of error also affect other tasks requiring both target approach in unseen scenes and precise interaction, providing a mechanism-based rationale for expecting similar complementarity.