- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
IntentWAM conditions world-action modeling on object-centric task intent. It extracts ORTS—task roles, endpoint scene graphs, and object dynamics—from robot or human demonstrations, grounds those roles to objects in the target scene, and encodes them as intent tokens that guide visual-future prediction and action generation. Its main contribution is a unified, embodiment-agnostic conditioning interface that allows one fixed policy to use same-robot, cross-robot, or human demonstrations via task-level demonstration–execution associations, without trajectory-specific pairing or temporal alignment. Validation includes held-out RoboTwin tasks under multiple demonstration sources and scene variations, real-world bimanual manipulation on held-out layouts, and component ablations, showing improved generalization over language-only and demonstration-conditioned baselines.
Critical strengths
One critical strength is the ORTS-based conditioning interface, which discards source actions and joint trajectories and represents demonstrations via task roles, endpoint scene graphs, and object motion, making the condition embodiment-agnostic. This is consequential because it allows one fixed execution policy to use same-robot, cross-robot, or human demonstrations through task-level associations, without trajectory-specific pairing or temporal alignment. A second strength is the validation across held-out RoboTwin tasks, multiple demonstration sources, scene perturbations, real-world held-out layouts, and component ablations, which substantiates that grounded object-level intent drives generalization over language-only and video-conditioned baselines.
Critical weaknesses
The critical weaknesese are listed as follows:
- Lack of causal isolation for the core interface. The comparisons against Zero-WAM and other baselines confound the ORTS/target-grounding interface with different checkpoints, source-specific fine-tuning, training mixtures, and additional fixed perception/language models. The component ablations are restricted to H demonstrations in standard UR5-WSG scenes, so they do not establish that the interface itself drives gains across R1/R2 or scene perturbations. This is consequential because the paper’s main claim is about the interface, not merely the full system.
- Real-world evidence is too limited and confounded. The real-world evaluation covers a single bimanual task/platform with very few trials per layout and no confidence intervals or significance tests. The cross-layout demonstration-reuse comparison is also confounded: the reused videos were seen during adaptation, while current-layout videos were not, so it does not cleanly isolate geometric generalization. This matters because held-out-layout real-world generalization is a central empirical pillar.
- Cross-embodiment claim is overstated. R2 source embodiments are already in the training pool, and execution is fixed to UR5-WSG; no new robot body is controlled zero-shot. Although source actions are discarded, controlling a new embodiment still requires that embodiment’s action data, channel mapping, normalization statistics, and adaptation/training. H also uses generated human videos rather than real human demonstrations. This is consequential because “embodiment-agnostic” applies mainly to the conditioning input, not to policy deployment, yet cross-embodiment transfer is a core claimed contribution.
Questions for authors
- Can you isolate the ORTS/grounding contribution with identical backbone, training data, checkpoint selection, and inputs?
- Can you report ORTS/grounding failure rates and decompose success by extraction, grounding, and execution failures?
- Can you clarify what exactly is required to deploy on a new embodiment not in the training pool?
- Can you provide confidence intervals/significance for the real-world results and a clean cross-layout demonstration-reuse control?
3 comments
Thank you for the detailed response and the new experiments. The matched controls, including the no demonstration and raw video variants, address my main concern on isolating the interface. The clean cross layout control and the narrowed embodiment claim also resolve my other concerns, so I will raise my score. Two minor points remain: please state which demonstration sources the matched controls cover, and note that the real world results rest on small trial counts.
Thank you for your careful review and constructive feedback. We will make both points more explicit in the revised manuscript.
1. Demonstration Sources. The matched controls in Table 6 are evaluated on the seven held-out RoboTwin tasks under standard scenes with UR5-WSG execution. Demonstration-conditioned variants use the same fixed generated human demonstration (H) per task, whereas the no-demonstration variant receives none. During training, demonstration-conditioned variants share the same demonstration-source mixture as Full (Appendix C.7). The R1/R2 results are separate system-level comparisons. We clarify this evaluation scope directly in the Table 6 caption.
2. Real-World Trial Counts. The main candle-lighting comparison uses 10 trials per method per layout, while the new cross-layout reuse control uses 10 trials per condition per held-out layout. Both experiments therefore include 40 trials across the four held-out layouts (L2–L5). The scrambled-egg evaluation uses 20 trials per method. We report these counts explicitly in the revised manuscript and candidly acknowledge in the Limitations section that the relatively small sample sizes and limited task coverage introduce uncertainty. Accordingly, we will characterize the real-world results as preliminary evidence under the evaluated conditions.
Thanks for the careful review and constructive feedback, which helped strengthen our evaluation and clarify our claims. Our point-by-point responses follow:
Response to Q1.
We added matched controls sharing backbone initialization, execution training data, optimization, and the checkpoint-selection rule (Table 6, Appendix C.7). Full and matched raw video use identical demonstrations, language, robot observations, and task-level pairing. No demonstration retains instruction-derived grounded roles.
Ablation results (Table 6 of the revised manuscript).
\hline \text{Variant} & \text{Success (\\%)}\\,\uparrow & \text{Drop (pp)} \\\\ \hline \text{Full} & \mathbf{47.97} & \text{—} \\\\ \text{w/o OD} & 23.95 & 24.02 \\\\ \text{w/o TG} & 18.33 & 29.64 \\\\ \text{w/o IFP} & 32.14 & 15.83 \\\\ \text{Matched raw video} & 12.33 & 35.64 \\\\ \text{No demonstration} & 6.33 & 41.64 \\\\ \hline \end{array}$$ OD: object dynamics; TG: target grounding; IFP: future-chunk prediction. Under matched inputs and training, Full outperforms raw video (47.97% vs. 12.33%), while removing target grounding lowers success to 18.33%. These controls support structured demonstration conditioning and explicit target grounding. The evidence concerns the complete interface, including fixed perception. **Response to Q2.** We added a **300-rollout audit** across seven held-out tasks: 100 per source (R1, R2, and H; Table 8, Appendix C.9). R2 uses Franka-Panda demonstrations. Failed rollouts are classified hierarchically: extraction errors; missing, ambiguous, or incorrect required-role bindings after extraction passes; or execution failures after both checks pass. **Outcome decomposition.** $$\begin{array}{lcccc} \hline \text{Source} & \text{Task success (\\%)} & \text{Extraction failure (\\%)} & \text{Grounding failure (\\%)} & \text{Execution failure (\\%)} \\\\ \hline \text{R1} & 48 & 4 & 8 & 40 \\\\ \text{R2} & 46 & 3 & 10 & 41 \\\\ \text{H} & 45 & 7 & 6 & 42 \\\\ \hline \end{array}$$ Failures with detected extraction or grounding errors account for **12–13%** of rollouts, while **40–42%** fail after both checks pass, making execution the largest remaining failure category. These are outcome categories, not standalone module error rates. **Response to Q3.** We agree that the scope should be stated more precisely. Our **“embodiment-agnostic”** claim concerns the demonstration-conditioning interface: ORTS excludes source actions and joint trajectories. Deployment on a new execution robot still requires robot-specific action data, compatible observation inputs, action-channel mapping, normalization statistics, and policy training or adaptation. We do not claim zero-shot control of an unseen robot. In RoboTwin, execution is fixed to UR5-WSG, and R2 source embodiments are included in the training pool. Our results therefore demonstrate transfer across demonstration sources within this setting. H demonstrations in simulation are generated; our real-world experiments use filmed human demonstrations. **Response to Q4.** We added **two-sided 95% Wilson confidence intervals** for the new real-world evaluations. Scrambled-egg results appear in Table 5, with an additional Zero-WAM baseline in Appendix B.9. **Extended scrambled-egg evaluation.** $$\begin{array}{lcc} \hline \text{Method} & \text{Successes/trials (\\%)} & \text{95\\% CI (\\%)} \\\\ \hline \text{Full IntentWAM} & \mathbf{8/20\\;(40.0)} & \text{21.9–61.3} \\\\ \text{Matched raw video} & 1/20\\;(5.0) & \text{0.9–23.6} \\\\ \text{No demonstration} & 0/20\\;(0.0) & \text{0.0–16.1} \\\\ \text{Without relational edges} & 5/20\\;(25.0) & \text{11.2–46.9} \\\\ \text{Zero-WAM} & 3/20\\;(15.0) & \text{5.2–36.0} \\\\ \hline \end{array}$$ For cross-layout reuse, we added a control comparing current-layout demonstrations with newly recorded $L_1$ demonstrations, **both excluded from training, adaptation, and model selection**. Both conditions use the same frozen IntentWAM checkpoint and execution settings, with 10 fresh rollouts at each of $L_2$–$L_5$ (40 per condition). These trials are separate from the original reuse experiment (Table 4, Appendix B.8). **Cross-layout reuse with unseen demonstrations.** $$\begin{array}{lcc} \hline \text{Demonstration} & \text{Successes/trials (\\%)} & \text{95\\% CI (\\%)} \\\\ \hline \text{Current-layout} & 28/40\\;(70.0) & \text{54.6–81.9} \\\\ \text{Reused }L_1 & 18/40\\;(45.0) & \text{30.7–60.2} \\\\ \hline \end{array}$$ Reusing unseen $L_1$ demonstrations achieves **45.0%** success versus **70.0%** with current-layout demonstrations, an observed decrease of **25.0 percentage points**. This control removes the mismatch in demonstration training exposure and shows lower observed reliability for cross-layout reuse. Our revised discussion explicitly acknowledges this limitation.