- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes WATCH, a cross-demonstration pre-training framework for world-action models. Given a reference human demonstration and its actions, together with a target demonstration’s current observation and state, the model jointly predicts the target’s future video and actions. Training pairs are constructed using task labels and trajectory-based matching across human data sources, aiming to learn transferable interaction representations without requiring paired human–robot demonstrations during this stage. The pretrained model is subsequently adapted to robot control, with or without reference demonstrations. Experiments across two backbones and three simulation benchmarks show improvements over within-video pre-training and co-training baselines, with the largest gains in reference-conditioned MimicDroid evaluation.
Critical strengths
-
The method achieves substantial improvements on the challenging MimicDroid evaluation levels under reference-conditioned control, particularly when transferring to unseen layouts.
-
The experiments demonstrate transfer benefits across different robot embodiments and backbones. Improvements also persist without reference demonstrations during robot post-training and evaluation, suggesting that the learned prior is useful beyond reference-conditioned execution.
-
The paper addresses a valuable problem: learning useful priors for robot control from human demonstrations while reducing reliance on paired human–robot data during pre-training.
Critical weaknesses
-
The largest performance gains are observed under reference-conditioned evaluation, whereas improvements over the within-video and co-training baselines are more modest in the context-free setting. While the former demonstrates the effectiveness of the overall system, a central claim concerns learning transferable representations. The authors should explain this discrepancy and clarify to what extent the gains reflect broadly transferable features versus an improved ability to exploit reference demonstrations.
-
A major motivation is that the proposed pre-training setting can leverage scalable human demonstration data without requiring paired human–robot demonstrations. However, the practical benefits of this reduced data requirement are not sufficiently established relative to prior cross-demonstration methods. The authors should provide clearer evidence of the resulting gains in data efficiency, collection cost, or downstream transfer. This distinction is important for assessing whether the work is primarily an exploratory extension of existing training paradigms or delivers a substantial practical advance.
-
The representation analysis in Fig. 5 may be confounded by differences in conditioning information. WATCH additionally receives the future actions of a matched reference, whereas the baselines do not. These inputs are directly relevant to motion retrieval and future-motion regression, making it difficult to attribute the gains to improved representations alone. An input-matched analysis would provide stronger support for the claimed transferability.
Questions for authors
See weaknesses.
9 comments
W2: Practical value of human-only pre-training
WATCH improves robot performance through human-only cross-demonstration pre-training, with target-robot post-training data. This stage requires no matched human–robot trajectories. Its practical value combines the collection efficiency reported in prior work with the transfer and scaling gains demonstrated here.
Data collection efficiency. Prior work reports substantially faster demonstration collection with human hands or handheld interfaces than with robot teleoperation:
| Study | Reported collection throughput |
|---|---|
| EgoMimic [1], ICRA 2025 | 1,400 human vs. 135 robot demonstrations/hour (≈10×), on continuous object-in-bowl collection. |
| DexWild [2], RSS 2025 | 201 human vs. 43 GELLO demonstrations/hour (reported 4.6×), across five tasks. |
| UMI [3], RSS 2024 | 111 handheld-gripper vs. 35 SpaceMouse demonstrations/hour (>3×), on cup arrangement. |
Transfer with fixed robot data. WATCH translates the opportunity offered by efficient human-data collection into downstream gains. With the human corpus and target-robot data fixed, it improves context-free success over within-video pre-training from 64.6% to 69.0% on MimicDroid, 61.6% to 65.4% on TableTop-Sim, and 58.3% to 63.6% on RoboCasa, without human references at deployment.
Comparison with a cross-demonstration method. Vid2Robot [4] learns cross-demonstration correspondence from human–robot pairs. Trained following its released recipe on the same MimicDroid human references and robot demonstrations, it reaches 47.6% in-context success, compared with 68.3% for WATCH (Table 2). WATCH obtains this gain by adding a pre-training stage on human–human pairs, with no additional robot data.
Scaling human versus robot pairs. The revised Fig. 9 compares human-pair and robot-pair pre-training on a shared collection-budget axis, with the backbone, target-robot data, and post-training protocol fixed. Under the central 8× cost assumption, human pairs reach higher success at the same or a lower estimated budget on every evaluated in-context and context-free level. On in-context L2, the full human corpus reaches 72.5%, compared with 49.0% for the largest robot-pair setting, which uses about twice the estimated collection budget.
Collection-budget analysis. Figure 9 expresses each budget as the amount of paired video multiplied by its assumed unit collection cost, normalized by the estimated cost of the full human corpus. The robot-to-human cost ratio is 8× for the central curves, with a 4×–16× sensitivity band (§F.3). Reported collection throughput [1–3] motivates these assumptions. These are data-acquisition estimates, not measured monetary ratios or total training costs.
Data accounting. Section C.3 reports 10.99M query–reference links and 10.0K paired window-hours. These include overlapping and reused windows, rather than 10.0K hours of independently collected footage.
We have moved the human-versus-robot collection-budget comparison into the main text (§4.3, Fig. 9), with the cost assumptions detailed in §F.3. Section C.3 documents the window-hour accounting, and Table 4 reports transfer with fixed target-robot data.
References
[1] Kareer et al. EgoMimic: Scaling Imitation Learning via Egocentric Video. ICRA 2025.
[2] Tao et al. DexWild: Dexterous Human Interactions for In-the-Wild Robot Policies. RSS 2025.
[3] Chi et al. Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots. RSS 2024.
[4] Jain et al. Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformers. RSS 2024.
W3: Input-matched representation analysis
We agree that the original analysis confounds learned representations with the additional reference input. We have therefore repeated the analysis with identical inputs for the original video model, the within-video model, and WATCH. All receive the same query observation, state history, and matched reference future actions, with reference video slots blank. As detailed in §E.5, the conditioning reference comes from the query’s own source or robot, while retrieval evaluates matching across sources or robots. The windows, layers 10–27, retrieval pairs, and regression protocol are shared.
| Model with matched inputs | Human retrieval AUC | Robot retrieval AUC | Human regression | Robot regression |
|---|---|---|---|---|
| Original video model | 0.53 | 0.61 | 0.10 | 0.03 |
| Within-video | 0.67 | 0.67 | 0.15 | 0.25 |
| WATCH | 0.79 | 0.73 | 0.18 | 0.28 |
With matched inputs, WATCH achieves the highest scores across all four metrics, improving human and robot retrieval AUC by 0.12 and 0.06 over within-video pre-training, respectively.
We have updated Fig. 5 in the main text with these results.
W1: Transfer with and without references
We view the in-context and context-free gains as two uses of the same learned correspondence, rather than as two separate abilities. Using a human reference for robot control requires relating the same motion across demonstrators, scenes, and embodiments, which is the transferable representation that WATCH aims to learn. In-context control uses this correspondence directly through the reference slots, whereas context-free control can use it only through the learned features. On in-context MimicDroid, WATCH improves over within-video pre-training by 17.1 points (51.2% to 68.3%) and over co-training by 8.3 points (60.0% to 68.3%), with the same reference protocol during robot post-training and evaluation (Table 4).
The in-context gain follows correspondence quality. The pairing comparisons indicate that this gain does not come from exposure to reference conditioning during pre-training. Random same-task pre-training uses the same reference slots and cross-source references as WATCH, yet reaches 38.8% on in-context MimicDroid, below the 40.9% obtained without pre-training. With motion-matched references, same-source top-1 reaches 58.6% and WATCH reaches 68.3% (Table 5). Under the same input format, in-context performance therefore follows the quality of the correspondence learned during pre-training.
The same correspondence is present without references. With identical feature-extraction inputs, WATCH improves human cross-source motion retrieval AUC from 0.67 to 0.79 and cross-robot retrieval AUC from 0.67 to 0.73 over within-video pre-training (W3). It also improves robot control when no reference is provided during robot post-training or evaluation:
Simulation: Table 4. Real robot: mean over three tasks and Easy/Hard conditions (Table 15). —: not evaluated. Gains are percentage points.
Without a reference, the learned correspondence benefits control only through the features, and the gains are smaller: 3.8 to 5.3 points over within-video pre-training and 1.9 to 4.4 points over co-training in simulation. Both baselines learn from the same human data, so these gains measure the contribution of cross-demonstration prediction beyond human-data learning. On the physical robot, WATCH improves over within-video pre-training by 13.3 points under the same robot post-training protocol (Table 15). Cosmos3 is already trained on human and robot action data under a within-video objective (§4.1). Applying WATCH to it improves context-free success from 49.7% to 63.4% on MimicDroid and from 65.8% to 72.0% on TableTop-Sim. The gain on RoboCasa is small (55.0% to 55.8%); there, the base model has already seen robot data and WATCH pre-training additionally includes RoboMIND (Table 3).
We added the discussion on the gains relative to the stronger human-data baselines and their interpretation across downstream settings (§4.3). Fig. 5 and §E.5 report the input-matched analysis, and §4.4 and Table 15 report the context-free physical-robot evaluation.