acceptodds
Under review as a conference paper at ICLR 2027

Learning Transferable World-Action Models from Task-Paired Human Videos

Abstract

World-action models inherit broad visual and physical priors from video pre-training. However, standard within-video next-frame prediction allows them to rely on appearance shared between the context and future, rather than learning interactions that transfer across scenes and embodiments. We introduce WATCH, a cross-video pre-training that learns from task-paired human demonstrations collected across diverse environments. Conditioned on the video and actions of one demonstration, the model jointly predicts another demonstration’s actions and future frames from its current observation. Predicting one demonstration from another encourages the model to learn task-relevant interactions that transfer across scenes and demonstrators. Our pre-training shows that these capabilities can be learned from readily scalable human–human pairs alone, without requiring matched human–robot demonstrations. Analyses before robot post-training show improved motion alignment across embodiments and better action and video prediction on unseen robots, with representations closer to those learned from robot pairs. After robot post-training, WATCH improves MimicDroid in-context success from 40.9% to 68.3% with Cosmos Policy and from 59.8% to 77.7% with Cosmos3, with the largest gains on the unseen levels. Gains persist under context-free post-training across robot embodiments. WATCH outperforms within-video pre-training and co-training baselines, supporting cross-video prediction as a scalable approach to learning transferable world-action representations.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Action Model, In-context Learning, Egocentric Video

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_jcv5OpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes WATCH, a cross-demonstration pre-training framework for world-action models. Given a reference human demonstration and its actions, together with a target demonstration’s current observation and state, the model jointly predicts the target’s future video and actions. Training pairs are constructed using task labels and trajectory-based matching across human data sources, aiming to learn transferable interaction representations without requiring paired human–robot demonstrations during this stage. The pretrained model is subsequently adapted to robot control, with or without reference demonstrations. Experiments across two backbones and three simulation benchmarks show improvements over within-video pre-training and co-training baselines, with the largest gains in reference-conditioned MimicDroid evaluation.

Critical strengths
  1. The method achieves substantial improvements on the challenging MimicDroid evaluation levels under reference-conditioned control, particularly when transferring to unseen layouts.

  2. The experiments demonstrate transfer benefits across different robot embodiments and backbones. Improvements also persist without reference demonstrations during robot post-training and evaluation, suggesting that the learned prior is useful beyond reference-conditioned execution.

  3. The paper addresses a valuable problem: learning useful priors for robot control from human demonstrations while reducing reliance on paired human–robot data during pre-training.

Critical weaknesses
  1. The largest performance gains are observed under reference-conditioned evaluation, whereas improvements over the within-video and co-training baselines are more modest in the context-free setting. While the former demonstrates the effectiveness of the overall system, a central claim concerns learning transferable representations. The authors should explain this discrepancy and clarify to what extent the gains reflect broadly transferable features versus an improved ability to exploit reference demonstrations.

  2. A major motivation is that the proposed pre-training setting can leverage scalable human demonstration data without requiring paired human–robot demonstrations. However, the practical benefits of this reduced data requirement are not sufficiently established relative to prior cross-demonstration methods. The authors should provide clearer evidence of the resulting gains in data efficiency, collection cost, or downstream transfer. This distinction is important for assessing whether the work is primarily an exploratory extension of existing training paradigms or delivers a substantial practical advance.

  3. The representation analysis in Fig. 5 may be confounded by differences in conditioning information. WATCH additionally receives the future actions of a matched reference, whereas the baselines do not. These inputs are directly relevant to motion retrieval and future-motion regression, making it difficult to attribute the gains to improved representations alone. An input-matched analysis would provide stronger support for the claimed transferability.

Questions for authors

See weaknesses.

9 comments

W1: Transfer with and without references

We view the in-context and context-free gains as two uses of the same learned correspondence, rather than as two separate abilities. Using a human reference for robot control requires relating the same motion across demonstrators, scenes, and embodiments, which is the transferable representation that WATCH aims to learn. In-context control uses this correspondence directly through the reference slots, whereas context-free control can use it only through the learned features. On in-context MimicDroid, WATCH improves over within-video pre-training by 17.1 points (51.2% to 68.3%) and over co-training by 8.3 points (60.0% to 68.3%), with the same reference protocol during robot post-training and evaluation (Table 4).

The in-context gain follows correspondence quality. The pairing comparisons indicate that this gain does not come from exposure to reference conditioning during pre-training. Random same-task pre-training uses the same reference slots and cross-source references as WATCH, yet reaches 38.8% on in-context MimicDroid, below the 40.9% obtained without pre-training. With motion-matched references, same-source top-1 reaches 58.6% and WATCH reaches 68.3% (Table 5). Under the same input format, in-context performance therefore follows the quality of the correspondence learned during pre-training.

The same correspondence is present without references. With identical feature-extraction inputs, WATCH improves human cross-source motion retrieval AUC from 0.67 to 0.79 and cross-robot retrieval AUC from 0.67 to 0.73 over within-video pre-training (W3). It also improves robot control when no reference is provided during robot post-training or evaluation:

Success rate (%)MimicDroidTableTop-SimRoboCasaReal robot
Within-video64.661.658.366.7
Co-training66.161.061.7—
WATCH69.065.463.680.0
Gain over within-video+4.4+3.8+5.3+13.3
Gain over co-training+2.9+4.4+1.9—

Simulation: Table 4. Real robot: mean over three tasks and Easy/Hard conditions (Table 15). —: not evaluated. Gains are percentage points.

Without a reference, the learned correspondence benefits control only through the features, and the gains are smaller: 3.8 to 5.3 points over within-video pre-training and 1.9 to 4.4 points over co-training in simulation. Both baselines learn from the same human data, so these gains measure the contribution of cross-demonstration prediction beyond human-data learning. On the physical robot, WATCH improves over within-video pre-training by 13.3 points under the same robot post-training protocol (Table 15). Cosmos3 is already trained on human and robot action data under a within-video objective (§4.1). Applying WATCH to it improves context-free success from 49.7% to 63.4% on MimicDroid and from 65.8% to 72.0% on TableTop-Sim. The gain on RoboCasa is small (55.0% to 55.8%); there, the base model has already seen robot data and WATCH pre-training additionally includes RoboMIND (Table 3).

We added the discussion on the gains relative to the stronger human-data baselines and their interpretation across downstream settings (§4.3). Fig. 5 and §E.5 report the input-matched analysis, and §4.4 and Table 15 report the context-free physical-robot evaluation.

W2: Practical value of human-only pre-training

WATCH improves robot performance through human-only cross-demonstration pre-training, with target-robot post-training data. This stage requires no matched human–robot trajectories. Its practical value combines the collection efficiency reported in prior work with the transfer and scaling gains demonstrated here.

Data collection efficiency. Prior work reports substantially faster demonstration collection with human hands or handheld interfaces than with robot teleoperation:

StudyReported collection throughput
EgoMimic [1], ICRA 20251,400 human vs. 135 robot demonstrations/hour (≈10×), on continuous object-in-bowl collection.
DexWild [2], RSS 2025201 human vs. 43 GELLO demonstrations/hour (reported 4.6×), across five tasks.
UMI [3], RSS 2024111 handheld-gripper vs. 35 SpaceMouse demonstrations/hour (>3×), on cup arrangement.

Transfer with fixed robot data. WATCH translates the opportunity offered by efficient human-data collection into downstream gains. With the human corpus and target-robot data fixed, it improves context-free success over within-video pre-training from 64.6% to 69.0% on MimicDroid, 61.6% to 65.4% on TableTop-Sim, and 58.3% to 63.6% on RoboCasa, without human references at deployment.

Comparison with a cross-demonstration method. Vid2Robot [4] learns cross-demonstration correspondence from human–robot pairs. Trained following its released recipe on the same MimicDroid human references and robot demonstrations, it reaches 47.6% in-context success, compared with 68.3% for WATCH (Table 2). WATCH obtains this gain by adding a pre-training stage on human–human pairs, with no additional robot data.

Scaling human versus robot pairs. The revised Fig. 9 compares human-pair and robot-pair pre-training on a shared collection-budget axis, with the backbone, target-robot data, and post-training protocol fixed. Under the central 8× cost assumption, human pairs reach higher success at the same or a lower estimated budget on every evaluated in-context and context-free level. On in-context L2, the full human corpus reaches 72.5%, compared with 49.0% for the largest robot-pair setting, which uses about twice the estimated collection budget.

Collection-budget analysis. Figure 9 expresses each budget as the amount of paired video multiplied by its assumed unit collection cost, normalized by the estimated cost of the full human corpus. The robot-to-human cost ratio is 8× for the central curves, with a 4×–16× sensitivity band (§F.3). Reported collection throughput [1–3] motivates these assumptions. These are data-acquisition estimates, not measured monetary ratios or total training costs.

Data accounting. Section C.3 reports 10.99M query–reference links and 10.0K paired window-hours. These include overlapping and reused windows, rather than 10.0K hours of independently collected footage.

We have moved the human-versus-robot collection-budget comparison into the main text (§4.3, Fig. 9), with the cost assumptions detailed in §F.3. Section C.3 documents the window-hour accounting, and Table 4 reports transfer with fixed target-robot data.

References

[1] Kareer et al. EgoMimic: Scaling Imitation Learning via Egocentric Video. ICRA 2025.
[2] Tao et al. DexWild: Dexterous Human Interactions for In-the-Wild Robot Policies. RSS 2025.
[3] Chi et al. Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots. RSS 2024.
[4] Jain et al. Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformers. RSS 2024.

W3: Input-matched representation analysis

We agree that the original analysis confounds learned representations with the additional reference input. We have therefore repeated the analysis with identical inputs for the original video model, the within-video model, and WATCH. All receive the same query observation, state history, and matched reference future actions, with reference video slots blank. As detailed in §E.5, the conditioning reference comes from the query’s own source or robot, while retrieval evaluates matching across sources or robots. The windows, layers 10–27, retrieval pairs, and regression protocol are shared.

Model with matched inputsHuman retrieval AUCRobot retrieval AUCHuman regressionRobot regression
Original video model0.530.610.100.03
Within-video0.670.670.150.25
WATCH0.790.730.180.28

With matched inputs, WATCH achieves the highest scores across all four metrics, improving human and robot retrieval AUC by 0.12 and 0.06 over within-video pre-training, respectively.

We have updated Fig. 5 in the main text with these results.

Reviewer_y6hQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces WATCH, a cross-video pre-training framework for learning transferable world-action representations from task-paired human demonstrations. The model predicts the future video and actions of one demonstration conditioned on another demonstration of the same task, encouraging motion and interaction representations that generalize across scenes and embodiments. Experiments on MimicDroid, TableTop-Sim, and RoboCasa show consistent improvements over within-video pre-training and co-training baselines, with particularly strong gains on unseen environments.

Critical strengths
  1. Meaningful pre-training formulation: WATCH uses task-paired human demonstrations to learn transferable motion representations without requiring matched human-robot data. The cross-video prediction objective provides a reasonable way to reduce reliance on scene-specific visual cues.

  2. Comprehensive experimental validation: The method shows substantial improvements across two backbones and multiple benchmarks. The comparisons with within-video pre-training and co-training, together with representation analyses and ablation studies, provide encouraging evidence for its effectiveness.

Critical weaknesses
  1. Limited novelty in the underlying learning mechanism: Cross-demonstration conditioning and cross-embodiment representation learning have been explored in prior work, including Vid2Robot (Jain et al., 2024) and Zhu et al. (2025). WATCH mainly extends these ideas to human-human paired pre-training using an existing joint video-action prediction architecture. The paper could better clarify what is fundamentally new beyond the pairing strategy and training formulation.

  2. Insufficient isolation of the cross-video learning mechanism: The paired demonstrations are selected using motion-based matching, which may itself provide strong supervision. Moreover, the same-source top-1 ablation changes both source diversity and reference selection. It remains unclear how much of the improvement comes specifically from cross-source prediction rather than motion-aligned pairing or reference quality.

[1] Vid2Robot: Jain, V., et al. (2024). Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformers. Robotics: Science and Systems (RSS 2024).

[2] UniSkill: Kim, H., et al. (2025). UniSkill: Imitating Human Videos via Cross-Embodiment Skill Representations. Conference on Robot Learning (CoRL 2025).

Questions for authors

1: What is the key methodological distinction between WATCH and prior cross-demonstration approaches such as Vid2Robot (Jain et al., 2024) and Zhu et al. (2025), beyond using human-human pairs for pre-training?

2: Could the authors further disentangle the contributions of cross-source diversity, motion-based pairing, and reference selection? In particular, would cross-video pre-training still provide comparable gains with randomly selected same-task references under matched training settings?

3 comments

W1 and Q1: Methodological contribution

The key methodological distinction is how cross-demonstration correspondence is supervised. WATCH trains a joint world-action predictor to use an interaction from one demonstration to predict both the future motion and visual outcome of another. This couples correspondence learning directly to action–video prediction before target-robot adaptation (§2.3).

Compared with Vid2Robot. Vid2Robot [1] learns robot-action prediction from a prompt video and current robot observations, with auxiliary contrastive losses for representation alignment. WATCH supervises the correspondence through joint prediction of the query’s future actions and frames. The model is trained to use the reference for both the query motion and its visual unfolding. This changes the prediction targets and the way correspondence enters the learning objective.

Compared with Zhu et al. (2025). UWM [2] couples action and video diffusion using actions and observations from the same trajectory. WATCH changes this within-trajectory prediction problem by conditioning the query future on the video and motion of a different demonstration. The reference can show the interaction in a different scene and viewpoint, so the model is encouraged to relate the shared interaction to the query’s own visual and geometric context.

Compared with MimicDroid and Zero-WAM. MimicDroid [3] also trains across human demonstrations without robot pairs, but predicts actions only and conditions on a demonstration at deployment. Zero-WAM [4] jointly predicts video and actions across demonstrations, but learns from human-to-robot pairs and executes tasks conditioned on a human video prompt. WATCH shares the cross-demonstration objective with these methods, and differs in using it to pre-train a prior that does not depend on a demonstration at test time.

Empirical consequence. Because the cross-demonstration objective serves as pre-training, its benefit carries over to context-free control, a setting that demonstration-conditioned formulations do not address. Without any reference during robot post-training or evaluation, WATCH improves Cosmos Policy from 49.3% to 69.0% on MimicDroid, from 59.6% to 65.4% on TableTop-Sim, and from 56.0% to 63.6% on RoboCasa (Table 3). On the physical humanoid it improves from 53.3% to 80.0% (Table 15). In the in-context setting, WATCH reaches 68.3% on MimicDroid, compared with 47.6% for Vid2Robot trained on the same robot data (Table 2).

Role of human–human pre-training. Human pairs let this prediction objective scale without paired robot data. Subsequent robot post-training retains the reference slots for in-context control or removes them for context-free control. Demonstrations therefore supervise a joint prediction prior that can support either deployment setting.

Appendix A and Table 6 compare WATCH with these methods in terms of training objective, prediction targets, data requirements, and test-time inputs.

References

[1] Jain et al. Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformers. RSS 2024.
[2] Zhu et al. Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets. RSS 2025.
[3] Shah et al. MimicDroid: In-Context Learning for Humanoid Robot Manipulation from Human Play Videos. arXiv:2509.09769, 2025.
[4] Zhou et al. Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalization. arXiv:2608.26103, 2026.

W2 and Q2: Pairing and reference selection

Our comparisons support combining motion-based pairing with a reference pool spanning sources. Motion matching is particularly important for in-context learning: random same-task references retain cross-source diversity but yield only 38.8% on MimicDroid, compared with 68.3% for WATCH, a 29.5-point difference. Motion-matched same-source references recover much of this performance, while WATCH improves further by using its broader reference pool.

All variants in the table below keep the backbone, the human-video corpus, the pre-training recipe, and the robot post-training fixed, and differ only in how references are selected (§4.3).

Pre-trainingCross-sourceMotion-matchedMimicDroid in-context L1 / L2 / L3 / MeanTableTop-Sim context-free Easy / Hard / Mean
None––75.0 / 29.4 / 18.3 / 40.969.6 / 49.6 / 59.6
Within-video––85.8 / 44.4 / 23.3 / 51.271.6 / 51.6 / 61.6
Random same-task✓✗75.8 / 31.5 / 9.2 / 38.870.8 / 48.8 / 59.8
Same-source top-1✗✓84.2 / 58.3 / 33.3 / 58.669.6 / 51.2 / 60.4
WATCH✓*✓91.7 / 72.5 / 40.8 / 68.376.8 / 54.0 / 65.4

* Cross-source indicates inclusion of cross-source references. WATCH samples from a pool containing both same-source and cross-source candidates.

Motion-based pairing. Removing motion matching produces a much larger performance drop in-context (68.3% to 38.8%) than context-free (65.4% to 59.8%). On unseen L3 layouts, in-context success falls from 40.8% to 9.2%. Random same-task pre-training does not ensure that reference motion corresponds to the query motion. Its poor in-context performance is consistent with our hypothesis that such pre-training develops a weaker ability to use fine-grained reference motion. Task agreement and cross-source diversity alone are therefore insufficient to recover WATCH’s in-context gains.

Cross-source references. With motion matching retained, same-source top-1 reaches 58.6% on MimicDroid and 60.4% on TableTop-Sim. WATCH reaches 68.3% and 65.4%, improvements of 9.7 and 5.0 points, respectively. This comparison changes two factors at once. Same-source top-1 restricts candidates to the query's own source and conditions on its single best-ranked reference, whereas WATCH samples uniformly from up to 15 motion-matched references, on average 4.2 from the same source and 6.4 from other sources (§C.2). We therefore attribute this difference to the WATCH reference set as a whole, and do not separate the contribution of cross-source references from that of sampling among several matched references. Together, the comparisons indicate that both motion correspondence and the broader reference pool contribute to the effectiveness of the proposed design.

We have added the random same-task comparison in §4.3 and Table 5. Section 4.3 describes the random and same-source top-1 variants, and §C.2 specifies WATCH’s candidate pool and sampling rule.

Dear Reviewer y6hQ,

Thank you again for your constructive feedback. We would like to follow up on our responses above, including the clarification of our methodological contribution and the requested random same-task pairing experiment.

As the discussion period draws to a close, we would be grateful to hear whether our responses have addressed your concerns. If any questions remain, we would be happy to provide further clarification.

Best regards, The Authors

Reviewer_yw5wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes WATCH, a cross-video pre-training framework that learns transferable world-action representations from task-paired human demonstrations. By conditioning on one demonstration to predict another's future actions and video frames, WATCH encourages the learning of task-relevant interactions across different environments and embodiments. Experiments demonstrate improvements in action/video prediction and downstream robot manipulation performance across multiple simulation benchmarks.

Critical strengths

The proposed cross-video pre-training objective is intuitive and well-motivated, offering an interesting approach to learning transferable representations from human demonstrations without requiring paired human–robot data.

Critical weaknesses

W1. Lack of Real-World Robot Evaluation

A central objective of this paper is to learn transferable world-action representations from human videos that generalize across different embodiments. However, all downstream robot control experiments are conducted in simulated environments, including MimicDroid, TableTop-Sim, and RoboCasa, without any validation on physical robots.

Real-world robotic manipulation involves challenges that may not be adequately captured in simulation, such as complex contact dynamics, perception noise, control latency, and uncertainties in physical object interactions. Consequently, it remains unclear whether the improvements achieved through the proposed pre-training method would translate to real-world robotic manipulation.

Notably, closely related work such as UniSkill [1] has demonstrated human-to-robot skill transfer through experiments on physical Franka robots. Although the research objectives and methodological approaches are not identical, both studies emphasize the transfer of knowledge from human demonstrations to robotic manipulation. Given that cross-embodiment transferability is a central claim of WATCH, I believe that at least a small-scale real-world evaluation would be important to substantiate its practical effectiveness.

W2. Restrictive Data Requirements and Limited Scalability

The paper highlights scalable pre-training from large-scale human video data, without requiring paired human–robot demonstrations, as one of its main contributions. However, the proposed training pipeline relies on considerably more structured information than raw RGB human videos alone.

Specifically, WATCH requires:

  • Task category annotations for human demonstrations.

  • 3D hand/wrist poses and fingertip trajectories.

  • Motion-based pairing across different demonstrations.

  • A shared Cartesian action representation that enables correspondence between human and robot actions.

These requirements impose substantially stronger constraints on the human video data compared to approaches such as UniSkill, which can learn skill representations from unpaired RGB human and robot videos without explicitly requiring 3D hand trajectories or task-matched demonstration pairs.

In particular, accurately recovering 3D hand poses from unconstrained, in-the-wild videos can be challenging due to variations in camera viewpoints, occlusions, and video quality. Such factors may introduce substantial estimation errors, yet the robustness of the proposed pairing procedure and pre-training objective to noisy or imperfect hand trajectories is not sufficiently investigated.

Furthermore, although the paper constructs cross-video pairs from different collection sources, the pre-training pairs are derived from a single structured human video dataset, EgoVerse. It therefore remains unclear whether the proposed method can be effectively extended to more diverse, uncurated human video collections, where reliable task annotations and 3D motion information may not be readily available.

Given that scalability is a key motivation of the proposed approach, I believe that further analysis of its sensitivity to imperfect motion estimates and task annotations, as well as its applicability to less structured human video datasets, would be valuable in substantiating the paper's scalability claims.

[1] Kim, Hanjung, et al. "Uniskill: Imitating human videos via cross-embodiment skill representations." arXiv preprint arXiv:2505.08787 (2025).

Questions for authors

Q1. Real-World Robot Evaluation

Could the authors provide a real-world evaluation on a physical Franka robot or another robotic platform? In particular, it would be valuable to compare the original Cosmos Policy, within-video pre-training, and WATCH under the same robot post-training data and evaluation protocol, including scenarios with unseen object configurations or environmental variations. Such an experiment would help establish whether the improvements observed in simulation translate to real-world robotic manipulation.

Q2. Robustness and Scalability to Less Structured Human Videos

How robust is the proposed method to incomplete or noisy task annotations? Could the authors provide an ablation study evaluating the impact of imperfect task labels on the pairing procedure and downstream performance? Additionally, could the authors investigate whether WATCH can be extended to uncurated RGB human videos without explicit 3D hand pose annotations? Such experiments would help clarify the practical scalability of the proposed pre-training framework beyond structured datasets such as EgoVerse.

7 comments

W1 and Q1: Real-world robot evaluation

We thank the reviewer for the suggestion. We conducted a physical-robot evaluation comparing Cosmos Policy, within-video pre-training, and WATCH with identical robot post-training data and evaluation protocols. WATCH achieves 80.0% mean success, compared with 66.7% for within-video pre-training and 53.3% for Cosmos Policy.

Setup. A humanoid performs whole-body manipulation using two arms, parallel grippers, and an egocentric stereo camera. The policy receives the left camera view at 224 × 224 and outputs base velocity, both end-effector poses in the torso frame, and gripper commands. All models use context-free post-training and execution, with no reference demonstration.

We use 20 teleoperated demonstrations for each of three tasks, totaling 60 episodes and approximately 11.5 minutes at 15 Hz. Each model is evaluated for 10 trials in each task and condition. Easy uses the post-training scene. Hard introduces an unseen drawer for Open Drawer, an unseen cup for PnP Drawer, and a stuffed toy replacing the tumbler for Take & Close.

Success rate (%)Open Drawer Easy / HardPnP Drawer Easy / HardTake & Close Easy / HardMean
Cosmos Policy90 / 8040 / 4060 / 1053.3
Within-video100 / 10060 / 6070 / 1066.7
WATCH100 / 10080 / 6090 / 5080.0

Results. WATCH improves mean success by 13.3 percentage points over within-video pre-training and by 26.7 points over Cosmos Policy. Across the three Hard conditions, the corresponding means are 70.0%, 56.7%, and 43.3%. The largest observed difference from within-video is on Take & Close Hard, where the robot must retrieve a deformable toy and close the drawer using the other arm: WATCH succeeds in 5/10 trials, compared with 1/10 for both baselines. These results extend the simulation evidence to physical humanoid manipulation with unseen objects and a different drawer. WATCH achieves the highest mean success in both Easy and Hard conditions under the same robot post-training protocol.

We have added the physical-robot evaluation in §4.4 and Fig. 10. Appendix G.6 reports the setup and evaluation protocol, per-condition results in Table 15, demonstration and rollout snapshots in Figs. 23 and 25, and real-robot video is in supplementary materials. We also plan to evaluate in-context post-training on the physical robot during the remaining rebuttal period and incorporate any results available by the rebuttal deadline into the revised manuscript.

W2: Data requirements

Our scalability argument rests on two practical advantages: human videos are easier to acquire than robot demonstrations or matched human–robot pairs, and the task and motion information needed to construct training pairs can be supplied by existing annotation and pose-estimation tools. Both support scaling action-supervised pre-training without scaling robot collection.

Collection efficiency. Human demonstrations avoid robot hardware and teleoperation during pre-training data collection. The revised main-text Fig. 9 compares human-pair and robot-pair pre-training on a collection-budget axis under the same target-robot post-training protocol (§4.3). Under the central 8× robot/human unit collection-cost assumption, human pairs achieve higher success at the same or a lower estimated budget on every evaluated level. Section F.3 explains this assumption and the 4–16× sensitivity range, motivated by reported collection throughput [1–3].

Obtaining the required information. Task labels can be derived automatically from free-form task descriptions, as they already are for MECKA, about half of our corpus (§G.5). Such descriptions can come from dataset annotations or from existing vision–language models. Off-the-shelf estimators such as HaWoR [4] recover hand motion and camera trajectories from RGB video. Pairing then runs automatically on the task and motion estimates, without manually annotating demonstration pairs. Existing egocentric datasets already provide these signals: EgoDex [5], for example, includes language annotations, 3D hand and finger poses, and camera transforms. These options make the required structure practical to obtain at scale.

Cartesian action space. Human and robot motion is expressed in end-effector coordinates with fingertip or gripper terms. This representation enables transfer without matched human–robot examples.

Imperfect labels and motion. The corpus already spans capture pipelines with different tracking noise. Aria tracks show the largest jitter and Scale tracks the most dropped frames, and together they make up half of the episodes (Table 12). Across noise quintiles within each source, cross-source coverage does not drop and the best pairing distance shows no trend with the noise (Spearman |ρ| ≤ 0.03 for Aria and Scale; §G.5). We report downstream results under corrupted task labels in Q2.

Generalization beyond EgoVerse. On held-out EgoDex, recorded with a different rig and excluded from all training manifests, WATCH achieves action MSE 0.019 versus 0.024 for within-video pre-training and [email protected] of 73.4% versus 69.0% (§E.2, Table 8). This cross-dataset prediction result supports expanding the training pool with additional human collections that provide, or can be processed to provide, the required information.

Comparison with UniSkill. UniSkill [6] and WATCH use human video for different learning objectives. UniSkill learns latent skill representations through inverse and forward skill dynamics, then conditions a robot policy on skills extracted from human video prompts. This avoids explicit wrist-trajectory labels and supports demonstration-conditioned imitation. WATCH focuses on pre-training the world-action model itself: human-only cross-demonstration prediction jointly supervises motion and its visual consequences. The resulting action–video prediction prior can be adapted to both in-context and context-free robot control. WATCH thus uses human experience to initialize the model’s predictive and control capabilities, without matched human–robot pairs during pre-training.

Appendix A and Table 6 clarify the comparison with prior work; §B states the requirements; §4.3 and Fig. 9 present the collection-budget comparison. Sections E.2, F.3, and G.5 provide cross-dataset results, cost assumptions, and tracking evidence.

References

[1] Kareer et al. EgoMimic: Scaling Imitation Learning via Egocentric Video. ICRA 2025.
[2] Tao et al. DexWild: Dexterous Human Interactions for In-the-Wild Robot Policies. RSS 2025.
[3] Chi et al. Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots. RSS 2024.
[4] Zhang et al. HaWoR: World-Space Hand Motion Reconstruction from Egocentric Videos. CVPR 2025.
[5] Hoque et al. EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video. arXiv:2505.11709, 2025.
[6] Kim et al. UniSkill: Imitating Human Videos via Cross-Embodiment Skill Representations. CoRL 2025.

Q2: Robustness and Scalability to Less Structured Human Videos

We tested imperfect task labels at both the pairing and downstream-policy levels, and evaluated RGB-based wrist estimates as a first step toward using videos without recorded hand trajectories.

Task-label robustness. We replace 25% or 50% of fine-grained labels with related classes from the same super-group, sampled by class frequency, and rebuild the pairs. Motion agreement over 3,000 queries is reported below (Table 13):

Pairing labelsSpeed correlationShape cosine
Original0.200.33
25% related-class confusion0.200.32
50% related-class confusion0.200.33
Super-group only (10 classes)0.200.34
Random pairs0.000.01

Motion agreement remains close to the original labels under both corruption levels, or with only the ten super-group labels. Random pairs yield near-zero agreement. These results support using imperfect task labels to select candidates before motion matching.

Downstream performance. With pairs rebuilt under 50% related-class label confusion, we repeat pre-training and robot post-training with the training recipe fixed (Table 14):

Pre-trainingMimicDroid in-context L1 / L2 / L3 / MeanTableTop-Sim context-free Easy / Hard / Mean
None75.0 / 29.4 / 18.3 / 40.969.6 / 49.6 / 59.6
Within-video85.8 / 44.4 / 23.3 / 51.271.6 / 51.6 / 61.6
WATCH original labels91.7 / 72.5 / 40.8 / 68.376.8 / 54.0 / 65.4
WATCH 50% confused93.3 / 61.9 / 33.3 / 62.873.2 / 56.0 / 64.6

WATCH retains its advantage over within-video pre-training: 62.8% versus 51.2% on MimicDroid and 64.6% versus 61.6% on TableTop-Sim. The tested label corruption thus preserves pairing agreement, and WATCH retains most of its downstream gain. The drop concentrates on in-context L2 and L3, where the policy must follow a reference of the same task, while TableTop-Sim is nearly unchanged.

RGB-based motion estimates. We assess RGB-estimated motion in two steps: its trajectory error against tracked motion, and its effect on reference selection. We compare smoothed HaWoR [1] estimates with tracked trajectories over 1,076 three-second windows (2,072 hand-windows). The table reports errors after offset removal. Each value is the median across per-window median errors.

SourceWrist position (m)Rotation (°)Fingertip position (m)
MECKA0.02170.016
Scale0.02460.016
Aria0.045110.018

Pairing quality. We compare reference selection using tracked and smoothed HaWoR motion on 50 queries, with an average of 21 candidate references per query. All selected pairs are evaluated using tracked trajectories, so the quality scores do not rely on the HaWoR estimates used for selection. This evaluation is smaller than and separate from the 3,000-query label-robustness analysis above, so absolute agreement scores are not comparable across the two.

Motion used for reference selectionSpeed correlation [95% CI]Shape cosine [95% CI]
Tracked0.067 [0.006, 0.125]0.202 [0.122, 0.283]
HaWoR (RGB, smoothed)0.125 [0.064, 0.191]0.206 [0.118, 0.293]
Random0.015 [−0.000, 0.030]0.001 [−0.018, 0.019]

Results. Within this smaller evaluation, RGB-based selection shows no observed loss of pair quality. Shape agreement is nearly unchanged (0.206 versus 0.202), and the confidence intervals for speed correlation overlap (0.125 versus 0.067). The pairing score does not directly optimize these metrics (Table 7), so selection with tracked motion does not bound them from above. Both methods yield higher agreement than random selection. We have not pre-trained WATCH on HaWoR-derived actions. Together with the trajectory errors above, these results indicate that RGB-estimated motion is a practical input for constructing pairs, and we leave pre-training on such estimates to future work.

Section G.5 reports the label-corruption, tracking-noise, and RGB-estimate analyses, with Tables 12–14.

References

[1] Zhang et al. HaWoR: World-Space Hand Motion Reconstruction from Egocentric Videos. CVPR 2025.

Comments on the paper

3 comments

Dear Reviewers and Area Chairs,

We thank the reviewers for their careful evaluation and constructive feedback. We have revised the manuscript to strengthen the physical-robot evidence, examine the learned representations and pairing design, and clarify the data requirements and practical value of human-video pre-training. The main changes are marked in blue in the revised paper.

  1. Physical-robot evaluation. We added a humanoid evaluation on three manipulation tasks with Easy and Hard conditions, including unseen objects and a different drawer. With identical robot post-training data and evaluation protocols, WATCH achieves 80.0% mean success, versus 66.7% for within-video pre-training and 53.3% for Cosmos Policy. All three use context-free execution (§4.4, Fig. 10, §G.6, Table 15).
  2. Representation analysis and transfer. We repeated the representation analysis with identical feature-extraction inputs. Human cross-source motion retrieval AUC improves from 0.67 to 0.79 over within-video pre-training (Fig. 5, §E.5). We also clarify how cross-demonstration correspondence can support the larger in-context gains and how its benefits extend to context-free control (§4.3).
  3. Data robustness and RGB requirements. We added label-corruption and coarse-grouping comparisons, downstream evaluation under 50% related-class label confusion, tracking-quality statistics, and an RGB-based wrist-pose assessment. The downstream advantage remains under the tested label corruption. The revision specifies the task and motion information required by the current pipeline (§B, §G.5, Tables 12–14).
  4. Pairing comparisons. We added the requested random same-task experiment and discuss it alongside same-source top-1 and WATCH. Random same-task pairing reaches 38.8% on in-context MimicDroid, versus 68.3% for WATCH. We clarify the candidate pools and sampling rules and interpret these comparisons as evidence for the complete pairing design (§4.3, Table 5).
  5. Prior work and collection cost. We expanded the comparison with UniSkill and other related methods, specifying differences in objectives and data requirements (§A, Table 6). The revised main-text Fig. 9 compares human-pair and robot-pair pre-training on a collection-budget axis (§4.3). We added collection-throughput evidence from EgoMimic, DexWild, and UMI, with the cost assumptions and sensitivity range detailed in §F.3.
Area_Chair_yaPEOpenReview ↗

Please review the authors’ rebuttal and update your assessment

Dear reviewers,

Thank you for your reviews, and thank you to the authors for their responses. Please read the authors’ rebuttal and discuss whether it addresses your main concerns. Please update your review and rating if warranted, and briefly explain your current assessment, including any concerns that remain unresolved. If your assessment is unchanged, please explain why.

Thank you for your continued engagement.

AuthorsOpenReview ↗

Update after the initial response (Nov 17–19)

Dear Reviewers and Area Chairs,

Following our initial responses, we conducted three additional evaluations to address the remaining questions about the contribution of the pre-training objective, transfer to physical robots, and reliance on recorded motion. We summarize the new evidence below for ease of reference.

1. Controlled comparison with a prior cross-demonstration objective. To address the request to separate the contribution of our formulation from differences in data and architecture, we implemented the Vid2Robot training formulation within our framework. Both methods use the same Cosmos-Predict2.5 backbone, EgoVerse human pairs, 8K pre-training iterations, and downstream data and protocol. The adapted Vid2Robot formulation predicts query actions with its two auxiliary alignment losses, without video prediction. Under this controlled comparison, WATCH achieves 68.3% versus 53.1% on in-context MimicDroid and 65.4% versus 52.2% on context-free TableTop-Sim. These results support the contribution of WATCH’s joint cross-demonstration action–video objective with benefits in both reference-conditioned and reference-free control (Appendix G.7, Table 17).

2. In-context transfer to physical-robot control. We extended the physical-robot evaluation to in-context control, using identical robot post-training data and evaluation protocols across models. On four settings absent from post-training, WATCH achieves 62.5% mean success, compared with 47.5% for within-video pre-training and 30.0% for Cosmos Policy. WATCH improves over within-video pre-training on changing the acting hand, grasping a different object, and changing the placement destination; both reach 100% on drawer closing. These results complement the context-free hardware evaluation, showing that the benefits of cross-demonstration pre-training extend to physical-robot control both with and without human references (§4.4, Appendix G.6, Table 16).

3. Pre-training with motion estimated from RGB throughout the pipeline. We replaced recorded hand tracking with HaWoR estimates in pairing, reference actions, and action targets for every pre-training episode. WATCH retains higher average success than within-video pre-training using recorded tracking: 59.4% versus 51.2% on MimicDroid and 62.8% versus 61.6% on TableTop-Sim. It demonstrates that WATCH can benefit from egocentric RGB demonstrations without recorded hand trajectories, given task annotations and estimated motion. Recorded tracking remains stronger overall, and we now explicitly scope our claims to egocentric video rather than uncurated, unlabeled web video (Appendix G.5, Table 14).

All three new evaluations are included in the revised manuscript. We thank the reviewers for the suggestions that motivated these experiments and helped sharpen both the evidence and the scope of our claims.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.