acceptodds
Under review as a conference paper at ICLR 2027

RoboTryOn: Synthesizing Dexterous Robot Demonstrations from Human Videos via Virtual Try-On

Abstract

Robot foundation models have shown remarkable performance in robot control, yet adapting them to a new task still relies heavily on teleoperated demonstrations. Egocentric human videos offer a scalable substitute, but retargeting human poses into robot actions or overlaying rendered robots on human leaves significant visual or action gaps for dexterous hands with high-DoF. To address this limitation, we propose RoboTryOn, which treats human-to-robot translation as virtual try-on: try-on keeps the person and swaps the garment, RoboTryOn keeps the demonstrated motion and swaps the human for the robot. Specifically, RoboTryOn synthesizes robot data using masks and hand skeletons as conditioning signals, while reference images specify the target robot embodiment. To maintain visual consistency, we translate selected keyframes autoregressively and use them to guide video synthesis conditioned on the skeleton. Additionally, we use an inverse dynamics model (IDM) that incorporates depth features to infer action labels from synthesized videos for policy learning. Our experiments across multiple humanoid embodiments show that policies trained with our synthesized data improve the average success rate on novel tasks, including challenging bimanual manipulation, from 2% to 60%, without any real robot demonstration of those tasks. Moreover, training with our synthetic data matches the performance of policies trained solely on teleoperated demonstrations at approximately a third of the collection cost, demonstrating the cost efficiency of our approach.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Learning, Human-to-Robot Transfer, Synthetic Data Generation

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_Aq1yOpenReview ↗
Rating
3
Confidence
3
Informativeness
2(−1)
Presentation
3
Soundness
2
Read review
Paper summary

RoboTryOn converts human videos into robot demonstrations through keyframe translation, video synthesis, and inverse dynamics action labeling. Experiments on two dexterous robots show substantial gains on tasks without task-specific robot demonstrations.

Critical strengths

The problem is important, and the real-robot improvements are substantial. Evaluation covers two embodiments and two policy backbones. The component ablations and action replay experiments provide useful evidence for the pipeline.

Critical weaknesses
  1. The success rates may not always be indicative of task completion. In section C.1, some tasks such as pouring involve only tilting the bottle above the cup, while brushing involves only reaching the marked area. This compromises the assertions that new skills have been learned.
  2. The generators and IDM employ other robot recordings apart from those in the policy but their task composition is unknown. It would therefore be hard to prove that the asserted novel skills are not part of any of the training phases.
  3. The comparisons involve changes in both visual inputs and action labels. The current ablations do not distinguish between the effect of video generation and action supervision. Evaluation of the policy employs videos from the robot perspective captured by humans.

Thanks the authors for their reply and explanations. I tend to maintain my score.

Questions for authors
  1. What are the binary full-task success rates, especially for bimanual tasks? Please include confidence intervals.
  2. Which tasks, objects, and skills appear in the additional robot recordings, including the GR-1 data?
  3. How do policies perform with synthesized videos and retargeted actions, or rendered videos labeled by the same IDM?
  4. How sensitive is performance to changes in the human camera position?
5 comments
AuthorsOpenReview ↗

Response to Reviewer Aq1y (1/3)

Dear reviewer Aq1y,

We sincerely appreciate your efforts in reviewing our manuscript. We respond to each comment in the following content. In the revised manuscript, we have marked the revisions with “blue”.


[W1-1/Q1] Clarify the full-task success rates

We clarify that all success rates reported in our paper are already binary full-task success rates, where a trial counts as a success only when the entire task is completed. During evaluation, we recorded the outcome of each subtask of the bimanual tasks. We mistakenly described this per-subtask recording as a scoring rule of 0.5 points per hand in Appendix C.1 of the original draft, and have corrected this writing error in our revised manuscript. Since all tables in our paper already report full-task success rates, no recomputation is needed. For the bimanual tasks, the table below summarizes the number of fully successful trials out of 28, with the per-subtask score that we have added in Appendix F.5 (Table 18) of our revised manuscript in parentheses.

\begin{array}{ll|ccc} \hline \text{Robot} & \text{Method} & \text{Box Convey} & \text{Air Fryer} & \text{PnP Ball in Cup} \newline \hline \text{OpenArm} & \text{Teleop only} & 0 \ (0) & 0 \ (0) & 0 \ (0) \newline \text{OpenArm} & \text{+ EgoMimic} & 0 \ (3) & 0 \ (0) & 0 \ (0) \newline \text{OpenArm} & \text{+ Masquerade} & 0 \ (5.5) & 0 \ (1) & 0 \ (0) \newline \text{OpenArm} & \textbf{+ RoboTryOn (Ours)} & \textbf{22} \ (\textbf{22}) & \textbf{17} \ (\textbf{19.5}) & \textbf{14} \ (\textbf{14.5}) \newline \hline \text{RB-Y1} & \text{Teleop only} & 0 \ (0) & 0 \ (0) & 0 \ (0) \newline \text{RB-Y1} & \text{+ EgoMimic} & 0 \ (2.5) & 0 \ (0) & 0 \ (0) \newline \text{RB-Y1} & \text{+ Masquerade} & 0 \ (3) & 0 \ (0) & 0 \ (0) \newline \text{RB-Y1} & \textbf{+ RoboTryOn (Ours)} & \textbf{9} \ (\textbf{12.5}) & 0 \ (0) & \textbf{15} \ (\textbf{15}) \newline \hline \end{array}


[W1-2/Q1] Confidence intervals of the full-task success rates

Thank you for the suggestion. We have added the 90% confidence intervals of the full-task success rates in our main results in Appendix F.2 (Table 13) of our revised manuscript, which we summarize in the table below. We estimate them using stratified bootstrapping [1].

\begin{array}{ll|ccc} \hline \text{Robot} & \text{Method} & \text{In-domain} & \text{Bimanual} & \text{Out-of-domain Avg.} \newline \hline \text{OpenArm} & \text{Teleop only} & 47.3 \pm 7.4 & 0.0 \pm 0.0 & 3.1 \pm 1.9 \newline \text{OpenArm} & \text{+ EgoMimic} & 50.0 \pm 7.6 & 0.0 \pm 0.0 & 5.1 \pm 2.5 \newline \text{OpenArm} & \text{+ Masquerade} & 67.9 \pm 6.7 & 0.0 \pm 0.0 & 8.2 \pm 3.1 \newline \text{OpenArm} & \textbf{+ RoboTryOn (Ours)} & \textbf{75.9} \pm 6.5 & \textbf{63.1} \pm 8.4 & \textbf{73.0} \pm 5.0 \newline \hline \text{RB-Y1} & \text{Teleop only} & 69.6 \pm 7.0 & 0.0 \pm 0.0 & 1.0 \pm 1.1 \newline \text{RB-Y1} & \text{+ EgoMimic} & 63.4 \pm 7.4 & 0.0 \pm 0.0 & 5.1 \pm 2.5 \newline \text{RB-Y1} & \text{+ Masquerade} & 63.4 \pm 7.5 & 0.0 \pm 0.0 & 7.7 \pm 3.0 \newline \text{RB-Y1} & \textbf{+ RoboTryOn (Ours)} & \textbf{79.5} \pm 6.2 & \textbf{28.6} \pm 7.1 & \textbf{46.9} \pm 5.2 \newline \hline \end{array}

[1] Agarwal et al., Deep Reinforcement Learning at the Edge of the Statistical Precipice, NeurIPS 2021


[W1-3] Clarify the success criteria of Pouring and Brushing

We clarify that the success criteria of Pouring and Brushing require the complete motion of each skill, not just reaching the target region. We have specified them in Appendix C.1 of our revised manuscript. For Pouring, where the bottle is capped, we evaluate the pouring motion. A trial is successful only when the robot holds the bottle (i) tilted by at least 45° from upright, (ii) toward the target cup, (iii) with the bottle mouth within 5 cm of the cup horizontally, and (iv) for at least 3 s. For Brushing, a trial is successful only when the robot sweeps over the region marked with black tape at least once while keeping the brush on the table. We evaluated all reported trials of both tasks under these criteria. We show failure and success examples of both tasks in Figure 10 of our revised manuscript. We have also corrected the success criterion of Air Fryer in Appendix C.1 to the one actually used in our evaluation.

AuthorsOpenReview ↗

Response to Reviewer Aq1y (2/3)

[W2/Q2] Clarify the tasks, objects, and skills in the additional robot data

We clarify that the additional OpenArm and RB-Y1 recordings contain no task and no object. They are free-motion data, in which the robot moves its arms and hands freely without contacting any object and without performing any task. We have added examples of the free-motion data in Figure 7 (Appendix B.2) of our revised manuscript. The GR-1 data are the public teleoperation dataset used in DreamDojo [1], which consists of tabletop tasks. Most of these tasks are pick-and-place, and the objects are diverse, including office supplies, kitchenware, and food.

We use the additional data only to train the generative models and the IDM, and never for policy training. Since the in-domain teleoperated demonstrations are limited in amount and diversity, the additional data teach the generative models the appearance of each robot across a wide range of arm and hand configurations, and teach the IDM the dynamics of each robot. To make clear how much of the training corpus the additional data occupy, we have broken down the training corpus by source in the table below and in Table 5 (Appendix B.2) of our revised manuscript. In Table 5, we have also corrected a miscount of the hours and episodes in the original draft.

\begin{array}{ll|cc|c} \hline \text{Robot} & \text{Source} & \text{Hours} & \text{Episodes} & \text{Policy training} \newline \hline \text{OpenArm} & \text{In-domain task demonstrations} & 1.51 & 200 & \checkmark \newline \text{OpenArm} & \text{Free-motion recordings} & 2.28 & 253 & \times \newline \hline \text{RB-Y1} & \text{In-domain task demonstrations} & 1.47 & 200 & \checkmark \newline \text{RB-Y1} & \text{Free-motion recordings} & 1.06 & 152 & \times \newline \hline \text{GR-1} & \text{GR00T teleoperation} & 2.50 & 1{,}180 & \times \newline \hline \end{array}

GR-1 is not our target embodiment. Nevertheless, since the tabletop tasks in the GR-1 data may raise a concern about fairness, we retrain all three components of RoboTryOn (the image model, the video model, and the IDM) using only the OpenArm and RB-Y1 data. As shown in the table below, the success rate remains close to that of the original RoboTryOn. For your information, we have added these results in Appendix F.8 of our revised manuscript.

\begin{array}{l|cc} \hline \text{Training data of RoboTryOn} & \text{In-domain} & \text{Out-of-domain} \newline \hline \text{OpenArm + RB-Y1 + GR-1 (original)} & 75.9 & 73.0 \newline \text{OpenArm + RB-Y1 (w/o GR-1)} & 75.0 & 74.0 \newline \hline \end{array}

[1] Gao et al., DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos, ICML 2026

AuthorsOpenReview ↗

Response to Reviewer Aq1y (3/3)

[W3-1/Q3] Disentangle the effects of visual inputs and action labels

We would like to point out that our original draft already analyzed the visual gap and the action gap separately in Section 4.3. RoboTryOn closes both gaps, as its frames lie among the real robot frames (Figure 4) and its action labels succeed about twice as often as retargeted ones in open-loop replay (Figure 6c). Nevertheless, to address your concern, we further separate the two at the policy level by pairing the videos of one method with the action labels of another. Regarding the rendered videos of the baselines, we observe that the IDM, which is trained on real robot videos, infers inaccurate actions from these rendered videos, which look different from real robot videos. We therefore pair the rendered videos with the action labels that the IDM infers from the videos synthesized by RoboTryOn for the same human videos. As shown in the table below, replacing our action labels with retargeted actions causes a larger drop in the out-of-domain success rate than replacing our videos, which is consistent with prior findings [1, 2]. At the same time, pairing our videos with our action labels achieves the highest success rate. We have added these results in Table 2c and Appendix F.3 of our revised manuscript. Furthermore, we have added an analysis of why retargeted actions lead to a low success rate and are hard to co-train with teleoperated actions in Appendix D of our revised manuscript.

\begin{array}{ll|cc} \hline \text{Video} & \text{Action} & \text{In-domain} & \text{Out-of-domain} \newline \hline \textbf{RoboTryOn} & \textbf{RoboTryOn} & \textbf{75.9} & \textbf{73.0} \newline \hline \text{RoboTryOn} & \text{Retargeting} & 69.6 & 11.2 \newline \text{EgoMimic} & \text{RoboTryOn} & 67.0 & 53.1 \newline \text{Masquerade} & \text{RoboTryOn} & 74.1 & 61.2 \newline \hline \end{array}

[1] Liu et al., EgoEngine: From Egocentric Human Videos to High-Fidelity Dexterous Robot Demonstrations, CoRL 2026

[2] Yuan et al., MotionTrans: Human VR Data Enable Motion-Level Learning for Robotic Manipulation Policies, ICRA 2026


[W3-2/Q4] Sensitivity to the human camera position

Thank you for the suggestion. To measure this sensitivity, we record the human videos of four out-of-domain tasks with a different camera, and find that RoboTryOn still outperforms all baselines. Specifically, this camera is mounted lower than the robot's head camera and has a narrower field of view (see Figure 13 of our revised manuscript), and we train the OpenArm policy in the same way as in our main experiments. As shown in the table below, the success rate drops from 80.4% to 57.1% on average, since the IDM is trained only on robot data captured from the fixed head camera of each robot. Nevertheless, it remains far above all baselines, which are trained on human videos recorded from the robot's camera position. This shows that the generative models of RoboTryOn are robust to the camera position, as they still translate these videos into plausible robot videos (Figure 13). We have added these results in Appendix F.4 of our revised manuscript. Building on this robustness, we have also shown that RoboTryOn applies to in-the-wild human videos recorded with moving cameras from EgoDex [1] and VITRA [2], where pre-training on the converted videos improves the success rate of the policy (Appendix F.6 of our revised manuscript).

\begin{array}{l|ccccc} \hline \text{Method} & \text{PnP Coffee Pot} & \text{PnP Bottle to Shelf} & \text{Pouring} & \text{Brushing} & \text{Average} \newline \hline \text{Teleop only} & 14.3 & 7.1 & 0.0 & 0.0 & 5.4 \newline \text{+ EgoMimic} & 14.3 & 0.0 & 7.1 & 14.3 & 8.9 \newline \text{+ Masquerade} & 21.4 & 7.1 & 7.1 & 21.4 & 14.3 \newline \text{+ RoboTryOn (Ours, robot view)} & 64.3 & 89.3 & 85.7 & 82.1 & 80.4 \newline \hline \text{+ RoboTryOn (Ours, different view)} & 35.7 & 78.6 & 71.4 & 42.9 & 57.1 \newline \hline \end{array}

[1] Hoque et al., EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video, ICLR 2026

[2] Li et al., Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videos, ICRA 2026

Reviewer_bMTROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RoboTryOn, a pipeline for synthesizing robot data from human video demonstrations. Using reference images to specify the target robot embodiment, RoboTryOn generates robot videos using masks and hand skeletons as conditioning signals. The corresponding robot actions are inferred using a separate inverse dynamics model that incorporates depth features from the generated robot video. Compared to prior work, RoboTryOn can learn novel tasks(e.g., bimanual manipulation) from human video alone. Moreover, training on RoboTryOn’s synthetic data matches the performance of policies trained on teleoperation data at a third of the collection cost, showing its scalability.

Critical strengths
  • The method is straightforward and easy to follow.
  • It does not require manual image editing during policy deployment.
  • It outperforms baselines on two different embodiments, across tasks with different type of generalization (i.e., novel object, novel goal, novel skill, bimanual). The authors also evaluated over 28 trials per task.
Critical weaknesses
  • Image model, video model, and IDM were trained on additional data on top of teleoperation demonstrations of the four pick-and-place tasks. Meanwhile, the baselines were not given with such data, and it is unclear to which extent the performance gain comes from the proposed pipeline or the additional data.
  • As mentioned, adding new embodiment requires teleoperation dataset of that robot, which is nontrivial.
  • Missing baselines.
    • EgoVLA [1] uses human video demonstration and robot teleoperation dataset for policy training. Specifically, human data is used for pretraining and robot data is used for finetuning. All datasets are used to predict future kinematic hand state, and a separate IDM model trained solely on robot data is used to further ground it to joint commands.
    • Do as I Do [2] (or other physics based retargeting works with 4D reconstruction) generates robot data from human videos by reconstructing the object and hand mesh, followed by physics based retargeting. While it is a very different pipeline, it is unclear which pipeline is better given that the evaluated tasks are simple pick and place.
  • Ablations. The authors should also compare w/o IDM itself. For simple tasks this could be enough.
  • The tasks are generally simple pick and place, it is unclear to which extent the proposed pipeline would offer benefit.
  • It requires retiming human video demonstration for better performance.

[1] Egovla: Learning vision-language-action models from egocentric human videos, Yang et al. IROS 2026.

[2] Do as I Do: Dexterous Manipulation Data from Everyday Human Videos, Pailwal et al., CoRL 2026.

Questions for authors
  • What happens if the baselines are also trained on additional data?
  • What happens when IDM is removed from the pipeline?
  • Given multiple components, what are the common failure cases?
10 comments
AuthorsOpenReview ↗

Response to Reviewer bMTR (1/3)

Dear reviewer bMTR,

We sincerely appreciate your efforts in reviewing our manuscript. We respond to each comment in the following content. In the revised manuscript, we have marked the revisions with “blue”.


[W1/Q1] Baselines trained with the additional robot data

We would like to clarify that RoboTryOn and the baselines share the same goal of human-to-robot transfer but take different approaches. RoboTryOn is learning-based for both the observations and the actions, and the additional task-agnostic data are used only to train its models. Since the in-domain teleoperated data lack diversity, these data teach the generative models the appearance of each robot and the IDM its dynamics (see our response to [W2-1/Q2] of Reviewer oD1A). To the best of our knowledge, existing methods that convert human videos into dexterous robot demonstrations all obtain the actions by retargeting, and such retargeting-based baselines have no place to use these data. Nevertheless, to address your concern, we add the additional OpenArm data directly to the policy training of all baselines. As shown in the table below, the additional data barely change the success rates of any baseline, and RoboTryOn still outperforms all of them. Since the additional data contain no task, they provide little benefit when used directly for policy training. Furthermore, RoboTryOn retrained without the GR-1 data [1] achieves a similar success rate (see our response to [W2-1/Q2] of Reviewer oD1A). For your information, we have broken down the training corpus by source in Table 5 (Appendix B.2) of our revised manuscript, where we have also corrected a miscount of the hours and episodes in the original draft.

\begin{array}{l|cc} \hline \text{Method} & \text{In-domain} & \text{Out-of-domain} \newline \hline \text{Teleop only} & 47.3 & 3.1 \newline \text{Teleop only + additional data} & 49.1 & 3.1 \newline \hline \text{EgoMimic} & 50.0 & 5.1 \newline \text{EgoMimic + additional data} & 49.1 & 6.1 \newline \hline \text{Masquerade} & 67.9 & 8.2 \newline \text{Masquerade + additional data} & 64.3 & 8.7 \newline \hline \textbf{RoboTryOn (Ours)} & \textbf{75.9} & \textbf{73.0} \newline \hline \end{array}

[1] Gao et al., DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos, ICML 2026


[W2] Cost of adding a new embodiment

As discussed in Appendix A of our original draft, adding a new embodiment requires task-agnostic robot data, and this cost is paid once per robot rather than once per task. This one-time cost is negligible compared with collecting task-specific teleoperation for every task. For example, collecting 50 teleoperated demonstrations for each of our 11 tasks on a new robot would take about 18 person-hours, based on Table 10 of our original draft with twice the cost for the bimanual tasks. In contrast, the free-motion recordings take only about 1.7 person-hours per robot on average, which is about 9% of this cost. This overhead becomes even more negligible as the number of tasks grows. These data are also easy to collect. Free-motion recordings require no objects, no scene reset, and no task success, and such data can also come from public datasets, as for GR-1, or from policy rollouts. Once these data are available, RoboTryOn synthesizes demonstrations of the target robot for new tasks from human demonstrations alone.

AuthorsOpenReview ↗

Response to Reviewer bMTR (2/3)

[W3-1] Comparison with EgoVLA

We note that EgoVLA [1] addresses a different problem from RoboTryOn and the baselines, as discussed in Section 2 of our original draft. Rather than converting human videos into robot demonstrations, EgoVLA pre-trains a policy on human videos in the human action space (wrist pose and MANO hand pose) and then post-trains it on robot data. We would also like to clarify that EgoVLA does not use a separate IDM. Its module that converts the predicted human actions to robot joints is a retargeting network that maps human hand poses to robot joints, not a model that infers actions from videos. Nevertheless, to address your concern, we compare RoboTryOn with EgoVLA using the same backbone (π0.5), following its training scheme of pre-training on the human videos of all 11 tasks and post-training on the in-domain teleoperated data. As shown in the table below, RoboTryOn outperforms EgoVLA, especially on the out-of-domain tasks. For your information, we have added these results in Appendix F.9 of our revised manuscript.

\begin{array}{l|cc} \hline \text{Method} & \text{In-domain} & \text{Out-of-domain} \newline \hline \text{EgoVLA} & 66.1 & 5.1 \newline \textbf{RoboTryOn (Ours)} & \textbf{75.9} & \textbf{73.0} \newline \hline \end{array}

[1] Yang et al., EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videos, IROS 2026


[W3-2] Comparison with Do as I Do

We believe that Do as I Do [1] is not a suitable baseline for our setting, since its physics-based retargeting reconstructs and tracks only a single rigid object per episode. It therefore cannot handle our tasks with multiple or articulated objects, such as PnP Ball in Cup and Air Fryer, and its per-frame 6D object tracking takes about 5–6 A6000 GPU-hours per episode, or roughly 3,000 GPU-hours for our 550 training episodes. Nevertheless, to compare the action labels of the two pipelines, we evaluate Do as I Do on the pick-and-place tasks it can handle. Since its monocular reconstruction leaves the metric scale ambiguous, we supply all the information available to us, including stereo depth, the ground-truth object size, the camera intrinsics, the direction of gravity relative to the camera, and the table height. We replay its action labels open loop on the same 66 episodes used in our action replay analysis (Figure 6 and Appendix D), which include 22 episodes each of PnP Bottle, PnP Box, and PnP Ball. As shown in the table below, the action labels of Do as I Do succeed in 43.9% of the replays on average, compared with 69.7% for RoboTryOn.

\begin{array}{l|ccc|c} \hline \text{Action labels} & \text{PnP Bottle} & \text{PnP Box} & \text{PnP Ball} & \text{Average} \newline \hline \text{Retargeting [2]} & 63.6 & 31.8 & 9.1 & 34.8 \newline \text{Do as I Do [1]} & 59.1 & 40.9 & 31.8 & 43.9 \newline \textbf{RoboTryOn (Ours)} & \textbf{77.3} & \textbf{63.6} & \textbf{68.2} & \textbf{69.7} \newline \hline \end{array}

[1] Paliwal et al., Do as I Do: Dexterous Manipulation Data from Everyday Human Videos, CoRL 2026

[2] Qin et al., AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System, RSS 2023

AuthorsOpenReview ↗

Response to Reviewer bMTR (3/3)

[W4/Q2] Ablation without the IDM

As shown in Figure 6c of our original draft, the IDM labels already succeed about twice as often as retargeted actions in open-loop replay (69.7% vs. 34.8%). Nevertheless, as you suggested, we have added the ablation without the IDM in policy learning, where the policy is trained on RoboTryOn videos with retargeted actions instead of the actions inferred by the IDM. As shown in the table below, removing the IDM drops the out-of-domain success rate from 73.0% to 11.2%. This result suggests that the IDM is essential to RoboTryOn, which is consistent with prior findings that executable actions matter more than visual realism when learning from human videos [1, 2]. We have added these results in Table 2c of our revised manuscript, together with the results of pairing the videos of the baselines with our action labels (see our response to [W2-2/Q3] of Reviewer oD1A). Furthermore, we have added an analysis of why retargeted actions lead to a low success rate in Appendix D of our revised manuscript.

\begin{array}{ll|cc} \hline \text{Video} & \text{Action} & \text{In-domain} & \text{Out-of-domain} \newline \hline \textbf{RoboTryOn} & \textbf{RoboTryOn} & \textbf{75.9} & \textbf{73.0} \newline \hline \text{RoboTryOn} & \text{Retargeting (w/o IDM)} & 69.6 & 11.2 \newline \hline \end{array}

[1] Liu et al., EgoEngine: From Egocentric Human Videos to High-Fidelity Dexterous Robot Demonstrations, CoRL 2026

[2] Yuan et al., MotionTrans: Human VR Data Enable Motion-Level Learning for Robotic Manipulation Policies, ICRA 2026


[W5] Task complexity beyond pick-and-place

We politely disagree that our tasks are mostly simple pick-and-place. While the four in-domain tasks with teleoperated data are pick-and-place, four of the seven out-of-domain tasks (57%) are not pick-and-place (Pouring, Brushing, Box Convey, and Air Fryer), and PnP Ball in Cup is a bimanual task. This highlights the strength of RoboTryOn: although the policy is co-trained only with teleoperated data of simple pick-and-place, it solves these diverse tasks from human videos. On these four tasks, RoboTryOn outperforms the baselines by a large margin on both robots (Table 11 of our original draft). Furthermore, we have added two new non-pick-and-place tasks, Water Plant and Open Bag, in Appendix F.7 of our revised manuscript, where RoboTryOn achieves an average success rate of 85.7% while the baselines reach at most 25.0%.


[W6] Necessity of retiming human videos

We kindly remark that RoboTryOn does not require retiming, as shown in Table 3 of our original draft. Without retiming, RoboTryOn achieves 73.0% on the out-of-domain tasks, which is higher than both retimed variants (70.4% and 42.9%). Retiming is instead an additional feature of RoboTryOn: since the motion is represented by the hand skeleton flow, we can retime it and synthesize the video again from the same keyframes, without collecting new demonstrations. This feature allows us to trade off the success rate against the execution time of the policy. Retiming to the slower pace of the teleoperated data raises the in-domain success rate from 75.9% to 85.7%, whereas retiming to twice that pace roughly halves the rollout time at the cost of a lower success rate. The appropriate pace depends on the task. For example, a human motion that is too fast for the robot to follow precisely can be slowed down, and a task where execution time matters can be sped up. This flexibility to adapt the pace of the synthesized demonstrations to each task is an advantage of RoboTryOn.


[Q3] Common failure cases of each component

Our original draft already presented failure cases of RoboTryOn in Figure 14. Following your suggestion, we further break them down by component in Figure 21 of our revised manuscript. The common failure cases of each component are as follows:

Generation: the image and video models occasionally fail to preserve objects in contact with the hand, so that an object disappears from the hand or the fingers pass through it. Action labeling: at the moment of contact, the hand pose given by the action labels may not fully align with the robot hand in the synthesized video. The hand can be shifted to one side of the object or stop short of it, which is enough to miss a fine-grained grasp.

Reviewer_czGNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RoboTryOn converts human videos into robot demonstrations through keyframe translation, skeleton-guided video generation, and inverse-dynamics action labeling with wrist correction. It requires no paired human–robot videos. Tests on two real robots, component ablations, and a second policy backbone show that the synthesized data improve policy learning.

Critical strengths
  1. Coarse human-motion guidance and learned robot actions reduce reliance on precise retargeting. Simulated replay supports the improvement in action labels.
  2. Policy gains hold on two robots with different hands and a second policy backbone. Ablations support the main design choices.
Critical weaknesses
  1. Precise contact remains difficult. Figure 14 shows incorrect generated contacts and inaccurate action labels. Section F.4.2 also identifies limited IDM precision in the RB-Y1 Air Fryer failures.
  2. The evaluated pipeline uses human videos recorded from the robot’s camera position, and wrist correction requires camera calibration. Policy learning from moving-camera videos remains untested.
Questions for authors

For the RB-Y1 Air Fryer task, do the generated videos show a valid grasp? This would help distinguish generation errors from the IDM precision limit discussed in Section F.4.2.

3 comments
AuthorsOpenReview ↗

Response to Reviewer czGN

Dear reviewer czGN,

We sincerely appreciate your efforts in reviewing our manuscript. We respond to each comment in the following content. In the revised manuscript, we have marked the revisions with “blue”.


[W1/Q1] Generation errors versus IDM precision in the RB-Y1 Air Fryer task

The generated videos show a valid grasp, and the failure on the RB-Y1 Air Fryer task comes mainly from the difficulty of this task for WujiHand2 [1]. We inspected all 50 generated episodes of this task, and 48 of them show a valid grasp. As noted in Appendix F.4.2 of our original draft, the long fingers of WujiHand2 hit the air fryer body as they bend toward the handle and push the air fryer away. This task is challenging even for a skilled human teleoperator, who succeeded in only 4 out of 20 trials when teleoperating WujiHand2 with MANUS gloves [2], which shows that the execution is difficult even without the IDM. In contrast, RoboTryOn achieves 60.7% on the same task on OpenArm with the Inspire RH56F1 hand [3], whose shorter fingers close around the handle without pushing the air fryer away.

As you mentioned, precise contact remains challenging for RoboTryOn, which we have discussed as a limitation in Appendix A of our original draft. For your information, we have added examples of IDM errors in Figure 21 of our revised manuscript, where the action labels are rendered in MuJoCo and overlaid on the synthesized videos.

[1] Wuji Technology, Wuji Hand 2, https://www.wuji.tech/en/hand2

[2] MANUS, Metagloves Pro, https://www.manus-meta.com/products/metagloves-pro

[3] Inspire Robots, RH56F1 Dexterous Hand, https://en.inspire-robots.com/product/rh56f1


[W2] Policy learning from moving-camera human videos

As discussed in Appendix A of our original draft, our human videos are recorded from the fixed viewpoint of the robot's head camera, and moving-camera videos are left as future work. This follows our robot setup, where the head and torso of each robot are fixed during manipulation. Nevertheless, we further show that RoboTryOn benefits policy learning even with in-the-wild human videos recorded with moving cameras. We translate 35 hours of egocentric human videos from EgoDex [1] and VITRA [2] into three embodiments (OpenArm, RB-Y1, and GR-1) with RoboTryOn, yielding 105 hours of robot data. Since the IDM is trained on robot data recorded with a fixed head camera and a fixed body, it cannot infer the neck and torso actions in these videos, so we use the synthesized data to pre-train π0.5 before post-training on 50 teleoperated demonstrations per task. As shown in the table below, pre-training improves the success rate on both PnP Banana and Open Air Fryer on OpenArm over 18 trials per task. For your information, we have added these results, examples of the synthesized videos, and further discussion in Appendix F.6 of our revised manuscript.

\begin{array}{lc|cc} \hline \text{Method} & \text{RoboTryOn Pre-training} & \text{PnP Banana} & \text{Open Air Fryer} \newline \hline \pi_{0.5} & \times & 55.6 & 77.8 \newline \pi_{0.5} & \checkmark & \textbf{72.2} & \textbf{88.9} \newline \hline \end{array}

[1] Hoque et al., EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video, ICLR 2026

[2] Li et al., Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videos, ICRA 2026

AuthorsOpenReview ↗

Further Discussion Before the Deadline

Dear Reviewer czGN,

Thank you once again for your time and thoughtful efforts in reviewing our paper. As the discussion period will conclude in about two days, we would like to gently remind you of our response. We have sincerely addressed your concerns, supported by the additional experimental results in our response and the revised manuscript. We would greatly appreciate it if you could let us know whether our response has resolved your concerns. If you have any further questions, we would be happy to address them before the discussion period ends.

Thank you very much,

Authors

AuthorsOpenReview ↗

Gentle Reminder Before the Discussion Deadline

Dear Reviewer czGN,

We sincerely appreciate the time and care you have devoted to reviewing our paper. As only about one day remains in the discussion period, we would like to follow up once more in case you have any remaining comments. We hope that our responses, together with the additional experimental results, have addressed your concerns. If you have any further concerns or questions, please feel free to let us know, and we will respond promptly.

With many thanks,

The Authors

Reviewer_oD1AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

RoboTryOn converts human manipulation videos into robot demonstrations through autoregressive keyframe translation, skeleton-guided video synthesis, and inverse-dynamics action labeling with wrist correction. The pipeline requires robot training data but no paired human–robot demonstrations. Experiments on two real robot platforms show substantial improvements on tasks without task-specific robot demonstrations, supported by component ablations, simulated action replay, and a collection-cost analysis.

Critical strengths
  1. The paper focus on a practically important problem. The paper tries to address the high cost of collecting dexterous robot demonstrations and evaluates the resulting policies on two robot platforms with different hand configurations. The substantial improvements in Table 1, provide encouraging evidence that the synthesized demonstrations are useful for policy learning.

  2. The paper introduces a coherent system design supported by real-world experiments. The pipeline combines temporally consistent visual translation, skeleton-guided motion generation, and action labeling with wrist correction. Component ablations demonstrate the importance of these choices, while simulated action replay provides evidence beyond policy performance.

Critical weaknesses
  1. The evaluation metric does not clearly establish complete task success and tasks are relatively too simple. Appendix C.1 assigns 0.5 points per hand on bimanual tasks, suggesting that the reported values may measure partial completion rather than full-task success. However, the main text and abstract describe them as success rates, including the headline 45.9% bimanual result. The appendix also lists full-task success conditions, leaving the actual scoring procedure ambiguous. Moreover, Pouring requires only tilting the bottle over the cup, and Brushing requires only reaching the designated region, which do not establish the intended physical outcomes. Since learning novel skills and bimanual tasks is central to the paper, the authors should report strict full-task success separately from partial progress and clarify the outcome criteria.

  2. The auxiliary training-data boundary and source of the gains remain insufficiently established. The paper reports additional OpenArm, RB-Y1, and GR-1 demonstrations for training the generative models and IDM, but does not sufficiently specify their tasks, objects, or bimanual behaviors. Their exclusion from policy fine-tuning alone does not establish that the target skills are unseen throughout the pipeline. Furthermore, RoboTryOn changes both visual observations and action labels relative to the baselines, so the comparisons do not isolate the contribution of generative translation from improved action supervision. A complete training-data breakdown and a controlled comparison using the same synthesized videos with retargeted versus IDM-generated actions would materially strengthen the generalization and attribution claims.

Questions for authors
  1. Do the bimanual values in Tables 1 and 11 and the abstract represent complete task success rates or average scores with 0.5 points per hand? Please report full-success counts and partial-completion scores separately, and recompute the seven-task averages using complete task success. Are there additional evaluations of the actual outcomes of Pouring and Brushing?

  2. Please provide the task and skill composition of all robot data used to train the generative models and IDM. Do any of these recordings contain the target objects, goals, skills, or bimanual behaviors? Does the statement that all teleoperated demonstrations are single-arm include the additional OpenArm, RB-Y1, and GR-1 data?

  3. What happens when the synthesized videos are held fixed and IDM labels are replaced with retargeted actions? And are there other controlled experiments separating the benefits of visual translation from those of improved action labels?

6 comments
AuthorsOpenReview ↗

Response to Reviewer oD1A (1/2)

Dear reviewer oD1A,

We sincerely appreciate your efforts in reviewing our manuscript. We respond to each comment in the following content. In the revised manuscript, we have marked the revisions with “blue”.


[W1-1/Q1] Clarify the rubric of the reported success rates

We clarify that all success rates reported in our paper are full-task success rates, where a trial counts as a success only when the entire task is completed. During evaluation, we recorded the outcome of each subtask of the bimanual tasks. We mistakenly described this per-subtask recording as a scoring rule of 0.5 points per hand in Appendix C.1 of the original draft, and have corrected this error in our revised manuscript. For completeness, we have also added the per-subtask scores in Appendix F.5 (Table 18) of our revised manuscript, which we summarize in the table below (full successes out of 28, with the per-subtask score in parentheses). Since all tables in our paper already report full-task success rates, no recomputation is needed.

\begin{array}{ll|ccc} \hline \text{Robot} & \text{Method} & \text{Box Convey} & \text{Air Fryer} & \text{PnP Ball in Cup} \newline \hline \text{OpenArm} & \text{Teleop only} & 0 \ (0) & 0 \ (0) & 0 \ (0) \newline \text{OpenArm} & \text{+ EgoMimic} & 0 \ (3) & 0 \ (0) & 0 \ (0) \newline \text{OpenArm} & \text{+ Masquerade} & 0 \ (5.5) & 0 \ (1) & 0 \ (0) \newline \text{OpenArm} & \textbf{+ RoboTryOn (Ours)} & \textbf{22} \ (\textbf{22}) & \textbf{17} \ (\textbf{19.5}) & \textbf{14} \ (\textbf{14.5}) \newline \hline \text{RB-Y1} & \text{Teleop only} & 0 \ (0) & 0 \ (0) & 0 \ (0) \newline \text{RB-Y1} & \text{+ EgoMimic} & 0 \ (2.5) & 0 \ (0) & 0 \ (0) \newline \text{RB-Y1} & \text{+ Masquerade} & 0 \ (3) & 0 \ (0) & 0 \ (0) \newline \text{RB-Y1} & \textbf{+ RoboTryOn (Ours)} & \textbf{9} \ (\textbf{12.5}) & 0 \ (0) & \textbf{15} \ (\textbf{15}) \newline \hline \end{array}


[W1-2/Q1] Clarify the success criteria of Pouring and Brushing

We clarify that the success criteria of Pouring and Brushing require the complete motion of each skill, not just reaching the target region. We have specified them in Appendix C.1 of our revised manuscript. For Pouring, where the bottle is capped, we evaluate the pouring motion. A trial is successful only when the robot holds the bottle (i) tilted by at least 45° from upright, (ii) toward the target cup, (iii) with the bottle mouth within 5 cm of the cup horizontally, and (iv) for at least 3 s. For Brushing, a trial is successful only when the robot sweeps over the region marked with black tape at least once while keeping the brush on the table. We evaluated all reported trials of both tasks under these criteria. We show failure and success examples of both tasks in Figure 10 of our revised manuscript. We have also corrected the success criterion of Air Fryer in Appendix C.1 to the one actually used in our evaluation. Furthermore, we have added two new tasks whose physical outcomes can be verified directly, Water Plant and Open Bag, in Appendix F.7 of our revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer oD1A (2/2)

[W2-1/Q2] Clarify the task and skill composition of the additional robot data

We clarify that the additional robot data contain none of the tasks we evaluate, and they are never used for policy training. They are used only to train the generative models and the IDM. Since the in-domain teleoperated demonstrations are limited in amount and diversity, we use the additional data to teach the generative models the appearance of each robot across a wide range of arm and hand configurations, and to teach the IDM the dynamics of each robot. We have also narrowed the statement that all teleoperated demonstrations are single-arm to the teleoperated task demonstrations in our revised manuscript.

For OpenArm and RB-Y1, we directly recorded free-motion data, in which the robot moves its arms and hands freely without contacting any object and without performing any task. These recordings also include bimanual motions. We have added examples of the free-motion data in Figure 7 (Appendix B.2) of our revised manuscript. For GR-1, we use the public teleoperation dataset used in DreamDojo [1], which consists of tabletop tasks. Most of these tasks are pick-and-place, and the objects are diverse, including office supplies, kitchenware, and food. To make clear how much of the training corpus the additional data occupy, we have broken down the training corpus by source in the table below and in Table 5 (Appendix B.2) of our revised manuscript. In Table 5, we have also corrected a miscount of the hours and episodes in the original draft.

\begin{array}{ll|cc|c} \hline \text{Robot} & \text{Source} & \text{Hours} & \text{Episodes} & \text{Policy training} \newline \hline \text{OpenArm} & \text{In-domain task demonstrations} & 1.51 & 200 & \checkmark \newline \text{OpenArm} & \text{Free-motion recordings} & 2.28 & 253 & \times \newline \hline \text{RB-Y1} & \text{In-domain task demonstrations} & 1.47 & 200 & \checkmark \newline \text{RB-Y1} & \text{Free-motion recordings} & 1.06 & 152 & \times \newline \hline \text{GR-1} & \text{GR00T teleoperation} & 2.50 & 1{,}180 & \times \newline \hline \end{array}

GR-1 is not our target embodiment. Nevertheless, since the tabletop tasks in the GR-1 data may raise a concern about fairness, we retrain all three components of RoboTryOn (the image model, the video model, and the IDM) using only the OpenArm and RB-Y1 data. As shown in the table below, the success rate remains close to that of the original RoboTryOn. For your information, we have added these results in Appendix F.8 of our revised manuscript.

\begin{array}{l|cc} \hline \text{Training data of RoboTryOn} & \text{In-domain} & \text{Out-of-domain} \newline \hline \text{OpenArm + RB-Y1 + GR-1 (original)} & 75.9 & 73.0 \newline \text{OpenArm + RB-Y1 (w/o GR-1)} & 75.0 & 74.0 \newline \hline \end{array}

[1] Gao et al., DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos, ICML 2026


[W2-2/Q3] Disentangle the benefits of visual translation and action labels

We kindly remark that our original draft already analyzed the visual gap and the action gap separately in Section 4.3. RoboTryOn closes both gaps, as its frames lie among the real robot frames (Figure 4) and its action labels succeed about twice as often as retargeted ones in open-loop replay (Figure 6c). Nevertheless, to address your concern, we further separate the two at the policy level by pairing the videos of one method with the action labels of another. As shown in the table below, replacing our action labels with retargeted actions causes a larger drop in the out-of-domain success rate than replacing our videos, which is consistent with prior findings [1, 2]. At the same time, the synthesized videos contribute in two ways. Pairing them with our action labels achieves the highest success rate. They also resemble the real robot videos that the IDM is trained on, which allows the IDM to infer accurate actions. We have added these results in Table 2c and Appendix F.3 of our revised manuscript. Furthermore, we have added an analysis of why retargeted actions lead to a low success rate and are hard to co-train with teleoperated actions in Appendix D of our revised manuscript.

\begin{array}{ll|cc} \hline \text{Video} & \text{Action} & \text{In-domain} & \text{Out-of-domain} \newline \hline \textbf{RoboTryOn} & \textbf{RoboTryOn} & \textbf{75.9} & \textbf{73.0} \newline \hline \text{RoboTryOn} & \text{Retargeting} & 69.6 & 11.2 \newline \text{EgoMimic} & \text{RoboTryOn} & 67.0 & 53.1 \newline \text{Masquerade} & \text{RoboTryOn} & 74.1 & 61.2 \newline \hline \end{array}

[1] Liu et al., EgoEngine: From Egocentric Human Videos to High-Fidelity Dexterous Robot Demonstrations, CoRL 2026

[2] Yuan et al., MotionTrans: Human VR Data Enable Motion-Level Learning for Robotic Manipulation Policies, ICRA 2026

AuthorsOpenReview ↗

Further Discussion Before the Deadline

Dear Reviewer oD1A,

Thank you once again for your time and thoughtful efforts in reviewing our paper. As the discussion period will conclude in about two days, we would like to gently remind you of our response. We have sincerely addressed your concerns, supported by the additional experimental results in our response and the revised manuscript. We would greatly appreciate it if you could let us know whether our response has resolved your concerns. If you have any further questions, we would be happy to address them before the discussion period ends.

Thank you very much,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.