- Rating
- 3
- Confidence
- 3
- Informativeness
- 2(−1)
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
RoboTryOn converts human videos into robot demonstrations through keyframe translation, video synthesis, and inverse dynamics action labeling. Experiments on two dexterous robots show substantial gains on tasks without task-specific robot demonstrations.
Critical strengths
The problem is important, and the real-robot improvements are substantial. Evaluation covers two embodiments and two policy backbones. The component ablations and action replay experiments provide useful evidence for the pipeline.
Critical weaknesses
- The success rates may not always be indicative of task completion. In section C.1, some tasks such as pouring involve only tilting the bottle above the cup, while brushing involves only reaching the marked area. This compromises the assertions that new skills have been learned.
- The generators and IDM employ other robot recordings apart from those in the policy but their task composition is unknown. It would therefore be hard to prove that the asserted novel skills are not part of any of the training phases.
- The comparisons involve changes in both visual inputs and action labels. The current ablations do not distinguish between the effect of video generation and action supervision. Evaluation of the policy employs videos from the robot perspective captured by humans.
Thanks the authors for their reply and explanations. I tend to maintain my score.
Questions for authors
- What are the binary full-task success rates, especially for bimanual tasks? Please include confidence intervals.
- Which tasks, objects, and skills appear in the additional robot recordings, including the GR-1 data?
- How do policies perform with synthesized videos and retargeted actions, or rendered videos labeled by the same IDM?
- How sensitive is performance to changes in the human camera position?
5 comments
Response to Reviewer Aq1y (2/3)
[W2/Q2] Clarify the tasks, objects, and skills in the additional robot data
We clarify that the additional OpenArm and RB-Y1 recordings contain no task and no object. They are free-motion data, in which the robot moves its arms and hands freely without contacting any object and without performing any task. We have added examples of the free-motion data in Figure 7 (Appendix B.2) of our revised manuscript. The GR-1 data are the public teleoperation dataset used in DreamDojo [1], which consists of tabletop tasks. Most of these tasks are pick-and-place, and the objects are diverse, including office supplies, kitchenware, and food.
We use the additional data only to train the generative models and the IDM, and never for policy training. Since the in-domain teleoperated demonstrations are limited in amount and diversity, the additional data teach the generative models the appearance of each robot across a wide range of arm and hand configurations, and teach the IDM the dynamics of each robot. To make clear how much of the training corpus the additional data occupy, we have broken down the training corpus by source in the table below and in Table 5 (Appendix B.2) of our revised manuscript. In Table 5, we have also corrected a miscount of the hours and episodes in the original draft.
\begin{array}{ll|cc|c} \hline \text{Robot} & \text{Source} & \text{Hours} & \text{Episodes} & \text{Policy training} \newline \hline \text{OpenArm} & \text{In-domain task demonstrations} & 1.51 & 200 & \checkmark \newline \text{OpenArm} & \text{Free-motion recordings} & 2.28 & 253 & \times \newline \hline \text{RB-Y1} & \text{In-domain task demonstrations} & 1.47 & 200 & \checkmark \newline \text{RB-Y1} & \text{Free-motion recordings} & 1.06 & 152 & \times \newline \hline \text{GR-1} & \text{GR00T teleoperation} & 2.50 & 1{,}180 & \times \newline \hline \end{array}
GR-1 is not our target embodiment. Nevertheless, since the tabletop tasks in the GR-1 data may raise a concern about fairness, we retrain all three components of RoboTryOn (the image model, the video model, and the IDM) using only the OpenArm and RB-Y1 data. As shown in the table below, the success rate remains close to that of the original RoboTryOn. For your information, we have added these results in Appendix F.8 of our revised manuscript.
\begin{array}{l|cc} \hline \text{Training data of RoboTryOn} & \text{In-domain} & \text{Out-of-domain} \newline \hline \text{OpenArm + RB-Y1 + GR-1 (original)} & 75.9 & 73.0 \newline \text{OpenArm + RB-Y1 (w/o GR-1)} & 75.0 & 74.0 \newline \hline \end{array}
[1] Gao et al., DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos, ICML 2026
Response to Reviewer Aq1y (3/3)
[W3-1/Q3] Disentangle the effects of visual inputs and action labels
We would like to point out that our original draft already analyzed the visual gap and the action gap separately in Section 4.3. RoboTryOn closes both gaps, as its frames lie among the real robot frames (Figure 4) and its action labels succeed about twice as often as retargeted ones in open-loop replay (Figure 6c). Nevertheless, to address your concern, we further separate the two at the policy level by pairing the videos of one method with the action labels of another. Regarding the rendered videos of the baselines, we observe that the IDM, which is trained on real robot videos, infers inaccurate actions from these rendered videos, which look different from real robot videos. We therefore pair the rendered videos with the action labels that the IDM infers from the videos synthesized by RoboTryOn for the same human videos. As shown in the table below, replacing our action labels with retargeted actions causes a larger drop in the out-of-domain success rate than replacing our videos, which is consistent with prior findings [1, 2]. At the same time, pairing our videos with our action labels achieves the highest success rate. We have added these results in Table 2c and Appendix F.3 of our revised manuscript. Furthermore, we have added an analysis of why retargeted actions lead to a low success rate and are hard to co-train with teleoperated actions in Appendix D of our revised manuscript.
\begin{array}{ll|cc} \hline \text{Video} & \text{Action} & \text{In-domain} & \text{Out-of-domain} \newline \hline \textbf{RoboTryOn} & \textbf{RoboTryOn} & \textbf{75.9} & \textbf{73.0} \newline \hline \text{RoboTryOn} & \text{Retargeting} & 69.6 & 11.2 \newline \text{EgoMimic} & \text{RoboTryOn} & 67.0 & 53.1 \newline \text{Masquerade} & \text{RoboTryOn} & 74.1 & 61.2 \newline \hline \end{array}
[1] Liu et al., EgoEngine: From Egocentric Human Videos to High-Fidelity Dexterous Robot Demonstrations, CoRL 2026
[2] Yuan et al., MotionTrans: Human VR Data Enable Motion-Level Learning for Robotic Manipulation Policies, ICRA 2026
[W3-2/Q4] Sensitivity to the human camera position
Thank you for the suggestion. To measure this sensitivity, we record the human videos of four out-of-domain tasks with a different camera, and find that RoboTryOn still outperforms all baselines. Specifically, this camera is mounted lower than the robot's head camera and has a narrower field of view (see Figure 13 of our revised manuscript), and we train the OpenArm policy in the same way as in our main experiments. As shown in the table below, the success rate drops from 80.4% to 57.1% on average, since the IDM is trained only on robot data captured from the fixed head camera of each robot. Nevertheless, it remains far above all baselines, which are trained on human videos recorded from the robot's camera position. This shows that the generative models of RoboTryOn are robust to the camera position, as they still translate these videos into plausible robot videos (Figure 13). We have added these results in Appendix F.4 of our revised manuscript. Building on this robustness, we have also shown that RoboTryOn applies to in-the-wild human videos recorded with moving cameras from EgoDex [1] and VITRA [2], where pre-training on the converted videos improves the success rate of the policy (Appendix F.6 of our revised manuscript).
\begin{array}{l|ccccc} \hline \text{Method} & \text{PnP Coffee Pot} & \text{PnP Bottle to Shelf} & \text{Pouring} & \text{Brushing} & \text{Average} \newline \hline \text{Teleop only} & 14.3 & 7.1 & 0.0 & 0.0 & 5.4 \newline \text{+ EgoMimic} & 14.3 & 0.0 & 7.1 & 14.3 & 8.9 \newline \text{+ Masquerade} & 21.4 & 7.1 & 7.1 & 21.4 & 14.3 \newline \text{+ RoboTryOn (Ours, robot view)} & 64.3 & 89.3 & 85.7 & 82.1 & 80.4 \newline \hline \text{+ RoboTryOn (Ours, different view)} & 35.7 & 78.6 & 71.4 & 42.9 & 57.1 \newline \hline \end{array}
[1] Hoque et al., EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video, ICLR 2026
[2] Li et al., Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videos, ICRA 2026
Response to Reviewer Aq1y (1/3)
Dear reviewer Aq1y,
We sincerely appreciate your efforts in reviewing our manuscript. We respond to each comment in the following content. In the revised manuscript, we have marked the revisions with “blue”.
[W1-1/Q1] Clarify the full-task success rates
We clarify that all success rates reported in our paper are already binary full-task success rates, where a trial counts as a success only when the entire task is completed. During evaluation, we recorded the outcome of each subtask of the bimanual tasks. We mistakenly described this per-subtask recording as a scoring rule of 0.5 points per hand in Appendix C.1 of the original draft, and have corrected this writing error in our revised manuscript. Since all tables in our paper already report full-task success rates, no recomputation is needed. For the bimanual tasks, the table below summarizes the number of fully successful trials out of 28, with the per-subtask score that we have added in Appendix F.5 (Table 18) of our revised manuscript in parentheses.
\begin{array}{ll|ccc} \hline \text{Robot} & \text{Method} & \text{Box Convey} & \text{Air Fryer} & \text{PnP Ball in Cup} \newline \hline \text{OpenArm} & \text{Teleop only} & 0 \ (0) & 0 \ (0) & 0 \ (0) \newline \text{OpenArm} & \text{+ EgoMimic} & 0 \ (3) & 0 \ (0) & 0 \ (0) \newline \text{OpenArm} & \text{+ Masquerade} & 0 \ (5.5) & 0 \ (1) & 0 \ (0) \newline \text{OpenArm} & \textbf{+ RoboTryOn (Ours)} & \textbf{22} \ (\textbf{22}) & \textbf{17} \ (\textbf{19.5}) & \textbf{14} \ (\textbf{14.5}) \newline \hline \text{RB-Y1} & \text{Teleop only} & 0 \ (0) & 0 \ (0) & 0 \ (0) \newline \text{RB-Y1} & \text{+ EgoMimic} & 0 \ (2.5) & 0 \ (0) & 0 \ (0) \newline \text{RB-Y1} & \text{+ Masquerade} & 0 \ (3) & 0 \ (0) & 0 \ (0) \newline \text{RB-Y1} & \textbf{+ RoboTryOn (Ours)} & \textbf{9} \ (\textbf{12.5}) & 0 \ (0) & \textbf{15} \ (\textbf{15}) \newline \hline \end{array}
[W1-2/Q1] Confidence intervals of the full-task success rates
Thank you for the suggestion. We have added the 90% confidence intervals of the full-task success rates in our main results in Appendix F.2 (Table 13) of our revised manuscript, which we summarize in the table below. We estimate them using stratified bootstrapping [1].
\begin{array}{ll|ccc} \hline \text{Robot} & \text{Method} & \text{In-domain} & \text{Bimanual} & \text{Out-of-domain Avg.} \newline \hline \text{OpenArm} & \text{Teleop only} & 47.3 \pm 7.4 & 0.0 \pm 0.0 & 3.1 \pm 1.9 \newline \text{OpenArm} & \text{+ EgoMimic} & 50.0 \pm 7.6 & 0.0 \pm 0.0 & 5.1 \pm 2.5 \newline \text{OpenArm} & \text{+ Masquerade} & 67.9 \pm 6.7 & 0.0 \pm 0.0 & 8.2 \pm 3.1 \newline \text{OpenArm} & \textbf{+ RoboTryOn (Ours)} & \textbf{75.9} \pm 6.5 & \textbf{63.1} \pm 8.4 & \textbf{73.0} \pm 5.0 \newline \hline \text{RB-Y1} & \text{Teleop only} & 69.6 \pm 7.0 & 0.0 \pm 0.0 & 1.0 \pm 1.1 \newline \text{RB-Y1} & \text{+ EgoMimic} & 63.4 \pm 7.4 & 0.0 \pm 0.0 & 5.1 \pm 2.5 \newline \text{RB-Y1} & \text{+ Masquerade} & 63.4 \pm 7.5 & 0.0 \pm 0.0 & 7.7 \pm 3.0 \newline \text{RB-Y1} & \textbf{+ RoboTryOn (Ours)} & \textbf{79.5} \pm 6.2 & \textbf{28.6} \pm 7.1 & \textbf{46.9} \pm 5.2 \newline \hline \end{array}
[1] Agarwal et al., Deep Reinforcement Learning at the Edge of the Statistical Precipice, NeurIPS 2021
[W1-3] Clarify the success criteria of Pouring and Brushing
We clarify that the success criteria of Pouring and Brushing require the complete motion of each skill, not just reaching the target region. We have specified them in Appendix C.1 of our revised manuscript. For Pouring, where the bottle is capped, we evaluate the pouring motion. A trial is successful only when the robot holds the bottle (i) tilted by at least 45° from upright, (ii) toward the target cup, (iii) with the bottle mouth within 5 cm of the cup horizontally, and (iv) for at least 3 s. For Brushing, a trial is successful only when the robot sweeps over the region marked with black tape at least once while keeping the brush on the table. We evaluated all reported trials of both tasks under these criteria. We show failure and success examples of both tasks in Figure 10 of our revised manuscript. We have also corrected the success criterion of Air Fryer in Appendix C.1 to the one actually used in our evaluation.