acceptodds
Withdrawn from ICLR 2027

H2R-Grounder: Paired-Data-Free Human-to-Robot Video Translation with Real-Robot Supervision

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Robots that learn manipulation skills from everyday human videos could acquire broad capabilities without tedious robot data collection. We propose a video-to-video translation framework that converts ordinary human–object interaction videos into motion-consistent robot manipulation videos with realistic and physically plausible robot–object interactions. Our approach does not require any paired human–robot videos for training – only a set of unpaired robot videos, making the system easy to scale. We introduce a transferable representation that bridges the embodiment gap: by inpainting the robot arm in training videos to obtain a clean background and overlaying a simple visual cue (a marker and arrow indicating the gripper’s position and orientation), we can condition a generative model to insert the robot arm back into the scene. At test time, we apply the same process to human videos (inpainting the person and overlaying human pose cues) and generate high-quality robot videos that mimic the human’s actions. We fine-tune a SOTA video diffusion model (Wan2.2) in an in-context learning manner to ensure temporal coherence and leveraging of its rich prior knowledge. Empirical results demonstrate that our approach achieves significantly more realistic and grounded robot motions compared to baselines, pointing to a promising direction for scaling up robot learning from unlabeled human videos.

Primary area:
generative models
Keywords:
Human-to-Robot Video Translation, Video Diffusion Models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_LUG9OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents H2R-Grounder, a framework for translating human interaction videos into robot manipulation videos without paired human–robot training data. It constructs a shared representation by removing the human or robot and overlaying 2D hand or gripper pose cues on the remaining video. A pretrained video generation model is then fine-tuned on robot videos using LoRA to reconstruct robot interactions from this representation. Experiments evaluate generation quality and motion consistency, with additional real-robot experiments exploring the utility of translated videos for policy learning.

Critical strengths
  1. The generated robot videos exhibit good visual quality, with generally consistent scene appearance and motion alignment with the human demonstrations.

  2. The downstream real-robot experiments show modest improvements in task success rates, suggesting potential benefits of the translated videos for policy learning.

  3. The paper is clearly written, and the proposed pipeline is straightforward and easy to understand.

Critical weaknesses
  1. The paper does not sufficiently establish why more visually realistic robot videos lead to better downstream policy performance. Since the utility of translated videos for robot learning is central to the paper’s motivation, the authors should provide a rigorous explanation and controlled comparisons.

  2. Although the mechanism for generating robot imagery differs, prior work such as Phantom [1] and H2R [2] has already explored robotizing human videos and demonstrated benefits for downstream robot learning. The authors should clarify the substantive differences and advantages of their approach over these methods, beyond improved visual realism.

  3. The claim of physically grounded generation is not sufficiently justified by the proposed conditioning representation. A 2D position marker and orientation arrow do not explicitly specify 3D contact configurations or kinematic feasibility. The authors should clarify how these sparse cues support physically valid robot–object interactions and provide evidence distinguishing visual plausibility from physical correctness.

[1] Lepert, M., Fang, J., & Bohg, J. (2025). Phantom: Training robots without robots using only human videos. arXiv preprint arXiv:2503.00779.

[2] Li, G., Lyu, Y., Liu, Z., Hou, C., Zhang, J., & Zhang, S. (2025). H2r: A human-to-robot data augmentation for robot pre-training from videos. arXiv preprint arXiv:2505.11920.

Questions for authors

See weaknesses.

Reviewer_19nsOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces H2R-Grounder, a video-to-video translation framework that converts videos of humans interacting with objects into robot manipulation videos. The key idea that differentiates this from prior work is an intermediate representation, H2Rep, which removes the robot/human from the video and adds a marker and arrow to indicate the pose of the gripper/hand. The authors train a video model to take this intermediate representation as input and output a video of the robot arm performing the manipulation task with its gripper aligned with the marker and arrow.

This approach does not require any paired robot-human data. Instead, the authors finetune Wan2.2 with LoRA on the DROID dataset (all Franka robot arm) by preprocessing all the data into the intermediate representation, which is used as input to the model. At inference time, they take in a human video, remove the human, add the visual indicator where the hand was, and run it through their finetuned video model.

The approach is primarily evaluated on DexYCB by human preferences and VLM (Gemini) scoring against video-editing (Kling, Runway Aleph) and robot animation (RoboMaster) baselines, with additional qualitative results on internet videos. An appendix experiment reports small, not statistically significant improvements on two real-robot tasks when the generated videos are used for auxiliary 2D trajectory pretraining of OpenVLA-OFT, followed by fine-tuning on real robot demonstrations.

Critical strengths

Strength 1: Novel, simple, and well-motivated approach Instead of rendering a robot into the scene, a video model learns from real robot data to insert a robot arm, conditioned on a representation that can be extracted from both human and robot videos. To my knowledge, this is a new way of crossing the embodiment gap. H2Rep is intuitive and requires neither paired data nor camera-to-robot calibration for the human videos. The overall system design is impressive and leverages existing tools effectively.

Strength 2: Promising qualitative results The generated videos appear much more realistic than the baselines’ results, particularly in physical plausibility, and the human preference study supports this. While they are not at the level of real robot data (see Weakness 2), the results are intriguing.

Critical weaknesses

Weakness 1: Insufficient evaluation and justification of H2Rep (the main technical contribution)

H2Rep is the main technical contribution of the paper, but its specific value is not clearly established. In Appendix Table 3, removing the pose indicator leads to a very small degradation (SSIM decreases from 0.82 to 0.80 and LPIPS increases from 0.22 to 0.23). The DROID reconstruction setting may also not be the right test for the pose indicator. The inpainted background still shows the manipulated objects moving, so the robot’s trajectory is partly recoverable even without the marker. While the qualitative example in Figure 6 is promising, it is unclear from this one example whether the pose indicator actually improves the generated video. The authors also do not quantify how accurately the generated robot follows the conditioning trajectory, particularly on human-to-robot transfer examples.

The authors do not compare against alternative intermediate representations, such as a position-only marker, multiple keypoints, or masks. A particularly relevant alternative is EgoMimic (Kareer et al., 2024). EgoMimic masks the human hand and the robot arm with SAM and overlays a red line indicating the end-effector direction (from the gripper to the elbow for the robot). Their ablation shows that this line contributes substantially to policy performance. The submission’s related work describes EgoMimic only as masking and omits the line overlay, even though it is the closest prior analogue to the marker and arrow in H2Rep. Because EgoMimic’s line also encodes part of the arm configuration, it might help constrain where the generated arm is placed. A more systematic analysis of why this particular representation works, when it fails, and how accurately the generated robot follows the conditioning would make a big difference.

The main comparisons are against general-purpose video models and a robot animation method. The general-purpose video models did not receive robot-specific DROID training, and from what I can tell, RoboMaster (the animation method) was trained on the Bridge dataset, not DROID. This makes it difficult to disentangle the benefits of H2Rep from the benefits of DROID-specific data.

Weakness 2: Limited evidence that the generated videos benefit robot learning

The paper’s motivation is robot learning from human videos, but the main text evaluates only perceptual quality (human preferences and VLM scores). In contrast, the rendering-based methods the paper criticizes (Phantom, Masquerade, H2R) are evaluated through downstream policy or pretraining performance.

The only downstream experiment is in Appendix E, and its improvement is small: roughly 6–7 percentage points on two real-world tasks, i.e., two additional successes out of 30 trials per task (18 to 20 and 14 to 16). With 30 trials, the standard error of each success rate is around 9 percentage points, so the difference is within noise, and no variance across seeds is reported. The experiment also does not isolate the benefit of translating human videos into robot videos. The translated videos are used only for a Stage-1 auxiliary loss whose targets are the 2D human hand positions from H2Rep, and these are available from the original human videos without any translation.

More fundamentally, the generated videos contain no action labels, as the authors acknowledge in Appendix E. They can therefore only serve as pretraining data, and the policy still has to be fine-tuned on real, action-labeled robot demonstrations. This also requires robot-side camera calibration. The training data construction projects DROID end-effector poses with calibrated cameras (Eq. 2), and Stage 2 of the downstream experiment projects the robot’s end-effector trajectory into the camera.

The paper argues that rendering-based methods require camera calibration, which hinders generalization to in-the-wild videos (Introduction, Section 4.2). This overstates the difference. As Appendix B acknowledges, rendering pipelines for egocentric video can reuse a fixed robot–camera transform across clips, so they need only robot-side calibration, which H2R-Grounder also requires. The remaining advantage applies mainly to third-person videos with arbitrary viewpoints: a rendering pipeline must choose a robot base pose for each video, whereas H2R-Grounder leaves this to the generator. This avoids the choice, but the generated arm’s base position and kinematics are then uncontrolled, which is also why no action labels can be derived. This trade-off should be discussed in the main text.

The paper also motivates H2R-Grounder through the shortcomings of rendering-based methods (Figure 2), but never compares against them. Section 4.2 excludes them because calibration is unavailable for in-the-wild videos, yet the authors’ own downstream setup (Appendix E) and the synchronized demonstrations in Appendix F use calibrated cameras. A rendering-based baseline in these settings, for both visual quality and downstream performance, would directly test the paper’s central claim.

Finally, in the supplementary videos, the generated robot is noticeably blurrier and noisier than real robot data, especially in the contact region between gripper and object. This is consistent with contact realism being the lowest of the decomposed physical plausibility scores (Table 6) and with the approach-direction failures in Appendix F. Since contact is what matters most for manipulation learning, this further limits the usefulness of the data (probably to only pretraining data). The reported 72% usability rate is based on the authors’ own manual inspection, with limited details about its criteria and no baselines. Plausibility alone does not demonstrate that the generated videos are useful for robot learning.

Questions for authors

Q1 (Additional experiment) Could you compare H2Rep with other simple representations, such as position-only cues, multiple keypoints, masks, or EgoMimic’s mask and line overlay? Could you also evaluate the pose-indicator ablation on human-to-robot transfer rather than DROID reconstruction?

Q2 (Additional experiment) How does using the H2Rep-generated videos compare against using the original human videos with the same auxiliary 2D objective?

Q3 (Additional experiment) Section 4.2 excludes rendering-based baselines because calibration is unavailable, but your Appendix E and F setups are calibrated. Could you compare against a rendering-based pipeline, such as those discussed in Section 4.2, in these settings, for both visual quality and downstream performance?

Q4 (Additional experiment) The current downstream evaluation (two tasks, 30 trials each, two additional successes per task by the proposed method) cannot distinguish between methods. Could you evaluate with more tasks or trials and report confidence intervals? For reference, roughly 80–90 trials per condition would detect a 20-point difference, whereas the current gap of about 7 points would need around 800. If possible, also evaluate in scenes or with objects that appear in the human videos but not in the robot demonstrations, since that is where human data should help most.

A clear improvement over the baselines in Q3 and Q4 would substantially change my assessment. Alternatively, the claims about robot learning should be scoped down to video generation.

Q5 (Clarification) Can you quantify how accurately the generated gripper follows the conditioning trajectory on DexYCB, e.g., by tracking the generated gripper and comparing it to the marker positions?

Q6 (Clarification) Could you clarify the human ranking protocol? How many videos did each participant rate? Were all four methods ranked jointly or pairwise? How were ties handled, and why do some columns of Table 1 sum to less than 100% (or higher than 100%)? Also, how does RoboMaster receive 0.8 on a 1–5 scale in Table 6?

Q7 (Future work) Is there any way to control the robot base position during generation, or to evaluate how kinematically feasible the generated arm configurations are? This seems like a necessary step toward recovering action labels from the generated videos.

Reviewer_iTVoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

H2R-Grounder translates human manipulation videos into robot videos without paired human–robot training demonstrations. Its shared representation, H2Rep, combines an inpainted background video with a 2D position-and-orientation indicator. The model is trained to reconstruct DROID robot videos from this representation using LoRA adaptation of Wan2.2, and applies the same representation to human videos at inference. Evaluation includes human and VLM assessments on 100 DexYCB videos, qualitative internet-video examples, DROID reconstruction ablations, and two real-robot tasks. The paper reports improved visual plausibility over the evaluated baselines and modest downstream policy improvements from geometric pretraining on translated videos.

Critical strengths
  1. A simple and practical cross-embodiment representation. Combining an inpainted scene video with a compact pose indicator allows training on robot demonstrations without collecting aligned human–robot pairs.

  2. Evaluation extends beyond visual examples. The paper includes pose and adaptation ablations, a human study, and a downstream real-robot experiment.

Critical weaknesses
  1. The evidence for physical grounding is limited by the evaluation design and reporting inconsistencies. The main evidence comes from human preferences and VLM ratings, while the DROID ablations use full-frame SSIM and LPIPS, which may be dominated by background preservation rather than gripper geometry or contact correctness. The quantitative human-video evaluation also covers only one DexYCB subject and one camera view.

  2. The downstream benefit is not yet attributable to robot-video translation itself. Appendix E reports gains from 60.0% to 66.7% and from 46.7% to 53.3%, corresponding to only two additional successes per task over 30 trials, without reported uncertainty or repeated training seeds. The proposed setting adds translated videos and geometric pretraining, while the baseline’s use of the auxiliary trajectory objective is unclear. Missing controls include the same training procedure using original human videos and a real-data-only policy with the same auxiliary objective. Without these comparisons, the improvement could arise from additional motion supervision rather than translating human appearance into robot appearance.

Questions for authors
  1. Does the Table 4 baseline use the same real-data trajectory objective and a comparable optimization budget? How does the proposed method compare with geometric pretraining on the original human videos using the same waypoint targets? Please report uncertainty or repeated-seed results for the policy comparison.

  2. How many clips did each participant assess, were all four methods ranked together, and how were ties, missing responses, and aggregation handled? Please reconcile Table 1’s motion-consistency total and Table 6’s out-of-range score. Please also provide the VLM checkpoint and evaluation prompts.

  3. Can you provide gripper-localized or trajectory-based measurements in addition to full-frame SSIM/LPIPS, and evaluate the pose ablation on human-to-robot transfer? This would help distinguish faithful motion transfer from plausible robot generation driven primarily by the supplied scene and object motion.

  4. Can the fine-tuned VACE variants be evaluated on the same DexYCB human-video test set? The current VACE comparison measures DROID reconstruction, while RoboMaster receives an initial frame and trajectories rather than the complete background video available to H2R-Grounder.

  5. Distinguish calibration-free human-video inference from training, which uses projected robot trajectories and camera calibration. How many unique DROID demonstrations were actually used in the 200-step training run, and were validation videos separated by trajectory or scene?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.