Weakness 1: Insufficient evaluation and justification of H2Rep (the main technical contribution)
H2Rep is the main technical contribution of the paper, but its specific value is not clearly established. In Appendix Table 3, removing the pose indicator leads to a very small degradation (SSIM decreases from 0.82 to 0.80 and LPIPS increases from 0.22 to 0.23). The DROID reconstruction setting may also not be the right test for the pose indicator. The inpainted background still shows the manipulated objects moving, so the robot’s trajectory is partly recoverable even without the marker. While the qualitative example in Figure 6 is promising, it is unclear from this one example whether the pose indicator actually improves the generated video. The authors also do not quantify how accurately the generated robot follows the conditioning trajectory, particularly on human-to-robot transfer examples.
The authors do not compare against alternative intermediate representations, such as a position-only marker, multiple keypoints, or masks. A particularly relevant alternative is EgoMimic (Kareer et al., 2024). EgoMimic masks the human hand and the robot arm with SAM and overlays a red line indicating the end-effector direction (from the gripper to the elbow for the robot). Their ablation shows that this line contributes substantially to policy performance. The submission’s related work describes EgoMimic only as masking and omits the line overlay, even though it is the closest prior analogue to the marker and arrow in H2Rep. Because EgoMimic’s line also encodes part of the arm configuration, it might help constrain where the generated arm is placed. A more systematic analysis of why this particular representation works, when it fails, and how accurately the generated robot follows the conditioning would make a big difference.
The main comparisons are against general-purpose video models and a robot animation method. The general-purpose video models did not receive robot-specific DROID training, and from what I can tell, RoboMaster (the animation method) was trained on the Bridge dataset, not DROID. This makes it difficult to disentangle the benefits of H2Rep from the benefits of DROID-specific data.
Weakness 2: Limited evidence that the generated videos benefit robot learning
The paper’s motivation is robot learning from human videos, but the main text evaluates only perceptual quality (human preferences and VLM scores). In contrast, the rendering-based methods the paper criticizes (Phantom, Masquerade, H2R) are evaluated through downstream policy or pretraining performance.
The only downstream experiment is in Appendix E, and its improvement is small: roughly 6–7 percentage points on two real-world tasks, i.e., two additional successes out of 30 trials per task (18 to 20 and 14 to 16). With 30 trials, the standard error of each success rate is around 9 percentage points, so the difference is within noise, and no variance across seeds is reported. The experiment also does not isolate the benefit of translating human videos into robot videos. The translated videos are used only for a Stage-1 auxiliary loss whose targets are the 2D human hand positions from H2Rep, and these are available from the original human videos without any translation.
More fundamentally, the generated videos contain no action labels, as the authors acknowledge in Appendix E. They can therefore only serve as pretraining data, and the policy still has to be fine-tuned on real, action-labeled robot demonstrations. This also requires robot-side camera calibration. The training data construction projects DROID end-effector poses with calibrated cameras (Eq. 2), and Stage 2 of the downstream experiment projects the robot’s end-effector trajectory into the camera.
The paper argues that rendering-based methods require camera calibration, which hinders generalization to in-the-wild videos (Introduction, Section 4.2). This overstates the difference. As Appendix B acknowledges, rendering pipelines for egocentric video can reuse a fixed robot–camera transform across clips, so they need only robot-side calibration, which H2R-Grounder also requires. The remaining advantage applies mainly to third-person videos with arbitrary viewpoints: a rendering pipeline must choose a robot base pose for each video, whereas H2R-Grounder leaves this to the generator. This avoids the choice, but the generated arm’s base position and kinematics are then uncontrolled, which is also why no action labels can be derived. This trade-off should be discussed in the main text.
The paper also motivates H2R-Grounder through the shortcomings of rendering-based methods (Figure 2), but never compares against them. Section 4.2 excludes them because calibration is unavailable for in-the-wild videos, yet the authors’ own downstream setup (Appendix E) and the synchronized demonstrations in Appendix F use calibrated cameras. A rendering-based baseline in these settings, for both visual quality and downstream performance, would directly test the paper’s central claim.
Finally, in the supplementary videos, the generated robot is noticeably blurrier and noisier than real robot data, especially in the contact region between gripper and object. This is consistent with contact realism being the lowest of the decomposed physical plausibility scores (Table 6) and with the approach-direction failures in Appendix F. Since contact is what matters most for manipulation learning, this further limits the usefulness of the data (probably to only pretraining data). The reported 72% usability rate is based on the authors’ own manual inspection, with limited details about its criteria and no baselines. Plausibility alone does not demonstrate that the generated videos are useful for robot learning.