acceptodds
Withdrawn from ICLR 2027

RoboEdit: Turning Human Manipulation Videos into Scalable Robot Experience

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Collecting robot hand-object interaction data is costly and embodiment-specific, yet abundant human-object videos remain unusable for robot training. We present RoboEdit, a human-to-robot video editing suite that transforms human manipulation videos into action-consistent, physically plausible robot videos with aligned 3D hand states. To enable scalable supervision, we introduce RoboEdit-ADC, an automatic pipeline that reconstructs and retargets 3D interactions from RGB videos across embodiments. This pipeline generates RoboEdit-14M, a large-scale dataset of 174K aligned video pairs (14M frames) spanning seven robot embodiments, diverse scenes, and interaction types. The core editing engine, RoboEdit-Trans, employs cross-embodiment adaptation modules to preserve temporal coherence while adapting appearance and motion. It further integrates a 3D Robot-State Decoder to recover per-frame hand states for structured motion supervision. Experiments show that RoboEdit achieves state-of-the-art editing quality and supports downstream robot control policies in real-world manipulation tasks. Ultimately, the RoboEdit suite unlocks the vast potential of unlabeled human videos, providing scalable, high-fidelity visual and 3D motion supervision for generalizable robot learning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Human-to-Robot Video Generation, Cross-Embodiment Transfer, Robot Manipulation, Robot Learning, Synthetic Robot Data, Embodied AI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hkopOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a human-to-robot video editing suite consisting of an automatic geometry/retargeting pipeline for constructing paired human–robot videos, a learned cross-embodiment video editor trained on these pairs, and a 3D robot-state decoder. The resulting dataset contains 14.1M paired frames across seven embodiments. The method is evaluated on video-editing metrics, robot-state reconstruction, and trajectory-conditioned downstream control.

Critical strengths

Large-scale and coherent system contribution. The paper integrates automatic human-object reconstruction, embodiment retargeting, video editing, and robot-state decoding into a unified pipeline and demonstrates it at substantial scale across multiple robot embodiments. Although the individual components are largely based on existing techniques, the resulting system enables a form of paired human–robot supervision that would otherwise be expensive to obtain. I view this system- and data-level capability as a substantive contribution independent of architectural novelty.

Potentially useful amortization of expensive geometric supervision. The paper introduces RoboEdit-Trans, a learned robot-conditioned video editor that amortizes part of the expensive per-video reconstruction, retargeting, physics refinement, and compositing pipeline used by RoboEdit-ADC. Once trained on the automatically curated paired data, RoboEdit-Trans can translate new human manipulation videos through model inference rather than repeatedly executing the full geometric pipeline for every example. I view this amortization of an expensive data-generation procedure into a reusable learned model as an important practical contribution toward scaling human-to-robot video supervision.

Critical weaknesses

Scalability beyond the current source distributions is not yet established.
While the paper demonstrates impressive scale in the amount of generated data, it remains unclear whether this scalability extends to substantially more unconstrained in-the-wild human videos. This limits how strongly I can interpret the broader claim of unlocking abundant human-video supervision.

The practical robotics utility of the recovered trajectories remains insufficiently characterized.
The real-robot experiments demonstrate that the recovered trajectories can provide useful references for downstream control, but the current evaluation does not clearly establish what level of robot-relevant information they reliably preserve. This makes the practical scope of the structured trajectory supervision difficult to assess.

Questions for authors

Scalability to more unconstrained human videos.
A central motivation of the paper is to leverage abundant human manipulation videos at scale. Could the authors clarify how well the proposed framework extends beyond the source distributions considered in the current experiments to more unconstrained in-the-wild videos? In particular, I would be interested in both (i) the robustness of the automatic data-construction pipeline under substantially greater variation in viewpoint, occlusion, scene complexity, objects, and interactions, and (ii) the generalization of the learned RoboEdit-Trans model to such inputs. Evidence or analysis along either of these axes would help better establish the practical scalability of the approach.

Robotic utility of the recovered trajectories.
I would appreciate a clearer characterization of the level of robotic supervision provided by the recovered trajectories at their current accuracy. The real-robot experiments show that these trajectories can serve as useful references for a downstream residual controller, but this leaves somewhat open how the trajectories themselves should be interpreted from a robot-learning perspective. Are they intended to function as approximately executable demonstrations, coarse motion references that require substantial corrective control, or structured supervision for higher-level aspects of the interaction? Any additional evidence or analysis characterizing which aspects of the recovered motion are reliable enough for downstream use would help clarify the practical scope of this contribution.

Reviewer_thtXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to turn abundant RGB human–object interaction videos into embodiment-specific robot experience for downstream robot learning. It introduces three components: RoboEdit-ADC, an automated pipeline that reconstructs 3D hand–object interactions and retargets and renders them across robot embodiments; RoboEdit-14M, a dataset containing approximately 174K aligned video pairs across seven robot embodiments; and RoboEdit-Trans, a model trained on this dataset to translate human interaction videos into aligned robot videos, from which a robot-state decoder recovers 3D robot-hand trajectories.

A central motivation is to make human videos useful for robot learning through the trajectories recovered from the generated robot videos. The paper evaluates video editing quality, includes architectural ablations, and demonstrates downstream execution using a controller conditioned on the decoded trajectories. My main concern is that the downstream evaluation provides feasibility evidence but leaves important questions about reliability and comparative utility unresolved.

Critical strengths
  • Relevant motivation and a coherent pipeline. The approach addresses the scarcity and embodiment specificity of robot interaction data by exploiting human videos. The automated data pipeline supplies the paired supervision needed to train the video translation model.
  • Substantial dataset contribution and video editing evaluation. The dataset covers seven robot embodiments. Comparisons with video editing baselines and architectural ablations support the effectiveness of the proposed modifications on the reported video-quality metrics.
  • Evidence of downstream feasibility. Simulation and real-world experiments show that a control system using the decoded trajectories as references can complete manipulation tasks. These results support the potential use of the generated trajectories for physical robot control.
Critical weaknesses

1. The simulation evaluation protocol is insufficiently specified.

Section 4.5 reports simulation success rates of 71% for Panda and 62% for XHand. Appendix G provides a per-rollout success criterion and describes training-time randomization. However, the paper does not clearly specify how many distinct source videos and reference trajectories underlie these results, how many evaluation rollouts are performed per trajectory, which perturbations are applied during evaluation, or how success is aggregated across trajectories and objects. The description of 512 parallel environments does not resolve these questions. Consequently, it is difficult to assess the coverage and robustness represented by the reported success rates.

2. The representativeness of the real-world results is unclear.

The paper states that real-world deployment uses simulation-validated trajectories and that each task–end-effector configuration achieves at least 50% success over ten trials. However, it does not report the size of the candidate trajectory pool, the fraction passing simulation validation, or the rule used to select trajectories for physical deployment. These results therefore demonstrate successful execution for the evaluated configurations, but do not establish how representative those configurations and trajectories are of the broader set of generated outputs.

3. Comparative downstream utility is not established.

The paper does not report execution-performance comparisons with other methods that recover robot trajectories from RGB human videos. It also does not compare ADC trajectories with Trans-decoded trajectories under matched execution conditions. The latter would be a particularly informative internal comparison: using the same source videos, embodiments, controller, and evaluation conditions would help establish how much execution performance is retained when replacing the ADC pipeline with Trans. The reported efficiency advantage and video-quality results do not, by themselves, resolve this question.

Taken together, the current experiments support downstream control feasibility, but provide insufficient evidence to fully assess the reliability of the generated trajectories across the evaluated population or their downstream utility relative to alternative trajectory sources.

Questions for authors
  • For the reported simulation success rates, could you specify the number of distinct source videos and reference trajectories, the number of evaluation rollouts per trajectory, the evaluation perturbations, and the aggregation procedure?
  • How many candidate trajectories were evaluated in simulation, what fraction passed validation, and how were trajectories selected for real-world deployment? Could you also provide the exact success counts and number of distinct deployed trajectories for each task–end-effector configuration?
  • Could you compare ADC trajectories and Trans-decoded trajectories using matched source videos, embodiments, controllers, and evaluation conditions? This would help quantify the execution-performance trade-off associated with the reported efficiency gains.
Reviewer_27m4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RoboEdit turns videos of human hands manipulating objects into videos of robot hands doing the same thing, together with the robot's 3D hand pose in every frame. RoboEdit-ADC uses off-the-shelf models to recover the hand, the object and the camera, retargets the motion to a robot hand with inverse kinematics and a physics-based clean-up, erases the human hand, and pastes in a rendered robot. Running it on about 24K clips from five lab datasets gives a dataset of 174K paired videos covering seven robot hands. RoboEdit-Trans is then fine-tuned on these pairs to do the conversion faster, and a decoder reads the robot hand pose back out of the edited video. The editor is compared with eight video editors, and a reinforcement-learning controller follows the decoded poses in simulation and on a real Franka arm.

Critical strengths
  • The system is complete and works end to end. From a single human video it produces a robot video and a 3D pose track, for seven different robot hands.

  • The dataset is large and built automatically, and the comparison table makes clear what it adds over earlier human-robot datasets. It would be a useful resource if it is released.

  • The editing benchmark is thorough. Eight baselines are run in two conditioning settings, and the paper openly reports the metrics on which it comes second.

Critical weaknesses
  • The paper's own ablation shows that its new modules barely matter. Fine-tuning the backbone alone already gives 0.9255 SSIM, and adding both proposed adapters raises this only to 0.9282. The large gap to the VACE baseline therefore comes from training on the new data, not from the new design.

  • The main evaluation checks how closely the editor reproduces the pipeline's own renders, and those renders are never checked for physical plausibility with numbers or human raters. No method built specifically for human-to-robot video, such as H2R-Grounder or Mitty, is compared.

  • The title promises robot experience, but no robot policy is ever trained on the edited videos. The real-robot test has no baseline and reports only "at least 50% success over 10 trials", and the paper never checks whether poses decoded from edited videos are any better than the poses the pipeline already produces directly.

  • All source videos come from five lab datasets with known objects and the pipeline is never tried on in-the-wild footage.

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.