acceptodds
Desk rejected at ICLR 2027

BeyondRetarget: Learning Executable Humanoid Motions Directly from Monocular Video

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

12 views

Abstract

Learning executable motions from human videos offers a scalable solution for humanoid robots to acquire demonstration motions. However, existing pipelines typically first construct an explicit human motion representation and then convert it into robot motions via motion retargeting. Although such methods can effectively leverage large volumes of existing human data for training, the substantial differences between humans and humanoid robots in locomotion mechanisms and joint degree-of-freedom configurations make motions generated by this human-representation-centric approach difficult to execute on robots. Furthermore, errors introduced during human motion estimation inevitably propagate to the retargeting stage and cannot be eliminated via joint optimization. We propose BeyondRetarget, an end-to-end framework that directly maps monocular RGB videos to robot motions. Discarding the explicit human representation, this framework learns robot-oriented implicit representations directly from visual observations, enabling the model to capture cross-morphology motion structures. To generate motions more suitable for robot execution, we further design a contact-aware motion optimization mechanism to improve temporal consistency and physical plausibility. Experiments show that BeyondRetarget significantly improves the accuracy and robustness of generated robot motions, while achieving higher execution success rates and lower latency in both simulation environments and real humanoid robots.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotics, Computer Vision, Embodied Intelligence, Humanoid Robot, Pose Estimation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_8etWOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents BeyondRetarget, a framework that directly predicts humanoid robot motion from monocular human videos without an explicit human-motion intermediate at inference. It learns a shared motion representation using morphology-aware geometric supervision and adapts to different robots through robot-specific output branches. Contact-aware refinement improves motion consistency and foot support. Experiments across multiple humanoid embodiments, including simulation and real-robot execution, demonstrate competitive motion quality and efficient inference compared with two-stage reconstruction-and-retargeting pipelines.

Critical strengths
  1. The motivation and problem: learning executable motions from human videos for humanoid robots is crucial and the proposed solution via end-to-end training is promising.

  2. The main experiment demonstrates persuasive improvement comparing with two-stage baselines, while the extended evaluation in appendix shows competitive cross-dataset performance.

  3. The shared motion representation and robot-specific output branches enable efficient adaptation to different humanoid embodiments, with experiments on eight robots supporting the flexibility of the approach.

Critical weaknesses
  1. Although the evaluation seems fair, could the authors clarify whether the two-stage baselines were adapted to MotionPRO and discuss how differences in training distributions may affect the observed improvements? Addressing the generalization limitations of these baselines is not necessary, but this clarification would help interpret the performance gap.

  2. The in-the-wild examples and cross-dataset evaluation are encouraging. Could the authors discuss the data and annotation requirements for extending training to web videos, particularly how human-motion and contact supervision could be obtained? The cost of such expansion is relevant to the framework’s scalability.

Questions for authors

See Weaknesses.

Reviewer_VqMtOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

BeyondRetarget predicts robot root poses and joint trajectories directly from monocular human video using frozen visual features, a temporal encoder, and robot-specific decoders. Training uses morphology-adjusted human motion annotations and differentiable forward kinematics, without paired robot trajectories. Contact prediction and geometric refinement improve foot stability. Evaluation covers eight robot kinematic models, with simulation and real-robot execution demonstrated on G1 using SONIC.

Critical strengths
  1. The supervision scheme supports learning robot trajectories from human annotations without paired robot joint-state labels. Adapting robot-specific decoder branches offers a practical way to reuse the learned visual representation across morphologies.
  2. On MotionPRO, the method reduces G1 morphology-aligned pose error to 26.36 mm versus 39.19 mm for the strongest estimated-motion baseline. The ablations also demonstrate that contact refinement reduces foot sliding.
Critical weaknesses
  1. The advantage over SONIC's existing video-control pipeline is not established. SONIC already supports recorded monocular video and live webcam control through GEM and its human-motion interface. The paper compares human reconstruction followed by explicit retargeting, but omits this native video-control baseline. SONIC also documents training on new embodiments, so producing explicit trajectories for multiple robots does not by itself establish a comparative advantage.
  2. The title emphasizes “Executable,” which is misleading. The proposed method generates references, while SONIC provides the execution controller. Execution is validated only on G1; the other seven robots have kinematic metrics and FK visualizations. Appendix I appropriately qualifies executability, but the evidence still does not establish executable motion generation across eight embodiments. Even on G1, Table 1 improves execution success from the strongest baseline's 342/362 to 347/362, approximately 1.4 percentage points.
  3. Online quality and downstream benefits remain unquantified. The online configuration uses past-only cross-window context, whereas offline inference accesses future windows (Appendix H). Table 4 reports runtime but no separate online pose accuracy or execution success rate. Consequently, the offline accuracy and online speed results do not establish that both are achieved together. The supplementary teleoperation examples provide qualitative evidence, but no controlled advantage over existing video teleoperation is shown. Similarly, the proposed use as training data for imitation learning is not evaluated through downstream policy performance.
  4. Attribution of the reported gains is also incomplete: the baseline protocol does not establish comparable MotionPRO adaptation or evaluate the same contact refinement on baseline trajectories. These controls are needed to distinguish benefits of direct robot prediction from those of training data and postprocessing.
Questions for authors
  1. How does BeyondRetarget compare with GEM followed by SONIC's native human-motion interface on the same videos, measuring executed motion quality, success rate, and latency?
  2. Which accuracy and execution results, if any, use the online configuration? Please report these separately and clarify how contact refinement operates causally, including any buffering delay.
  3. Do the gains persist when the strongest two-stage baseline receives comparable MotionPRO adaptation and the same contact refinement? This would materially clarify the contribution of removing the explicit human-motion intermediate.
Reviewer_rT8nOpenReview ↗
Rating
3(+1)
Confidence
2(−2)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BeyondRetarget, a framework that directly predicts executable humanoid robot motions from monocular human RGB videos, avoiding the explicit human motion reconstruction and retargeting stages during inference. The method combines a shared robot-oriented motion representation, unified human-robot geometric supervision, and contact-aware kinematic refinement. Experiments on MotionPRO and RICH, together with multi-robot evaluations, physics simulations, and real-robot demonstrations, assess its motion accuracy, robustness, and practical applicability.

Critical strengths
  1. Clear and Meaningful Reformulation of the Video-to-Robot Pipeline The paper proposes a meaningful departure from the conventional two-stage video-to-robot pipeline commonly adopted in humanoid motion learning. By eliminating the explicit human motion reconstruction and retargeting stages during inference, the proposed framework directly predicts robot motions from monocular RGB videos. This is a clear and practically relevant contribution, as it simplifies the inference pipeline and potentially avoids error propagation across separate stages. I appreciate this task-oriented reformulation and consider it one of the paper's primary strengths.
  2. Extensive Experimental Evaluation and Practical Validation The paper provides a relatively comprehensive experimental evaluation, demonstrating clear performance advantages over existing two-stage baselines on the primary MotionPRO benchmark. Beyond pose accuracy, the evaluation includes temporal consistency, physical plausibility, motion executability, and inference efficiency. The experiments also cover multiple humanoid embodiments and include both physics simulation and real-robot demonstrations. These results provide valuable evidence of the framework's practical applicability and demonstrate substantial experimental effort.
Critical weaknesses
  1. Insufficient Differentiation from Established Humanoid Locomotion and Motion Tracking Techniques and Inadequate Validation of the Core Innovation

-I have concerns about the novelty of several methodological contributions. In particular, the proposed Unified Human-Robot Supervision is closely related to morphology adaptation in H2O and non-uniform local scaling in GMR. Similarly, the contact-based root correction and motion stabilization in Contact-aware Kinematic Refinement resemble techniques used in PBHC (KungfuBot, NeurIPS 2025).

-Given that these are representative works in humanoid locomotion and motion tracking, I would expect a clearer discussion of the technical differences. However, the paper does not sufficiently distinguish its proposed components from these prior methods, despite presenting them as methodological contributions. Although GMR is cited, its relationship to the proposed supervision mechanism remains unclear.

-More importantly, the ablation studies mainly validate components related to prior work, rather than the paper's central idea of direct -video-to-robot prediction. While comparisons with two-stage baselines are provided, differences in architecture and supervision make it difficult to isolate the benefit of removing the intermediate human motion representation.

I would like the authors to clarify the technical differences from H2O, GMR, and PBHC, and provide stronger evidence for the independent benefit of direct prediction.

  1. Concerns Regarding Proxy Ground Truth and the Fairness of Quantitative Evaluation

I have two concerns about the experimental evaluation.

-First, the primary MAMPJPE evaluation relies on proxy ground truth obtained by morphologically aligning SMPL annotations to the target robot. The proposed method is trained using the same alignment procedure, while the baselines are not explicitly optimized against this particular proxy target. Could this introduce an evaluation bias, where the improvements partly reflect better fitting to the chosen proxy rather than better robot motion fidelity? Although simulation and real-robot experiments provide additional evidence, the validity of this evaluation reference needs further justification.

-Second, BeyondRetarget achieves slightly worse MAMPJPE than GVHMR→NMR on RICH (67.970 vs. 65.745 mm), which the authors attribute to differences in training data diversity. However, since BeyondRetarget is trained and primarily evaluated on MotionPRO, could the same distribution differences also contribute to its large improvements over GVHMR on MotionPRO?

-Furthermore, GVHMR reports a training cost of approximately 13 hours on two RTX 4090 GPUs, suggesting that a more controlled comparison may be feasible. Why were the baselines not adapted to MotionPRO, or the proposed method trained on a broader dataset? I would appreciate clarification of these experimental choices and their computational constraints.

Questions for authors

Q1. Novelty and Technical Distinctions

Could the authors clarify the technical differences between the proposed Unified Human-Robot Supervision and the morphology adaptation techniques in H2O and GMR? Similarly, how does the proposed Contact-aware Kinematic Refinement differ from the contact-based motion correction mechanisms in PBHC (KungfuBot)? Which specific components should be regarded as the main methodological innovations beyond these established techniques?

Q2. Controlled Validation of Direct Prediction

The central contribution appears to be replacing the conventional two-stage pipeline with direct video-to-robot prediction. Could the authors provide a more controlled comparison that isolates the benefit of removing the explicit SMPL intermediate representation, ideally under comparable training data, supervision, and model capacity?

Q3. Validity of Proxy Ground Truth

Given that the proposed method is trained using morphology-aligned SMPL targets constructed with the same alignment procedure used in MAMPJPE evaluation, while the baselines are not explicitly optimized against this particular proxy target, could the reported performance gains partly reflect better fitting to the chosen evaluation reference rather than genuine improvements in robot motion fidelity? Could the authors provide independent validation or comparisons using alternative evaluation references to address this potential bias?

Q4. Training Distribution and Baseline Fairness

The authors attribute the slightly worse MAMPJPE on RICH to differences in training data diversity and model capacity. Could these same differences also contribute to BeyondRetarget's substantial advantage on MotionPRO, where it is trained and primarily evaluated? Have the authors considered adapting GVHMR to MotionPRO or evaluating the proposed method under more comparable training distributions? If not, what are the main practical or computational constraints?

I would be open to revising my rating upward if the authors can convincingly address these concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.