- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper presents ViT-H, a video-to-humanoid framework that directly generates Unitree G1 reference motions from monocular human videos, avoiding explicit human motion reconstruction and inference-time retargeting. A video-conditioned diffusion model generates motions, which are executed by whole-body trackers; tracking outcomes provide feedback for DPO post-training. An autoregressive variant enables real-time teleoperation. Experiments show gains over reconstruction-retargeting baselines in motion quality and executability, with ablations on tracking feedback, data composition, and model scaling.
Critical strengths
- The generation-and-tracking formulation is meaningful: it optimizes motion generation with downstream robot executability rather than only human-motion reconstruction or visual similarity.
- The evaluation is comprehensive, covering motion quality and executability, with useful ablations, cross-tracker tests, and real-time teleoperation. The results support the practical value of the framework.
Critical weaknesses
- The DPO-specific contribution is not isolated. Equation (12) combines DPO with supervised fine-tuning (SFT) on preferred motions. The comparisons with no post-training and the ablations in Figure 3 support the effectiveness of the overall procedure, but do not establish how much DPO adds beyond fitting the selected high-quality samples. A focused SFT-only ablation would clarify the contribution of pairwise preference learning.
- The framework depends on the motion tracker used for feedback, so the learned improvement may partly reflect tracker preferences. More evaluations on diverse trackers for feedback such as BeyondMimic and GMT would strengthen these claims.
- Physical and task-level evaluation is limited. Quantitative results rely mainly on simulation tracking, with little evidence on disturbances, contact dynamics, energy efficiency, or long-horizon stability. The current framework is also closer to motion imitation than task-oriented behavior involving objects, environment reasoning, and manipulation.
Questions for authors
- Why are there synthetic data in the training set, but synthetic data is not used in the experimental evaluation? Meanwhile, on the video-only test set, what are the common failed actions?
- Before the evaluation, RICH and EMDB underwent action screening and calculated the success rate based on 250-frame segments. How many sequences and which action types were retained before and after the screening? Can the 100% success rate on RICH be maintained if longer continuous sequences are used?
6 comments
W2. Tracker dependence (2/5)
The reviewer writes:
The framework depends on the motion tracker used for feedback, so the learned improvement may partly reflect tracker preferences.
We clarify that Fig. 5 evaluates ScaleBFM, HoloMotion, and Humanoid-GPT, none of which provides feedback for the original post-training (PT). All three benefit from post-training.
To examine whether the gains also hold with a different feedback source, we repeated DPO using ScaleBFM and evaluated with Ours , HoloMotion, and Humanoid-GPT on the same video-only test set. We chose ScaleBFM because it is a more recent general motion tracker than GMT. BeyondMimic trains a separate policy for each reference motion, so it cannot evaluate newly generated motions during post-training.
Table R10a. Offline cross-tracker transfer with ScaleBFM feedback.
| Evaluation tracker | w/o PT | w/ PT (ScaleBFM) | ||
|---|---|---|---|---|
| Success rate ↑ | Trk. MPJPE ↓ | Success rate ↑ | Trk. MPJPE ↓ | |
| Ours | 74.8 | 59.0 | 78.3 | 53.2 |
| HoloMotion | 65.9 | 76.6 | 71.0 | 70.5 |
| Humanoid-GPT | 46.4 | 109.0 | 54.2 | 103.4 |
Table R10b. Online cross-tracker transfer with ScaleBFM feedback.
| Evaluation tracker | w/o PT | w/ PT (ScaleBFM) | ||
|---|---|---|---|---|
| Success rate ↑ | Trk. MPJPE ↓ | Success rate ↑ | Trk. MPJPE ↓ | |
| Ours | 49.6 | 71.7 | 61.5 | 64.4 |
| HoloMotion | 49.7 | 88.3 | 58.9 | 82.1 |
| Humanoid-GPT | 41.1 | 131.5 | 45.6 | 124.2 |
Success improves and Trk. MPJPE decreases for all three evaluation trackers in both modes. Cross-tracker gains therefore also hold when ScaleBFM provides the DPO feedback, supporting transfer beyond the preferences of one tracker. We will include this comparison in Sec. 4.2. Incorporating feedback from multiple trackers remains future work [C,E].
W3. Physical evaluation and task scope (3/5)
The reviewer writes:
Quantitative results rely mainly on simulation tracking, with little evidence on disturbances, contact dynamics, energy efficiency, or long-horizon stability.
Although robustness, contact and energy are not claimed contributions, we evaluate them to provide further evidence of physical execution. We added these evaluations in MuJoCo (Tab. R11). All methods use . Baselines include the original and adapted (†, Sec. C.4) versions of GVHMR + GMR and GEM + GMR.
To evaluate disturbance robustness, contact dynamics, energy efficiency, and long-horizon stability, we follow the evaluation approaches in [1-4]. Push Succ. (%) measures tracking success under external disturbances [1], with five 70 N horizontal pushes to the pelvis per trial. Foot slip (cm/s) measures sliding speed during ground contact, adapting the contact-displacement evaluation in PhysPT [2]. Abs. mech. power (W) measures average absolute joint mechanical power, adapting the positive-power evaluation in Walk These Ways [3]. Long Succ. (%) measures uninterrupted full-sequence tracking success, following PHC's sequence-level evaluation [4].
Table R11. Physical evaluation on EMDB and RICH.
| Method | EMDB | RICH | ||||||
|---|---|---|---|---|---|---|---|---|
| Push Succ. ↑ | Foot slip ↓ | Abs. mech. power ↓ | Long Succ. ↑ | Push Succ. ↑ | Foot slip ↓ | Abs. mech. power ↓ | Long Succ. ↑ | |
| GVHMR + GMR | 58.8 | 3.3 | 87.0 | 21.2 | 87.2 | 2.0 | 61.3 | 80.5 |
| GVHMR† + GMR | 68.3 | 3.3 | 90.6 | 34.6 | 82.0 | 2.1 | 66.6 | 78.9 |
| GEM + GMR | 70.4 | 3.4 | 87.3 | 30.8 | 86.3 | 2.1 | 63.0 | 79.7 |
| GEM† + GMR | 69.4 | 3.5 | 92.5 | 25.0 | 85.5 | 2.0 | 65.3 | 81.2 |
| Ours (online) w/o PT | 72.6 | 3.0 | 87.2 | 44.2 | 84.7 | 2.1 | 60.1 | 63.9 |
| Ours (online) w/ PT | 88.5 | 2.5 | 77.5 | 75.0 | 89.0 | 1.8 | 50.4 | 78.2 |
| Ours (offline) w/o PT | 88.9 | 2.8 | 84.9 | 59.6 | 100.0 | 1.5 | 50.6 | 86.5 |
| Ours (offline) w/ PT | 90.2 | 2.7 | 84.3 | 80.8 | 100.0 | 1.5 | 50.2 | 91.0 |
Post-training (PT) improves or maintains push success and improves full-sequence success. Both generators show similar or lower foot slip and lower mechanical power. We will include these evaluations in App. D and their metric definitions in Sec. C.3.
Additionally, our real-world demonstrations further validate our method's ability to execute tasks over long horizons, with seven loco-manipulation examples and up to 82 seconds of video teleoperation [C,E].
The reviewer also writes:
The current framework is also closer to motion imitation than task-oriented behavior involving objects, environment reasoning, and manipulation.
We clarify that our task is to convert human motion in monocular videos into whole-body humanoid actions, as defined in the first sentence of Sec. 3. Object-goal planning, environment reasoning, and dexterous manipulation are outside this scope and were not claimed capabilities. We will make this task boundary more explicit in Sec. 3 [C].
[1] van Marum et al., Revisiting Reward Design and Evaluation for Robust Humanoid Standing and Walking, IROS 2024.
[2] Zhang et al., PhysPT: Physics-aware Pretrained Transformer for Estimating Human Dynamics from Monocular Videos, CVPR 2024.
[3] Margolis and Agrawal, Walk These Ways: Tuning Robot Control for Generalization with Multiplicity of Behavior, CoRL 2022.
[4] Luo et al., Perpetual Humanoid Control for Real-time Simulated Avatars, ICCV 2023.
We thank the reviewer for their thoughtful comments and suggestions.
W1. DPO vs. SFT-only (1/5)
The reviewer writes:
Following this suggestion, we post-trained the models with only the SFT term of Eq. (12), setting λDPO=0 while keeping the preferred samples, checkpoint, trainable layers, and training schedule fixed. DPO (Ours) uses both loss terms. Tab. R9 compares both variants with the Base model before post-training on the video-only test set in Fig. 3(a)(b), using πtrack.
Table R9. Base, SFT-only, and DPO on the video-only test set.
SFT-only provides limited gains. Adding DPO improves success and lowers Trk. MPJPE for both generators. Fitting the preferred samples alone does not reproduce these gains, showing why DPO post-training is necessary. We will add this ablation to Sec. 4.2 alongside Fig. 3 [E].