acceptodds
Under review as a conference paper at ICLR 2027

ViT-H: Generative Video-To-Humanoid Control with Motion Tracking Feedback

14 views

Abstract

Learning humanoid control from monocular human videos offers a scalable way to acquire diverse whole-body behaviors. Yet, existing reconstruction-retargeting-tracking pipelines are limited in system-level scalability by separately optimized intermediate stages and cannot directly optimize reference motion executability using downstream tracking feedback. We present ViT-H, a video-to-humanoid control framework following a generation-and-tracking paradigm. ViT-H directly generates humanoid reference motions from videos with a diffusion model, bypassing human motion reconstruction and retargeting, and uses motion-tracking outcomes to post-train the generator with direct preference optimization. We further introduce an autoregressive variant that enables real-time humanoid video teleoperation with sub-500 ms latency. Extensive experiments on the Unitree G1 demonstrate that ViT-H outperforms state-of-the-art reconstruction-retargeting-tracking pipelines in both motion quality and executability. We further show that ViT-H scales favorably with model size and with both pre-training and post-training data scale, while motion-tracking feedback consistently improves executability across different motion trackers. We will release our code and processed datasets.

Primary area:
applications to robotics, autonomy, planning
Keywords:
humanoid robotics, humanoid control, motion generation, motion tracking, post-training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FHCMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents ViT-H, a video-to-humanoid framework that directly generates Unitree G1 reference motions from monocular human videos, avoiding explicit human motion reconstruction and inference-time retargeting. A video-conditioned diffusion model generates motions, which are executed by whole-body trackers; tracking outcomes provide feedback for DPO post-training. An autoregressive variant enables real-time teleoperation. Experiments show gains over reconstruction-retargeting baselines in motion quality and executability, with ablations on tracking feedback, data composition, and model scaling.

Critical strengths
  1. The generation-and-tracking formulation is meaningful: it optimizes motion generation with downstream robot executability rather than only human-motion reconstruction or visual similarity.
  2. The evaluation is comprehensive, covering motion quality and executability, with useful ablations, cross-tracker tests, and real-time teleoperation. The results support the practical value of the framework.
Critical weaknesses
  1. The DPO-specific contribution is not isolated. Equation (12) combines DPO with supervised fine-tuning (SFT) on preferred motions. The comparisons with no post-training and the ablations in Figure 3 support the effectiveness of the overall procedure, but do not establish how much DPO adds beyond fitting the selected high-quality samples. A focused SFT-only ablation would clarify the contribution of pairwise preference learning.
  2. The framework depends on the motion tracker used for feedback, so the learned improvement may partly reflect tracker preferences. More evaluations on diverse trackers for feedback such as BeyondMimic and GMT would strengthen these claims.
  3. Physical and task-level evaluation is limited. Quantitative results rely mainly on simulation tracking, with little evidence on disturbances, contact dynamics, energy efficiency, or long-horizon stability. The current framework is also closer to motion imitation than task-oriented behavior involving objects, environment reasoning, and manipulation.
Questions for authors
  1. Why are there synthetic data in the training set, but synthetic data is not used in the experimental evaluation? Meanwhile, on the video-only test set, what are the common failed actions?
  2. Before the evaluation, RICH and EMDB underwent action screening and calculated the success rate based on 250-frame segments. How many sequences and which action types were retained before and after the screening? Can the 100% success rate on RICH be maintained if longer continuous sequences are used?
6 comments

We thank the reviewer for their thoughtful comments and suggestions.

W1. DPO vs. SFT-only (1/5)

The reviewer writes:

A focused SFT-only ablation would clarify the contribution of pairwise preference learning.

Following this suggestion, we post-trained the models with only the SFT term of Eq. (12), setting λDPO=0\lambda_{\mathrm{DPO}}=0 while keeping the preferred samples, checkpoint, trainable layers, and training schedule fixed. DPO (Ours) uses both loss terms. Tab. R9 compares both variants with the Base model before post-training on the video-only test set in Fig. 3(a)(b), using πtrack\pi_{\mathrm{track}}.

Table R9. Base, SFT-only, and DPO on the video-only test set.

Training objectiveOurs (offline)Ours (online)
Success rate ↑Trk. MPJPE ↓Success rate ↑Trk. MPJPE ↓
Base74.859.049.671.7
SFT-only on preferred samples76.557.155.768.2
DPO (Ours)80.054.263.865.2

SFT-only provides limited gains. Adding DPO improves success and lowers Trk. MPJPE for both generators. Fitting the preferred samples alone does not reproduce these gains, showing why DPO post-training is necessary. We will add this ablation to Sec. 4.2 alongside Fig. 3 [E].

W2. Tracker dependence (2/5)

The reviewer writes:

The framework depends on the motion tracker used for feedback, so the learned improvement may partly reflect tracker preferences.

We clarify that Fig. 5 evaluates ScaleBFM, HoloMotion, and Humanoid-GPT, none of which provides feedback for the original post-training (PT). All three benefit from post-training.

To examine whether the gains also hold with a different feedback source, we repeated DPO using ScaleBFM and evaluated with Ours πtrack\pi_{\mathrm{track}}, HoloMotion, and Humanoid-GPT on the same video-only test set. We chose ScaleBFM because it is a more recent general motion tracker than GMT. BeyondMimic trains a separate policy for each reference motion, so it cannot evaluate newly generated motions during post-training.

Table R10a. Offline cross-tracker transfer with ScaleBFM feedback.

Evaluation trackerw/o PTw/ PT (ScaleBFM)
Success rate ↑Trk. MPJPE ↓Success rate ↑Trk. MPJPE ↓
Ours πtrack\pi_{\mathrm{track}}74.859.078.353.2
HoloMotion65.976.671.070.5
Humanoid-GPT46.4109.054.2103.4

Table R10b. Online cross-tracker transfer with ScaleBFM feedback.

Evaluation trackerw/o PTw/ PT (ScaleBFM)
Success rate ↑Trk. MPJPE ↓Success rate ↑Trk. MPJPE ↓
Ours πtrack\pi_{\mathrm{track}}49.671.761.564.4
HoloMotion49.788.358.982.1
Humanoid-GPT41.1131.545.6124.2

Success improves and Trk. MPJPE decreases for all three evaluation trackers in both modes. Cross-tracker gains therefore also hold when ScaleBFM provides the DPO feedback, supporting transfer beyond the preferences of one tracker. We will include this comparison in Sec. 4.2. Incorporating feedback from multiple trackers remains future work [C,E].

W3. Physical evaluation and task scope (3/5)

The reviewer writes:

Quantitative results rely mainly on simulation tracking, with little evidence on disturbances, contact dynamics, energy efficiency, or long-horizon stability.

Although robustness, contact and energy are not claimed contributions, we evaluate them to provide further evidence of physical execution. We added these evaluations in MuJoCo (Tab. R11). All methods use πtrack\pi_{\mathrm{track}}. Baselines include the original and adapted (†, Sec. C.4) versions of GVHMR + GMR and GEM + GMR.

To evaluate disturbance robustness, contact dynamics, energy efficiency, and long-horizon stability, we follow the evaluation approaches in [1-4]. Push Succ. (%) measures tracking success under external disturbances [1], with five 70 N horizontal pushes to the pelvis per trial. Foot slip (cm/s) measures sliding speed during ground contact, adapting the contact-displacement evaluation in PhysPT [2]. Abs. mech. power (W) measures average absolute joint mechanical power, adapting the positive-power evaluation in Walk These Ways [3]. Long Succ. (%) measures uninterrupted full-sequence tracking success, following PHC's sequence-level evaluation [4].

Table R11. Physical evaluation on EMDB and RICH.

MethodEMDBRICH
Push Succ. ↑Foot slip ↓Abs. mech. power ↓Long Succ. ↑Push Succ. ↑Foot slip ↓Abs. mech. power ↓Long Succ. ↑
GVHMR + GMR58.83.387.021.287.22.061.380.5
GVHMR† + GMR68.33.390.634.682.02.166.678.9
GEM + GMR70.43.487.330.886.32.163.079.7
GEM† + GMR69.43.592.525.085.52.065.381.2
Ours (online) w/o PT72.63.087.244.284.72.160.163.9
Ours (online) w/ PT88.52.577.575.089.01.850.478.2
Ours (offline) w/o PT88.92.884.959.6100.01.550.686.5
Ours (offline) w/ PT90.22.784.380.8100.01.550.291.0

Post-training (PT) improves or maintains push success and improves full-sequence success. Both generators show similar or lower foot slip and lower mechanical power. We will include these evaluations in App. D and their metric definitions in Sec. C.3.

Additionally, our real-world demonstrations further validate our method's ability to execute tasks over long horizons, with seven loco-manipulation examples and up to 82 seconds of video teleoperation [C,E].

The reviewer also writes:

The current framework is also closer to motion imitation than task-oriented behavior involving objects, environment reasoning, and manipulation.

We clarify that our task is to convert human motion in monocular videos into whole-body humanoid actions, as defined in the first sentence of Sec. 3. Object-goal planning, environment reasoning, and dexterous manipulation are outside this scope and were not claimed capabilities. We will make this task boundary more explicit in Sec. 3 [C].

[1] van Marum et al., Revisiting Reward Design and Evaluation for Robust Humanoid Standing and Walking, IROS 2024.

[2] Zhang et al., PhysPT: Physics-aware Pretrained Transformer for Estimating Human Dynamics from Monocular Videos, CVPR 2024.

[3] Margolis and Agrawal, Walk These Ways: Tuning Robot Control for Generalization with Multiplicity of Behavior, CoRL 2022.

[4] Luo et al., Perpetual Humanoid Control for Real-time Simulated Avatars, ICCV 2023.

Reviewer_JdtHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ViT-H, a video-to-humanoid control framework that directly generates humanoid reference motions using a diffusion model, bypassing explicit human motion reconstruction and retargeting at inference time. It further uses motion-tracking feedback and DPO to improve executability. Experiments demonstrate improved motion quality and tracking performance over existing pipelines, including cross-tracker evaluations and real-world demonstrations.

Critical strengths
  1. Strong empirical performance. ViT-H achieves substantial improvements in motion accuracy and tracking success over reconstruction-retargeting baselines, supported by extensive simulation experiments and real-world demonstrations.

  2. Effective tracking-feedback optimization. DPO post-training improves motion executability, with gains transferring across multiple unseen trackers. This provides meaningful evidence that tracking feedback can improve generated reference motions beyond a specific controller.

Critical weaknesses
  1. Insufficient evidence for the claimed advantage of bypassing reconstruction and retargeting. The training motions are themselves largely obtained through GMR retargeting. Although ViT-H outperforms existing pipelines, the comparisons do not isolate whether the improvement comes from direct robot-space generation or differences in model architecture and training objectives.

  2. Unclear trade-off between motion fidelity and executability. For video-only post-training data, preference rewards primarily measure tracking performance without directly evaluating fidelity to the demonstrated motion. Improved executability may therefore partly result from generating simpler or less dynamic motions rather than more faithfully executable versions of the original actions. The existing evaluations do not sufficiently distinguish these possibilities.

Questions for authors
  1. Controlled comparison: Can the authors compare direct robot-space generation against human-motion generation followed by GMR, using the same diffusion architecture, video inputs, and training data? This would help isolate the benefit of eliminating the intermediate human-motion representation.

  2. Motion fidelity: Does DPO post-training preserve the original motion's amplitude, velocity, and dynamic characteristics, particularly for challenging motions? A quantitative comparison before and after post-training would help determine whether tracking improvements come at the cost of motion fidelity.

3 comments

We thank the reviewer for their thoughtful comments and suggestions.

W1 / Q1. Direct robot-space prediction (1/3)

The reviewer writes:

The training motions are themselves largely obtained through GMR retargeting.

We clarify that bypassing reconstruction and retargeting refers to inference. We use an adapted GMR to prepare training targets, with multiple initialization seeds and quality filtering to improve their quality (Sec. C.1). We will make this distinction explicit in the data-preparation description [C].

The reviewer asks:

Can the authors compare direct robot-space generation against human-motion generation followed by GMR, using the same diffusion architecture, video inputs, and training data?

Following this suggestion, we trained new models that output human motion, using the same diffusion architecture, video inputs, training data, and schedule as our method. Only the output representation was changed; the predicted human motions were then retargeted to G1 with GMR. Tab. R6 and Tab. R7 compare this variant with direct G1 prediction before post-training (PT), in both modes. Post-trained results are also included for reference.

Table R6. Motion quality for human-space and robot-space prediction.

MethodEMDBRICH
MPJPE ↓PA-MPJPE ↓RTE ↓Jitter ↓Pen. ↓MPJPE ↓PA-MPJPE ↓RTE ↓Jitter ↓Pen. ↓
GVHMR + GMR73.354.43.118.42.351.644.94.67.71.7
GVHMR† + GMR71.449.13.516.51.669.651.54.410.32.2
GEM + GMR62.338.44.619.61.356.549.44.711.92.2
GEM† + GMR72.951.04.521.72.070.351.24.420.52.2
Ours (online) human output + GMR, w/o PT79.464.67.323.70.555.450.36.621.70.5
Ours (online) w/o PT55.436.68.325.30.244.534.56.824.40.3
Ours (online) w/ PT55.536.86.418.50.144.336.16.317.40.2
Ours (offline) human output + GMR, w/o PT64.340.43.117.81.151.443.04.310.52.1
Ours (offline) w/o PT52.934.02.611.20.238.728.54.47.20.2
Ours (offline) w/ PT53.334.32.510.60.138.429.44.37.20.1

W1 / Q1. Direct robot-space prediction (continued, 2/3)

Table R7. Executability for human-space and robot-space prediction.

MethodEMDBRICH
Succ. ↑MPJPE ↓PA-MPJPE ↓Trk. MPJPE ↓Trk. PA-MPJPE ↓Succ. ↑MPJPE ↓PA-MPJPE ↓Trk. MPJPE ↓Trk. PA-MPJPE ↓
GVHMR + GMR61.7101.259.847.529.890.364.647.236.428.9
GVHMR† + GMR71.494.557.044.730.084.790.858.253.434.7
GEM + GMR73.094.943.043.225.490.371.751.037.929.7
GEM† + GMR73.0100.058.143.929.490.390.955.152.534.2
Ours (online) human output + GMR, w/o PT71.4112.972.356.535.281.991.357.752.234.0
Ours (online) w/o PT76.985.841.334.023.187.576.040.934.321.6
Ours (online) w/ PT88.584.040.729.416.498.269.840.027.617.3
Ours (offline) human output + GMR, w/o PT75.289.242.242.924.790.371.950.538.429.7
Ours (offline) w/o PT89.575.440.032.018.6100.062.034.326.318.2
Ours (offline) w/ PT90.275.139.731.518.0100.062.034.625.917.7

In the offline setting, human output + GMR improves over the adapted baselines on most metrics. The casual online setting is comparable to these offline baselines. In both settings, direct G1 prediction is clearly better, so the main gain comes from robot-space prediction. The architecture and training objectives contribute, but play a secondary role. We will include this controlled comparison in Sec. 4.1 [E].

W2 / Q2. Motion dynamics before and after post-training (3/3)

The reviewer asks:

Does DPO post-training preserve the original motion's amplitude, velocity, and dynamic characteristics, particularly for challenging motions?

To address this question, we compared both generators before and after post-training (PT) on EMDB and RICH. Tab. R8a and Tab. R8b evaluate generated motions and motions executed by πtrack\pi_{\mathrm{track}} against GT. Velocity and acceleration errors measure dynamic differences (mm/frame and mm/frame² at 50 fps); NPSS compares temporal frequency structure.

Table R8a. Dynamic fidelity of generated motions.

MethodEMDBRICH
Vel. error ↓Acc. error ↓NPSS ↓Vel. error ↓Acc. error ↓NPSS ↓
Ours (online) w/o PT19.42.74.17.62.220.8
Ours (online) w/ PT19.32.32.57.11.616.9
Ours (offline) w/o PT18.51.93.76.51.021.3
Ours (offline) w/ PT18.31.83.46.51.019.8

Table R8b. Dynamic fidelity after tracking.

MethodEMDBRICH
Vel. error ↓Acc. error ↓NPSS ↓Vel. error ↓Acc. error ↓NPSS ↓
Ours (online) w/o PT20.01.94.17.21.022.0
Ours (online) w/ PT19.91.92.77.01.019.5
Ours (offline) w/o PT19.21.93.76.70.921.3
Ours (offline) w/ PT19.11.83.66.60.919.6

The metrics remain comparable after post-training, as does spatial accuracy in the paper's Tab. 1. Damped or simplified motions would increase these errors against GT, which we do not observe.

We clarify that the video-only test set contains highly dynamic, challenging motions such as gymnastics and martial arts, as shown in the Supp. video. We conducted a human study for both online and offline models on this test set, following the protocol in the paper (Sec. C.2). Twenty participants with computer vision or robotics backgrounds watched the input video alongside the generated motions before and after PT, and chose which better transfers the human motion to G1.

Table R8c. Human preference on the video-only test set.

MethodPrefer w/ PTPrefer w/o PT
Ours (online)76.223.8
Ours (offline)58.741.3

As shown in Tab. R8c, participants prefer the post-trained motions for both models. Together with the comparisons before and after post-training in the Supp. video, these results show that PT improves tracking without sacrificing motion fidelity, including on challenging motions. We will add these comparisons to Sec. 4.1 and describe the expanded human study in Sec. C.2 [C,E].

Reviewer_s4YnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ViT-H, a generative video-to-humanoid control framework that directly predicts humanoid reference motions from monocular videos, bypassing the conventional human motion reconstruction and retargeting stages. The framework adopts a generation-and-tracking paradigm, where a video-conditioned diffusion model generates humanoid motions and a whole-body motion tracker executes them. To improve motion executability, the authors introduce a post-training approach that leverages tracking feedback to construct preference pairs and optimize the motion generator through DPO. An autoregressive variant is further developed to support real-time video teleoperation. Experiments on Unitree G1 demonstrate improvements in motion quality and executability over existing reconstruction-retargeting-tracking pipelines, along with favorable scaling behavior and generalization of post-training gains across different motion trackers.

Critical strengths

A well-motivated framework for video-to-humanoid control. The paper proposes a generation-and-tracking framework that directly predicts humanoid motions from monocular videos, bypassing the conventional reconstruction and retargeting stages. More importantly, it incorporates downstream motion-tracking feedback into the generator's post-training through DPO, explicitly optimizing motion executability rather than focusing solely on motion reconstruction accuracy. This is a meaningful and practically relevant design for humanoid control.

Critical weaknesses

Potential trade-off between motion fidelity and executability. The tracking-based rewards primarily measure how accurately the robot executes the generated reference motions, rather than how faithfully those motions reproduce the input video. This may favor simpler, easier-to-track motions at the expense of challenging motion details. Although the paper reports comparable average motion accuracy before and after post-training, it does not sufficiently investigate this potential trade-off, particularly for challenging motions and video-only datasets without 3D ground truth.

Questions for authors

Compute Efficiency of Tracking-Feedback Post-Training. Given that both pre-training and post-training data scaling improve motion executability, how does tracking-feedback-based DPO compare with simply allocating the same additional computational budget to pre-training? A compute-matched comparison would help establish whether the proposed post-training procedure offers benefits beyond increasing the scale of supervised training.

3 comments

We thank the reviewer for their thoughtful comments and suggestions.

W1. Motion fidelity (1/2)

The reviewer writes:

This may favor simpler, easier-to-track motions at the expense of challenging motion details.

To address this concern, we introduce three additional metrics: velocity error, acceleration error, and NPSS (Normalized Power Spectrum Similarity). We use them alongside the paper's existing metrics to assess whether motion details are preserved. Velocity and acceleration errors measure dynamic differences in mm/frame and mm/frame² at 50 fps; NPSS compares temporal frequency structure. For both online and offline models, we evaluate generated motions (Tab. R1) and motions executed by πtrack\pi_{\mathrm{track}} (Tab. R2) against GT on EMDB and RICH.

Table R1. Dynamic fidelity of generated motions.

MethodEMDBRICH
Vel. error ↓Acc. error ↓NPSS ↓Vel. error ↓Acc. error ↓NPSS ↓
Ours (online) w/o PT19.42.74.17.62.220.8
Ours (online) w/ PT19.32.32.57.11.616.9
Ours (offline) w/o PT18.51.93.76.51.021.3
Ours (offline) w/ PT18.31.83.46.51.019.8

Table R2. Dynamic fidelity after tracking.

MethodEMDBRICH
Vel. error ↓Acc. error ↓NPSS ↓Vel. error ↓Acc. error ↓NPSS ↓
Ours (online) w/o PT20.01.94.17.21.022.0
Ours (online) w/ PT19.91.92.77.01.019.5
Ours (offline) w/o PT19.21.93.76.70.921.3
Ours (offline) w/ PT19.11.83.66.60.919.6

As shown in Tab. R1 and Tab. R2, all three metrics remain comparable after post-training (PT). Damped or simplified motions would increase these errors against GT, which we do not observe. Together with the original metrics, these results show that post-training improves success rates without sacrificing motion details [E].

The reviewer also writes:

it does not sufficiently investigate this potential trade-off, particularly for challenging motions and video-only datasets without 3D ground truth.

We clarify that the video-only test set contains highly dynamic, challenging motions such as gymnastics and martial arts, as shown in the Supp. video. We conducted a human study for both online and offline models, following the protocol in the paper (Sec. C.2). Twenty participants with computer vision or robotics backgrounds watched the input video alongside the generated motions before and after post-training, and chose which better transfers the human motion to G1.

Table R3. Human preference on the video-only test set.

MethodPrefer w/ PTPrefer w/o PT
Ours (online)76.223.8
Ours (offline)58.741.3

As shown in Tab. R3, participants prefer the post-trained motions for both models. Together with the comparisons before and after post-training in the Supp. video, these results show that post-training preserves motion fidelity even on challenging motions. We will include these comparisons in Sec. 4.1 and update the study description in Sec. C.2 [C,E].

Q1. Compute efficiency (2/2)

The reviewer asks:

Given that both pre-training and post-training data scaling improve motion executability, how does tracking-feedback-based DPO compare with simply allocating the same additional computational budget to pre-training?

Post-training (PT) is lightweight relative to pre-training, with schedules of 28k and 730k steps, respectively. Following the reviewer's suggestion, we continued supervised training from the same pre-trained checkpoint for 90k steps, more than three times the post-training schedule. This control tests whether additional supervised training can account for the gains. Tab. R4 and Tab. R5 report the metrics from the paper's Tab. 1 and Tab. 2.

Table R4. Motion quality with additional supervised training.

MethodEMDBRICH
MPJPE ↓PA-MPJPE ↓RTE ↓Jitter ↓Pen. ↓MPJPE ↓PA-MPJPE ↓RTE ↓Jitter ↓Pen. ↓
Ours (online) w/o PT55.436.68.325.30.244.534.56.824.40.3
Ours (online) +90k supervised updates55.536.87.925.30.244.534.96.724.20.3
Ours (online) w/ PT55.536.86.418.50.144.336.16.317.40.2
Ours (offline) w/o PT52.934.02.611.20.238.728.54.47.20.2
Ours (offline) +90k supervised updates53.134.22.611.50.238.828.84.37.20.2
Ours (offline) w/ PT53.334.32.510.60.138.429.44.37.20.1

Table R5. Executability with additional supervised training.

MethodEMDBRICH
Succ. ↑MPJPE ↓PA-MPJPE ↓Trk. MPJPE ↓Trk. PA-MPJPE ↓Succ. ↑MPJPE ↓PA-MPJPE ↓Trk. MPJPE ↓Trk. PA-MPJPE ↓
Ours (online) w/o PT76.985.841.334.023.187.576.040.934.321.6
Ours (online) +90k supervised updates76.285.641.034.022.885.875.640.633.421.7
Ours (online) w/ PT88.584.040.729.416.498.269.840.027.617.3
Ours (offline) w/o PT89.575.440.032.018.6100.062.034.326.318.2
Ours (offline) +90k supervised updates89.575.340.231.919.0100.062.034.526.318.2
Ours (offline) w/ PT90.275.139.731.518.0100.062.034.625.917.7

Additional supervised training leaves motion quality largely unchanged and success similar. DPO post-training preserves motion quality while improving executability. Even with more supervised updates, executability does not improve, so the gains come from tracking feedback rather than additional training. We will add this comparison to Sec. 4.3.

Additionally, we clarify that the video-only post-training data have no 3D labels and cannot be used for supervised training, and post-training with makes them usable [C,E].

Thank you for the authors' response. I will maintain my positive rating.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of Rebuttal

We thank all reviewers for their constructive comments and suggestions. The reviewers recognize our framework as well-motivated, meaningful, and practically relevant (s4Yn, FHCM), highlighting its focus on downstream motion executability (s4Yn, JdtH, FHCM) and effective tracking-feedback optimization (JdtH). They further acknowledge the strong empirical performance (JdtH) and comprehensive evaluation (FHCM), including substantial improvements over baselines (JdtH), generalization across unseen trackers (JdtH), extensive simulation experiments and real-world demonstrations (JdtH), and useful ablations and real-time teleoperation (FHCM). These results support the practical value of our framework (s4Yn, FHCM). We address the reviewers' concerns below and will revise the paper with clarifications [C] and additional experiments [E].

New experiments in this rebuttal:

  • Dynamic-fidelity metrics and a human study before and after post-training (s4Yn, JdtH)
  • Continued supervised training as a control for post-training (s4Yn)
  • Human-space vs. robot-space prediction with the same architecture and data (JdtH)
  • SFT-only ablation of the post-training objective (FHCM)
  • Post-training with an alternative feedback tracker (FHCM)
  • Push robustness, foot slip, mechanical power, and long-horizon success (FHCM)
  • Evaluation on longer sequences (FHCM)
AuthorsOpenReview ↗

Follow-up on Reviewer Responses

Dear Reviewers,

We thank Reviewer s4Yn for their response!

We understand that the CVPR deadline has just passed and that this is a particularly busy time for everyone. However, with less than 24 hours remaining before the author-reviewer discussion closes, we would greatly appreciate it if the other reviewers could take a look at our rebuttal and let us know whether their concerns have been addressed. We are also happy to clarify further if needed.

Thank you for your time!

AuthorsOpenReview ↗

Follow-up on Reviewer Responses

Dear Reviewers,

We thank reviewers s4Yn and FHCM for their responses and positive feedback!

With less than 5 hours remaining in the discussion period, we would appreciate it if reviewer JdtH could take a look at our rebuttal and share any remaining concerns.

Thank you for your time!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.