acceptodds
Under review as a conference paper at ICLR 2027

FlowHMR: Physically Plausible Motion Capture from Video

12 views

Abstract

We present FlowHMR, a framework for recovering global 3D human motion from monocular video. Previous methods commonly learn to recover human poses and trajectories through geometric supervision. However, geometric reconstruction ob- jectives do not explicitly account for the dynamics and contact constraints of human motion. Our approach models the video-conditioned distribution of global human motion using Flow Matching. We train the model on a large collection of videos paired with 3D motion annotations and further optimize it through reinforcement learning, jointly rewarding reconstruction fidelity and accurate tracking in simu- lation. To evaluate motion recovery performance, we introduce Wild-4K, a large and diverse evaluation set of internet videos. Experiments show that FlowHMR outperforms state-of-the-art methods in overall motion quality as judged by human evaluators and in simulation tracking performance. Our code will be made publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human Motion Generation, Human Motion Capture, Digital Human

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dEHvOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper combines video-conditioned flow matching with physics-based RL to recover body and hand motion and global trajectory from monocular video. It also introduces Wild-4K, a large and diverse evaluation set for assessing motion recovery quality and simulation tracking performance. The experiments show that it outperforms state-of-the-art methods.

Critical strengths
  1. The paper is clearly written and well organized, making the method and experiments easy to follow.
  2. The proposed framework combines video-conditioned flow matching with physics-based post-training, jointly modeling body pose, hand articulation, and global trajectories in world coordinates for motion recovery from monocular video. And it outperforms state-of-the-art methods in overall motion quality as judged by human evaluators.
  3. The introduced physics-based reinforcement learning approach balances executability and reconstruction fidelity. The ablations show why optimizing tracking alone is insufficient.
  4. Wild-4K provides a useful evaluation set covering diverse in-the-wild motions, including challenging actions and occlusions.
Critical weaknesses
  1. Table 1 uses the proposed Wild-4K dataset, and Table 2 uses the internal Studio dataset. I suggest moving the full baseline comparison on RICH into the main paper, as it provides important evidence of reconstruction accuracy on a public benchmark. Additional results on EMDB would further strengthen the evaluation.
  2. Table 2 reports RTE and Jitter in mm and mm/s², whereas Table D.2 uses % and m/s³. The authors should clarify whether these differences reflect distinct definitions or labeling errors.
  3. Although joint body and hand recovery is a stated contribution, hand-specific quantitative results mainly come from the internal Studio dataset. Could the authors report body and hand reconstruction errors separately on RICH, if its hand annotations support reliable evaluation?
Questions for authors

See the weaknesses. I would be willing to raise my score if the authors address most of my concerns.

4 comments

We thank the reviewer for the constructive feedback.

Q1: RICH and EMDB evaluation

We will move the full RICH comparison into the main paper and add the EMDB-2 evaluation to the supplementary material.

EMDB-2 and training data coverage. EMDB-2 is a standard benchmark for global motion recovery with moving cameras. It includes motions involving non-level terrain or vehicles, such as walking on slopes and skateboarding. These scenarios are not covered by our training data and incur larger reconstruction errors. The following table compares default VGGT-Ω inference on the full set and a subset excluding these scenarios. WA and W are measured in mm, and RTE in %.

Evaluation setWA ↓W ↓RTE ↓
Full EMDB-297.82233.313.60
Subset excluding motions on non-level terrain or vehicles65.28179.991.81

We also explore camera estimation and PnP alignment to handle these scenarios. We replace the input camera rotation R and use the corresponding camera, detected keypoints, and predicted body motion for PnP alignment. However, this approach depends on accurate camera estimation: failures in camera translation estimation can cause the recovered human trajectory to drift. We therefore do not use it as our default inference procedure. Our default uses only VGGT-Ω rotations as camera conditions and directly predicts motion in world coordinates. The following results compare camera configurations on the full EMDB-2 set.

Method / SettingWA ↓W ↓RTE ↓
GVHMR111.0276.52.0
GENMO74.3202.11.2
DuoMo66.0167.11.1
FlowHMR, default VGGT-Ω97.82233.313.60
FlowHMR, VIPE + PnP alignment62.12160.121.97
FlowHMR, DROID-SLAM + PnP alignment69.90187.992.20
FlowHMR, GT camera + PnP alignment54.19136.990.68

3DPW and motion representation. FlowHMR directly predicts motion in world coordinates and does not directly output camera-space motion, so camera-space evaluation on 3DPW requires additional alignment. We tested a GVHMR-style representation that estimates camera-space motion and camera parameters, making it straightforward to overlay the motion on the input image and compute metrics such as MPJPE. Both FlowHMR variants below are evaluated before RL; SR denotes PHC+ tracking success rate.

Method / RepresentationRICH WA ↓RICH W ↓Wild-4K SR (%) ↑
DuoMo52.8678.2856.71
FlowHMR, paper representation (w/o RL)55.7088.3778.79
FlowHMR, GVHMR-style (w/o RL)49.179.071.02

The GVHMR-style variant has lower RICH WA and W than the pretrained model in the paper, and its WA is also better than DuoMo's. However, its Wild-4K tracking success is lower, its motions are less stable, and its overall visual quality is worse than that of the paper version. We therefore adopt the current world-coordinate representation and emphasize global motion metrics and human evaluation.

Q2: Units of RTE and Jitter

Thank you for pointing this out. RTE divides the aligned root trajectory error by the total ground-truth path length and multiplies the result by 100, so it is reported in %. Jitter measures the third temporal derivative of joint positions (jerk), computed in m/s³ and divided by 10 for reporting. Keeping this numerical scale, the column should be labeled Jitter (10 m/s³). We will correct the unit labels and clarify the reporting scale in Table 2, Table D.2, and the related tables.

Q3: Separate body and hand evaluation

We follow GVHMR's body24 protocol for RICH. The MPJPE metrics in Table D.2 therefore evaluate body joints and exclude fingers, enabling consistent comparison with other global motion capture methods.

Our model retains finger joints during training and prediction. Figure B.1 separately evaluates finger quality on Wild-300, where our pretrained model achieves the highest mean finger rating among the compared methods (3.55/5). Tables 2 and B.1 also separately report hand rotation errors (MPJRE_hand) on Studio. We will clarify the scope of these evaluations in the revision.

AuthorsOpenReview ↗

Follow-up on Our Response to Reviewer dEHv

Dear Reviewer dEHv,

Thank you again for your thoughtful feedback. We have posted a response to your comments with additional clarifications and experimental results, and we have updated the manuscript in light of the reviewers’ feedback.

With the discussion deadline approaching, we would appreciate it if you could take a look at our response and let us know if any concerns remain. We would be glad to address any follow-up questions before the deadline. If the new evidence and clarifications address your concerns, we would be grateful if you would revisit your overall assessment in light of the revision.

Thank you for your valuable time and consideration!

The authors' response has addressed most of my concerns. I have decided to raise my score to 3.

We sincerely thank the reviewer for the thoughtful questions and valuable feedback, which have been very helpful in improving our paper.

Reviewer_QM38OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents FlowHMR, a framework for recovering global 3D human motion (body, hands, and world-space trajectory) from monocular video. Motion recovery is formulated as conditional generation: an MMDiT-based Flow Matching model is pretrained on ~3,000 hours of synthetic rendered video, conditioned on frozen SAM 3D Body features and camera rotations. The model is then post-trained with MixGRPO using two rewards: a fidelity reward against the ground-truth annotation, and a tracking reward that measures how well a fixed PHC+ controller follows the generated motion in simulation. The authors also introduce Wild-4K, an evaluation set of 4,182 internet video clips without ground truth. On Wild-4K, FlowHMR outperforms GVHMR, PromptHMR, GENMO, and DuoMo in blinded pairwise human evaluation and in PHC+ tracking success rate (82.47% vs. 62.82% for the strongest baseline). Ablations cover data scale, generative vs. regression formulation, model size, and post-training strategies and reward combinations.

Critical strengths

Physical executability matters for downstream uses such as character animation and humanoid learning from video, and geometric reconstruction objectives alone do not ensure it. Bringing simulation feedback into a video-conditioned motion generator is a natural and relevant direction.

Critical weaknesses

Training reward and evaluation metric share the same controller. The tracking reward (Eq. 4) and the headline Success Rate (Table 1, Eq. 13) both come from the same frozen PHC+ controller with the same 0.6 threshold. The model is thus optimized directly on its main evaluation metric, and Table 3 shows how easily this metric is exploited (tracking-only GRPO: 99.62% SR, 255.67 mm WA-MPJPE). The only independent-controller evaluation (SONIC, Appendix D.4) has its own problems. It covers only 3,247 of 4,182 clips with no explanation for the rest, and reports results on a 615-clip "success-union" subset. This means all methods fail on ~81% of clips, and conditioning on the union of successes biases the comparison.

Questions for authors

Q1:What are the SONIC success rates on the full set of 3,247 (or 4,182) clips, and why were the remaining clips excluded from the SONIC evaluation? Q2:Does RL post-training reduce motion amplitude (e.g., joint velocity distributions, root displacement) compared with the pretrained model? Since pairwise seed MPJPE drops from 57.25 to 22.86 mm (Table C.2), it would help to know whether the tracking gains partly come from more conservative motions.

4 comments

We thank the reviewer for the constructive feedback.

Q1: SONIC evaluation set and success rates

Wild-4K includes complex motions such as gymnastics and fitness exercises. In our evaluation, many SONIC tracking failures occurred on large-amplitude motions. SONIC is designed for the Unitree G1. In BONES-SEED, the publicly released subset of its motion data, the Locomotion package accounts for 52.38% of clips (74,488/142,220), including walking, jogging, jumping, climbing, and crawling. Its official training pipeline excludes certain acrobatic motions. Tracking success therefore depends on both reconstruction quality and the motion capabilities of the robot and controller.

Because Wild-4K lacks ground-truth 3D motion, we cannot use ground-truth tracking to identify motions that SONIC can execute. We attempted evaluation on all 4,182 clips. After excluding clips with retargeting or other processing failures for any of the five original methods, the common evaluation set contained 3,247 clips. Tracking failures within this set were included in the denominator.

At least one of the five original methods tracked successfully on 615 clips, while all five failed on the remaining 2,632. Table D.3 reported success rates on this 615-clip subset. We now report results on both the subset and the full common evaluation set:

MethodSR on 615 clips (%) ↑SR on 3,247 clips (%) ↑
PromptHMR44.078.35
GENMO48.789.24
DuoMo51.069.67
GVHMR54.1510.26
FlowHMR w/o RL57.2310.84
FlowHMR77.7214.72

Both columns use the same success counts, divided by 615 and 3,247, respectively.

SONIC is not used to compute our training rewards. Physics-based post-training improves success by 20.49 percentage points on the 615-clip subset and 3.88 percentage points on the full common evaluation set. We will include both evaluations and clarify the selection protocol in the revised manuscript.

Q2: Motion amplitude and reconstruction fidelity after RL

Mean limb speed and motion amplitude decrease slightly after RL, while root movement is largely preserved. We compare the pretrained and RL models on the same 500 Wild-4K clips, using their direct motion predictions before simulation tracking:

Metric (mean)Before RLAfter RLRelative change
Limb joint speed¹ (m/s)0.20660.1960−5.13%
Limb motion amplitude (RMS)¹ (cm)8.818.59−2.50%
Root path length (m)1.9671.933−1.73%
Horizontal root path length (m)1.67141.6708−0.04%
Net root displacement (m)0.9380.933−0.53%

¹ Limb metrics are computed in the local pelvis frame after removing root translation and full 3D rotation at each frame. For RMS amplitude, we compute each joint's root-mean-square displacement from its temporal mean position within the sequence, then average over 21 joints.

The pairwise seed MPJPE in Table C.2 measures variation between predictions from different random samples for the same video. Its reduction after RL indicates more consistent outputs.

Reconstruction metrics on RICH also show small changes (Table D.2): WA-MPJPE increases from 55.70 to 56.68 mm, W-MPJPE changes from 88.37 to 88.43 mm, and RTE decreases from 1.92% to 1.82%.

We will add these motion amplitude statistics in the revision and move the RICH and SONIC evaluation tables into the main paper.

AuthorsOpenReview ↗

Follow-up on Our Response to Reviewer QM38

Dear Reviewer QM38,

Thank you again for your thoughtful feedback. We have posted a response to your comments with additional clarifications and experimental results, and we have updated the manuscript in light of the reviewers’ feedback.

With the discussion deadline approaching, we would appreciate it if you could take a look at our response and let us know if any concerns remain. We would be glad to address any follow-up questions before the deadline. If the new evidence and clarifications address your concerns, we would be grateful if you would revisit your overall assessment in light of the revision.

Thank you for your valuable time and consideration!

Thank you for the authors' response. I will maintain my positive rating.

Thank you very much for your thoughtful follow-up and for taking the time to review our revisions and additional experiments. We are glad that our responses have adequately addressed your concerns, and we sincerely appreciate your positive assessment and constructive feedback, which have helped improve the clarity and completeness of our paper.

Reviewer_iD6vOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1(−1)
Read review
Paper summary

The paper presents FlowHMR, a method that recovers global 3D human motion from monocular video. It works in two stages:

  • A video-conditioned flow-matching model (MMDiT on frozen SAM 3D Body features plus camera rotations) is pretrained on about 3,000 hours of synthetic video, rendered from about 650 hours of motion assets.
  • The model is post-trained with MixGRPO, using two rewards: a fidelity reward (joint error against the ground-truth annotation) and a tracking reward from a fixed PHC+ controller in Isaac Gym.

The authors also introduce Wild-4K, a set of 4,182 internet clips without ground truth. They use it for simulation success rate, physical metrics, and blinded pairwise human evaluation. FlowHMR reports higher PHC+ success than GVHMR, PromptHMR, GENMO, and DuoMo (82.5% vs. at most 62.8%) and wins most human comparisons.

Critical strengths
  • Using both reconstruction (Recon.) and simulation tracking (Sim.) is intuitive and well motivated. It keeps the motion aligned to the video while making it physically trackable. The reward ablation in Table 3 shows this clearly.
  • Using GRPO-style post-training with physics feedback to improve video pose estimation is an interesting direction for the HMR community.
  • Wild-4K could be a useful benchmark for physical plausibility on in-the-wild video.
  • The appendix ablations are thorough: SFT vs. GRPO, the kinematic proxy reward, and sampling variability.
Critical weaknesses
  1. Novelty is limited. Optimizing a motion generator with GRPO using simulator-tracking plus alignment rewards has already been done for text-to-motion (RLPF, Yue et al. 2025; PhysMoDPO), and GRPO has been applied to motion recovery before (MotionGRPO). The main change here is moving from text conditioning to video conditioning, with a ground-truth fidelity reward. I still like the idea, but the methodological contribution is incremental.

  2. Most of the gain comes from pretraining, not RL, and the comparisons aren’t fair. Without RL, the model already reaches 78.8% PHC+ success, versus at most 62.8% for the baselines. Post-training adds only 3.7 points. In the human evaluation against Ours w/o RL, 59% of judgments are ties (26% wins, 15% losses).

    • So the headline gains over prior work come mainly from the pretraining setup: 3,000 hours of (likely proprietary) data, a strong frozen SAM 3D Body encoder, and a 0.5B MMDiT. None of these are shared with the baselines.
    • The paper should explain why the generator is so much stronger than GENMO or GVHMR even before RL. Methods should also be compared on the same training data, for example by training FlowHMR on BEDLAM/BEDLAM2 only, or by retraining one baseline on the same data.
    • Otherwise, it isn’t possible to tell how much comes from the method and how much from the data and the encoder.
  3. Post-training is never applied to other models. GENMO and DuoMo are both diffusion-based, and the proposed RL stage should transfer to them directly. Applying it to at least one of them would separate the effect of post-training from the closed-source training data. I would expect post-training to help these weaker models even more, which would strengthen the paper’s claim.

  4. The motion capture data is unspecified. Appendix A.1 mentions only “motion capture collections and professional animation assets.” Which datasets are these? Are they public? The Studio evaluation set is also internal. This hurts reproducibility and makes the data-scale ablation (200h vs. 3,000h) hard to interpret.

  5. Standard benchmarks are missing. RICH appears only in the appendix, and on RICH, DuoMo is better on WA-MPJPE, W-MPJPE, and RTE. 3DPW and EMDB (especially EMDB-2 for global trajectories) are not reported. I understand these datasets aren’t enough for a full physics evaluation, but they should be included for completeness and for comparison with the literature.

  6. The world-frame representation is under-explained. The motion representation follows Kimodo, which assumes canonicalized motion (gravity-aligned and heading-normalized). Here, Appendix B.1 defines the sequence frame by the initial camera orientation. Several questions follow:

    • How is gravity alignment obtained when the first camera has pitch or roll?
    • Where do the relative camera rotations come from at test time (VGGT-Ω?), and how sensitive is the model to errors in them?
    • How is the ground plane defined for simulation (“upright-start alignment”)?
  7. The same controller is used for reward and evaluation. PHC+ provides the training reward and the main evaluation metric, so the model may overfit to that specific controller. The SONIC results in D.4 help, but they use a 615-clip subset and don’t include Ours w/o RL. Please report the RL vs. no-RL comparison under SONIC.

  8. The evaluation protocol has weaknesses.

    • Each human-evaluation pair is judged by a single annotator, about 130 pairs per baseline, with different videos for each baseline and no inter-annotator agreement reported.
    • The physical metrics are medians of sub-millimetre values (for example, penetration of 0.60 vs. 0.06 mm), with a 5 mm tolerance. Means or high percentiles would be more informative.
    • Wild-4K has no ground truth, so it can’t measure reconstruction accuracy.
  9. The fidelity reward needs ground truth. RL post-training therefore runs only on synthetic, annotated data, not on the in-the-wild videos the method targets. Have the authors considered a 2D reprojection or keypoint fidelity reward, which would allow post-training on unlabeled real video?

Questions for authors

Will Wild-4K, including the VGGT-Ω camera estimates and evaluation scripts, be released? Which motion capture and animation sources make up the 650 hours? Are any of them public? How do baselines perform when trained on the same data, and how does FlowHMR perform when trained only on public data (e.g., BEDLAM/BEDLAM2 + AMASS)? Does your post-training improve GENMO or DuoMo? How are gravity direction, camera rotations and the ground plane handled at test time (see weakness 6)? What are the 3DPW and EMDB results? How does Ours w/o RL perform under SONIC on the same subset?

Overall, using GRPO with joint fidelity and physics rewards to improve video-based motion capture is interesting and well executed. However, the novelty is limited, and the main gains appear to come from proprietary data and a stronger encoder rather than from the proposed post-training. The experiments need fairer comparisons before the paper’s claims are supported. I am open to raising my score if the authors address data fairness, apply post-training to other models, use standard benchmarks, and provide a world-frame explanation.

6 comments

We thank the reviewer for the detailed and constructive feedback.

Q1: Novelty

Our contributions center on a video motion capture system built on large, diverse data and a complete training pipeline. The RL stage uses an existing optimization algorithm.

1. Large-scale, diverse data and a video motion capture model. We construct approximately 3,000 hours of synthetic video from around 650 hours of source motion, covering diverse motions, cameras, appearances, and occlusions. We develop a video-conditioned Flow Matching model that jointly recovers the body, hands, and global trajectory, achieving better overall motion quality in human evaluation and better simulation tracking performance among the compared methods.

2. A training pipeline that balances reconstruction fidelity and physical executability. We believe that limited data coverage and reconstruction capability may constrain the gains from post-training. When the reconstructed motion deviates substantially from the input video, these errors are difficult to correct through RL using feedback from physical priors. Sufficient pretraining is therefore an important foundation for effective physics-based post-training. We first establish motion recovery capability across diverse scenarios through large-scale data and 3D supervision, then combine reconstruction and simulation tracking rewards for post-training. Table 3 shows that the joint reward achieves better reconstruction and tracking performance than kinematic proxy rewards and SFT on projected targets.

Q2: Pretraining gains and fair comparisons

We agree that strong pretraining is an important source of the overall improvement. Table 2 examines data scale, model size, and generative versus regression formulations. We add the following comparisons to control for training data, image encoder, and model size.

All three motion capture networks are trained from scratch on BEDLAM1+2 without RL, using a global batch size of 128, 20,000 training steps, and approximately 0.2B parameters, and are evaluated on RICH. The training data pipeline for GENMO is based on GVHMR_BEDLAM2. The standard configuration includes additional motion-only data from AMASS. Since FlowHMR is conditioned on image features and does not use 2D keypoints as inputs or for supervision, we remove this additional motion-only data from all three configurations.

MethodWA ↓W ↓PA ↓RTE ↓Jitter ↓FS ↓
GENMO (HMR2)141.41241.4047.876.1119.228.96
FlowHMR (HMR2)115.90192.8351.224.9013.518.13
FlowHMR (SAM 3D Body)101.46163.8746.903.2417.987.98

WA, W, and PA denote WA-MPJPE, W-MPJPE, and PA-MPJPE, respectively. These metrics and FS are measured in mm; RTE is reported in %.

With the same training data and HMR2 encoder, FlowHMR reduces WA, W, and RTE by 18.0%, 20.1%, and 19.8% relative to GENMO. Its advantage in global motion recovery therefore does not rely entirely on private data or the SAM 3D Body encoder.

The human evaluation assesses overall motion quality and consistency with the input video using predictions before simulation (see Q8). Most comparisons between the models before and after RL show no clear overall preference, while preferences for the RL model outnumber those for the pretrained model. This is consistent with our goal of preserving reconstruction quality while improving executability. Tracking success also depends on controller capability: under SONIC, which is not used for the training reward, RL improves success by 20.49 percentage points (see Q7).

Q3: Applying post-training to another model

We add a post-training experiment on GENMO using only the public datasets BEDLAM1, BEDLAM2, and AMASS. Each step uses a global batch of 16 clips, with 8 candidates per clip, giving 128 candidate motions in total. We train for 2,000 steps using a PHC tracking reward and a reconstruction fidelity reward.

MethodWild-4K success rate (%) ↑
GENMO w/o RL58.8
GENMO + RL59.5

The success rate increases by 0.7 percentage points, showing that the post-training scheme can be applied to GENMO and improve tracking on in-the-wild videos using public training data. We observe that some tracking failures are accompanied by substantial motion reconstruction errors.

We believe that limited data coverage and reconstruction capability may constrain the gains from post-training. When the reconstructed motion deviates substantially from the input video, these errors are difficult to correct through RL using feedback from physical priors. Sufficient pretraining is therefore an important foundation for effective physics-based post-training.

Q4: Data sources and reproducibility

The approximately 650 hours of source motion come primarily from motion assets accumulated internally over time. We render these assets to construct approximately 3,000 hours of paired video and motion data. The 200-hour setting in Table 2 is randomly sampled from these 3,000 hours. Studio is an internal optical motion capture dataset used primarily for model development and ablation studies.

We will release the source video links and clip time ranges for Wild-4K, along with VGGT-Ω camera estimates, image features, and evaluation scripts. We will not directly redistribute the original internet videos because we do not hold redistribution rights.

Q5: Public benchmarks and reconstruction accuracy

We will move the full RICH comparison into the main paper and add the EMDB-2 evaluation to the supplementary material.

EMDB-2 and training data coverage. EMDB-2 is a standard benchmark for global motion recovery with moving cameras. It includes motions involving non-level terrain or vehicles, such as walking on slopes and skateboarding. These scenarios are not covered by our training data and incur larger reconstruction errors. The following table compares default VGGT-Ω inference on the full set and a subset excluding these scenarios. WA and W are measured in mm, and RTE in %.

Evaluation setWA ↓W ↓RTE ↓
Full EMDB-297.82233.313.60
Subset excluding motions on non-level terrain or vehicles65.28179.991.81

We also explore camera estimation and PnP alignment to handle these scenarios. We replace the input camera rotation R and use the corresponding camera, detected keypoints, and predicted body motion for PnP alignment. However, this approach depends on accurate camera estimation: failures in camera translation estimation can cause the recovered human trajectory to drift. We therefore do not use it as our default inference procedure. Our default uses only VGGT-Ω rotations as camera conditions and directly predicts motion in world coordinates. The following results compare camera configurations on the full EMDB-2 set.

Method / SettingWA ↓W ↓RTE ↓
GVHMR111.0276.52.0
GENMO74.3202.11.2
DuoMo66.0167.11.1
FlowHMR, default VGGT-Ω97.82233.313.60
FlowHMR, VIPE + PnP alignment62.12160.121.97
FlowHMR, DROID-SLAM + PnP alignment69.90187.992.20
FlowHMR, GT camera + PnP alignment54.19136.990.68

3DPW and motion representation. FlowHMR directly predicts motion in world coordinates and does not directly output camera-space motion, so camera-space evaluation on 3DPW requires additional alignment. We tested a GVHMR-style representation that estimates camera-space motion and camera parameters, making it straightforward to overlay the motion on the input image and compute metrics such as MPJPE. Both FlowHMR variants below are evaluated before RL; SR denotes PHC+ tracking success rate.

Method / RepresentationRICH WA ↓RICH W ↓Wild-4K SR (%) ↑
DuoMo52.8678.2856.71
FlowHMR, paper representation (w/o RL)55.7088.3778.79
FlowHMR, GVHMR-style (w/o RL)49.179.071.02

The GVHMR-style variant has lower RICH WA and W than the pretrained model in the paper, and its WA is also better than DuoMo's. However, its Wild-4K tracking success is lower, its motions are less stable, and its overall visual quality is worse than that of the paper version. We therefore adopt the current world-coordinate representation and emphasize global motion metrics and human evaluation.

Q6: World coordinates, camera conditioning, and simulation alignment

Coordinates and gravity. The training frame retains the annotated world's up axis, with heading defined by the horizontal projection of the initial camera's viewing direction. Camera pitch/roll therefore do not tilt the frame. We remove only the initial horizontal translation and retain body orientation.

Camera conditions. Training rotations are relative to the first frame; default test rotations come from VGGT-Ω. Q5 compares different input rotations and corresponding PnP alignment cameras on EMDB-2.

Simulation alignment. For simulation on horizontal ground, we convert Y-up motions to Z-up and apply PHC's fixed skeleton rotation (upright-start alignment). The reference sequence receives one vertical offset based on the lowest mesh point over its first 30 frames; PHC applies its standard height adjustment at initialization.

We will release the coordinate processing, inference, and simulation evaluation code.

Q7: RL gains under an independent controller

We add FlowHMR w/o RL to the SONIC evaluation in the original Table D.3. SONIC is not used to compute the training reward, and this evaluation uses the original subset of 615 clips.

MethodSuccess rate (%) ↑
PromptHMR44.07
GENMO48.78
DuoMo51.06
GVHMR54.15
FlowHMR w/o RL57.23
FlowHMR (ours)77.72

RL increases success from 57.23% to 77.72%, a gain of 20.49 percentage points. On this subset, the tracking improvements therefore transfer to a controller that was not used for training. We will include the SONIC results and the comparison before and after RL in the revised main paper and update Table D.3.

Q8: Human evaluation and physical metrics

Human evaluation. We sample comparison pairs without replacement from a shared pool of randomly selected Wild-4K videos. Annotators view the same input video and two predictions before simulation, with hidden method identities, randomized order, and win/tie/loss judgments. Wild-300 separately evaluates pretrained models through independent blind ratings by at least five professional animation artists following Table B.2. RICH and EMDB-2 provide evaluation with 3D ground truth.

Physical metrics. We follow PhysDiff, including the 5 mm tolerance for penetration and floating. We report the median of per-sequence mean errors, so aggregated values can be below 5 mm.

Q9: 3D supervision and post-training on real videos

Our final model uses RL on synthetic data with 3D ground truth, without 2D keypoint supervision, to balance reconstruction fidelity, temporal stability, and physical executability. Two observations motivate this choice.

1. Fidelity constraints are necessary. Sim. only (Table 3) reaches 99.62% success but tends to generate stationary motions that deviate from the input video.

2. 2D supervision introduced temporal and alignment issues. As discussed in Q5, our model predicts motion directly in world coordinates, so additional camera alignment is needed before projection onto the input video. In our early experiments, we explored RL on 20,000 internet videos using Sapiens2-5B keypoints and PnP alignment with estimated cameras and predicted 3D motion. PCK improved, but temporal stability decreased, and some sequences showed misalignment and body tilt after PnP alignment.

We agree that post-training on real-world videos is a promising direction for future work, particularly given the cost and coverage limitations of captured and synthetic motion data.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We sincerely thank all reviewers for their valuable feedback. During the rebuttal, we conducted several additional experiments to address the main concerns:

  1. Fair comparisons. We conducted controlled comparisons with GENMO and further trained FlowHMR for 500 epochs using the same public datasets and closely aligned training settings. FlowHMR achieves better global motion accuracy, lower foot sliding, and lower jitter than the released GENMO baseline under the same local evaluation protocol.

  2. RL post-training. We applied our RL strategy to GENMO and observed improved tracking performance. We also clarified the body-shape retargeting procedure used during RL.

  3. Additional evaluations. We added results on RICH and EMDB-2, examined different camera and motion representations, and clarified the evaluation protocols.

  4. Physical plausibility. We evaluated our method using the independent SONIC controller, where RL improves tracking success by 20.49 percentage points on the 615-clip subset. We also analyzed reconstruction accuracy and motion characteristics before and after RL.

We are grateful that Reviewer dEHv raised their rating to Weak Accept after reviewing our additional experiments, while Reviewer QM38 maintained their positive assessment. We also thank Reviewer iD6v for their detailed follow-up questions, which motivated us to conduct more rigorous comparisons with GENMO and complete the additional 500-epoch training experiment.

We thank all reviewers again for their constructive suggestions, which have helped us improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.