- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper combines video-conditioned flow matching with physics-based RL to recover body and hand motion and global trajectory from monocular video. It also introduces Wild-4K, a large and diverse evaluation set for assessing motion recovery quality and simulation tracking performance. The experiments show that it outperforms state-of-the-art methods.
Critical strengths
- The paper is clearly written and well organized, making the method and experiments easy to follow.
- The proposed framework combines video-conditioned flow matching with physics-based post-training, jointly modeling body pose, hand articulation, and global trajectories in world coordinates for motion recovery from monocular video. And it outperforms state-of-the-art methods in overall motion quality as judged by human evaluators.
- The introduced physics-based reinforcement learning approach balances executability and reconstruction fidelity. The ablations show why optimizing tracking alone is insufficient.
- Wild-4K provides a useful evaluation set covering diverse in-the-wild motions, including challenging actions and occlusions.
Critical weaknesses
- Table 1 uses the proposed Wild-4K dataset, and Table 2 uses the internal Studio dataset. I suggest moving the full baseline comparison on RICH into the main paper, as it provides important evidence of reconstruction accuracy on a public benchmark. Additional results on EMDB would further strengthen the evaluation.
- Table 2 reports RTE and Jitter in mm and mm/s², whereas Table D.2 uses % and m/s³. The authors should clarify whether these differences reflect distinct definitions or labeling errors.
- Although joint body and hand recovery is a stated contribution, hand-specific quantitative results mainly come from the internal Studio dataset. Could the authors report body and hand reconstruction errors separately on RICH, if its hand annotations support reliable evaluation?
Questions for authors
See the weaknesses. I would be willing to raise my score if the authors address most of my concerns.
4 comments
Follow-up on Our Response to Reviewer dEHv
Dear Reviewer dEHv,
Thank you again for your thoughtful feedback. We have posted a response to your comments with additional clarifications and experimental results, and we have updated the manuscript in light of the reviewers’ feedback.
With the discussion deadline approaching, we would appreciate it if you could take a look at our response and let us know if any concerns remain. We would be glad to address any follow-up questions before the deadline. If the new evidence and clarifications address your concerns, we would be grateful if you would revisit your overall assessment in light of the revision.
Thank you for your valuable time and consideration!
The authors' response has addressed most of my concerns. I have decided to raise my score to 3.
We sincerely thank the reviewer for the thoughtful questions and valuable feedback, which have been very helpful in improving our paper.
We thank the reviewer for the constructive feedback.
Q1: RICH and EMDB evaluation
We will move the full RICH comparison into the main paper and add the EMDB-2 evaluation to the supplementary material.
EMDB-2 and training data coverage. EMDB-2 is a standard benchmark for global motion recovery with moving cameras. It includes motions involving non-level terrain or vehicles, such as walking on slopes and skateboarding. These scenarios are not covered by our training data and incur larger reconstruction errors. The following table compares default VGGT-Ω inference on the full set and a subset excluding these scenarios. WA and W are measured in mm, and RTE in %.
We also explore camera estimation and PnP alignment to handle these scenarios. We replace the input camera rotation R and use the corresponding camera, detected keypoints, and predicted body motion for PnP alignment. However, this approach depends on accurate camera estimation: failures in camera translation estimation can cause the recovered human trajectory to drift. We therefore do not use it as our default inference procedure. Our default uses only VGGT-Ω rotations as camera conditions and directly predicts motion in world coordinates. The following results compare camera configurations on the full EMDB-2 set.
3DPW and motion representation. FlowHMR directly predicts motion in world coordinates and does not directly output camera-space motion, so camera-space evaluation on 3DPW requires additional alignment. We tested a GVHMR-style representation that estimates camera-space motion and camera parameters, making it straightforward to overlay the motion on the input image and compute metrics such as MPJPE. Both FlowHMR variants below are evaluated before RL; SR denotes PHC+ tracking success rate.
The GVHMR-style variant has lower RICH WA and W than the pretrained model in the paper, and its WA is also better than DuoMo's. However, its Wild-4K tracking success is lower, its motions are less stable, and its overall visual quality is worse than that of the paper version. We therefore adopt the current world-coordinate representation and emphasize global motion metrics and human evaluation.
Q2: Units of RTE and Jitter
Thank you for pointing this out. RTE divides the aligned root trajectory error by the total ground-truth path length and multiplies the result by 100, so it is reported in %. Jitter measures the third temporal derivative of joint positions (jerk), computed in m/s³ and divided by 10 for reporting. Keeping this numerical scale, the column should be labeled Jitter (10 m/s³). We will correct the unit labels and clarify the reporting scale in Table 2, Table D.2, and the related tables.
Q3: Separate body and hand evaluation
We follow GVHMR's body24 protocol for RICH. The MPJPE metrics in Table D.2 therefore evaluate body joints and exclude fingers, enabling consistent comparison with other global motion capture methods.
Our model retains finger joints during training and prediction. Figure B.1 separately evaluates finger quality on Wild-300, where our pretrained model achieves the highest mean finger rating among the compared methods (3.55/5). Tables 2 and B.1 also separately report hand rotation errors (MPJRE_hand) on Studio. We will clarify the scope of these evaluations in the revision.