- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents DreamHMR, a framework for recovering complete SMPL-X body-and-hand motion from videos, with a particular focus on partial visibility and occlusion. The core idea is to repurpose representations from pretrained video-generation models for explicit human geometry through a global-to-local pipeline. The method is instantiated with four video backbones and evaluated on standard HMR benchmarks as well as held-out BEDLAM/EgoBody partial-observation settings. On 3DPW, EMDB-1, and RICH, the Wan-5B configuration obtains the lowest reported errors across all nine metrics in Table 1.
Critical strengths
- DreamHMR demonstrates strong results across several dimensions rather than relying on a single benchmark. On 3DPW, EMDB-1, and RICH, Wan-5B is best on all reported PA-MPJPE, MPJPE, and PVE metrics, while smaller DreamHMR configurations are also competitive.
- In Table 3, replacing RGB supervision with combined canonical coloring and shading reduces occlusion body/hand MPJPE from 120.0/68.4 to 106.3/57.8 mm. Likewise, Table 4 shows a relatively clean progression from a flat-query decoder to structured part/joint states, local sampling, evidence gating, and explicit evidence supervision. These experiments make the paper considerably more convincing than a result in which gains could only be attributed to a larger video backbone.
Critical weaknesses
- The dedicated “occlusion” evaluation primarily measures artificial partial observation rather than natural occluder interference The method is motivated partly by the problem that local features can correspond to foreground occluders or background rather than the target person, and the evidence-gating mechanism is designed to suppress such unreliable local evidence. This is a reasonable test of partial visibility, and the results are strong. But it is not equivalent to the more difficult situation emphasized in the motivation, where the target joint is spatially close to an actual foreground object, another person, or confusing background texture. In the artificial-mask setting, a large fraction of the problem is determining whether an anatomical region is inside or outside an explicitly defined observation region.
- The central benefit of video-generation pretraining is not fully isolated at convergence: does video-generation pretraining produce a fundamentally better representation for HMR, or primarily faster optimization/sample efficiency? The main DreamHMR training schedule is substantially longer and includes 10,000 geometry-adaptation steps followed by 47,500 parameter-recovery steps. Without full-budget random-vs-pretrained curves or a converged matched control, the strongest interpretation of the paper's central premise is not yet established.
Questions for authors
- For Wan-1.3B and/or Wan-5B, what happens if the randomly initialized and video-pretrained models are trained to the full parameter-recovery budget or to convergence, rather than being compared after 10,000 task-training steps?
- Can the authors report the evidence-gating ablation on a subset containing natural foreground occlusion, for example RICH/EgoBody frames stratified by visible-body fraction or by overlap with scene/foreground occluders?
4 comments
Dear Reviewer N76j,
Thank you again for your thoughtful review. We extended both initialization runs to 57,500 steps to answer your question about longer training, and tested evidence gating under natural occlusion.
As the discussion period draws to a close, we would appreciate hearing whether our response and additional results address your concerns. If any questions remain, we would be happy to clarify them.
Best regards,
The authors
Thanks for authors' detailed responses. My main concerns have been addressed and I also read other reviewers' comments. I hope authors will open source in future. So I will rise my score to '3 Weak acceptance'.
Dear Reviewer N76j,
Thank you for your positive assessment and for the constructive feedback throughout the discussion. We are glad that the full-budget initialization comparison and the natural-occlusion evaluation of evidence gating address your concerns.
We also appreciate your interest in our code. We plan to release the code and pretrained models publicly.
Best regards, The authors
Response to Reviewer N76j
We thank the reviewer for the constructive feedback and address the concerns below.
W1 / Q2: Evidence gating under natural occlusion
Thank you for highlighting the importance of evaluating evidence gating under natural foreground occlusion. We have added an comparison on 3DPW-OCC using natural occluders, and evidence gating reduces body MPJPE, PA-MPJPE and PVE for both backbones. These improvements show that our methods do not overfit to specific occlusion settings, but achieves generalization, which is brought about by the video generation backbone and evidence gating.
This evaluation retains real occluders in the input images and applies no synthetic masks. Comparing ungated and evidence-gated variants in this setting directly tests whether weighting local updates improves reconstruction when the observed RGB contains natural occlusion.
3DPW-OCC body reconstruction (mm, ↓).
Evidence gating reduces MPJPE from 57.91 to 54.72 mm for Wan-1.3B and from 55.04 to 53.65 mm for Wan-5B. The complete Wan-5B model also achieves the lowest error across all three metrics among the compared methods.
We will distinguish the constructed partial-observation setting from natural-occlusion evaluation in the revision, add these results to Section 4.3, and document the evaluation protocol in Appendix B.
W2 / Q1: Initialization at the full budget
Thank you for asking whether the pretraining advantage persists with longer task training; we have completed the paired initialization experiment through the full 57.5k-step budget. Video-pretrained Wan-1.3B has 1.87/2.22 mm lower MPJPE on 3DPW/3DPW-OCC at the final checkpoint and lower error at every intermediate checkpoint. It also surpasses the random run's full-budget accuracy by 30k steps.
Comparison between random and pretrained initialization under full budget. Both Wan-1.3B runs use the same architecture and training recipe, and only the DiT initialization differs (Random versus video Pretrained). We evaluate body MPJPE (mm, ↓) on 3DPW and 3DPW-OCC at checkpoints fixed in advance at 20k, 30k, 40k, and 57.5k total task-training steps. In the table, Δ = Pretrained − Random, so negative values favor pretraining; † marks pretrained checkpoints that already outperform the random run at the full budget.
The experimental results support the following conclusion.
Video-generation pretraining therefore improves both the optimization path and the representation for HMR. The revision adds this full-budget comparison to Appendix D and references it from Section 4.3.