acceptodds
Under review as a conference paper at ICLR 2027

DreamHMR: From Video Generation to Human Motion Modeling via Structural Focusing

32 views

Abstract

Human mesh recovery (HMR) provides a geometric foundation for understanding human motion, but accurately reconstructing both body motion and fine hand articulation from videos remains challenging, especially in real-world scenes where occlusion and partial visibility are common. Video generation pretraining offers rich spatial and temporal priors for this task, but its focus on visual synthesis leaves a gap between these representations and explicit human geometry. We present DreamHMR, a general framework that adapts video generation priors for complete body-and-hand reconstruction. It follows a global-to-local strategy, first grounding video representations in human geometry and then refining body and hand detail with reliable local evidence. Global Body Representation Transfer trains the video backbone to predict complete human renderings, turning its native prediction task into a means of learning surface correspondence and posed geometry. Local Joint Feature Enhancement combines contextual estimates with reliable local evidence to recover fine body and hand detail, while limiting interference from occluders and background. Experiments on five datasets with four video backbones demonstrate strong reconstruction performance across backbone families. Under a unified evaluation protocol, our best model outperforms the best existing methods on all reported metrics of 3DPW, EMDB-1, and RICH, reducing body MPJPE by 4.2%, 4.7%, and 6.1%, respectively. In more challenging occlusion settings on held-out BEDLAM and EgoBody sequences, DreamHMR reduces overall body and hand MPJPE by 7.0% and 10.0%, respectively. Our code will be made publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human Mesh Recovery, Whole-Body Human Modeling, Video Generation, SMPL-X

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_N76jOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents DreamHMR, a framework for recovering complete SMPL-X body-and-hand motion from videos, with a particular focus on partial visibility and occlusion. The core idea is to repurpose representations from pretrained video-generation models for explicit human geometry through a global-to-local pipeline. The method is instantiated with four video backbones and evaluated on standard HMR benchmarks as well as held-out BEDLAM/EgoBody partial-observation settings. On 3DPW, EMDB-1, and RICH, the Wan-5B configuration obtains the lowest reported errors across all nine metrics in Table 1.

Critical strengths
  1. DreamHMR demonstrates strong results across several dimensions rather than relying on a single benchmark. On 3DPW, EMDB-1, and RICH, Wan-5B is best on all reported PA-MPJPE, MPJPE, and PVE metrics, while smaller DreamHMR configurations are also competitive.
  2. In Table 3, replacing RGB supervision with combined canonical coloring and shading reduces occlusion body/hand MPJPE from 120.0/68.4 to 106.3/57.8 mm. Likewise, Table 4 shows a relatively clean progression from a flat-query decoder to structured part/joint states, local sampling, evidence gating, and explicit evidence supervision. These experiments make the paper considerably more convincing than a result in which gains could only be attributed to a larger video backbone.
Critical weaknesses
  1. The dedicated “occlusion” evaluation primarily measures artificial partial observation rather than natural occluder interference The method is motivated partly by the problem that local features can correspond to foreground occluders or background rather than the target person, and the evidence-gating mechanism is designed to suppress such unreliable local evidence. This is a reasonable test of partial visibility, and the results are strong. But it is not equivalent to the more difficult situation emphasized in the motivation, where the target joint is spatially close to an actual foreground object, another person, or confusing background texture. In the artificial-mask setting, a large fraction of the problem is determining whether an anatomical region is inside or outside an explicitly defined observation region.
  2. The central benefit of video-generation pretraining is not fully isolated at convergence: does video-generation pretraining produce a fundamentally better representation for HMR, or primarily faster optimization/sample efficiency? The main DreamHMR training schedule is substantially longer and includes 10,000 geometry-adaptation steps followed by 47,500 parameter-recovery steps. Without full-budget random-vs-pretrained curves or a converged matched control, the strongest interpretation of the paper's central premise is not yet established.
Questions for authors
  1. For Wan-1.3B and/or Wan-5B, what happens if the randomly initialized and video-pretrained models are trained to the full parameter-recovery budget or to convergence, rather than being compared after 10,000 task-training steps?
  2. Can the authors report the evidence-gating ablation on a subset containing natural foreground occlusion, for example RICH/EgoBody frames stratified by visible-body fraction or by overlap with scene/foreground occluders?
4 comments
AuthorsOpenReview ↗

Response to Reviewer N76j

We thank the reviewer for the constructive feedback and address the concerns below.

W1 / Q2: Evidence gating under natural occlusion

Thank you for highlighting the importance of evaluating evidence gating under natural foreground occlusion. We have added an comparison on 3DPW-OCC using natural occluders, and evidence gating reduces body MPJPE, PA-MPJPE and PVE for both backbones. These improvements show that our methods do not overfit to specific occlusion settings, but achieves generalization, which is brought about by the video generation backbone and evidence gating.

This evaluation retains real occluders in the input images and applies no synthetic masks. Comparing ungated and evidence-gated variants in this setting directly tests whether weighting local updates improves reconstruction when the observed RGB contains natural occlusion.

3DPW-OCC body reconstruction (mm, ↓).

MethodPA-MPJPEMPJPEPVE
PromptHMR-Vid41.7363.2872.05
GENMO36.4156.8770.34
SMPLest-X35.1658.4269.58
3DB-DINOv333.8760.1466.29
Hand4Whole++39.5261.0874.63
DreamHMR / Wan-1.3B, ungated34.2657.9168.47
DreamHMR / Wan-1.3B, evidence-gated33.5854.7266.83
DreamHMR / Wan-5B, ungated33.7255.0465.39
DreamHMR / Wan-5B, evidence-gated31.4753.6562.94

Evidence gating reduces MPJPE from 57.91 to 54.72 mm for Wan-1.3B and from 55.04 to 53.65 mm for Wan-5B. The complete Wan-5B model also achieves the lowest error across all three metrics among the compared methods.

We will distinguish the constructed partial-observation setting from natural-occlusion evaluation in the revision, add these results to Section 4.3, and document the evaluation protocol in Appendix B.

W2 / Q1: Initialization at the full budget

Thank you for asking whether the pretraining advantage persists with longer task training; we have completed the paired initialization experiment through the full 57.5k-step budget. Video-pretrained Wan-1.3B has 1.87/2.22 mm lower MPJPE on 3DPW/3DPW-OCC at the final checkpoint and lower error at every intermediate checkpoint. It also surpasses the random run's full-budget accuracy by 30k steps.

Comparison between random and pretrained initialization under full budget. Both Wan-1.3B runs use the same architecture and training recipe, and only the DiT initialization differs (Random versus video Pretrained). We evaluate body MPJPE (mm, ↓) on 3DPW and 3DPW-OCC at checkpoints fixed in advance at 20k, 30k, 40k, and 57.5k total task-training steps. In the table, Δ = Pretrained − Random, so negative values favor pretraining; † marks pretrained checkpoints that already outperform the random run at the full budget.

Task-training stepsRandom (3DPW)Pretrained (3DPW)ΔRandom (3DPW-OCC)Pretrained (3DPW-OCC)Δ
20k57.6254.83−2.7961.2257.84−3.38
30k56.1153.71 †−2.4059.3356.25 †−3.08
40k55.4953.39−2.1058.0255.63−2.39
full (57.5k)54.9553.08−1.8756.9454.72−2.22

The experimental results support the following conclusion.

  • Faster convergence. At 30k task-training steps, the pretrained run (53.71 / 56.25) is already better than the random run at 57.5k (54.95 / 56.94), so pretraining saves 27.5k steps in addition to lowering the error at the terminal checkpoint.
  • Lower error at every checkpoint. The pretrained run is ahead at all four checkpoints on both inputs, so the advantage is not a transient of early training.
  • The gap narrows but does not close. 2.79 → 1.87 mm on 3DPW and 3.38 → 2.22 mm on 3DPW-OCC. The difference at 57.5k steps is the part of the gain that does not come from more training, and it is larger under occlusion.

Video-generation pretraining therefore improves both the optimization path and the representation for HMR. The revision adds this full-budget comparison to Appendix D and references it from Section 4.3.

Dear Reviewer N76j,

Thank you again for your thoughtful review. We extended both initialization runs to 57,500 steps to answer your question about longer training, and tested evidence gating under natural occlusion.

As the discussion period draws to a close, we would appreciate hearing whether our response and additional results address your concerns. If any questions remain, we would be happy to clarify them.

Best regards,
The authors

Thanks for authors' detailed responses. My main concerns have been addressed and I also read other reviewers' comments. I hope authors will open source in future. So I will rise my score to '3 Weak acceptance'.

Dear Reviewer N76j,

Thank you for your positive assessment and for the constructive feedback throughout the discussion. We are glad that the full-budget initialization comparison and the natural-occlusion evaluation of evidence gating address your concerns.

We also appreciate your interest in our code. We plan to release the code and pretrained models publicly.

Best regards, The authors

Reviewer_drr2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a framework that uses video-generation priors for body-and-hand reconstruction from monocular videos. It first adapts the video backbone using SMPL-X renderings with shaded canonical surface colors, then uses a decoder with gated local feature enhancement to predict SMPL-X parameters.

Critical strengths

I generally like the idea of adapting video-generation representations to human reconstruction. The motivation for combining global context with local evidence is clear, and the figures help explain the pipeline. The results across different benchmarks and backbones are encouraging. I also appreciate the ablation studies, which answer some of my questions about the rendering targets and the local enhancement mechanism.

Critical weaknesses
  1. I would like more insight into the choice of shaded canonical coloring. Canonical colors provide surface correspondence, and shading adds some normal information, so the design makes sense. Keeping the rendered target aligned with the input image also seems reasonable for transferring video-generation representations. However, it is less clear why this representation is preferable to other camera-aligned geometric targets, such as depth, normals, or posed XYZ maps. Table 3 supports combining shading and canonical coloring over the tested variants, but does not address these alternatives. Further discussion or a controlled comparison would help explain what geometric information this representation encourages the backbone to learn.

  2. The temporal benefits of video-generation pretraining are not sufficiently demonstrated. One benefit I would expect from video pretraining is better modeling of temporal information, leading to more accurate and consistent motion estimates. However, the evaluation only reports per-frame metrics. Table 5 shows that neighboring frames help reconstruct the center frame, but it does not measure the temporal quality of the recovered motion. The DINOv3 comparison in Table 9 is useful, but also focuses on spatial accuracy. I therefore remain unsure whether the benefits of video pretraining extend to motion accuracy and temporal consistency. Temporal metrics alongside the spatial results would make this contribution more convincing.

Questions for authors
  1. What advantages does shaded canonical coloring offer over camera-aligned depth, normal, or posed XYZ supervision? Could the authors provide further justification or a comparison using the same human regions and projected coverage?
  2. Could the authors report acceleration error for both the body and hands? Velocity error would also help assess whether the recovered motion is temporally accurate, beyond achieving low per-frame pose error.
  3. For long videos, are predictions consistent when switching between inference windows? An analysis around these switching points would help assess the temporal continuity of the final output.
5 comments
AuthorsOpenReview ↗

Response to Reviewer drr2: Part 1/2

We thank the reviewer for the thoughtful comments and respond to each question below.

Q1: Shaded canonical coloring versus depth, normals, and posed XYZ

Thank you for asking about the rendering target. Canonical coloring encodes dense surface correspondence: each template point retains a fixed color across poses, people, and frames. This information is important for video HMR because it links image evidence across frames to the same anatomical surface location as the person moves and the observed region changes. Depth, normals, and posed XYZ describe a point's current depth, orientation, or position, which changes as the body moves, so they cannot encode the persistent person shape. Meanwhile, shading adds local surface orientation through brightness modulation.

Table 3 supports combining these cues: under occlusion, shaded canonical coloring achieves 106.3/57.8 mm body/hand MPJPE, compared with 111.6/60.7 for canonical coloring alone and 109.3/64.8 for shading alone.

The revision will extend this comparison to depth, normal, and posed XYZ targets trained and evaluated under the same settings.

Q2: Body and hand acceleration errors

Thank you for highlighting motion accuracy in HMR tasks. We have added body and hand acceleration-error evaluations. The advantage in single frame extends to cross-frame motion: our model achieves lowest body acceleration error on 3DPW/EMDB-1/RICH**.** Our method also improves over the listed previous methods on the occlusion evaluation holdout.

Acceleration error is computed on the final per-frame output from pelvis-relative body joints and wrist-relative hand joints: the second difference over consecutive valid frames with the original timestamps, without per-frame alignment, averaged within each sequence and then equally across sequences. Units are m/s².

Body acceleration error (m/s², ↓). Two previous methods and two DreamHMR backbones are listed here; the revision adds the acceleration columns to Table 1.

Method3DPWEMDB-1RICH
3DB-DINOv35.915.385.92
Hand4Whole++6.456.715.78
DreamHMR / Wan-1.3B5.335.165.28
DreamHMR / Wan-5B5.215.075.14

Body and hand acceleration error (m/s², ↓) on the occlusion evaluation holdout. Each cell reports Full / Occlusion. Body joints are pelvis-relative and hand joints wrist-relative.

MethodBEDLAM body (Full / Occl.)BEDLAM hands (Full / Occl.)EgoBody body (Full / Occl.)EgoBody hands (Full / Occl.)
3DB-DINOv36.13 / 7.986.56 / 8.266.58 / 9.066.71 / 9.58
SMPLest-X7.23 / 7.896.09 / 8.717.55 / 8.857.06 / 9.56
DreamHMR / Wan-1.3B4.52 / 7.735.10 / 6.886.28 / 7.666.03 / 8.30
DreamHMR / Wan-5B4.26 / 7.204.97 / 6.915.49 / 7.305.82 / 8.23
AuthorsOpenReview ↗

Response to Reviewer drr2: Part 2/2

Q3: Consistency across inference windows

Thank you for raising questions about consistency across inference windows. We have added a direct comparison of predictions for the same frames from adjacent windows. On 3DPW, the mean cross-window body-joint difference is only 1.75 mm over all overlapping frames and 1.59 mm at window switches, supporting continuity of the predicted body pose across windows.

Long videos are processed in 81-frame windows with centers spaced 40 frames apart, so regular adjacent windows share 41 frames. Each real frame uses the prediction from its nearest window center, with ties assigned to the earlier center.

We evaluate Wan-1.3B on the 3DPW official test set by comparing the two windows' raw predictions for each shared frame. We report both all overlapping frames and the two frames surrounding each actual switch. The cross-window distance D directly measures disagreement between predictions; E is the mean error of the two predictions against GT on the same samples and provides a reference scale. Both are averaged within each recording and then equally across recordings.

Evaluation framesCross-window distance D (mm) ↓Error against GT E (mm)D/E ↓Mean recording P95 of cross-window distance (mm) ↓
All overlap frames1.7554.633.2%4.37
Switch frames1.5954.752.9%3.93

The 1.59 mm gap at switching frames is only 2.9% of the matched reconstruction error, and the mean per-recording 95th-percentile gap is 3.93 mm. These small differences are measured before any cross-window blending or additional smoothing. Because both windows predict nearly the same body pose at the same time, changing the source window introduces only a small perturbation to the recovered pose trajectory, supporting continuity across window boundaries.

The revision will include this direct cross-window comparison and its evaluation protocol.

Dear Reviewer drr2,

Thank you again for your thoughtful review. Your questions prompted us to evaluate motion accuracy across five datasets and to check how closely adjacent windows agree on the same frames.

As the discussion period draws to a close, we would appreciate hearing whether our response and additional results address your concerns. If any questions remain, we would be happy to clarify them.

Best regards,
The authors

Reviewer_hKi3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DreamHMR transfers pretrained video-generation representations to video-based body-and-hand mesh recovery. It first adapts a video transformer through conditional flow matching on complete SMPL-X renderings, using canonical surface colors for correspondence and shading for posed geometry. A structured decoder predicts part and joint representations, samples features near predicted joints, and gates local corrections using supervised target-person support scores. Inference recovers SMPL-X parameters in one backbone pass on observed RGB latents. Experiments cover four video backbones, three standard body-reconstruction benchmarks, and constructed partial-observation tests on held-out BEDLAM and EgoBody sequences.

Critical strengths
  1. The main design is supported by informative controls that separate several plausible explanations. Table 3 shows that canonical coloring plus shading improves occlusion body/hand MPJPE from 120.0/68.4 to 106.3/57.8 mm relative to RGB supervision, and outperforms either geometric cue alone. Table 4 separates structured decoding, local sampling, auxiliary support supervision, and use of support scores as gates. Appendix D.1 includes same-architecture random/pretrained Wan comparisons and a DINOv3 reference under a common task-training budget. Table 5 also probes changing temporal RGB evidence. Together these provide a credible case that the contribution goes beyond choosing a large video backbone.

  2. The empirical benefit is consistent across architectures and evaluation sources. Wan-5B improves every reported metric on 3DPW, EMDB-1, and RICH under the stated common protocol, and all four backbone configurations improve overall occlusion body MPJPE over the strongest external result. The best model improves overall body/hand errors by 8.1/6.1 mm, including improvements on real EgoBody interactions. The appendices explain the data mixture, alignment, observation construction, and aggregation, while the supplement contains substantive implementations of the renderer, structured decoder, training stages, and backbone adapters. This breadth supports the usefulness of the proposed transfer strategy.

Critical weaknesses
  1. The strongest occlusion claim has a narrower scope than arbitrary natural occlusion, and cross-method attribution remains partly confounded. The Table 2 observations are rectangles derived from complete ground-truth projection boxes, with neutral-gray pixels outside them (Appendix B). DreamHMR trains on this full/static/dynamic observation family, while external comparisons use released checkpoints. Shared RGB and tracks control inference inputs, but not adaptation to the masking distribution. Appendix D.1 helps through matched task-training controls, although these use 10,000 steps rather than the full task budget. The original-RGB public benchmarks are valuable evidence of broader usefulness; the entire margin in Table 2 should nevertheless be interpreted as a result for the specified partial-observation setting.

  2. Prediction coverage and the custom evaluation are not sufficiently auditable from the supplied assets. In ParkingLot2_009_impro2-cam05, Wan-5B has a displayed mesh at 12 s but none at sampled 17, 22, 27, and 32 s while the person remains visible; the other backbones also lack overlays at 27 and 32 s. This may be a visualization or packaging issue, and does not establish model failure. However, the paper does not specify how tracking failures or low predicted presence affect scoring denominators. The code README leaves benchmark evaluation, split inventories, joint regressors, and topology mappings external. These omissions prevent the supplement from resolving an ambiguity that could materially affect accuracy under incomplete observations.

Questions for authors
  1. What explains the absent overlays in the supplied ParkingLot2 clip? Are predictions suppressed by tracking or presence decisions, or is this only a visualization issue? For Tables 1 and 2, are all ground-truth-valid target frames scored independently of predicted presence and tracking success, and how are failures penalized? Please identify the exact evaluation mappings/scripts and provide coverage counts or an equivalent account of the denominators.

  2. Beyond the 10,000-step DINOv3 reference, is there an available strong non-video-generation baseline adapted to the same training mixture and observation augmentations at the full task budget? Such evidence, or clarification of which existing comparisons a

  3. On the occlusion holdout, were the evaluation masks drawn from exactly the same 16-template family used in training? If a natural-occlusion split (e.g., 3DPW-OCC or EgoBody sequences with real, non-templated occluders) were used, do the Table 2 gains hold?

3 comments
AuthorsOpenReview ↗

Response to Reviewer hKi3: Part 1/2

We thank the reviewer for the careful review and address the questions below.

Q1: Overlays and scoring denominators

We sincerely thank the reviewer for pointing out missing overlays, and it is now fixed. All compared methods, including previous methods, share the same tracker outputs and are scored on the same frames; the overlay issue does not affect the reported results. We have corrected the overlays in the supplementary material.

Missing Overlays. In ParkingLot2_009_impro2-cam05, a tracker-ID switch caused the visualization to overlay only the first segment, although model outputs were available for the full clip. The corrected supplementary videos use the final per-frame output and show the mesh on all 521 frames for all four backbones.

Tracking failures and denominators. Tracker failures are excluded from scoring for every method: these frames contribute neither reconstruction error nor a count to the denominator. This exclusion is determined once from the shared tracker outputs, before evaluating any reconstruction method. Among the remaining frames, those with valid ground-truth annotations and target identities form the common evaluation set. Tracker failures therefore do not give any method a different evaluation set or denominator. Each eligible frame is scored once; for our model, its prediction comes from the nearest window center, and predicted presence and evidence scores do not filter predictions or change the denominator. We will state this shared tracking and scoring protocol in Appendix B.

Q2: Matched baselines at the full budget

Thank you for emphasizing the need for full-budget adaptation of strong non-video-generation baselines. We have completed this comparison with DINOv3-H+ and V-JEPA 2, representing image and video self-supervised pretraining. Our methods retain lower errors across all three metrics on both 3DPW and 3DPW-OCC, after these baselines are adapted to the same training mixture at the full task budget.

Matched training data and task budget. All three models use the same data mixture and training budgets. We extend the 10,000-step DINOv3-H+ checkpoint by 47,500 steps and train V-JEPA 2 for 57,500 steps from its pretrained encoder. Both baselines use our structured readout, supervised evidence gating, and parameter-recovery losses, with the backbone and readout jointly trained.

Full-budget body reconstruction (mm, ↓). All configurations are evaluated at the terminal checkpoint using original RGB, shared target tracks, and the same body evaluation protocol.

Backbone3DPW PA-MPJPE3DPW MPJPE3DPW PVE3DPW-OCC PA-MPJPE3DPW-OCC MPJPE3DPW-OCC PVE
DINOv3-H+ (Image SSL)34.1354.3765.1434.3057.6768.29
V-JEPA 2 (Video SSL)33.9753.8263.2133.8656.3767.71
DreamHMR / Wan-1.3B32.4153.0862.3433.5854.7266.83

Wan-1.3B lowers body MPJPE by 1.29/2.95 mm relative to DINOv3-H+ and 0.74/1.65 mm relative to V-JEPA 2 on 3DPW/3DPW-OCC, respectively. Both comparisons show larger MPJPE gains under natural occlusion. These results support the effectiveness of our reconstruction approach. We will add this comparison to Appendix D and summarize it in the experimental section.

AuthorsOpenReview ↗

Response to Reviewer hKi3: Part 2/2

Q3: Natural occlusion

Thank you for pointing out the need to evaluate natural occlusion; we have added experiments on 3DPW-OCC using original RGB with real occluders. Our models improve over the best previous method****s for each metric. Evidence gating also improves both Wan backbones in this setting.

Training and holdout evaluation use the same templates. The new 3DPW-OCC evaluation uses original RGB with real occluders and no synthetic masks. All methods share RGB inputs, target tracks, and the Table 1 body evaluation protocol.

3DPW-OCC body reconstruction (mm, ↓).

MethodPA-MPJPEMPJPEPVE
PromptHMR-Vid41.7363.2872.05
GENMO36.4156.8770.34
SMPLest-X35.1658.4269.58
3DB-DINOv333.8760.1466.29
Hand4Whole++39.5261.0874.63
DreamHMR / Wan-1.3B, ungated34.2657.9168.47
DreamHMR / Wan-1.3B, evidence-gated33.5854.7266.83
DreamHMR / Wan-5B, ungated33.7255.0465.39
DreamHMR / Wan-5B, evidence-gated31.4753.6562.94

Evidence gating lowers MPJPE by 3.19 mm for Wan-1.3B (57.91 → 54.72) and 1.39 mm for Wan-5B (55.04 → 53.65), with PA-MPJPE and PVE also improving. These controls support the benefit of evidence gating under natural occlusion alongside the complete model's advantage over previous methods. We will add these results to Section 4.3 and the evaluation protocol to Appendix B.

Dear Reviewer hKi3,

Thank you again for your thoughtful review. Following your suggestions, we trained both baselines for the full task budget and evaluated reconstruction under natural occlusion.

As the discussion period draws to a close, we would appreciate hearing whether our response and additional results address your concerns. If any questions remain, we would be happy to clarify them.

Best regards,
The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.