acceptodds
Withdrawn from ICLR 2027

WildHSR: Self-Supervised Metric Feed-Forward 4D People-Scene Reconstruction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

3D foundation models recover video cameras and geometry in one forward pass, but some of the strongest are up to scale. Joint people-scene reconstruction then requires two missing outputs: metric scale and persistent person identity. We ask whether one up-to-scale foundation representation can support both through lightweight adaptation. Exact metric labels are scarce, but unlabeled in-the-wild video is abundant. We use people in curated web video to initialise the solution: a posed metric body and 2D keypoints give an approximate, closed-form scale pseudo-label. These pseudo-labels pretrain a Scale Readout, which is then fine-tuned together with a lightweight adapter using exact metric supervision from standard real-video training splits. At inference, the Scale Readout predicts metric scale from foundation-model tokens, without the ruler or its teachers. For person identity, we probe the pretrained foundation model alone and find evidence that its intermediate query-key features encode person correspondence across frames. In most evaluated moving-person clips, a mid-layer token prefers that person over the vacated location and other people. A tiny projection reads this correspondence; together with metric pelvis motion and proposal confidence, it drives dustbin-aware Sinkhorn association of per-frame bodies. WildHSR combines both readouts to reconstruct metric cameras, scene and people from monocular video. On EMDB-2, WildHSR is the first feed-forward method in the published comparison to beat the best optimization-based WA-MPJPE and RTE while leading feed-forward methods on all three world-frame metrics. On RICH, it leads feed-forward people-and-scene methods on WA-MPJPE and W-MPJPE. The complete pipeline runs at 10.1 fps on one GPU.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D foundation models, self-supervised learning, human mesh recovery, human-scene reconstruction, monocular video, representation probing People in unlabeled video teach an up-to-scale 3D foundation model to predict metric scale, and we find the model already tells people apart without any identity training. With both, we reconstruct metric cameras, scene and people from monocular video in one feed-forward pass.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zDLnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

WildHSR reconstructs metric humans, scenes, and camera trajectories from monocular video while maintaining person identities. It combines human-derived scale pseudo-labels, lightweight adaptation of a 3D foundation model, pelvis-based human-scene fusion, and temporal association using intermediate Q/K features. The reconstruction results are strong, particularly on EMDB-2, but the contribution of the identity mechanism to the final system remains insufficiently established.

Critical strengths
  1. Strong reconstruction results. The method achieves 66.3 mm WA-MPJPE and 0.9% RTE on EMDB-2 (Table 1), outperforming the methods listed on these metrics. The 3DPW results also indicate competitive local body accuracy (Table 4).

  2. Convincing evidence for scale pretraining. Turning approximate body measurements into offline supervision is a practical way to exploit unlabeled video without running the teachers at inference. Table 3 provides meaningful support: removing ITW pretraining increases WA-MPJPE from 66.3 to 116.3 mm and RTE from 0.9% to 3.7%, despite retaining exact metric fine-tuning.

  3. Interesting correspondence analysis. Figures 2 and 5 provide evidence that intermediate features of the pretrained geometry model retain correspondence information for moving people. This is a useful observation that motivates reusing the backbone for association.

Critical weaknesses
  1. The final-system benefit of the identity readout is unclear. Table 5 evaluates association components rather than the final end-to-end pipeline, as acknowledged in the text. It also labels the motion/confidence baseline as DP, leaving unclear whether the association backend is held fixed across variants. The combined method has lower IDF1 than Human3R [Chen et al., ICLR 2026] (82.1 vs. 87.4), despite substantially fewer aggregate ID switches. Since identity readout is a central contribution, a controlled comparison within the final pipeline is needed: keep proposals, training budget, and association backend fixed, and compare no identity features, early/image features, and intermediate Q/K features. Reporting IDF1/HOTA and per-sequence results would establish where the proposed representation helps.

  2. Essential training details and contribution attribution remain incomplete. Section 3.5 lists the loss categories without fully specifying the combined objective, loss weights, or the contact term's construction and gradient flow. The provided adapter and readout settings are helpful, but the Pelvis Readout/fusion configuration and stage-specific training schedule also need clarification. In particular, Table 3 establishes the value of the training curriculum, while leaving unclear which modules are updated in each stage. A compact specification of the objectives, module configurations, and trainable modules/data/steps per stage would make the method reproducible and clarify the source of its gains.

Questions for authors

See weaknesses.

Reviewer_yfxCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces WildHSR, a method for reconstructing people, camera motion, and scene geometry from a monocular video in a shared metric coordinate system. It uses people as approximate scale references to generate training labels from web videos, then fine-tunes the scale predictor with exact metric supervision. the authors also extract person-correspondence features from the geometry backbone and combines them with motion and confidence for tracking.They show human-reconstruction results on EMDB-2, RICH, and 3DPW datasets. The training data ablation of removing web video pretraining substantially worsens performance despite retaining supervised fine-tuning. Overall the main reconstruction contribution is helpful, although the evidence for reliable tracking and accurate scene geometry is less complete.

Critical strengths

The human scale learning could be interesting. Using people as approximate measuring references in web videos provides a useful way to learn metric scale without requiring exact labels for every training clip. The additional models used to generate these labels are unnecessary at inference. Table 3 convincingly shows that this training stage adds value beyond supervised fine-tuning: removing it increases trajectory error. The method shows reconstruction results on EMDB-2, WildHSR leads the listed feed-forward methods on all three global-motion metrics and outperforms the strongest listed optimization-based results on WA-MPJPE and trajectory error. It also improves local body accuracy on 3DPW. Together, these results suggest that the gains in global placement do not come at the expense of body reconstruction quality.

Critical weaknesses

Human scale is less ambiguous than other stationary features in the image. Learning scale could be interesting but humans or unanimatable objects add a lot of variance into the reconstruction. The body ruler scale estimates depend directly on the metric torso dimensions predicted by 4DHumans. How sensitive are these estimates to errors in body size and proportions? Since repeated readings from the same person can remain consistently biased, could the authors report scale errors across body size groups and the upper tail of the error distribution, in addition to the median?

The main benchmarks evaluate human joints and trajectories, while the tracking experiment uses fixed proposals. What ground-truth evaluation validates the complete system’s scene geometry, camera motion, and end-to-end identity tracking? can you use multi camera systems or lidar data to validate the metric accuracy with respect to baseline?

Table 5 reports better HOTA and fewer identity switches than Human3R. Human3R also performs better on eight of the thirteen sequences, and one crowded sequence accounts for much of its identity switch count. Furthermore, this is a component experiment using fixed body proposals, rather than an end-to-end evaluation. The fourteen-person example is visually encouraging, but only two people are annotated. This limits how confidently the paper can claim reliable identity preservation in crowded videos. we need better and larger evaluation benchmark

The broader scene-reconstruction claim is less thoroughly validated than the human-motion results. most quantitative evaluations measure body accuracy, trajectories, or association. These results are strong, but they do not directly establish accurate surrounding geometry. The system also applies scale correction during window stitching, making it difficult to separate the contribution of the learned scale predictor from later alignment. Direct scene and camera measurements, together with results before and after stitching, would clarify this.

Questions for authors

Are end-to-end tracking results available for the final pipeline, including per-sequence IDF1 and HOTA? How do these compare with the controlled experiment in Table 5?

How much does window stitching contribute to metric accuracy? Can you provide scale or trajectory errors before and after this step, particularly for sequences with little camera translation?

Reviewer_rLGLOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This work proposes WildHSR, which jointly reconstructs the scene and estimates camera and SMPL-X body parameters. Its Scale Readout recovers metric scale, while its Identity Readout identifies multiple people.

Critical strengths

The proposed pipeline outperforms the baselines, demonstrating the effectiveness of the overall approach.

Critical weaknesses
  • The writing needs substantial revision. Many sentences are difficult to follow. Figures 3 and 4 also contain scattered or misaligned arrows and other elements, which makes the pipeline harder to understand.
  • Although the method achieves strong results, it is unclear how much of the improvement comes from the proposed modules rather than the VGGT-ω and Multi-HMR backbones. The strongest baseline, UniSH, uses different scene and human mesh reconstruction models, making it difficult to isolate the contribution of the proposed approach. The authors should add ablation studies of the new modules. For example, they could evaluate the Scale Readout by replacing it with a body-ruler approach or a simple MLP-based regressor.
Questions for authors
  • Could the authors compare the proposed method with HARMONI, which uses a similar architecture?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.