acceptodds
Under review as a conference paper at ICLR 2027

Scene and Human in One World: Reconstruction in a Feedforward Pass

Abstract

Reconstructing humans in dynamic scenes from monocular video remains challenging due to scale ambiguity and human–scene misalignment. We observe that a monocular point map is defined only up to a single global scale, which scene geometry alone cannot resolve, whereas the parametric human body has a statistically well-constrained real-world size. We introduce SHOW, a mask-promptable feed-forward framework that reconstructs a designated person together with the surrounding scene in one shared metric space, in which the human body acts as the metric reference that determines the scale of the normalized point map. Mask prompting and auxiliary DensePose supervision make the geometry features discriminative on human regions, and a unified decoder regresses body pose, shape, and global translation together with the scale factor of the scene, so that scale, placement, and scene geometry are optimized as a single system instead of being aligned post hoc. SHOW therefore produces a human reconstruction that is already metrically consistent with the reconstructed scene. On 3DPW, EMDB, and RICH, it improves metric-scale consistency and human–scene alignment over prior joint human–scene feed-forward methods while remaining competitive on global human motion estimation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Computer vision, Computer graphics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xJ3mOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SHOW, a mask-promptable feed-forward framework for joint human–scene reconstruction in a shared metric space. It enhances VGGT with mask prompting and DensePose supervision to extract human-aware geometry features, and employs a unified decoder to predict SMPL-X parameters, global translation, and scene scale. Joint training further enforces human–scene consistency. Experiments show improved human–scene consistency and competitive human reconstruction and global motion estimation performance.

Critical strengths
  1. The paper introduces two complementary metrics, HS-CF and HS-V, to evaluate geometric alignment and scale consistency between reconstructed humans and scenes, providing a useful evaluation perspective beyond conventional human reconstruction metrics.
  2. The ablation studies demonstrate the effectiveness of key components, including mask prompting, ray encoding, DensePose supervision, and joint fine-tuning, providing empirical support for the proposed framework.
Critical weaknesses
  1. Limited Training Data and Generalization. SHOW requires paired 3D human and scene ground-truth annotations and is only fine-tuned on synthetic BEDLAM2.0 data. Although joint training improves human–scene alignment, both human and scene reconstruction performance degrade compared to their respective pretrained models, which can benefit from larger and more diverse training datasets. This reliance on limited synthetic data also raises concerns about OOD generalization to diverse real-world Internet videos.
  2. The backbones and baselines are relatively outdated. Human reconstruction lacks comparisons with GEM, while scene reconstruction lacks comparisons with Pi3X, DA3, and VGGT-Ω.
  3. Inconsistent Human–Scene Alignment Improvements. Human3R substantially outperforms SHOW on HS-CF across both datasets, suggesting that the alignment improvements are not consistent across different metrics.
  4. Lack of Metric-scale Evaluation. Although the paper claims metric-scale human–scene alignment, it lacks direct evaluation of absolute metric-scale accuracy.
Questions for authors
  1. Why is PromptHMR not included in Table 2?
  2. SHOW performs substantially worse than Human3R on HS-CF across both 3DPW and EMDB-1, but significantly better on HS-V. Could the authors provide qualitative visualizations comparing representative cases to illustrate this discrepancy?
  3. How does the alignment loss improve global human placement, given that it updates the point map and scale rather than the human mesh? Does any explicit geometric gradient propagate from this loss to the human translation prediction? How can the authors demonstrate that scene geometry actively refines human placement?
3 comments
AuthorsOpenReview ↗

Response to Reviewer xJ3m

We thank R3 for a constructive review; we address the points below. We will add the clarifications and additional results to the revision.

W1 — Limited training data and generalization

Both pretrained models for human mesh recovery or scene reconstruction were trained on far larger corpora with more compute than we have, but our model uses BEDLAM2.0 alone. They also tuned hyper-parameters, network structure and training objectives speciallized to their tasks and metric. Our primary objective is to optimize the HS-CF and HS-V metric. Our target is not the single-branch improvement but the internal consistency between the two outputs. We accept the drop as a limitation and expect it to close as the supervised training datasets approach the pretraining scale.

As for OOD generalisation: in-the-wild supervision generated by optimization mehtod proposed by JOSH or UniSH's video annotation, added to training datasets. We acknowledge that we have not included this in our current work and note it as our future work; because it is also outside our current work's focus, which is inner consistency between the two output branches, not generalization to unseen or rare scenarios.

W2 — Outdated backbones and baselines

We add the scene comparisons, including the models named by the reviewer, on DyCheck, with a common metric (camera pose AUC@30 and point-map accuracy):

MethodCamera pose (DyCheck AUC@30) ↑\uparrowPoint map (AbsRel) ↓\downarrow
MonST3R11.50.068
MapAnything6.10.049
MegaSaM26.80.042
VGGT21.00.055
Pi323.30.041
DA332.10.039
VGGT-Ω35.90.039
SHOW (scene branch)19.20.062

SHOW's scene branch is initialized from VGGT, so VGGT is the only backbone-matched baseline. The other methods use stronger backbones; with one of them, SHOW's scene metrics would improve as well.

W3 - Inconsistent Human–Scene Alignment Improvements

We acknowledge that Human3r can reach better performance on HS-CF. The main reason is that it human3r use pretrained CUT3R as backbone, which is trained on large-scale metric scene reconstruction datasets. Our method is trained on BEDLAM2.0 only and does not depend on outside semantic metric information. The main contribution of our work is to improve the internal consistency between the body and the scene, which is measured by HS-CF and HS-V metrics. We will clarify this in the text.

W4 — Metric-scale evaluation

Estimating metric scale from monocular video is ambiguous. Monocular video does not provide a unique solution for scale estimation. Thus our method try to improve the internal consistency between the body and the scene and use HS-CF and HS-V metric. Estimating and evaluating absolute scene scale and geoemtry from the scene alone is a separate problem (e.g., MoGe, MapAnything), which focus more on regress the scale from scene semantics feature from large metric dataset. This is outside our current work's focus.

For human metric mesh recovery, we report World-aligned MPJPE and MPJPE on EMDB-2 and RICH in Table 2. For relative scene reconstruction, we report camera pose AUC@30 and point-map accuracy on Re10K in Supplementary Material Table 3. Plus our HS-CF and HS-V metrics, these metric can valid the metric reconstruction performance of our method.

AuthorsOpenReview ↗

Cont. Response to Reviewer xJ3m

Q1 — PromptHMR in Table 2

PromptHMR itself is per-frame and camera-space, so the entry is its video version, PromptHMR-Vid: it adds a temporal transformer over the per-frame tokens and estimates the metric world frame with DROID-SLAM and monocular depth, following TRAM. Its world results are reported on EMDB-2 only:

MethodOpt. FreeSceneEMDB-2 (WA / W / RTE ↓)RICH (WA / W / RTE ↓)
SLAHMR✗✗326.9 / 776.1 / 10.2132.2 / 237.1 / 6.4
TRAM✗✗76.4 / 222.4 / 1.4127.8 / 238.0 / 6.0
PromptHMR-Vid✗✗71.0 / 216.5 / 1.3–
JOSH✗✓68.9 / 174.7 / 1.389.0 / 132.5 / 3.0
SHOW✓✓109.1 / 262.3 / 2.1107.3 / 172.7 / 2.2

Opt. Free: feed-forward (optimization-free); Scene: joint 3D scene reconstruction. All rows other than PromptHMR-Vid are from Table 2; PromptHMR-Vid is as reported in the PromptHMR paper. We will add PromptHMR-Vid to Table 2.

Q2 — Qualitative comparison of the HS-CF/HS-V discrepancy

Both are already visible: Fig. 4 shows Human3R's body sinking below the ground plane while its HS-CF stays low, and UniSH drifting laterally while its HS-V stays low. We will stress this in the caption and provide more examples in the supplementary material.

Q3 — Does the alignment loss propagate to translation?

With L_align = chamfer(s · P[m], sg(V_vis)) the target is detached, so the gradient reaches s and the point map, and τ receives none. This also holds in the final joint stage, where both branches are fine-tuned: τ appears only inside the stop-gradient, and the point map does not depend on the decoder, so L_align still has no path to it; τ is trained there by the translation term of L_hmr.

Placement improves indirectly, through the features. In stage 2, scene decoder is freezed and scale head is trained. This stage helps optimize scale output to align human and scene. In stage 3, geometry encoder and scene decoder are also fine-tuned; L_align pulls the human-region point map onto the body and continues to optimize scale output to help metric point map match human mesh. Also, location token in SMPL-X decoder learns how to read the geometry features from geometry encoder by cross attention to better estimate human translation. You can refer the ablation's RTE column for evidence: 3.3% without joint modeling, 2.9% without the ray encoder, 2.3% full, 1.9% end-to-end.

Thank you for the authors' response. I have carefully reviewed the rebuttal. However, I feel that my main concerns have not been fully addressed, particularly regarding the complexity of joint training, the limited training data, the inconsistent improvements in human–scene alignment, and the generalization ability of the method. Therefore, I will maintain my rating of Weak Reject.

Reviewer_nkurOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SHOW reconstructs a target person and the surrounding scene from monocular video and mask prompts in a single forward pass. It combines human-aware geometry adaptation with a decoder that jointly predicts body parameters, translation, and scene scale. The results demonstrate competitive global motion estimation among joint feed-forward methods and improved human–scene agreement relative to UniSH.

Critical strengths
  1. Well-motivated integration of human and scene features. Conditioning geometry features on the target mask and combining them with human priors and camera-ray encodings is a sensible design. Table 4 supports the importance of these components: removing mask prompting increases HS-CF₅ from 9.44 to 14.12, while removing the ray encoder increases HS-V₅ from 0.028 to 0.734.

  2. Competitive results within the relevant comparison group. Table 2 shows favorable global motion results on EMDB-2 and RICH among the reported joint feed-forward methods. Table 3 also shows lower HS-CF than UniSH on both datasets. These results indicate practical value in reconstructing humans and scenes without per-sequence optimization.

Critical weaknesses
  1. The evaluation does not adequately establish absolute metric scene accuracy. Section 4.2 interprets HS-V as exposing absolute scale errors, but it compares two predictions without ground truth. If the predicted human and human-region point cloud coincide, HS-V is zero even when both share an incorrect scale. HS-CF is likewise insufficient because it is invariant to common rescaling. Table 3 therefore measures agreement between representations rather than their absolute correctness; Table 2 evaluates human motion but does not directly validate scene geometry. This also weakens the interpretation in Section 4.3 that Human3R's higher HS-V demonstrates an incorrect metric scale. Please supplement these results with scene depth or point-map errors without scale alignment and a direct scale-error measure.

  2. The distinction from prior work is overstated. The introduction contrasts SHOW's shared metric space with prior unified pipelines. However, UniSH [Li et al., CVPR 2026] already fuses human and geometry features, predicts an explicit scene scale, and rescales the scene point map for human–scene alignment. SHOW's trainable, mask-conditioned geometry and unified body/scale/location decoder are plausible contributions, but shared metric reconstruction and explicit scale prediction are not sufficient distinctions by themselves. The paper should clarify this positioning and provide a controlled comparison that isolates the proposed architecture from differences in backbone, human prior, and training data.

  3. The ablations do not isolate the proposed scale-calibration mechanism. Section 3.3 combines direct scale supervision, temporal regularization, and alignment to the predicted body, yet Table 4 does not separate their contributions. Moreover, the end-to-end variant receives 100 additional epochs, so the HS-CF₅ improvement from 9.44 to 6.21 cannot be attributed solely to joint optimization. Since surface alignment directly encourages agreement with the predicted human, improved consistency need not imply more accurate scene geometry. Equal-budget comparisons and separate ablations of the scale, temporal, and alignment losses, evaluated against ground-truth geometry, would make the central argument more convincing.

Questions for authors

See weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer nkur

We thank R2 for the constructive suggestions. The following addresses the points raised:

W1 — Absolute metric accuracy of the scene

Apart from HS-CF and HS-V, we also report scene reconstruction metrics (camera pose AUC@30 and point-map accuracy on Re10K) in Supplementary Table 3 (reproduced below), and human mesh recovery metrics (PA-MPJPE on EMDB-1 and 3DPW) in Table 1 of the main paper. The predicted human mesh is at real-world metric scale, with a plausible human height of about 1.5–1.9 m. For the scene, we use the mask to extract the human-region point map. HS-V compares the spatial variance of the predicted human mesh with that of the human-region point map. If neither output is degenerate and the two variances match (HS-V = 0), the human-region point map has the same spatial variance as the metric human mesh, so the human region of the scene is at the body's metric scale. Thus the case the reviewer describes (human and scene coinciding at a common wrong scale while HS-V = 0) cannot occur.

MethodCamera pose (AUC@30) ↑Point-map acc. ↓
VGGT85.30.901
MASt3R76.40.968
DUSt3R67.71.167
SHOW (scene branch)80.70.954

W2 — The difference from UniSH

We agree that shared metric reconstruction and an explicit scale are not sufficient distinctions by themselves. Beyond the difference in the geometry backbone and the human prior, the main difference is that UniSH keeps the scene and human branches fully separate, while our method couples them and optimizes them in a joint space.

UniSH uses Pi3 as the scene backbone and decodes scene geometry from its features. It uses CameraHMR to encode human features and regresses SMPL parameters from them. The two parts are fully separate and share no features, so neither branch influences or benefits the other.

After the two feature sets are extracted, UniSH uses a lightweight AlignNet to predict the scale and translation of the human, with scene features as K,V and human features as Q. During training, both branches are frozen and only AlignNet is optimized.

Our method instead uses the geometry model and the human prior model to encode scene and human features. Cross-attention then identifies the human-region geometry features, which helps both human placement and scene scale estimation, since the metric human body provides a reference for the scene scale. Our camera ray encoder and full end-to-end tuning further improve this joint optimization.

The key difference is that our two branches are combined and optimized in a joint space, and the mask prompt guides the human branch to read depth information from the geometry features for better alignment. In UniSH, only very lightweight AlignNet is trainable, borrowing features from two branch for prediction.

Table 4 is the controlled comparison within the model: the backbone, human prior and training data are fixed. The architecture module we newly added changes. Removing joint modeling raises HS-V from 0.028 to 0.312 and HS-CF from 9.44 to 17.32; removing mask prompting raises HS-CF to 14.12; removing the ray encoder raises HS-V to 0.734.

W3 — Ablating the scale-calibration mechanism

Direct scale supervision loss could not be removed to do ablation study because it provides the sole source of supervision for the scale output. Temporal regularization loss will not improve HS-CF an HS-V metrics because it is used for regularizing the scale output to be temporally smooth for better visual quality. The ablation for alignment loss is following:

VariantEpochsHS-CF₅ ↓RTE ↓
full1509.442.3
full + e2e w/o L_align2508.252.1
full + e2e2506.211.9

We also add the extra 100 epochs of the end-to-end ablation experiment, freezing the backbone as it continues stage 2 training for the end-to-end stage 3 ablation. We add the epoch-matched control: the full model continues the stage-2 schedule for the same 100 extra epochs, with the geometry branch still frozen. It stays close to the 150-epoch model and is clearly worse than e2e on HS-CF₅ (9.28 vs. 6.21) and RTE (2.4 vs. 1.9), so the gain is not from the extra epochs. It requires fine-tuning the geometry branch together with the decoder.

VariantEpochsHS-CF₅ ↓RTE ↓
full1509.442.3
full + 100 epochs of stage 22509.282.4
full + e2e2506.211.9

Thank you for the detailed responses. I will keep my rating unchanged.

Reviewer_DwuxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a feed-forward method that reconstructs a person and the scene around them from a monocular video, in the same metric space. The main idea is that a predicted point map is only correct up to an unknown scale, while the human body has a known real-world size, so the body can be used to fix the scale of the scene.

To do this, the authors fine-tune VGGT with a mask of the target person and an auxiliary DensePose loss, so that the geometry features know where the person is. A separate decoder then predicts the SMPL-X body together with a scale factor for the scene. Training is done only on synthetic data (BEDLAM2.0), and a final joint stage pulls the predicted body and the rescaled point map into agreement.

The method is evaluated on 3DPW, EMDB and RICH for body accuracy and global motion. The authors also introduce two new metrics that measure how well the body and the scene agree with each other, motivated from the descrepancy I mention above. The results indicate that human–scene agreement is better over Human3R and UniSH, and is on par with them on the standard body metrics.

Critical strengths

The authors notice a misalignment of point-maps and bodies in human+scene reconstruction and address it with their methods. body-to-pointmap alignment loss directly addresses this and proposed evaluations address this as well. The authors also propose metrics that showcase this problem and report results of their method and competing methods.

The authors show visualisations as supplementary material that shows this misalignment for existing methods and shows that their method is superior.

With only Bedlam2.0 they achieve quite competitive results against methods trained also on real, metric data.

Critical weaknesses

The main claim is that the human and the scene end up in the same, correct metric space, but this is never fully tested against ground truth. The two new metrics (HS-CF and HS-V) only compare the predicted body with the model’s own predicted point map, and the joint training stage directly optimizes this kind of agreement, especially HS-CF. Table 2 does give ground-truth evidence for the human trajectory, but the scene side is never checked. I understand that 3DPW and EMDB have no scene ground truth, but RICH has scene scans, held-out BEDLAM2.0 data has exact depth, and the metric camera trajectories in EMDB-2 could be evaluated without scale alignment. The results of other methods are already metric, so a metric improvement in body scene alignment is not well justified with the current results. Furthermore HS-V should be reflecting as their description metric errors, however Human3r whcih should have wrong metric gets same world aligned MPJPE. A very simple basline of using the center of the body pixels to align the scale of the world to the body is not used in any of the comparisons. While world space reconstruction of bodies are not improved, such baseline could be very strong in my view.

The second issue is that body accuracy drops compared to the components the method is built on. The method directly uses PromptHMR’s backbone and the same kind of mask prompt, but it is clearly worse than PromptHMR, by about 4 to 9 mm on 3DPW and 4 to 5 mm on EMDB. On EMDB it is also worse than Human3R on MPJPE and PVE.

Questions for authors

Did the authors consider direct scene + human agreement metric on RICH or SLOPER4D dataset as opposed to self similarity metrics proposed in the paper?

The human–scene consistency comparison (Table 3) only includes Human3R and UniSH. JOSH, although optimization-based, would be a useful reference here, since it is much stronger on global motion in Table 2 and explicitly optimizes human–scene alignment. It would show how close the feed-forward approach gets. A runtime comparison in the main paper would also help support the case for the feed-forward approach.

As discussed, simply combining "up to a scale" world reconstruction systems, such as VGGT based on pixels to metric body reconstruction systems like prompt-HMR, such comparison is completely missing, but would be the right baseline in my view to justify the paper.

Some of the main components are directly taken from the prompthmr, what leads to the performance drop in authors view wrt this system for the human mesh reconstruction task?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Dwux

We thank R1 for a detailed review and for the two concrete suggestions. The following addresses the points raised, and we will add the clarifications and additional results to the revision.

W1.1 — On checking the scene against ground truth

The scene branch is evaluated on Re10K camera pose and point-map accuracy. This table shows no catastrophic forgetting from the auxiliary DensePose supervision (Supplementary Table 3). We will move it into the main paper.

MethodCamera pose (AUC@30) ↑Point-map acc. ↓
VGGT85.30.901
MASt3R76.40.968
DUSt3R67.71.167
SHOW (scene branch)80.70.954

Estimating metric scale from monocular video is ambiguous. Monocular video does not provide a unique solution for scale estimation. Thus our method try to improve the internal consistency between the body and the scene and use HS-CF and HS-V metric. Estimating and evaluating absolute scene scale and geoemtry from the scene alone is a separate problem (e.g., MoGe, MapAnything).

W1.2 — The pixel-centroid baseline

We now include an adapted version: normalized point maps from raw VGGT and human mesh from PromptHMR, aligned by an optimization of human-region chamfer distance. Its HS-CF is lower than ours, since HS-CF is the quantity it minimizes; its HS-V remains far higher.

MethodHS-CF₅ (3DPW) ↓HS-V₅ (3DPW) ↓
VGGT + PromptHMR, post-hoc fit0.561.342
UniSH7.650.028
SHOW5.690.030

W2 — Body accuracy

The drop is expected and is not the objective we optimise. PromptHMR is specialised for HMR and trained end-to-end on large, diverse real data until convergence, with hyper-parameters, model structure and training loss tuned for that task; we fine-tune on BEDLAM2.0 alone which is much smaller than PromptHMR training datasets. From stage 3, scene objective that trades some human mesh recovery accuracy for global consistency. PromptHMR produces a camera-space body and no scene; SHOW produces body, scene point map, metric scale and cameras in one pass, which has more functions and aims at different task than PromptHMR. Among methods that reconstruct the scene we lead all six 3DPW cells.

Q1 — RICH and SLOPER4D

The scans have holes at depth discontinuities and at range. Also the scene is scanned by lidar and is need to transform to the frame based point map. A raw scene chamfer is dominated by each method's filtering choices and the confidence magnitude it estimates for each point, making the comparison challenging. We are also more interested how human region point cloud align with human mesh, rather than points far away. This is the reason why we choose to use HS-CF and HS-V metrics. Because this metric can better reflect the alignment quality between the human and the scene.

A metric that is meaningful in this context is penetration and foot–floor contact restricted to the subject's neighbourhood, we have reported it in our limitation section and will try to address contact-related metrics in future work.

Q2 — JOSH and runtime

We will add the JOSH as baseline in our consistency comparison. The baseline evaluation results are as follows and you can refer to Supplementary Material Table 1 for runtime evaluation results for JOSH.

Method3DPW HS-V₅ ↓HS-CF₅ ↓
JOSH0.0250.36
Human3R0.6521.30
UniSH0.0287.65
SHOW0.0305.69

Q3

Covered by W1.3 and the table there.

Q4

Covered by W2.

Thank you for the response and the new experiments.

The additional results help, however my main concern remains, even elevated. The simple post-hoc baseline gets a much lower HS-CF than SHOW (0.56 vs 5.69). This is an order of magnitude difference. Also, JOSH is better on both of your metrics. I am also unsure why the post-hoc baseline has such a high HS-V: if the point map is scaled to the metric body, the two should match almost by construction.

On metric scale, the paper’s main claim is that the body fixes the scale of the scene, but this is still not checked against ground truth. For these reasons I will keep my rating of weak reject.

Comments on the paper

1 comment

Dear Reviewers and Area Chair,

Thank you for your time and constructive feedback. Our work contributes a mask-prompted joint human and scene reconstruction model with output modality alignment. In our rebuttal, we clarified the methodology and added evaluations addressing the concerns about dataset choice and generalization, metric-scale scene evaluation, proposed metric design, and more evaluation baselines including simple optimization-based and the latest state-of-the-art work.

As the discussion period approaches its end, we would greatly appreciate your feedback on whether our responses address your concerns. We would be happy to provide any further clarification or discuss remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.