- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents SHOW, a mask-promptable feed-forward framework for joint human–scene reconstruction in a shared metric space. It enhances VGGT with mask prompting and DensePose supervision to extract human-aware geometry features, and employs a unified decoder to predict SMPL-X parameters, global translation, and scene scale. Joint training further enforces human–scene consistency. Experiments show improved human–scene consistency and competitive human reconstruction and global motion estimation performance.
Critical strengths
- The paper introduces two complementary metrics, HS-CF and HS-V, to evaluate geometric alignment and scale consistency between reconstructed humans and scenes, providing a useful evaluation perspective beyond conventional human reconstruction metrics.
- The ablation studies demonstrate the effectiveness of key components, including mask prompting, ray encoding, DensePose supervision, and joint fine-tuning, providing empirical support for the proposed framework.
Critical weaknesses
- Limited Training Data and Generalization. SHOW requires paired 3D human and scene ground-truth annotations and is only fine-tuned on synthetic BEDLAM2.0 data. Although joint training improves human–scene alignment, both human and scene reconstruction performance degrade compared to their respective pretrained models, which can benefit from larger and more diverse training datasets. This reliance on limited synthetic data also raises concerns about OOD generalization to diverse real-world Internet videos.
- The backbones and baselines are relatively outdated. Human reconstruction lacks comparisons with GEM, while scene reconstruction lacks comparisons with Pi3X, DA3, and VGGT-Ω.
- Inconsistent Human–Scene Alignment Improvements. Human3R substantially outperforms SHOW on HS-CF across both datasets, suggesting that the alignment improvements are not consistent across different metrics.
- Lack of Metric-scale Evaluation. Although the paper claims metric-scale human–scene alignment, it lacks direct evaluation of absolute metric-scale accuracy.
Questions for authors
- Why is PromptHMR not included in Table 2?
- SHOW performs substantially worse than Human3R on HS-CF across both 3DPW and EMDB-1, but significantly better on HS-V. Could the authors provide qualitative visualizations comparing representative cases to illustrate this discrepancy?
- How does the alignment loss improve global human placement, given that it updates the point map and scale rather than the human mesh? Does any explicit geometric gradient propagate from this loss to the human translation prediction? How can the authors demonstrate that scene geometry actively refines human placement?
3 comments
Cont. Response to Reviewer xJ3m
Q1 — PromptHMR in Table 2
PromptHMR itself is per-frame and camera-space, so the entry is its video version, PromptHMR-Vid: it adds a temporal transformer over the per-frame tokens and estimates the metric world frame with DROID-SLAM and monocular depth, following TRAM. Its world results are reported on EMDB-2 only:
| Method | Opt. Free | Scene | EMDB-2 (WA / W / RTE ↓) | RICH (WA / W / RTE ↓) |
|---|---|---|---|---|
| SLAHMR | ✗ | ✗ | 326.9 / 776.1 / 10.2 | 132.2 / 237.1 / 6.4 |
| TRAM | ✗ | ✗ | 76.4 / 222.4 / 1.4 | 127.8 / 238.0 / 6.0 |
| PromptHMR-Vid | ✗ | ✗ | 71.0 / 216.5 / 1.3 | – |
| JOSH | ✗ | ✓ | 68.9 / 174.7 / 1.3 | 89.0 / 132.5 / 3.0 |
| SHOW | ✓ | ✓ | 109.1 / 262.3 / 2.1 | 107.3 / 172.7 / 2.2 |
Opt. Free: feed-forward (optimization-free); Scene: joint 3D scene reconstruction. All rows other than PromptHMR-Vid are from Table 2; PromptHMR-Vid is as reported in the PromptHMR paper. We will add PromptHMR-Vid to Table 2.
Q2 — Qualitative comparison of the HS-CF/HS-V discrepancy
Both are already visible: Fig. 4 shows Human3R's body sinking below the ground plane while its HS-CF stays low, and UniSH drifting laterally while its HS-V stays low. We will stress this in the caption and provide more examples in the supplementary material.
Q3 — Does the alignment loss propagate to translation?
With L_align = chamfer(s · P[m], sg(V_vis)) the target is detached, so the gradient reaches s and the point map, and τ receives none. This also holds in the final joint stage, where both branches are fine-tuned: τ appears only inside the stop-gradient, and the point map does not depend on the decoder, so L_align still has no path to it; τ is trained there by the translation term of L_hmr.
Placement improves indirectly, through the features. In stage 2, scene decoder is freezed and scale head is trained. This stage helps optimize scale output to align human and scene. In stage 3, geometry encoder and scene decoder are also fine-tuned; L_align pulls the human-region point map onto the body and continues to optimize scale output to help metric point map match human mesh. Also, location token in SMPL-X decoder learns how to read the geometry features from geometry encoder by cross attention to better estimate human translation. You can refer the ablation's RTE column for evidence: 3.3% without joint modeling, 2.9% without the ray encoder, 2.3% full, 1.9% end-to-end.
Thank you for the authors' response. I have carefully reviewed the rebuttal. However, I feel that my main concerns have not been fully addressed, particularly regarding the complexity of joint training, the limited training data, the inconsistent improvements in human–scene alignment, and the generalization ability of the method. Therefore, I will maintain my rating of Weak Reject.
Response to Reviewer xJ3m
We thank R3 for a constructive review; we address the points below. We will add the clarifications and additional results to the revision.
W1 — Limited training data and generalization
Both pretrained models for human mesh recovery or scene reconstruction were trained on far larger corpora with more compute than we have, but our model uses BEDLAM2.0 alone. They also tuned hyper-parameters, network structure and training objectives speciallized to their tasks and metric. Our primary objective is to optimize the HS-CF and HS-V metric. Our target is not the single-branch improvement but the internal consistency between the two outputs. We accept the drop as a limitation and expect it to close as the supervised training datasets approach the pretraining scale.
As for OOD generalisation: in-the-wild supervision generated by optimization mehtod proposed by JOSH or UniSH's video annotation, added to training datasets. We acknowledge that we have not included this in our current work and note it as our future work; because it is also outside our current work's focus, which is inner consistency between the two output branches, not generalization to unseen or rare scenarios.
W2 — Outdated backbones and baselines
We add the scene comparisons, including the models named by the reviewer, on DyCheck, with a common metric (camera pose AUC@30 and point-map accuracy):
SHOW's scene branch is initialized from VGGT, so VGGT is the only backbone-matched baseline. The other methods use stronger backbones; with one of them, SHOW's scene metrics would improve as well.
W3 - Inconsistent Human–Scene Alignment Improvements
We acknowledge that Human3r can reach better performance on HS-CF. The main reason is that it human3r use pretrained CUT3R as backbone, which is trained on large-scale metric scene reconstruction datasets. Our method is trained on BEDLAM2.0 only and does not depend on outside semantic metric information. The main contribution of our work is to improve the internal consistency between the body and the scene, which is measured by HS-CF and HS-V metrics. We will clarify this in the text.
W4 — Metric-scale evaluation
Estimating metric scale from monocular video is ambiguous. Monocular video does not provide a unique solution for scale estimation. Thus our method try to improve the internal consistency between the body and the scene and use HS-CF and HS-V metric. Estimating and evaluating absolute scene scale and geoemtry from the scene alone is a separate problem (e.g., MoGe, MapAnything), which focus more on regress the scale from scene semantics feature from large metric dataset. This is outside our current work's focus.
For human metric mesh recovery, we report World-aligned MPJPE and MPJPE on EMDB-2 and RICH in Table 2. For relative scene reconstruction, we report camera pose AUC@30 and point-map accuracy on Re10K in Supplementary Material Table 3. Plus our HS-CF and HS-V metrics, these metric can valid the metric reconstruction performance of our method.