- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents InHabit, an automated framework that uses image foundation models to generate large-scale 3D human–scene interaction (HSI) data via a render–generate–lift pipeline. It renders 3D scene views, uses a vision–language model to propose context-appropriate interactions, synthesizes humans with an image-editing model, and lifts them into 3D SMPL-X bodies using geometry-aware optimization with physical constraints.
Applying this pipeline to Habitat-Matterport3D, the authors build InHabitants, a dataset of roughly 78,000 images across 6,600 rooms paired with SMPL-X bodies and complete 3D scene geometry, covering diverse single- and multi-person interactions.
Placement comparisons and a perceptual user study demonstrate the plausibility and naturalness of the generated interactions, while experiments on human–scene reconstruction, 3D human generation, and contact estimation show that training on InHabitants improves downstream performance over several existing datasets.
Critical strengths
1. A scalable and well-motivated approach to generating context-aware human–scene interaction data.
The paper addresses an important limitation of existing HSI datasets: the difficulty of achieving both large-scale scene coverage and diverse, semantically meaningful interactions. The proposed render–generate–lift framework provides a practical way to transfer interaction knowledge from 2D foundation models into known 3D environments.
2. Convincing downstream evaluations demonstrating the utility of the generated data.
The paper evaluates InHabitants across three distinct downstream tasks rather than relying solely on qualitative examples or geometric plausibility metrics. The reported improvements are substantial in several settings, particularly for contact-aware human–scene reconstruction.
Critical weaknesses
1. Limited reliability of the reconstructed 3D humans and contact annotations.
Although InHabit produces visually plausible human–scene interactions, some qualitative results exhibit noticeable artifacts in the lifted SMPL-X bodies, including severe body–scene penetrations and inaccurate hand poses. These issues are particularly concerning for fine-grained interactions, where small errors in hand or finger articulation can lead to incorrect contact annotations. While the authors introduce geometry-aware optimization and quality filtering, their effectiveness in handling such errors is not sufficiently quantified. Since the generated data is used to supervise downstream reconstruction and contact estimation tasks, these inaccuracies may affect the reliability of the learned interactions.
2. Inconsistency between generated images and the underlying 3D scene geometry.
The image-editing model can produce plausible interactions that are not fully supported by the original 3D scene. In the supplementary webpage (Samples from InHabitants, first column, rows 5 and 10), some interaction-relevant objects, such as a cable or faucet, appear in the generated images but are absent from the corresponding 3D scene visualizations. Such discrepancies raise concerns about whether the lifted human poses and contact annotations accurately reflect the intended interactions. Furthermore, the limited geometric detail of scanned scenes may omit small but essential objects, such as handles or other contact surfaces. Although the authors acknowledge scene-editing artifacts and introduce quality-control filters, the robustness of the pipeline to these geometry and semantic mismatches remains unclear.
Questions for authors
1. How robust is the 3D lifting process to inaccuracies in SMPL-X reconstruction, particularly for fine-grained contacts?
Some qualitative examples exhibit substantial body–scene penetration or inaccurate hand articulation. Could the authors clarify how frequently such errors occur in the final dataset after quality filtering, and whether they affect the contact annotations used for downstream training? An evaluation of contact accuracy, particularly for hand–object interactions, would help establish the reliability of the generated supervision.
2. How does InHabit handle interactions involving objects or geometric details that are missing from the original 3D scene?
In the supplementary webpage (Samples from InHabitants, first column, rows 5 and 10), interaction-relevant objects such as a cable or faucet appear in the generated images but are not visible in the corresponding 3D scene visualizations. How does the pipeline ensure meaningful 3D contact supervision when the geometry required for the interaction is missing or inaccurately represented? Could the authors clarify whether such cases are detected and filtered, and how they affect the quality of the resulting dataset?
4 comments
We look forward to hearing whether this addresses your concerns
Dear Reviewer 55DP,
Thank you again for the review. In our global response, we provide new evidence for the quality of our labels on GT benchmarks in addition to our strong downstream performance.
In our reply above, we further show that our contacts are more accurate than those of the widely used DECO, including hands. We also clarify that our scope is human-scene interaction and not fine-grained hand manipulation, and have updated the limitations accordingly.
We look forward to your response on whether this addresses your concerns, and are glad to clarify further questions before the end of discussion.
Best regards, The authors.
Thank you for the detailed response and additional experiments.
I appreciate the new evaluations of lifting accuracy, contact annotations, and body–scene penetration. I also acknowledge that the generated dataset improves several downstream models, demonstrating the practical value of the proposed pipeline.
However, my main concerns remain. Although the paper introduces a novel pipeline for automatically generating large-scale human–scene interaction data, the technical components largely build upon existing methods, and I do not see sufficient effort devoted to addressing the nontrivial quality issues in the generated data. Even among the qualitative examples presented in the paper and supplementary materials, there are noticeable artifacts, including inaccurate SMPL-X reconstructions and inconsistencies between generated images and the original 3D scene geometry. The authors acknowledge that some of these errors can pass the proposed filtering process, yet the current lifting and filtering designs do not appear to adequately address them.
While I recognize the value of the automated pipeline and its demonstrated downstream benefits, I believe that more substantial technical effort toward resolving these visible failure cases is essential, particularly given that producing reliable 3D annotations is central to the proposed framework.
Therefore, I maintain my original rating.
Dear Reviewer 55DP,
Thank you for acknowledging that our paper introduces a novel pipeline and improves several downstream models, demonstrating the practical value of the proposed pipeline. Regarding the label quality of our data our position is the following: even foundational vision datasets such as ImageNet, COCO, and Cityscapes have imperfect labeling (as shown in [1–3]), yet were instrumental for progress in computer vision. Our HSI dataset is the first to provide realistic images of 3D bodies interacting in diverse 3D scenes at our scale. Our experiments demonstrate that our data improves state-of-the-art models across different tasks with the current quality assurance in place, validating for the first time that fully automatic 3D HSI construction works in practice. Furthermore, our data provides fertile ground for methods that explicitly reason in 3D: most notably, GRAFT—whose 3D scene-aware formulation depends on diverse 3D human–scene pairs—benefits greatly from our dataset to reach state-of-the-art performance, whereas existing alternatives fall short (PROX has limited scene diversity, BEDLAM lacks interactions). Thank you again for your feedback.
References
[1] Beyer et al., "Are we done with ImageNet?", arXiv 2020.
[2] Gupta et al., "LVIS: A Dataset for Large Vocabulary Instance Segmentation", CVPR 2019.
[3] Zlateski et al., "On the Importance of Label Quality for Semantic Segmentation", CVPR 2018.
Thank you for the careful look at the samples. We measured the cases you point to and penetration is close to that of captured ground truth, and our hand contacts are far more accurate than those of DECO, a widely used image-based contact estimator. Like the captured datasets PROX and RICH, InHabit targets full-body interaction with static scenes.
"How frequently do such errors occur in the final dataset?" At a rate close to captured ground truth in comparable indoor scenes: on average 1.78% of body vertices lie 2–15 cm inside the scene, against 1.38% for the GT bodies of PROX.
Our fit deliberately tolerates a small amount of penetration to account for soft body deformation. For example, a person sitting on a soft sofa or bed would naturally sink in, so their fit ends up slightly inside the mesh. This is also shown by the non-zero penetration on PROX. Some penetration comes from the image itself: if the editor draws a hand inside a cushion, the lifting stays faithful to the image.
"Do they affect the contact annotations?" Yes, locally: a penetrating or misposed hand can produce a missing or spurious contact label. All downstream results in the paper were obtained with these labels included, so the gains hold with this noise present: DECO's contact F1 on DAMON goes from 0.512 to 0.534 (Tab. 4), GRAFT reaches 0.594 on PROX versus 0.492 when trained on PROX itself, and adding InHabitants to BEDLAM improves Human3R on every metric (Tab. 2).
"An evaluation of contact accuracy, particularly for hand–object interactions." In a static scene, the hand contacts that can be evaluated are those with scene surfaces. We measured them against GT by applying our lifting to real images from RICH and PROX.
Our lifted contacts outperform those of PhySIC and of DECO, which initializes our fitting. For the hands the gain over DECO is the largest. On RICH, hand contacts are even more accurate than the body overall. Manipulation of handheld objects and fine finger articulation lie outside this static-scene setting; we now state this in the limitations (App. B).
You're right: in both examples, the image editor added an object that isn't in the 3D scene. These are scene changes, a limitation we discuss in App. B.
"Are such cases detected and filtered?" Partly. Our depth-based filter catches scene changes near the person, like added or moved furniture (Suppl. C.5, Fig. 9); it accounts for 37.2% of all rejections. A cable or a faucet is too thin to change the depth map much, so the filter misses them.
"How do they affect the quality of the dataset?" The effect is local to the contact label of the hand involved. Training GRAFT, Human3R and DECO on our data improves them, despite the noisy samples.
We have expanded the limitations (App. B) to name such small scene changes explicitly. Modeling scene changes in 3D, which App. B lists as future work, would remove this kind of error.