acceptodds
Under review as a conference paper at ICLR 2027

InHabit: Scaling 3D Human-Scene Interaction Data with Image Foundation Models

Abstract

Embodied AI and human-centric 3D scene understanding require data of people meaningfully interacting with diverse environments, yet such data remains scarce. Real-world capture has steadily pushed toward more scenes, but is bound to stu- dios and at most around a hundred scenes, while synthetic datasets scale easily but place people without regard to what the scene affords. We argue that useful human–scene interaction data requires diversity in both scenes and interactions, and we show this with our proposed data engine InHabit. InHabit is a fully auto- matic engine that renders an existing 3D scene, uses 2D foundation models to pro- pose and depict context-appropriate interactions, and lifts them into SMPL-X bod- ies, where the known scene geometry makes lifting well-constrained. Applied to Habitat-Matterport3D, InHabit produces InHabitants, which pairs photorealistic images with SMPL-X bodies and complete scene geometry across ∼6.6k rooms, about 60× more scenes than the largest captured dataset. We illustrate the quality and value of such large-scale human–scene interaction data through a variety of experiments. In a perceptual study, our interactions are preferred over two prior placement methods in 78% of cases. Across three tasks—1) contact estimation, 2) human–scene reconstruction, and 3) generating humans in scenes—training with InHabitants outperforms training on synthetic data with exact ground truth (BEDLAM), motion-capture data (TRUMANS), and real RGB-D captures (PROX); e.g., GRAFT trained on InHabitants reaches a contact F1 of 0.594 on PROX, versus 0.492 when trained on PROX itself.

Primary area:
datasets and benchmarks
Keywords:
human scene interaction, generative modeling, human scene reconstruction, synthetic data

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_55DPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents InHabit, an automated framework that uses image foundation models to generate large-scale 3D human–scene interaction (HSI) data via a render–generate–lift pipeline. It renders 3D scene views, uses a vision–language model to propose context-appropriate interactions, synthesizes humans with an image-editing model, and lifts them into 3D SMPL-X bodies using geometry-aware optimization with physical constraints.

Applying this pipeline to Habitat-Matterport3D, the authors build InHabitants, a dataset of roughly 78,000 images across 6,600 rooms paired with SMPL-X bodies and complete 3D scene geometry, covering diverse single- and multi-person interactions.

Placement comparisons and a perceptual user study demonstrate the plausibility and naturalness of the generated interactions, while experiments on human–scene reconstruction, 3D human generation, and contact estimation show that training on InHabitants improves downstream performance over several existing datasets.

Critical strengths

1. A scalable and well-motivated approach to generating context-aware human–scene interaction data.

The paper addresses an important limitation of existing HSI datasets: the difficulty of achieving both large-scale scene coverage and diverse, semantically meaningful interactions. The proposed render–generate–lift framework provides a practical way to transfer interaction knowledge from 2D foundation models into known 3D environments.

2. Convincing downstream evaluations demonstrating the utility of the generated data.

The paper evaluates InHabitants across three distinct downstream tasks rather than relying solely on qualitative examples or geometric plausibility metrics. The reported improvements are substantial in several settings, particularly for contact-aware human–scene reconstruction.

Critical weaknesses

1. Limited reliability of the reconstructed 3D humans and contact annotations.

Although InHabit produces visually plausible human–scene interactions, some qualitative results exhibit noticeable artifacts in the lifted SMPL-X bodies, including severe body–scene penetrations and inaccurate hand poses. These issues are particularly concerning for fine-grained interactions, where small errors in hand or finger articulation can lead to incorrect contact annotations. While the authors introduce geometry-aware optimization and quality filtering, their effectiveness in handling such errors is not sufficiently quantified. Since the generated data is used to supervise downstream reconstruction and contact estimation tasks, these inaccuracies may affect the reliability of the learned interactions.

2. Inconsistency between generated images and the underlying 3D scene geometry.

The image-editing model can produce plausible interactions that are not fully supported by the original 3D scene. In the supplementary webpage (Samples from InHabitants, first column, rows 5 and 10), some interaction-relevant objects, such as a cable or faucet, appear in the generated images but are absent from the corresponding 3D scene visualizations. Such discrepancies raise concerns about whether the lifted human poses and contact annotations accurately reflect the intended interactions. Furthermore, the limited geometric detail of scanned scenes may omit small but essential objects, such as handles or other contact surfaces. Although the authors acknowledge scene-editing artifacts and introduce quality-control filters, the robustness of the pipeline to these geometry and semantic mismatches remains unclear.

Questions for authors

1. How robust is the 3D lifting process to inaccuracies in SMPL-X reconstruction, particularly for fine-grained contacts?

Some qualitative examples exhibit substantial body–scene penetration or inaccurate hand articulation. Could the authors clarify how frequently such errors occur in the final dataset after quality filtering, and whether they affect the contact annotations used for downstream training? An evaluation of contact accuracy, particularly for hand–object interactions, would help establish the reliability of the generated supervision.

2. How does InHabit handle interactions involving objects or geometric details that are missing from the original 3D scene?

In the supplementary webpage (Samples from InHabitants, first column, rows 5 and 10), interaction-relevant objects such as a cable or faucet appear in the generated images but are not visible in the corresponding 3D scene visualizations. How does the pipeline ensure meaningful 3D contact supervision when the geometry required for the interaction is missing or inaccurately represented? Could the authors clarify whether such cases are detected and filtered, and how they affect the quality of the resulting dataset?

4 comments

Thank you for the careful look at the samples. We measured the cases you point to and penetration is close to that of captured ground truth, and our hand contacts are far more accurate than those of DECO, a widely used image-based contact estimator. Like the captured datasets PROX and RICH, InHabit targets full-body interaction with static scenes.

W1 / Q1: How often do penetration and hand errors occur after filtering, do they affect the contact labels, and how accurate are hand contacts?

"How frequently do such errors occur in the final dataset?" At a rate close to captured ground truth in comparable indoor scenes: on average 1.78% of body vertices lie 2–15 cm inside the scene, against 1.38% for the GT bodies of PROX.

Penetration ↓GT bodiesOur lifting
PROX1.38%1.61%
RICH0.06%1.05%
InHabitants (ours)–1.78%

Our fit deliberately tolerates a small amount of penetration to account for soft body deformation. For example, a person sitting on a soft sofa or bed would naturally sink in, so their fit ends up slightly inside the mesh. This is also shown by the non-zero penetration on PROX. Some penetration comes from the image itself: if the editor draws a hand inside a cushion, the lifting stays faithful to the image.

"Do they affect the contact annotations?" Yes, locally: a penetrating or misposed hand can produce a missing or spurious contact label. All downstream results in the paper were obtained with these labels included, so the gains hold with this noise present: DECO's contact F1 on DAMON goes from 0.512 to 0.534 (Tab. 4), GRAFT reaches 0.594 on PROX versus 0.492 when trained on PROX itself, and adding InHabitants to BEDLAM improves Human3R on every metric (Tab. 2).

"An evaluation of contact accuracy, particularly for hand–object interactions." In a static scene, the hand contacts that can be evaluated are those with scene surfaces. We measured them against GT by applying our lifting to real images from RICH and PROX.

Contact F1 vs. GTRICH, allRICH, handsPROX, allPROX, hands
DECO (image only)0.4280.3290.5270.213
PhySIC0.6230.7790.4790.508
Ours0.7710.8220.7420.645

Our lifted contacts outperform those of PhySIC and of DECO, which initializes our fitting. For the hands the gain over DECO is the largest. On RICH, hand contacts are even more accurate than the body overall. Manipulation of handheld objects and fine finger articulation lie outside this static-scene setting; we now state this in the limitations (App. B).

W2 / Q2: Objects that are in the image but missing from the 3D scene (cable, faucet).

You're right: in both examples, the image editor added an object that isn't in the 3D scene. These are scene changes, a limitation we discuss in App. B.

"Are such cases detected and filtered?" Partly. Our depth-based filter catches scene changes near the person, like added or moved furniture (Suppl. C.5, Fig. 9); it accounts for 37.2% of all rejections. A cable or a faucet is too thin to change the depth map much, so the filter misses them.

"How do they affect the quality of the dataset?" The effect is local to the contact label of the hand involved. Training GRAFT, Human3R and DECO on our data improves them, despite the noisy samples.

We have expanded the limitations (App. B) to name such small scene changes explicitly. Modeling scene changes in 3D, which App. B lists as future work, would remove this kind of error.

AuthorsOpenReview ↗

We look forward to hearing whether this addresses your concerns

Dear Reviewer 55DP,

Thank you again for the review. In our global response, we provide new evidence for the quality of our labels on GT benchmarks in addition to our strong downstream performance.

In our reply above, we further show that our contacts are more accurate than those of the widely used DECO, including hands. We also clarify that our scope is human-scene interaction and not fine-grained hand manipulation, and have updated the limitations accordingly.

We look forward to your response on whether this addresses your concerns, and are glad to clarify further questions before the end of discussion.

Best regards, The authors.

Thank you for the detailed response and additional experiments.

I appreciate the new evaluations of lifting accuracy, contact annotations, and body–scene penetration. I also acknowledge that the generated dataset improves several downstream models, demonstrating the practical value of the proposed pipeline.

However, my main concerns remain. Although the paper introduces a novel pipeline for automatically generating large-scale human–scene interaction data, the technical components largely build upon existing methods, and I do not see sufficient effort devoted to addressing the nontrivial quality issues in the generated data. Even among the qualitative examples presented in the paper and supplementary materials, there are noticeable artifacts, including inaccurate SMPL-X reconstructions and inconsistencies between generated images and the original 3D scene geometry. The authors acknowledge that some of these errors can pass the proposed filtering process, yet the current lifting and filtering designs do not appear to adequately address them.

While I recognize the value of the automated pipeline and its demonstrated downstream benefits, I believe that more substantial technical effort toward resolving these visible failure cases is essential, particularly given that producing reliable 3D annotations is central to the proposed framework.

Therefore, I maintain my original rating.

Dear Reviewer 55DP,

Thank you for acknowledging that our paper introduces a novel pipeline and improves several downstream models, demonstrating the practical value of the proposed pipeline. Regarding the label quality of our data our position is the following: even foundational vision datasets such as ImageNet, COCO, and Cityscapes have imperfect labeling (as shown in [1–3]), yet were instrumental for progress in computer vision. Our HSI dataset is the first to provide realistic images of 3D bodies interacting in diverse 3D scenes at our scale. Our experiments demonstrate that our data improves state-of-the-art models across different tasks with the current quality assurance in place, validating for the first time that fully automatic 3D HSI construction works in practice. Furthermore, our data provides fertile ground for methods that explicitly reason in 3D: most notably, GRAFT—whose 3D scene-aware formulation depends on diverse 3D human–scene pairs—benefits greatly from our dataset to reach state-of-the-art performance, whereas existing alternatives fall short (PROX has limited scene diversity, BEDLAM lacks interactions). Thank you again for your feedback.

References
[1] Beyer et al., "Are we done with ImageNet?", arXiv 2020.
[2] Gupta et al., "LVIS: A Dataset for Large Vocabulary Instance Segmentation", CVPR 2019.
[3] Zlateski et al., "On the Importance of Label Quality for Semantic Segmentation", CVPR 2018.

Reviewer_BEDrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces InHabit, an automatic pipeline for generating diverse and context-aware 3D human–scene interaction (HSI) data. A VLM first proposes scene-appropriate interactions. Then an image-editing model synthesizes interacting humans in rendered images, and an optimization-based reconstruction procedure produces SMPL-X representations constrained by the known 3D scene geometry. The proposed pipeline is applied to HM3D to construct InHabitants. The authors demonstrate its utility through human–scene reconstruction, scene-conditioned human generation, and contact estimation. They additionally conduct a user study and several ablations to evaluate interaction realism, geometric consistency, and data diversity.

Critical strengths
  • The studied problem is important and practically motivated. Efficient and scalable synthetic data pipelines for human-scene interaction has been a long-pursued goal of the community.

  • The demonstrated downstream effectiveness is convincing.

  • Extensive ablations is provided, validating the effectiveness of different components of the pipeline.

Critical weaknesses
  • The core contribution of the submission is closely related to existing methods like GenZI, ComA, and GenHSI. Especially, the difference between the proposed pipeline and GenHSI should be better covered.

  • If the paper could isolate the effects of affordance-driven prompting, action-driven prompting, PoseScript augmentation, and the stronger image-editing model, it will be much more convincing. For now, it remains unclear how much of the observed interaction diversity comes from the proposed prompting strategies rather than the capabilities of the underlying foundation model. A component-wise ablation would help establish the methodological contribution.

  • The geometric accuracy and physical validity of the generated annotations remain insufficiently validated. The quantitative lifting evaluation in Table 6 relies on coarse VLM-derived contact labels rather than independent ground-truth 3D annotations. A small independently annotated benchmark would significantly strengthen the claim that the pipeline produces accurate 3D HSI supervision rather than merely plausible-looking configurations.

  • Some downstream comparisons require more careful interpretation. The training datasets differ substantially in size, scene distribution, annotation characteristics, and interaction coverage. Consequently, the results do not fully isolate which properties of InHabitants explain its advantages.

  • Some improvements involve trade-offs. For example, in Table 2, GRAFT trained on InHabitants achieves better contact F1 and PA-MPJPE than the PROX-trained counterpart, but worse V2S (184.36 versus 166.51).

  • The diversity ablation does not completely disentangle its underlying factors. Table 7 suggests that interaction diversity contributes more to downstream performance than scene diversity. However, comparing feet-only interactions against all interaction types also changes the distribution of contact regions, poses, and potentially object categories. Therefore, the reported improvement cannot be attributed exclusively to interaction diversity. I find the conclusion plausible and interesting, but the current claim should be carefully reconsidered.

  • Evaluation of semantic interaction correctness may favor the proposed generation pipeline. The comparison in Sec. 4.4 uses interaction prompts and locations sampled from the proposed affordance-aware generation process. This may favor InHabit, particularly when compared with approaches using different conditioning interfaces. Additionally, the semantic contact evaluation in Table 5 uses LLM-generated pseudo-ground-truth body-part labels. Such labels can measure agreement with semantic expectations but cannot independently verify precise physical contacts.

Questions for authors
  • Can the authors provide an ablation of affordance-driven prompting, action-driven prompting, and PoseScript augmentation, ideally using the same scenes and image-editing backbone? This would clarify which components are responsible for interaction diversity and downstream gains.

  • Is it possible to evaluate the lifting stage against independently verified 3D poses or contacts on a small controlled subset? In particular, how reliable are the generated hand–object and body–object contacts, beyond their agreement with VLM-derived semantic labels?

  • How many generated samples are rejected by each quality-control filter? A breakdown by interaction category would help determine whether the filtering process disproportionately removes difficult or contact-rich interactions.

5 comments

Thank you for the constructive review. We add the prompting ablation and the ground-truth evaluation of the lifting that you asked for (Suppl. J, K).

W1: Relation to GenZI, ComA and GenHSI.

The key distinction is that InHabit produces an image, a body fitted to that image and situated in a complete 3D scene, which are consistent with each other. This allows us to use the generated data as training data for downstream tasks.
In GenHSI the image model and the lifting only consider the target object and not the full scene. Thus, the image model misses crucial scene context to create a plausible interaction and the lifting can result in a physically implausible interaction once merged with the whole scene. InHabit adds the person to the complete scene image and lifting is done against the complete scene to create plausible interactions. The latter matters for accuracy (table below).
GenZI generates a body but none of the inpaints match it. ComA considers only isolated objects and not full scenes.

Refinement and complete scan, evaluated on RICH and PROX (MPJPE in mm, contact F1):

RICH MPJPE ↓RICH F1 ↑PROX MPJPE ↓PROX F1 ↑
InHabit (full fit, complete scan)118.80.771100.40.742
alignment only, no refinement171.60.664138.10.698
estimated depth instead of the scan405.50.574154.60.617
GenHSI's fitting stage‡194.60.679187.20.646

‡ Our implementation (code not released).

W2 / Q1: Ablation of the prompting strategies.

We ran it on the same 300 rendered views, with the same image-editing model and the same number of prompts per setting, measuring pose diversity (APD) and interaction diversity (entropy of the body parts in contact, singly and in combination). Simple prompt ("Add a person to this image") isolates the contribution of the image model.

PromptingAPD ↑Part entropy ↑Pattern entropy ↑
Simple Prompt0.1910.7490.248
PoseScript only0.3760.7750.336
Scene affordance0.3280.8440.382
+ action0.3390.8680.402
+ action + PoseScript (ours)0.3620.8540.389

The simple prompt gives far less diversity compared to our prompting. Our prompting nearly doubles pose diversity (0.191 to 0.362) and increases both interaction measures. The components are complementary: PoseScript prompts give the most varied poses but less varied interactions than our affordance prompts, since they do not target scene parts; affordance and action prompts supply the interactions; the full setting combines both.

W3 / Q2: Independent validation of the lifting; hand and body contacts.

We evaluate the lifting stage on RICH and PROX with metrics computed against their GT bodies and contacts. We measure MPJPE in mm, scene penetration as percentage of body vertices 2-15cm inside the scene and contact F1 score.

RICH MPJPE ↓RICH Pen. ↓PROX MPJPE ↓PROX Pen. ↓
PhySIC471.65.62%218.53.61%
Ours118.81.05%100.41.61%
GT bodies–0.06%–1.38%

MPJPE in mm; Pen.: body vertices 2–15 cm inside the scene. Contact F1 against GT, by body part (overall: 0.771 on RICH, 0.742 on PROX):

Contact F1 ↑RICH bodyRICH handsRICH feetPROX bodyPROX handsPROX feet
DECO (image only)0.4280.3290.6280.5270.2130.775
PhySIC0.6230.7790.4730.4790.5080.369
Ours0.7710.8220.7500.7420.6450.851

Our contacts are the most accurate for every body part on both benchmarks, ahead of PhySIC and of DECO, the widely used image-based contact estimator whose prediction our fit starts from.

Q3: Rejections per quality-control filter.

Of 78,111 generated images, 65,229 (83.5%) are kept, 12,361 (15.8%) are rejected and 521 (0.7%) fail in processing. Among the rejected images (2,484 fail more than one filter, so the shares exceed 100%):

FilterRemovesShare of rejected images
Visibilitystrongly truncated people58.6%
Depth consistencyscene changes near the person37.2%
Volumeimplausibly small bodies26.4%

By interaction category, assigned from the prompt of every generated image:

InteractionShare of generatedShare of keptRejected
Hand interaction (reach, touch, open, push)36%31%27%
Whole-body (sitting, lying, kneeling, leaning)30%32%13%
Several people20%21%9%
Pose description (PoseScript)12%14%3%
Other2%2%13%

Filtering leaves the interaction mix almost unchanged: every category has nearly the same share before and after. Whole-body, contact-rich interactions are not filtered out; they gain share. Hand interactions go from 36% to 31%, because for prompts about reaching or touching, the image editor more often shows only part of the person, and the visibility filter removes those images.

W4: Downstream comparisons differ in size and distribution.

We agree that the training sets differ in several respects at once. However, we can rule out size and domain.
Size does not explain the gain: a 22k subset of InHabitants, a quarter of PROX's 88k images, reaches contact F1 0.577 on PROX, against 0.492 for training on PROX itself (Tab. 7).
Neither does domain match, since on RICH, where neither training set is in-domain, GRAFT improves from 0.378 to 0.578 (Tab. 2).
But our evidence points towards diversity. At fixed size, restricting InHabitants to feet-only interactions drops F1 from 0.577 to 0.468, below PROX-trained GRAFT, and using a quarter of the scene views drops it to 0.546 (Tab. 7; see also W6).

Our experiments do not separate scene distribution from annotation type. We now state this in Suppl. F.

W5: GRAFT: worse V2S than PROX training.

Yes. The trade-off is confined to the in-domain case: on PROX, PROX-trained GRAFT keeps the better V2S (166.5 vs 184.4), while training on InHabitants gives better contact F1 (0.594 vs 0.492) and PA-MPJPE (49.7 vs 72.5). On RICH, which neither model was trained on, InHabitants is better on all three: V2S 224.7 vs 270.1, F1 0.578 vs 0.378, PA-MPJPE 46.3 vs 84.4.

W6: Interaction diversity ablation changes distribution of contact regions, poses, etc.

In this ablation, "interaction diversity" means the range of body–scene configurations: which body parts are in contact with the scene, and in which poses. Going from feet-only contact to all body parts widens exactly that range, so the change in contact regions and poses is precisely the factor we want to vary. The ablation does not separate the object categories involved. We now define the term in Suppl. F and limit the conclusion to it.

W7: Semantic evaluation may favour InHabit (own prompts, LLM pseudo-GT).

While the initial prompts come from the affordance-aware generation process, we do rewrite them such that they can be used by GenZI and POSA++ while keeping the described interaction. In addition, the physical contact accuracy of our lifting is now measured against ground truth as presented in W3.

AuthorsOpenReview ↗

We look forward to hearing whether this addresses your concerns

Dear Reviewer BEDr,

Thank you again for your review. In our global response, we clarify our contribution as a scalable dataset pipeline, and how we improve upon prior work.

We further show on benchmarks that our lifting outperforms prior formulations by a large margin, show the effectiveness of our prompting strategy, and report that our quality filter does not significantly alter the interaction distribution.

We would be glad to hear whether these address your concerns and are happy to clarify further questions before the end of the discussion.

Best Regards, The authors.

Reviewer_yGgcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents InHabit, an automated pipeline for generating 3D human–scene interaction data in known scenes. A VLM proposes activities, an image-editing model inserts humans into rendered views, and scene-constrained optimization fits SMPL-X bodies using keypoints, estimated depth, and contact cues. The resulting dataset, InHabitants, contains approximately 78K images across 6.6K rooms. The authors evaluate placement quality against GenZI and POSA++, and demonstrate the dataset’s utility for human–scene reconstruction, contact estimation, and scene-conditioned human generation.

Critical strengths
  1. The pipeline produces paired images, human meshes, and scene geometry across substantially diverse environments than existing captured HSI datasets, without new per-instance HSI annotation. The visual results show varied, contextually appropriate interactions. The interactive 3D user study, with 78.1% preference over the included baselines, provides useful evidence of placement quality.

  2. The downstream experiments demonstrate practical value beyond the visual examples. Training GRAFT on InHabitants improves PROX contact F1 from 0.492 to 0.594 compared with training on PROX, and the contact-estimation and scene-conditioned generation experiments provide additional evidence of useful supervision. These results support the dataset contribution despite the limited methodological novelty.

Critical weaknesses
  1. The methodological contribution is insufficiently distinguished from closely related work. GenZI already combines scene-conditioned image generation with 3D human fitting. Missing relevant work includes GenHSI (Li et al., 2026), which uses image inpainting, HMR initialization, contact-aware placement, and explicit human-scale optimization. Its static 3D keyframe generation is directly relevant despite the final video output. FunHSI (Liu et al., 2026) also combines Gemini-based image synthesis, functional contact reasoning, and body refinement, and should be discussed. Together with the standard fitting objective inherited from PROX/PhySIC, these precedents leave the specific methodological advance unclear. The description of GenZI in Sec. 3.1 as lacking visual scene understanding should also be corrected.

  2. The experiments do not establish how much of the improvement comes from the proposed procedure versus its pretrained components. The placement comparisons change both the image generator and the lifting method. Tab. 6 demonstrates the benefits of explicit scale and complete geometry within InHabit, but does not compare against existing contact-aware fitting with the same generated images, initialization, and geometric inputs. Another relevant missing baseline is SAM 3D Body (Yang et al., 2026), using the known camera intrinsics and global alignment to the scene, followed separately by full refinement. This would test whether the proposed fitting remains consequential with a strong contemporary HMR model. The downstream gains establish data utility, but leave this attribution unresolved.

Questions for authors
  1. What specific capability or improvement distinguishes InHabit from GenHSI and FunHSI under comparable scene inputs? Can the authors provide a comparison that controls the image-generation model, or fixes the generated images and reconstruction cues when comparing fitting procedures?

  2. On the same generated images, how does SAM 3D Body with known intrinsics and global scene alignment compare with the current pipeline? How much additional improvement in contact correctness and penetration does the full refinement provide with this initialization?

4 comments

Thank you for the precise review. We now cite and discuss both GenHSI and FunHSI in the paper.

W1 / Q1: What distinguishes InHabit from GenHSI and FunHSI?

The aim of InHabit is to generate HSI data with 3D bodies interacting in 3D scenes with a corresponding RGB image. This requires accurate alignment between images and lifted 3D bodies, which we achieve with two design choices:

  • Complete scan. With single-image estimated depth (GenHSI's kind of input), our own fit degrades from MPJPE 118.8 to 405.5 mm on RICH and from 100.4 to 154.6 on PROX; with single-view measured depth, to 168.0 and 147.1.
  • Full fit to image and scene. A rigid fit with frozen pose, as in GenHSI, is clearly worse (table below).

How the methods differ:

  • GenHSI works from one image: estimated depth, plus one object completed by an image-to-3D model. It optimises scale, translation and global rotation, but keeps the HMR pose and shape fixed, resulting in inaccurate contacts due to local disagreements between the scene and estimated human pose.
  • FunHSI needs several posed RGB-D views of the scene. Its body is initialised from the generated image, but the optimisation that follows does not enforce image fidelity, so the body may drift from the image. This can be observed in the FunHSI paper in their Figure 9 (feet visibly drift from image configuration; the optimisation changes its contact distance from 0.61 to 0.18, its Tab. 2).
  • InHabit refines pose, shape, translation and scale with 2D keypoint and depth terms against the complete scan, so the body stays tied to the image it is paired with, as image-based training data requires.

Q1 (cont.): Fix the images and cues when comparing fitting procedures

The rows below fix images, cues, scene and initial body (NLF), and vary only the fitting:

RICH MPJPE ↓RICH F1 ↑RICH Pen. ↓PROX MPJPE ↓PROX F1 ↑PROX Pen. ↓
GenHSI-style194.60.6792.94%187.20.6462.74%
Ours118.80.7711.05%100.40.7421.61%

Results: Our full fit outperforms the GenHSI-style fit on all three metrics (table above), with the same complete scan. GenHSI-style is our implementation of GenHSI's fitting stage on these inputs (code not released; details in Suppl. J.1). FunHSI cannot be applied to these inputs, since its contact term depends on its own VLM-derived contact graph and scene segmentation. These are real images, where image and scene agree. For the generated data, where they can disagree, see the error rates under 55DP.

Ablating explicit scale: The explicit scale factor matters when the initial body size is off. On RICH, fitting without the per-human scale raises MPJPE from 118.8 to 129.8 mm and penetration from 1.05% to 1.53%, with contact F1 unchanged. On PROX the initial sizes are already close. In the released data, the fitted scales deviate from 1 as much as on RICH and more than twice as much as on PROX.

W1: The description of GenZI in Sec. 3.1. should be corrected

You are right; we have corrected it in Sec. 3.1. GenZI inpaints humans into 16 rendered views independently and fits one body (pose and shape) to their consensus against the scene. The views need not agree, so no single image matches that body. InHabit fits each body to one edited view, so every body comes with the image it was fitted to, and it needs no user-given prompt or 3D location. It is also cheaper per sample: about 25 GPU-seconds of lifting on an RTX 2080, against about 3 minutes on an A100 reported for GenZI's fitting.

W2 / Q2: Fitting baseline: SAM 3D Body with known intrinsics and global alignment, then full refinement.

We ran this on RICH and PROX, where pose and contact can be checked against ground truth and compare it with NLF as the initial model:

MethodRICH MPJPE ↓RICH F1 ↑RICH Pen. ↓PROX MPJPE ↓PROX F1 ↑PROX Pen. ↓
Align SAM 3D Body182.00.6283.89%136.80.6852.92%
+ refine123.30.7481.07%97.30.7411.50%
Align NLF171.60.6643.56%138.10.6982.80%
+ refine (Ours)118.80.7711.05%100.40.7421.61%

Both SAM 3D Body and NLF predictions benefit from the refinement step, improving MPJPE, contact F1 and penetrations, showing the value of the refinement even with a strong contemporary HMR model.

AuthorsOpenReview ↗

We look forward to hearing whether this addresses your concerns

Dear Reviewer yGgc,

We thank you again for your review. In our global response we clarify that our contribution is a scalable recipe for generating human-scene interaction data and the resulting dataset, and how we differ from and improve upon prior work.

In the reply above, we further show that in a matched-input setting, our formulation clearly outperforms GenHSI-style optimization on PROX and RICH, and that our refinement is effective even over aligned SAM 3D Body.

We would be glad to know whether this addresses your concerns, and are happy to clarify any further questions before the end of the discussion period.

Best Regards, The authors.

Thank you for the response and the additional experiments. The controlled comparison against the GenHSI-style fitting and the evaluation with SAM 3D Body address my concern in W2, showing that the refinement stage brings clear gains even with a stronger HMR initialization. I also appreciate the added discussion of GenHSI and FunHSI, as well as the correction regarding GenZI.

However, my main concern remains the limited novelty of the overall approach. While the resulting dataset is useful for downstream tasks, the pipeline itself, i.e. combining VLM prompting, 2D inpainting, and scene-constrained SMPL-X fitting on existing 3D scan, is still largely an incremental combination of existing paradigms. Therefore, I will maintain my original rating.

Dear Reviewer yGgc,

Thank you for the reply, and for confirming that the new comparisons address W2 and that the dataset is useful for downstream tasks.

On novelty: our main contribution is the dataset. We show that with existing components, we are able to generate HSI data automatic at scale and improve state-of-the-art models. To our knowledge, no prior pipeline of this kind has been shown to produce data that advances state-of-the-art models, which ours does on three tasks. As with BEDLAM, BEDLAM 2.0 and AnyHand, we believe this is the measure for such a contribution.

Thank you again for your feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response by Authors

We thank the reviewers for their thoughtful feedback. We are encouraged that they find the problem important and the approach well motivated (BEDr, 55DP), the generated interactions varied and contextually appropriate (yGgc), and the downstream results a strength (all three).

Our contribution is a recipe to generate human–scene interaction data automatically at scale (78k generated images lifted in about 19 h on 32 consumer GPUs), the resulting dataset, and extensive evaluation on downstream tasks. InHabit is distinct from GenHSI, GenZI, FunHSI and PhySIC in these three aspects (per-method details under yGgc and BEDr):

  • Automatic at scale. No human input is required: a VLM proposes the interactions and based on that the image editing model inpaints the human.
  • Faithful to the image. Pose and shape are fitted to one image against the complete scene, so every body comes with the image it was fitted to. GenHSI keeps the estimated pose fixed, FunHSI's optimisation does not enforce image fidelity, and GenZI fits one body to a consensus of 16 views.
  • More accurate. On real images with GT bodies, our lifting outperforms PhySIC and our implementation of GenHSI's fitting stage on all three metrics (table below).

Together, these make generated images usable as training data for 3D human–scene interaction, which to our knowledge has not been shown before.

As with dataset papers such as BEDLAM (CVPR 2023), BEDLAM 2.0 (NeurIPS 2025) and AnyHand (NeurIPS 2026), we think the test of such data is whether training on it helps. Training on InHabitants advances state-of-the-art methods on three tasks. It is particularly effective for human-scene reconstruction: GRAFT trained on it reaches contact F1 0.594 on PROX, against 0.492 when trained on PROX itself, and adding it to BEDLAM improves Human3R on every metric (Tab. 2).

Quality of the lifted labels. The quality of our data is shown by the results on downstream tasks. We also measure dataset-wide body-scene penetration (1.78% of body vertices 2–15 cm inside the scene; see 55DP). Finally, we evaluate our lifting stage separately on real images from RICH and PROX, against GT scans, bodies and contact. The NLF rows share images, cues, scene and initial body (NLF is our pipeline's HMR model); only the fitting differs.

RICH MPJPE ↓RICH F1 ↑RICH Pen. ↓PROX MPJPE ↓PROX F1 ↑PROX Pen. ↓
PhySIC471.60.6235.62%218.50.4793.61%
NLF + alignment171.60.6643.56%138.10.6982.80%
NLF + GenHSI-style stage‡194.60.6792.94%187.20.6462.74%
NLF + ours118.80.7711.05%100.40.7421.61%
GT bodies0.06%1.38%
  • MPJPE in mm. F1: contact against GT contact. Pen.: body vertices 2–15 cm inside the scene (deeper values are unreliable, as the scans are not watertight). Alignment: translation only. Ours: full refinement.
  • PhySIC is run in its original setting, with its own initialisation and estimated scene.
  • ‡ Our implementation of GenHSI's fitting stage on these inputs (code not released).

Compared with alignment alone, our refinement lowers MPJPE by 31% (RICH) and 27% (PROX), raises contact F1 by 0.107 and 0.044, and cuts penetration by about 70% and over 40%, on PROX to near the GT bodies themselves. PA-MPJPE (not shown) is up to 1.9 mm higher for ours than for the alignment and ‡ rows, which keep the initial pose. The gains are similar with SAM 3D Body as the initial model (see yGgc).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.