acceptodds
Withdrawn from ICLR 2027

LIVIN: Benchmarking Spatial and Embodied Intelligence in Digital Twins of Lived-In Homes

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Realistic household simulation must capture not only diverse environments but also the lived-in object arrangements and spatial constraints that shape robot motion and interaction. Existing resources often trade off scale, real-world correspondence, and interaction readiness, leaving a gap in faithful, interactive replicas of how real homes are actually arranged. To this end, we introduce LIVIN, a benchmark for embodied interaction and 3D perception built on digital twins of 30 diverse lived-in homes. These replicas preserve observed room layouts, furniture configurations, and everyday belongings. To construct them, we design a human-in-the-loop workflow comprising instance recognition, architectural reconstruction, and object generation and placement, with intermediate results reviewed and corrected by humans against the source observations at each stage. We evaluate four tasks in LIVIN: 3D detection, 3D reconstruction, navigation, and loco-manipulation. Our evaluations show that current methods remain challenged by the dense object arrangements, occlusions, limited free space, and constrained interaction regions found in realistic lived-in homes. We hope LIVIN will help advance embodied AI in real-world homes, from 3D scene understanding to robotic interaction, and ultimately bring embodied intelligence into everyday home environments.

Primary area:
datasets and benchmarks
Keywords:
Household Digital Twins, 3D Perception, Humanoid Navigation, Loco-Manipulation, Embodied AI Benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iz3POpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper propose a novel benchmarking dataset for robotics tasks that is generated using real world grounding. The paper emphasized that it "preserved layouts, furniture, everyday belongings, and articulated fixtures of individual lived-in homes, retaining naturally occurring spatial constraints that jointly affect navigation and manipulation". And this real world scene structure is the selling point of such a dataset.

The method used for constructing the benchmark is quite timely as it takes advantage of the recent improvement in agentic AI (with the release of Astra) to to automate many parts of the benchmark creation process that would have made this effort infeasible earlier. The paper described how such an agentic system can work, what tools it needs, and the human feedback still needed to ensure the reconstruction is of a high quality.

Finally, the paper ran baseline methods on their proposed benchmark to show that it is still challenging to the current SoTA methods for both manipulation and navigation tasks.

Critical strengths

The paper took good advantage of the recent progress in agentic AI to create a high quality dataset that prioritizes fidelity to real life environments. The quality of the dataset is convincing and the experiments done to create an initial baselines across different task is much appreciated.

Overall, the contribution claims are well supported and this dataset could be valuable for the robotics community

Critical weaknesses

The main weakness in the paper is that it did not demonstrate why object density and the " real world scene structure" captured matters for robotics benchmark. The paper did not evaluate against existing benchmarks and datasets (e.g. InternScenes) to show what analysis or insight can be derived that is not possible with existing data. For example, the claim on ln 430 about how GPT-6 Astra make unintended contact on the way to reach a cup could support the above stated intuition, but only if this type of failures would not be captured in SR metric in e.g. InternScenes.

In general, the lack of comparison to existing benchmark and in depth analysis on what is the actual gap (rather than conceptual) in available metrics/datasets does not make the motivation very convincing.

Questions for authors

Is there any evidence or argument that supports the intuition that more object density and the " real world scene structure" is something that holds the field back in making progress in the robotics domain in existing literature?

For future work, please evaluate also on existing benchmarks and show what insight is not available without using the dataset being contributed.

1 comment

We thank the reviewer for the constructive comments and suggestions. We take these concerns seriously and have carefully addressed them through additional analysis, as detailed below.

We clarify why real-world scene structure and object density matter for robotics evaluation, and demonstrate the value of LIVIN as a benchmark through comparisons with existing datasets. As noted in Internscenes, "The advancement of Embodied AI heavily relies on large-scale, simulatable 3D scene datasets characterized by scene diversity and realistic layouts". MetaScenes similarly emphasizes the importance of high-quality 3D scenes for ``sim-to-real transfer''. These works support the motivation for preserving real-world geometry and object arrangements in simulatable environments.

However, as discussed in our response to Reviewer uhLi, MetaScenes and InternScenes-Real2Sim rely heavily on asset replacement, which may introduce geometric discrepancies, while LIVIN focuses on faithful reconstruction of instance-specific geometry and real household layouts.

Furthermore, neither MetaScenes nor InternScenes evaluates humanoid navigation or loco-manipulation, whereas LIVIN examines these tasks under the spatial constraints of real household environments, better reflecting the locomotion demands of human activities in everyday homes.

LIVIN also preserves architectural details, including 45 doorway thresholds that require humanoid robots to perform stepping motions beyond planar navigation. An example is shown in the right panel of Fig.3. Such threshold-crossing challenges are not evaluated in MetaScenes or InternScenes.

High object density is common in lived-in homes, where furniture and everyday belongings often create narrow passages that constrain robot movement. Although InternScenes identifies narrow passages as an embodiment-dependent challenge, LIVIN provides a more detailed analysis of this issue using a humanoid robot. In Appendix E.3, we show that switching to CAT, a collision-aware humanoid traversal controller, allows the robot to navigate narrow passages requiring sideways movement that the original GR00T controller cannot traverse. As also discussed in our response to Reviewer uhLi, LIVIN has a higher object density than InternScenes and MetaScenes, and our cross-dataset evaluation shows substantially more navigation collisions on LIVIN (see the additional experiments in Appendix E.4 of the revised manuscript). These results highlight the importance of embodiment-aware control for humanoid navigation in real household environments. They also provide insights for future work on developing navigation policies that explicitly account for the robot’s body geometry and locomotion capabilities when traversing lived-in homes.

Reviewer_uhLiOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a benchmark for embodied tasks built on digital-twin scenes of real, inhabited homes. It contains 30 homes, 319 rooms, and 12,808 instances, including 1,559 articulated objects. It builds the virtual scenes from several sources of information: floor plans, posed multi-view images, and metric-scale scans. The pipeline then applies instance recognition, object generation and placement, and manual review. The paper also evaluates 3D detection, 3D reconstruction, robot navigation, and mobile manipulation. The main contribution is not a new algorithm. Instead, it is a resource of household scenes with dense everyday objects and interactive structures, used to evaluate various embodied tasks and existing methods. The experiments show that existing methods still have clear limitations in areas such as 3D reconstruction and navigation.

Critical strengths
  1. The paper focuses on scenes in real household environments, where occlusions and restricted manipulation are unavoidable. Unlike obstacles added by hand, these challenges arise naturally and reasonably. This motivation helps bridge the gap between idealized and real-world environments.
  2. The scene construction pipeline is fairly complete. It includes deduplication, collision detection, and manual correction, and the overall process is reasonable and well developed.
  3. The benchmark's evaluation is fairly comprehensive, covering 3D detection, 3D reconstruction, robot navigation, and mobile manipulation. The experiments also include multiple methods and metrics.
Critical weaknesses
  1. The paper does not show how digital-twin fidelity is ensured. The paper repeatedly states that the virtual scenes are faithful replicas of real-world scenes, and the pipeline relies heavily on real scans. However, it never reports how much the virtual scenes actually differ from the real ones. If that difference is large, it would undermine the core motivation behind Strength 1, which is grounding the benchmark in real scenes. It would also call into question the reliability of the evaluation results.
  2. The paper does not justify attributing failures to the complexity of real residential scenes. The paper attributes many failures of existing methods on specific tasks to the complexity of real scenes. However, complexity is only one property of the constructed scenes, so it cannot be concluded that complexity is the main factor affecting the results. Distortions introduced during scene construction, or inappropriate settings for integrating existing methods, could also substantially affect the final results.
  3. The core contribution is not sufficiently distinct. Several existing works, such as MetaScenes, InternScenes, and SceneSmith (some are cited in this paper), also start from real-world scenes. They propose methods for building simulated scenes and then evaluate downstream tasks on those scenes. Compared with these works, the core advantage of this paper is unclear.
  4. The scale and scalability are open to question. In the actual evaluation of existing methods, the navigation, mobile manipulation, and reconstruction experiments are too small in scale. This leads to considerable uncertainty in the reported results and raises doubts about how well these tasks scale.
Questions for authors
  1. The paper reports a questionnaire survey on scene quality but does not describe its setup. Could the authors explain how the survey was conducted, who the participants were, and whether the experimental settings were consistent and reasonable?
  2. Compared with other two-stage works that combine scene construction with downstream task validation, such as MetaScenes and InternScenes (cited in this paper), does this paper offer a more fundamental innovation?
  3. Could the authors provide further experiments showing how much the constructed scenes differ from the real-world scans, along with a deeper analysis of why the experiments failed?
2 comments

We thank the reviewer for the constructive comments and suggestions. We take these concerns seriously and have carefully addressed them through additional experiments and analysis, as detailed below.

Weakness 1, Question 3: Digital-Twin Fidelity.

We agree that quantitative comparisons with real-world observations are important for validating the fidelity of LIVIN. As also noted in our response to Reviewer xjQ9, we have added scene-level Chamfer Distance, depth MAE, and object penetration ratio to evaluate the reconstructed scenes across all 30 homes. The results show a CD of 0.134 m a depth MAE of 0.282 m, and a penetration ratio of 1.69%. These results have been added to the final paragraph of Appendix A.

Weakness 2, Question 3: Failure Attribution

As reported above, our fidelity evaluation shows that LIVIN scenes closely match their real-world counterparts. We further analyze the causes of failure through additional detection, reconstruction, and navigation experiments.

For detection, we further compare the original LIVIN scenes with simplified versions in which small household objects are removed. All original baselines show substantial performance gains on the simplified scenes. Fine-tuning EFM3D on LIVIN further improves recall on both the full and simplified scenes.

For reconstruction, we further compare the full evaluation set with a subset excluding instances with an occlusion ratio of at least 25%. All five reconstruction methods show higher reconstruction errors on the full set than on the filtered subset. Appearance metrics likewise improve on the filtered subset.

These results show that simplifying LIVIN scenes improves detection and reconstruction performance, confirming scene complexity as a major challenge. We have replaced the previous clutter analysis in Appendix D with these findings, which are now presented in Appendix D.2

For navigation, our failure analysis in Appendix E.3 shows that method-specific weaknesses, such as unrecovered blockage and goal-completion mismatches, contribute substantially to failures.

We further examine the impact of scene complexity through cross-dataset experiments. We evaluate the same methods on InternScenes and MetaScenes, which have lower object densities of 1.30 and 1.63 objects/m2\text{objects}/m^2, compared with 3.70 objects/m2\text{objects}/m^2 in LIVIN. We use identical robot and controller settings with comparable tasks across all datasets. Overall, LIVIN generally yields lower navigation success rates and consistently higher collision counts than the comparison datasets. These results provide further evidence that LIVIN's dense household arrangements contribute substantially to navigation difficulty. What's more, the better performance of the same methods on other datasets also suggests that our integration settings are generally appropriate. The additional experiments are reported in Appendix E.4 in the revision.

Weakness 3, Question 2: Core Novelty and Distinction from Prior Work

Unlike works such as SceneSmith, which generate plausible scenes from natural-language prompts, LIVIN reconstructs specific real homes from real-world observations, allowing each digital twin to be directly compared with its corresponding physical environment.

MetaScenes and InternScenes-Real2Sim rely substantially on asset selection and replacement, which may introduce geometric discrepancies from the original scans and alter passage clearances, reachability, and contact geometry. In contrast, LIVIN aims to faithfully reproduce individual objects in terms of geometry, scale, position, and orientation, while preserving the original layouts and articulated structures.

Regarding spatial scope, MetaScenes and InternScenes-Real2Sim only provide room-level or region-level replicas. LIVIN reconstructs complete real homes while preserving their original room connectivity, supporting cross-room tasks within the same household.

InternScenes-Synthetic includes multi-room environments, but these are designer-created rather than reconstructed from specific real-world reference scenes.

Weakness 4: Evaluation Scale and Scalability

We acknowledge that the current evaluation scale has some limitations. However, our evaluation is comparable in scale to several existing studies. For example, SIMPLE[1] reports 16 whole-body tasks across 50 scenes. To reduce evaluation variability, we evaluate each navigation and loco-manipulation task under three random seeds, following the multi-seed evaluation practice adopted in RoboDojo[2]. For 3D detection and reconstruction, we also run inference three times and report the average results. As high-fidelity digital twins of complete real homes remain limited, we focus on an initial evaluation of spatial understanding and humanoid embodied tasks in these environments. We will expand the dataset and task coverage in future work.

[1] Wei S, Ni Z, Liu J, Zhao Z, Ye J, Jing H, Xia J, Liu X, Leong M, Heng L, Huang D, Wang Y. SIMPLE: Simulation-Based Policy Learning and Evaluation for Humanoid Loco-manipulation. arXiv preprint arXiv:2606.08278. 2026 Jun 6.

[2] Chen T, Chen Y, Li Z, Tang J, Su K, Wan W, Chen B, Lu H, et al. RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies. arXiv preprint arXiv:2607.04434. 2026 Jul 5.

Question 1: Scene Quality Survey Setup

We conducted a survey involving 4848 participants, most of whom had a background in 3D vision. The questionnaire comprised 2020 single-choice questions, each presenting one indoor-scene image from each of the six datasets and asking participants to select the best image.

The questionnaire assessed three aspects of scene quality using the following questions:

  • Physical plausibility: “Which image shows the most physically plausible object placement?”
  • Realism: “Which scene most closely resembles a real, lived-in indoor environment?”
  • Visual detail quality: “Which scene has the best visual detail quality?”

To ensure comparable views, we first selected 2020 reference viewpoints from LIVIN. We then use GPT-6 Astra to identify candidate views with similar compositions in the other datasets, from which human reviewers selected the closest matches. All images were rendered with consistent ambient lighting, and each comparison set contained scenes of similar room types.

Ethics Concerns: We thank the reviewer for raising this concern. We obtained permission from the homeowners to use the existing source observations for constructing LIVIN. To protect privacy, we plan to release only the reconstructed 3D scenes after privacy screening. The original scans, reference images, and floorplans will not be released.

Reviewer_xjQ9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LIVIN introduces digital twins of 30 lived-in homes, containing 319 rooms, 12,808 object instances, and 1,559 articulated objects. Its human-in-the-loop construction workflow combines instance identification, architectural reconstruction, code-based modeling of articulated objects, and image-conditioned generation of assets, followed by human correction. The scenes support four evaluations: 3D detection, 3D reconstruction, humanoid navigation, and loco-manipulation. The paper compares scene statistics and human preferences against existing datasets and evaluates multiple released models without LIVIN-specific training. The results reveal persistent challenges in cluttered, spatially constrained homes, highlighting LIVIN's value as a benchmark for spatial and embodied intelligence.

Critical strengths
  • The paper presents a scene construction pipeline covering instance identification, architectural reconstruction, object generation and placement, articulation modeling, and human verification, with qualitatively strong results in detailed lived-in homes.

  • The comprehensive evaluations across 3D detection, reconstruction, humanoid navigation, and loco-manipulation demonstrate the scene data's usefulness for both spatial understanding and physical interaction.

Critical weaknesses

Although the paper reports subjective user-study evaluations, it lacks objective quantitative assessments of dataset construction quality, including physical plausibility, geometric accuracy, and rendered-image fidelity. In particular, systematic comparisons between the reconstructed scenes and the source observations (floorplans, metric scans, and posed multi-view images) are missing.

Questions for authors
  • Appendix F.4 states that physical properties are estimated by GPT-6 Astra and manually corrected. What evidence guides these corrections, and what validation and acceptance criteria establish the final scenes' physical validity and simulation readiness?

  • What human review time and computational and asset-generation resources are needed per home? Which stages rely most on human intervention, and what are the main obstacles to automating their checks and corrections efficiently?

1 comment

We thank the reviewer for the constructive comments and suggestions. We take these concerns seriously and have carefully addressed them through additional experiments and analysis, as detailed below.

Weakness 1: Lack of Quantitative Scene Fidelity Evaluation.

Regarding the quantitative evaluation of scene construction quality, we have added three metrics to assess the fidelity and reconstruction quality of LIVIN scenes. Specifically, we measure scene-level Chamfer Distance between reconstructed scenes and their corresponding metric scans, depth MAE between depth maps rendered from reconstructed scenes and reference scans under the same camera poses, and the object penetration ratio. Across all 30 homes, we obtain a CD of 0.134 m, a depth MAE of 0.282 m, and a penetration ratio of 1.69%.

For rendered-image fidelity, RGB-based metrics are sensitive to lighting, which our pipeline does not estimate. We therefore use depth MAE to assess geometric reconstruction accuracy in rendered 2D views.

Regarding floorplan fidelity, we construct the floor layouts directly from the reference floorplans at a 1:1 scale and keep them fixed during subsequent reconstruction stages. Thus, the original floorplan structure is preserved throughout the reconstruction process.

We have added these quantitative results and metric definitions to the final paragraph of Appendix A.

Question 1: Physical Property Validation.

Following ManiTwin[1], we initialize physical properties with GPT-6 Astra and manually review objects for physical plausibility, considering their geometry, material cues, and simulation results. Review feedback is consolidated to correct implausible estimates and refine the VLM prompts and filtering thresholds.

[1]Wang K, Chen T, Liu J, et al. ManiTwin: Scaling Data-Generation-Ready Digital Object Dataset to 100K[J]. arXiv preprint arXiv:2603.16866, 2026.

Question 2: Human Review, Construction Costs, and Automation Challenges.

For reviewers without prior experience in 3D vision, each home requires approximately 4 hours of human review.

We use four NVIDIA RTX 3090 GPUs to accelerate rendering for scene inspection and refinement. Each home requires approximately 393 Rodin-generated meshes, with 7% additional generations for corrections, costing approximately $315 per home.

Most human intervention occurs during non-articulated object generation and placement, as the large number of objects leads to frequent penetrations, floating geometry, and pose errors.

Automating human checks and corrections remains difficult for two reasons. First, GPT-6 Astra sometimes fails to identify incorrect object orientations when comparing rendered scenes with reference images. Second, geometric differences between generated and real objects can cause automatic collision correction to shift objects substantially from their observed positions. Human review is therefore needed to correct these errors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.