- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper propose a novel benchmarking dataset for robotics tasks that is generated using real world grounding. The paper emphasized that it "preserved layouts, furniture, everyday belongings, and articulated fixtures of individual lived-in homes, retaining naturally occurring spatial constraints that jointly affect navigation and manipulation". And this real world scene structure is the selling point of such a dataset.
The method used for constructing the benchmark is quite timely as it takes advantage of the recent improvement in agentic AI (with the release of Astra) to to automate many parts of the benchmark creation process that would have made this effort infeasible earlier. The paper described how such an agentic system can work, what tools it needs, and the human feedback still needed to ensure the reconstruction is of a high quality.
Finally, the paper ran baseline methods on their proposed benchmark to show that it is still challenging to the current SoTA methods for both manipulation and navigation tasks.
Critical strengths
The paper took good advantage of the recent progress in agentic AI to create a high quality dataset that prioritizes fidelity to real life environments. The quality of the dataset is convincing and the experiments done to create an initial baselines across different task is much appreciated.
Overall, the contribution claims are well supported and this dataset could be valuable for the robotics community
Critical weaknesses
The main weakness in the paper is that it did not demonstrate why object density and the " real world scene structure" captured matters for robotics benchmark. The paper did not evaluate against existing benchmarks and datasets (e.g. InternScenes) to show what analysis or insight can be derived that is not possible with existing data. For example, the claim on ln 430 about how GPT-6 Astra make unintended contact on the way to reach a cup could support the above stated intuition, but only if this type of failures would not be captured in SR metric in e.g. InternScenes.
In general, the lack of comparison to existing benchmark and in depth analysis on what is the actual gap (rather than conceptual) in available metrics/datasets does not make the motivation very convincing.
Questions for authors
Is there any evidence or argument that supports the intuition that more object density and the " real world scene structure" is something that holds the field back in making progress in the robotics domain in existing literature?
For future work, please evaluate also on existing benchmarks and show what insight is not available without using the dataset being contributed.
We thank the reviewer for the constructive comments and suggestions. We take these concerns seriously and have carefully addressed them through additional analysis, as detailed below.
We clarify why real-world scene structure and object density matter for robotics evaluation, and demonstrate the value of LIVIN as a benchmark through comparisons with existing datasets. As noted in Internscenes, "The advancement of Embodied AI heavily relies on large-scale, simulatable 3D scene datasets characterized by scene diversity and realistic layouts". MetaScenes similarly emphasizes the importance of high-quality 3D scenes for ``sim-to-real transfer''. These works support the motivation for preserving real-world geometry and object arrangements in simulatable environments.
However, as discussed in our response to Reviewer uhLi, MetaScenes and InternScenes-Real2Sim rely heavily on asset replacement, which may introduce geometric discrepancies, while LIVIN focuses on faithful reconstruction of instance-specific geometry and real household layouts.
Furthermore, neither MetaScenes nor InternScenes evaluates humanoid navigation or loco-manipulation, whereas LIVIN examines these tasks under the spatial constraints of real household environments, better reflecting the locomotion demands of human activities in everyday homes.
LIVIN also preserves architectural details, including 45 doorway thresholds that require humanoid robots to perform stepping motions beyond planar navigation. An example is shown in the right panel of Fig.3. Such threshold-crossing challenges are not evaluated in MetaScenes or InternScenes.
High object density is common in lived-in homes, where furniture and everyday belongings often create narrow passages that constrain robot movement. Although InternScenes identifies narrow passages as an embodiment-dependent challenge, LIVIN provides a more detailed analysis of this issue using a humanoid robot. In Appendix E.3, we show that switching to CAT, a collision-aware humanoid traversal controller, allows the robot to navigate narrow passages requiring sideways movement that the original GR00T controller cannot traverse. As also discussed in our response to Reviewer uhLi, LIVIN has a higher object density than InternScenes and MetaScenes, and our cross-dataset evaluation shows substantially more navigation collisions on LIVIN (see the additional experiments in Appendix E.4 of the revised manuscript). These results highlight the importance of embodiment-aware control for humanoid navigation in real household environments. They also provide insights for future work on developing navigation policies that explicitly account for the robot’s body geometry and locomotion capabilities when traversing lived-in homes.