- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces ESI-Bench, a benchmark for embodied spatial intelligence that evaluates whether agents can actively acquire evidence through perception, locomotion, and manipulation to resolve spatial uncertainty. Built in OmniGibson using BEHAVIOR-1K assets, it contains 3,081 task instances across 10 categories and 29 subcategories, covering geometry, occlusion, containment, physical dynamics, reflection, counting, navigation, and temporal reasoning. The main contributions are its broad task taxonomy, interactive evaluation protocol, and diagnostic comparisons between passive single-view, passive multi-view, active exploration, and expert-trajectory observations. Task construction combines LLM-generated proposals, simulator-based verification, and human review. Experiments evaluate multiple multimodal models, 3D-augmented baselines, and human participants. The reported results suggest that task-directed exploration generally improves over random passive observations, while substantial gaps remain in evidence acquisition and action selection. Additional analyses examine reconstruction errors, premature commitment, and differences between human and model exploration strategies.
Critical strengths
Broad coverage organized around a consequential evaluation gap. ESI-Bench evaluates whether agents can choose actions that reveal task-relevant evidence, extending spatial evaluation beyond reasoning over fixed observations. Its 10 categories and 29 subcategories cover diverse uncertainties—including occlusion, containment, reflection, geometry, and physical dynamics—that require different combinations of observation, movement, and manipulation. This breadth makes it a valuable testbed for systematically assessing a foundational component of embodied intelligence. Diagnostic comparisons that make failures more informative. Comparing autonomous exploration with passive observations and expert-trajectory observations helps distinguish difficulty acquiring useful evidence from difficulty interpreting that evidence. The substantial active-to-expert gaps on several tasks suggest that evidence acquisition deserves attention alongside perceptual reasoning. This diagnostic value strongly supports acceptance: the benchmark can guide the development of exploration strategies rather than merely provide another aggregate ranking.
Critical weaknesses
- Inconsistent reporting of experimental evidence. Appendix G and the main text discuss GPT-6-based 3D results as completed experiments, whereas the caption of Table 11 explicitly describes these results as “hypothetical planning values, not experimental measurements.” Consequently, the conclusions supported by this comparison cannot currently be treated as empirically validated. This discrepancy requires clarification and correction because reliable experimental reporting is essential to assessing the benchmark’s findings.
- Limited validation of applicability beyond simulation. All evaluations use simulated environments and high-level discrete actions. The authors reasonably justify this setting through access to accurate ground truth and controlled comparisons, but provide no real-world validation showing whether model rankings, exploration benefits, or failure patterns persist. This matters particularly for tasks involving transparency, reflection, contact, and physical interaction, where rendering and execution conditions can affect which actions provide useful evidence. This limitation does not invalidate the controlled benchmark, but constrains the broader conclusions about embodied spatial intelligence.
Questions for authors
- Clarification of experimental status. Appendix G and the main text describe the GPT-6 3D comparison as an experiment, but Table 11 labels its results as hypothetical planning values rather than experimental measurements. Which results were actually measured, and which conclusions remain supported after excluding hypothetical values? Please clarify whether this issue is confined to Table 11.
- Clarification of validation within simulation. For transparency, reflection, contact, and liquid-capacity tasks, what checks establish consistency between the rendered observations, simulator-derived labels, and high-level action outcomes? In particular, does human verification explicitly check that the decisive physical relation is visually recoverable through the available actions? Existing audit results or representative verified trajectories would help assess whether these tasks reliably measure evidence acquisition under the stated simulation abstraction.
3 comments
Response to Reviewer bJyT (2/2)
Q2.2: Clarification of validation within simulation. Human Audit & Verificaction.
We thank the reviewer for this question. We clarify the consistency among rendered observations, simulation states and action outcomes below. We further add our human audit on these perspectives. Our supplementary already provides detailed task-specific validation procedures (Appendix P), human verification protocols and quantitative audit results (Appendix F.1–F.3, Tables 4–7), and additional qualitative examples (Appendices D–E) that directly address some of these concerns.
Simulator-level validation. We explicitly cross-check simulator-derived physical states against rendered RGB observations and segmentation masks/pixels, ensuring that ground-truth labels are supported by observable visual evidence and that high-level action outcomes are consistent with both, which are further examined by humans.
- Physical Contact: We obtain object positions and bounding boxes from the simulator to determine whether objects are touching. We then examine rendered instance-segmentation masks, checking pixel adjacency for contact and visible separation for non-contact cases, to verify consistency between the physical relation and visual evidence.
- Material Transparency: We use simulator
Insidestates and 3D bounding boxes to determine whether an object is contained within a transparent container. We cross-check these states against rendered observations and object visibility, including views acquired after camera movement, to verify that the inside/outside relationship is visually distinguishable. - Liquid Volume: We obtain particle-based liquid volumes and container capacities from the simulator. Following pouring actions, we check the resulting particle transfer or overflow states and compare them with rendered observations to ensure that the visible liquid behavior supports the capacity label.
- Specular Reflection: We derive object positions, mirror geometry, and reflection visibility from simulator states and geometric ray calculations. We then compare predicted reflected-object visibility with rendered observations, checking whether objects expected to appear in the mirror are visually observable and consistent with occlusion constraints.
Human Verification and Consistency Audit. Following the simulator-level checks described above, we also examine our existing human verification records. We assess whether simulator-derived states agree with rendered observations, whether high-level actions produce consistent outcomes, and whether the ground-truth trajectories provide sufficient visual evidence to recover the correct answers. The results are summarized below.
| Subcategory | Instances Audited | Simulator–Visual Consistency (%) | Action-Outcome Consistency (%) | GT Evidence Recoverability (%) |
|---|---|---|---|---|
| Material Transparency | 50 | 94 | 100 | 96 |
| Specular Reflection | 50 | 92 | 98 | 94 |
| Physical Contact | 50 | 98 | 100 | 100 |
| Liquid Volume | 50 | 94 | 94 | 100 |
These results complement our automated validation procedures, providing additional evidence that simulator-derived labels, rendered observations, and action outcomes are mostly consistent.
Additional existing audit / verfication results in the original draft. Three independent human annotators assess correctness, answerability, and non-triviality using rendered trajectories and simulator metadata (Appendix F.1). Importantly, answerability explicitly requires that the decisive physical relation be visually recoverable through available actions within the step budget. Invalid, ambiguous, or unobservable instances are rejected. Appendix F.2–F.3 (Tables 4–7) reports inter-annotator agreement, rejection statistics, and quantitative verification scores. Answerability reaches 90.5% for Physical Structure, 92.1% for Specular Reflection, and 91.4% for Spatial Relations among simulator-filtered task proposals.
Simulation-to-real consistency. ESI-Bench builds upon BEHAVIOR-1K and OmniGibson, which provide highly photorealistic environments and physically interactive objects. Although simulation cannot perfectly reproduce real-world physics, our additional real-world evaluations discussed above exhibit similar challenges in active evidence acquisition.
Thank you again for your thoughtful questions that have helped us clarify key aspects of our work! We hope our detailed responses have addressed your concerns, and therefore have reinforced your confidence in our work. If so, would you kindly consider raising your score? If you have any additional questions, we would be happy to address them during the rebuttal window. Thanks again!
Follow-Up on Rebuttal
Dear Reviewer,
Thanks again for your suggestions to strengthen this work! As the rebuttal period is approaching the end soon, we want to know if our response has answered your questions and addressed your concerns. If no, we are more than happy to provide further modifications. If yes, would you kindly consider raising the score?
Thanks again for your truly constructive and insightful feedback.
Best, Authors
Response to Reviewer bJyT (1/2)
We appreciate the positive and constructive comments from you, which are crucial for improving the paper! We have carried out additional experiments, including one real-world robot experiment to resolve your concerns.
Since GPT-6 was released shortly before the submission deadline, so we first ran a pilot on a small subset of tasks and used it to estimate and fill in placeholder values for Table 11. Thus, the original draft indicates final results will be updated when the full-scale run is complete. However, before submission, we completed all experiments and replaced the small-scale run numbers with actual measurements from the full evaluation, but inadvertently forgot to update the table caption. We confirm that all reported results are experimentally measured and valid, and this issue is confined to the outdated Table 11 caption. All results reported in the paper come from full evaluation runs, and no conclusion in the paper relies on pilot or estimated values. All experimental code and evaluation scripts are available and reproducible at our github link.
We thank the reviewer for raising this important concern. We conducted additional real-world robotic experiments across seven ESI-Bench subcategories: Spatial Distance, Geometric Configuration, and Counting with Occlusion, Transparency, Physical Contact, Physical Containment and Inclined Plane (20 instances each). The experimental setup and qualitative demonstrations are available at https://anonymous.4open.science/w/esi-bench2-0/#real-world-experiments.
Real-World Experimental Setup. For Spatial Distance, Geometric Configuration, and Counting with Occlusion, Transparency, Physical Contact, we use the I2RT Linear Bot, a mobile robotic platform supporting omnidirectional movement, vertical adjustment, and manipulation. For Physical Containment and Inclined Plane, we use Franka arm. We adopt the same high-level action space as ESI-Bench, including movement (
move_forward,move_backward,move_left,move_right,move_up,move_down,pick up,put down,pour,turn left,turn right,turn up,turn down). This allows us to evaluate the same embodied spatial reasoning capabilities in physical environments.Quantitative & Qualitative Results. We report the performance across the seven real-world subcategories below, alongside their corresponding simulation results.
You can also find qualitative results here https://anonymous.4open.science/w/esi-bench2-0/#real-world-experiments.
Consistent Findings Across Simulation and the Real World. Quantitatively, our real-world experiments exhibit trends consistent with simulation. Passive Single-View and Active Exploration are generally more challenging in the real world due to increased visual complexity, environmental variability, and imperfect physical execution. Interestingly, GT Multi-View sometimes achieves higher accuracy in the real world, potentially because real-world observations avoid certain simulation rendering artifacts and provide clearer visual evidence. Nevertheless, the substantial performance gap between Active Exploration and GT Multi-View remains highly consistent across simulation and the real world. Our qualitative examples reveal failure patterns consistent with our simulation findings. Models frequently perform extensive exploration but fail to acquire sufficient spatial evidence, committing prematurely to incorrect answers with high confidence. These observations, together with the consistent accuracy trends, suggest that the fundamental bottleneck is not specific to simulated environments, but lies in the models' inability to effectively close the perception–action loop. Therefore, our simulation environment provides a reproducible and controlled testbed that also captures key challenges of real-world embodied spatial intelligence, enabling systematic evaluation and diagnosis of perception–action failures without the cost and variability of physical experiments.
We will expand our real-world evaluations to additional spatial categories and robotic platforms.