acceptodds
Under review as a conference paper at ICLR 2027

ESI-Bench: Towards Embodied Spatial Intelligence that Closes the Perception-Action Loop

Abstract

Spatial intelligence unfolds through a perception–action loop: agents act to acquire observations, and reason about how observations vary as a function of action. Rather than passively processing what is seen, they actively uncover what is unseen - occluded structure, dynamics, containment, and functionality that cannot be resolved from passive sensing alone. We move beyond prior formulations of spatial intelligence that assume oracle observations by recasting the observer as an actor. We introduce ESI-Bench, a comprehensive benchmark for embodied spatial intelligence spanning 10 task categories and 29 subcategories built on OmniGibson, grounded in Spelke’s core knowledge systems. Agents must decide what abilities to deploy — perception, locomotion, and manipulation — and how to sequence them to actively accumulate task-relevant evidence. We conduct extensive experiments on state-of-the-art MLLMs and find that active exploration substantially outperforms passive counterparts, with agents spontaneously discovering emergent spatial strategies without explicit instructions, while passive multi-view often adds noise rather than signal despite consuming far more images. Most failures stem not from weak perception but from action blindness: poor action choices lead to poor observations, which in turn drive cascading errors. While explicit 3D grounding stabilizes reasoning on depth-sensitive tasks, imperfect reconstruction proves more harmful than 2D baselines by distorting spatial relations. Human studies further reveal that unlike humans who seek falsifying viewpoints and revise beliefs under contradiction, models commit prematurely with high confidence regardless of evidence quality, exposing a metacognitive gap that neither better perception nor more embodied interaction alone can close.

Primary area:
datasets and benchmarks
Keywords:
spatial intelligence

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_bJyTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces ESI-Bench, a benchmark for embodied spatial intelligence that evaluates whether agents can actively acquire evidence through perception, locomotion, and manipulation to resolve spatial uncertainty. Built in OmniGibson using BEHAVIOR-1K assets, it contains 3,081 task instances across 10 categories and 29 subcategories, covering geometry, occlusion, containment, physical dynamics, reflection, counting, navigation, and temporal reasoning. The main contributions are its broad task taxonomy, interactive evaluation protocol, and diagnostic comparisons between passive single-view, passive multi-view, active exploration, and expert-trajectory observations. Task construction combines LLM-generated proposals, simulator-based verification, and human review. Experiments evaluate multiple multimodal models, 3D-augmented baselines, and human participants. The reported results suggest that task-directed exploration generally improves over random passive observations, while substantial gaps remain in evidence acquisition and action selection. Additional analyses examine reconstruction errors, premature commitment, and differences between human and model exploration strategies.

Critical strengths

Broad coverage organized around a consequential evaluation gap. ESI-Bench evaluates whether agents can choose actions that reveal task-relevant evidence, extending spatial evaluation beyond reasoning over fixed observations. Its 10 categories and 29 subcategories cover diverse uncertainties—including occlusion, containment, reflection, geometry, and physical dynamics—that require different combinations of observation, movement, and manipulation. This breadth makes it a valuable testbed for systematically assessing a foundational component of embodied intelligence. Diagnostic comparisons that make failures more informative. Comparing autonomous exploration with passive observations and expert-trajectory observations helps distinguish difficulty acquiring useful evidence from difficulty interpreting that evidence. The substantial active-to-expert gaps on several tasks suggest that evidence acquisition deserves attention alongside perceptual reasoning. This diagnostic value strongly supports acceptance: the benchmark can guide the development of exploration strategies rather than merely provide another aggregate ranking.

Critical weaknesses
  1. Inconsistent reporting of experimental evidence. Appendix G and the main text discuss GPT-6-based 3D results as completed experiments, whereas the caption of Table 11 explicitly describes these results as “hypothetical planning values, not experimental measurements.” Consequently, the conclusions supported by this comparison cannot currently be treated as empirically validated. This discrepancy requires clarification and correction because reliable experimental reporting is essential to assessing the benchmark’s findings.
  2. Limited validation of applicability beyond simulation. All evaluations use simulated environments and high-level discrete actions. The authors reasonably justify this setting through access to accurate ground truth and controlled comparisons, but provide no real-world validation showing whether model rankings, exploration benefits, or failure patterns persist. This matters particularly for tasks involving transparency, reflection, contact, and physical interaction, where rendering and execution conditions can affect which actions provide useful evidence. This limitation does not invalidate the controlled benchmark, but constrains the broader conclusions about embodied spatial intelligence.
Questions for authors
  1. Clarification of experimental status. Appendix G and the main text describe the GPT-6 3D comparison as an experiment, but Table 11 labels its results as hypothetical planning values rather than experimental measurements. Which results were actually measured, and which conclusions remain supported after excluding hypothetical values? Please clarify whether this issue is confined to Table 11.
  2. Clarification of validation within simulation. For transparency, reflection, contact, and liquid-capacity tasks, what checks establish consistency between the rendered observations, simulator-derived labels, and high-level action outcomes? In particular, does human verification explicitly check that the decisive physical relation is visually recoverable through the available actions? Existing audit results or representative verified trajectories would help assess whether these tasks reliably measure evidence acquisition under the stated simulation abstraction.
3 comments
AuthorsOpenReview ↗

Response to Reviewer bJyT (1/2)

We appreciate the positive and constructive comments from you, which are crucial for improving the paper! We have carried out additional experiments, including one real-world robot experiment to resolve your concerns.

 

W1&Q1: Clarification of experimentals.

Since GPT-6 was released shortly before the submission deadline, so we first ran a pilot on a small subset of tasks and used it to estimate and fill in placeholder values for Table 11. Thus, the original draft indicates final results will be updated when the full-scale run is complete. However, before submission, we completed all experiments and replaced the small-scale run numbers with actual measurements from the full evaluation, but inadvertently forgot to update the table caption. We confirm that all reported results are experimentally measured and valid, and this issue is confined to the outdated Table 11 caption. All results reported in the paper come from full evaluation runs, and no conclusion in the paper relies on pilot or estimated values. All experimental code and evaluation scripts are available and reproducible at our github link.

W2&Q2.1: Limited validation of applicability beyond simulation.

We thank the reviewer for raising this important concern. We conducted additional real-world robotic experiments across seven ESI-Bench subcategories: Spatial Distance, Geometric Configuration, and Counting with Occlusion, Transparency, Physical Contact, Physical Containment and Inclined Plane (20 instances each). The experimental setup and qualitative demonstrations are available at https://anonymous.4open.science/w/esi-bench2-0/#real-world-experiments.

Real-World Experimental Setup. For Spatial Distance, Geometric Configuration, and Counting with Occlusion, Transparency, Physical Contact, we use the I2RT Linear Bot, a mobile robotic platform supporting omnidirectional movement, vertical adjustment, and manipulation. For Physical Containment and Inclined Plane, we use Franka arm. We adopt the same high-level action space as ESI-Bench, including movement (move_forward, move_backward, move_left, move_right, move_up, move_down, pick up, put down, pour, turn left, turn right, turn up, turn down). This allows us to evaluate the same embodied spatial reasoning capabilities in physical environments.

Quantitative & Qualitative Results. We report the performance across the seven real-world subcategories below, alongside their corresponding simulation results.

SubcategorySim. Passive SingleSim. ActiveSim. GT Multi-ViewReal Passive SingleReal ActiveReal GT Multi-View
Spatial Distance66.466.483.655.060.085.0
Geometric Configuration31.623.057.220.025.055.0
Counting with Occlusion3.320.056.70.010.040.0
Transparency43.164.291.725.050.0100.0
Rigid Containment52.560.095.045.055.0100.0
Inclined Plane59.082.091.855.085.0100.0
Physical Contact52.570.090.045.075.090.0

You can also find qualitative results here https://anonymous.4open.science/w/esi-bench2-0/#real-world-experiments.

Consistent Findings Across Simulation and the Real World. Quantitatively, our real-world experiments exhibit trends consistent with simulation. Passive Single-View and Active Exploration are generally more challenging in the real world due to increased visual complexity, environmental variability, and imperfect physical execution. Interestingly, GT Multi-View sometimes achieves higher accuracy in the real world, potentially because real-world observations avoid certain simulation rendering artifacts and provide clearer visual evidence. Nevertheless, the substantial performance gap between Active Exploration and GT Multi-View remains highly consistent across simulation and the real world. Our qualitative examples reveal failure patterns consistent with our simulation findings. Models frequently perform extensive exploration but fail to acquire sufficient spatial evidence, committing prematurely to incorrect answers with high confidence. These observations, together with the consistent accuracy trends, suggest that the fundamental bottleneck is not specific to simulated environments, but lies in the models' inability to effectively close the perception–action loop. Therefore, our simulation environment provides a reproducible and controlled testbed that also captures key challenges of real-world embodied spatial intelligence, enabling systematic evaluation and diagnosis of perception–action failures without the cost and variability of physical experiments.

We will expand our real-world evaluations to additional spatial categories and robotic platforms.

AuthorsOpenReview ↗

Response to Reviewer bJyT (2/2)

Q2.2: Clarification of validation within simulation. Human Audit & Verificaction.

We thank the reviewer for this question. We clarify the consistency among rendered observations, simulation states and action outcomes below. We further add our human audit on these perspectives. Our supplementary already provides detailed task-specific validation procedures (Appendix P), human verification protocols and quantitative audit results (Appendix F.1–F.3, Tables 4–7), and additional qualitative examples (Appendices D–E) that directly address some of these concerns.

Simulator-level validation. We explicitly cross-check simulator-derived physical states against rendered RGB observations and segmentation masks/pixels, ensuring that ground-truth labels are supported by observable visual evidence and that high-level action outcomes are consistent with both, which are further examined by humans.

  • Physical Contact: We obtain object positions and bounding boxes from the simulator to determine whether objects are touching. We then examine rendered instance-segmentation masks, checking pixel adjacency for contact and visible separation for non-contact cases, to verify consistency between the physical relation and visual evidence.
  • Material Transparency: We use simulator Inside states and 3D bounding boxes to determine whether an object is contained within a transparent container. We cross-check these states against rendered observations and object visibility, including views acquired after camera movement, to verify that the inside/outside relationship is visually distinguishable.
  • Liquid Volume: We obtain particle-based liquid volumes and container capacities from the simulator. Following pouring actions, we check the resulting particle transfer or overflow states and compare them with rendered observations to ensure that the visible liquid behavior supports the capacity label.
  • Specular Reflection: We derive object positions, mirror geometry, and reflection visibility from simulator states and geometric ray calculations. We then compare predicted reflected-object visibility with rendered observations, checking whether objects expected to appear in the mirror are visually observable and consistent with occlusion constraints.

Human Verification and Consistency Audit. Following the simulator-level checks described above, we also examine our existing human verification records. We assess whether simulator-derived states agree with rendered observations, whether high-level actions produce consistent outcomes, and whether the ground-truth trajectories provide sufficient visual evidence to recover the correct answers. The results are summarized below.

SubcategoryInstances AuditedSimulator–Visual Consistency (%)Action-Outcome Consistency (%)GT Evidence Recoverability (%)
Material Transparency509410096
Specular Reflection50929894
Physical Contact5098100100
Liquid Volume509494100

These results complement our automated validation procedures, providing additional evidence that simulator-derived labels, rendered observations, and action outcomes are mostly consistent.

Additional existing audit / verfication results in the original draft. Three independent human annotators assess correctness, answerability, and non-triviality using rendered trajectories and simulator metadata (Appendix F.1). Importantly, answerability explicitly requires that the decisive physical relation be visually recoverable through available actions within the step budget. Invalid, ambiguous, or unobservable instances are rejected. Appendix F.2–F.3 (Tables 4–7) reports inter-annotator agreement, rejection statistics, and quantitative verification scores. Answerability reaches 90.5% for Physical Structure, 92.1% for Specular Reflection, and 91.4% for Spatial Relations among simulator-filtered task proposals.

Simulation-to-real consistency. ESI-Bench builds upon BEHAVIOR-1K and OmniGibson, which provide highly photorealistic environments and physically interactive objects. Although simulation cannot perfectly reproduce real-world physics, our additional real-world evaluations discussed above exhibit similar challenges in active evidence acquisition.

 

Thank you again for your thoughtful questions that have helped us clarify key aspects of our work! We hope our detailed responses have addressed your concerns, and therefore have reinforced your confidence in our work. If so, would you kindly consider raising your score? If you have any additional questions, we would be happy to address them during the rebuttal window. Thanks again!

AuthorsOpenReview ↗

Follow-Up on Rebuttal

Dear Reviewer,

Thanks again for your suggestions to strengthen this work! As the rebuttal period is approaching the end soon, we want to know if our response has answered your questions and addressed your concerns. If no, we are more than happy to provide further modifications. If yes, would you kindly consider raising the score?

Thanks again for your truly constructive and insightful feedback.

Best, Authors

Reviewer_mGmnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ESI-BENCH, a comprehensive benchmark for embodied spatial intelligence spanning 10 task categories, 29 subcategories and 3,081 task instances.

Critical strengths

The experimental comparisons are extensive.

The paper compares settings involving a single view, random multiple views, active exploration, and observations from reference trajectories. These comparisons provide valuable controls for analyzing the relationships among additional observations, evidence selection, and spatial reasoning. The results show that active exploration yields substantial improvements on multiple tasks.

The benchmark covers a broad range of tasks and has potential value for evaluation.

The benchmark covers 10 major categories and 29 subcategories, bringing problems involving occlusion, containment, mirror reflections, and physical dynamics into a unified interactive environment. It focuses on tasks in which the initial passive observation is insufficient and additional evidence must be acquired, helping assess agents' ability to select viewpoints and interaction strategies.

The authors consider data generation biases and the risk of shortcuts.

The question-only and metadata-only controls examine whether question wording and object-category priors, respectively, are sufficient to predict the answers. Together with simulation-based checks and manual review, these analyses provide some support for assessing the quality of the automatically generated tasks.

Critical weaknesses

The primary success metric does not sufficiently distinguish evidence-supported correct answers from answers that are correct by chance.

The paper primarily evaluates performance using final-answer accuracy. However, tasks with binary answers or a limited set of options may be answered correctly by guessing without sufficient evidence. Although the paper includes some analyses of trajectory behavior, it does not explicitly quantify how many successful answers are grounded in sufficient evidence.

Questions for authors

How many distinct scenes, rooms, object combinations, and layout templates do the 3,081 instances actually cover? What proportion of the instances involve only changes to the agent's initial position, object substitutions within the same layout, or substantive changes to spatial relationships or reasoning requirements, respectively? These statistics would help distinguish the number of instances from the actual diversity of environments and reasoning problems.

Please clarify the observation sequence lengths and frame selection rules used in the GT Passive setting, including whether all trajectory frames are retained, whether frames are subsampled, and whether the observation budget is matched to that of the Active setting. Please also report the average number of steps taken by humans during active exploration on successful tasks. Ideally, include the distribution of step counts across all tasks and the proportion of tasks that reach the budget limit, to facilitate a comparison of exploration efficiency between humans and models.

Are free-text answers processed through answer extraction, normalization of synonymous expressions, and object reference matching, or are they evaluated using exact string matching? How does the evaluator handle tasks with multiple semantically equivalent answers or valid action sequences? Please explain the scoring rules and distinguish formatting errors from semantic errors.

4 comments
AuthorsOpenReview ↗

Response to Reviewer mGmn (1/3)

We are extremely grateful for your thorough and insightful feedback, which has significantly enhanced the quality of our paper! Following your valuable suggestions, we have conducted additional experiments to address all your comments.

 

W1: The primary success metric does not sufficiently distinguish evidence-supported correct answers from answers that are correct by chance.

We thank the reviewer for this important comment. Our supplementary already includes Question-Only and Metadata-Only baselines in Appendix F.4 (Table 8) to evaluate whether models can exploit linguistic or dataset biases without visual observations. To further address this concern, we additionally introduce a Random Guessing baseline that selects answers randomly from the valid answer choices, providing an explicit chance-level reference.

Task CategoryRandom GuessingQuestion-OnlyMetadata-OnlyPassive SingleActive
Perceptual Grounding24.726.128.332.866.3
Physical Structure30.431.535.050.764.1
Physical Dynamics49.849.251.051.981.2
Specular Reflection46.447.249.547.250.4
Spatial Relations29.730.835.145.454.8
Metric Comparison48.749.250.848.564.1
Cognitive Mapping48.147.650.153.957.9
Enumerative Perception23.624.326.113.022.6
Temporal Understanding34.835.738.137.453.0
Action Sequencing25.424.829.344.254.5

Random Guessing and Question-Only achieve similar performance, with Question-Only slightly outperforming chance in most categories. In contrast, Active Exploration generally achieves substantially higher accuracy, suggesting that interactive visual evidence contributes meaningfully beyond chance-level guessing and question-specific priors.

W2: how many successful answers are grounded in sufficient evidence.

Good suggestion! To directly evaluate whether correct predictions are supported by sufficient visual evidence, we conduct a trajectory-level evidence audit, assessing whether the observations collected during active exploration provide sufficient information to justify the final answer. We report the percentage of correct predictions supported by strict sufficient visual evidence below.

Task CategoryEvidence-Supported Correct Answers (%)
Perceptual Grounding98.2
Physical Structure95.4
Physical Dynamics99.0
Specular Reflection89.6
Spatial Relations88.1
Metric Comparison86.8
Cognitive Mapping84.3
Enumerative Perception81.4
Temporal Understanding91.9
Action Sequencing98.6

These results suggest that most correct predictions are supported by sufficient visual evidence.

AuthorsOpenReview ↗

Response to Reviewer mGmn (2/3)

Q1: How many distinct scenes, rooms, object combinations, and layout templates do the 3,081 instances actually cover? What proportion of the instances involve only changes to the agent's initial position, object substitutions within the same layout, or substantive changes to spatial relationships or reasoning requirements, respectively? These statistics would help distinguish the number of instances from the actual diversity of environments and reasoning problems.

We thank the reviewer for this question. ESI-Bench covers 51 interactive 3D scenes and over 300 rooms across 3,081 instances. Object combinations are rarely repeated, resulting in over 1,900 distinct object combinations. We do not reuse a fixed set of layout templates. Instead, GPT-5 proposes object placements and agent positions for each instance, subject to task-specific spatial constraints, followed by randomized placement and physical validation. While we may provide task-specific guidance on favorable layout characteristics (e.g., placing objects behind obstacles to create occlusion), substantial randomness is retained in the generated configurations. Therefore, each instance is generated with a new spatial configuration, rather than simply changing the agent's initial position or substituting objects in an existing layout.

Q2: Please clarify the observation sequence lengths and frame selection rules used in the GT Passive setting, including whether all trajectory frames are retained, whether frames are subsampled, and whether the observation budget is matched to that of the Active setting. Please also report the average number of steps taken by humans during active exploration on successful tasks. Ideally, include the distribution of step counts across all tasks and the proportion of tasks that reach the budget limit, to facilitate a comparison of exploration efficiency between humans and models.

Thank you for this point. In the GT Passive setting, we retain all observation frames along the ground-truth trajectory, with exactly one frame per action and no frame subsampling, following the same action–observation protocol as the Active setting. Importantly, when designing the benchmark, we deliberately set the maximum step budget (30 steps) substantially higher than the number of steps required by the ground-truth trajectories. Therefore, GT Passive and Active share the same maximum observation budget, as the ground-truth trajectories require far fewer steps than the 30-step limit.

We have added the average number of steps taken by humans and models during successful active exploration, along with the percentage of trials exhausting the maximum step budget (30 steps), broken down across all 10 task categories. The results and visualizations are available at https://anonymous.4open.science/w/esi-bench2-0/#step-budget.

The results reveal a clear gap between models and humans. Humans generally require fewer exploration steps to reach correct answers and rarely exhaust the step budget, whereas models often explore less efficiently, particularly in Metric Comparison, Cognitive Mapping, and Enumerative Perception. We observe that models often drift progressively farther from their initial positions and away from viewpoints that could provide task-relevant evidence, which explains why they frequently exhaust the entire step budget without acquiring sufficient evidence to answer the question correctly. As exemplified in Figure 4(b) of the main paper, suboptimal actions lead to uninformative observations, which in turn induce increasingly ineffective actions, creating a self-reinforcing cycle of misguided exploration that continues until the step budget is exhausted.

AuthorsOpenReview ↗

Response to Reviewer mGmn (3/3)

Q3: Are free-text answers processed through answer extraction, normalization of synonymous expressions, and object reference matching, or are they evaluated using exact string matching? How does the evaluator handle tasks with multiple semantically equivalent answers or valid action sequences? Please explain the scoring rules and distinguish formatting errors from semantic errors.

Thank you for proposing the question. We clarify our evaluation protocol below.

1. Answer Format and Evaluation Protocol. As described in Sec. 3.1, we explicitly specify the required answer format in each question/prompt. Different tasks require different structured outputs:

  • Yes/no questions: Binary answers for relational or physical reasoning tasks.
  • Candidate selection: Choosing between predefined candidates (e.g., {obj A} or {obj B}, {room A} or {room B}). For some tasks, we explicitly provide these candidate options as hints in the question to constrain possible answers and avoid object-reference ambiguity.
  • Counting: Integer answers specifying the number of objects.
  • Stacking and object ordering: Structured strings specifying the object order (e.g., obj B -> obj A -> obj C).
  • Action sequencing: Structured sequences of actions (e.g., putting {obj B} on {obj A}; putting {obj C} on {obj D}).

We further clarify the scoring rules:

  • Normalization: Lowercase, normalize whitespace and punctuation, and remove special characters.
  • Synonym handling: Separate synonym matching is generally unnecessary since questions already constrain responses to predefined candidate labels and formats.
  • Structured parsing: Simple answers are compared directly after normalization, while compound formats (e.g., object orderings, spatial relationships, and action sequences) are parsed using regular expressions before scoring.
  • Invalid outputs: As detailed in Appendix M, invalid or malformed intermediate actions are handled by the action parser. Unparseable intermediate outputs consume an exploration step without changing the observation, and the trajectory continues.

Given these explicit answer constraints, normalization procedures, and structured parsing rules, we found almost zero formatting errors in the LLM-generated final answers across our evaluations. Therefore, incorrect final answers reflect semantic errors rather than formatting failures.

2. Single Ground Truth and Multiple Valid Solutions. As described in Appendix F.2 (Tables 5–6), we explicitly filter out tasks with multiple valid ground-truth answers during benchmark construction. For example, containment tasks are designed to have only one suitable container situation, while navigation tasks have a unique shortest path. Our human verification pipeline additionally rejects instances where multiple answers are equally plausible under the Answerability criterion. Consequently, each retained task has a unique, objectively verifiable ground-truth answer.

Importantly, a unique ground-truth answer does not imply a unique interaction trajectory. Models may follow multiple valid sequences of perception, locomotion, and manipulation actions to acquire the evidence needed for the same answer. Indeed, as demonstrated in Sec. 4.2 and Appendix H, agents spontaneously discover distinct exploration strategies that lead to the same correct answer. The ground-truth trajectories serve only as oracle references for comparison, not as prescribed execution paths. We evaluate the correctness of the final answer, not whether the agent follows a particular action trajectory.

We will incorporate these scoring details into the revised manuscript to further clarify the evaluation protocol.

 

We once again thank your efforts in reviewing our paper! Your valuable comments really help improve our paper, and inspire us a lot! We hope our response has addressed your concerns. If so, could you kindly consider raising your score? If you have more questions, don't hesitate to let us know in the rebuttal window! Thanks again!

AuthorsOpenReview ↗

Follow-Up on Rebuttal

Dear Reviewer,

Thanks again for your suggestions to strengthen this work! As the rebuttal period is approaching the end soon, we want to know if our response has answered your questions and addressed your concerns. If no, we are more than happy to provide further modifications. If yes, would you kindly consider raising the score?

Thanks again for your truly constructive and insightful feedback.

Best, Authors

Reviewer_7HHsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates spatial intelligence through a perception-action loop. To this end, the authors propose ESI-BENCH, an embodied spatial intelligence benchmark that spans 10 task categories and 29 subcategories grounded in Spelke’s core knowledge systems. Solving these tasks requires inferring a sequence of actions for perception, locomotion, and manipulation. They instantiate embodied agents using two families of models, 2D+VLMs and 3D+VLMs, evaluate them on ESI-BENCH, and report several interesting findings. Finally, they discuss the gap between models and humans in spatial reasoning.

Critical strengths
  1. A study of spatial intelligence through the perception-action loop: they move beyond passive spatial reasoning to active exploration of the environment
  2. Meaningful and comprehensive task design: they curate a set of embodied spatial reasoning tasks following Spelke’s core knowledge systems that identify four faculties of spatial intelligence. The spatial tasks have undergone human evaluation to ensure they require embodied interaction and manipulation.
  3. A lot of experiments and analysis: they evaluate 2D+VLM and 3D+LLM, which are two model families that instantiate an embodied agent, under passive, active, and oracle observations, respectively.
Critical weaknesses
  1. Active reasoning through a perception-action loop is not new [1]. In this work, the authors employ it to study embodied spatial reasoning, but ESPIRE also appears to implement a similar loop that enables active perception (see Table 6 in [2]). It would be helpful to clarify this paper’s distinct contributions.
  2. While some findings from ESI-BENCH are interesting (for example, imperfect 3D representations may be more harmful than 2D baselines, and models tend to commit prematurely with high confidence), it is unclear what actionable insights they offer for model development. For example, given the finding that poor actions lead to poor observations, which further result in worse actions, can a better perception-action mechanism mitigate this cycle, and is this a fundamental issue with pretrained VLMs/LLMs?
  3. The paper lacks real-world robotic evaluations, so it is unclear whether the findings from the simulated ESI-BENCH environment transfer to the real world. In the embodied domain, developing VLAs that unify reasoning and action has become a trend, but the paper lacks a comparison with such models.

[1] PhysVLM-AVR: Active Visual Reasoning for Multimodal Large Language Models in Physical Environments, https://arxiv.org/abs/2510.21111

[2] ESPIRE: A DIAGNOSTIC BENCHMARK FOR EMBODIED SPATIAL REASONING, https://arxiv.org/abs/2603.13033

Questions for authors

see "Weaknesses"

6 comments
AuthorsOpenReview ↗

Response to Reviwer 7HHs (1/3)

Thank you for your insightful and constructive comments! We have added additional experiments, including one comprehensive real-world robot experiment and modified our paper according to your comments.

 

W1: Active reasoning through a perception-action loop is not new [1]. In this work, the authors employ it to study embodied spatial reasoning, but ESPIRE also appears to implement a similar loop that enables active perception (see Table 6 in [2]). It would be helpful to clarify this paper’s distinct contributions.

We thank the reviewer for raising this point. We would like to emphasize that our work is fundamentally different from these prior works, as we clarify below.

  • Differences with ESPIRE:

    • We acknowledge that ESPIRE is a very good paper and have already cited and compared with it in the original manuscript. In ESPIRE, the notion of "spatial" primarily refers to spatial grounding: identifying objects, resolving spatial relationships, and predicting 6-DoF poses for robotic pick-and-place execution. Its central motivation is that accurate spatial grounding and localization can improve robotic task execution. Although Table 6 touches upon occlusion, these are primarily considered in the context of improving grounding and execution. For example, the task "Find a book at 12 o'clock of the cheval mirror from the table, and grab it" also tells the agent where the book is, and requires the agent to localize the correct object before grasping it. ESPIRE does not systematically study the perception–action loop itself, particularly how perception changes as a function of action and how agents can strategically select actions to acquire previously unknown spatial information (e.g., where is the book exactly), nor does it comprehensively evaluate the broader range of spatial reasoning capabilities.

    • In contrast, ESI-Bench explicitly studies embodied spatial intelligence through the perception–action loop. We want to build the correlation between spatial perception and spatial actions better. Its tasks require information-seeking spatially-related actions to resolve hidden spatial and physical uncertainties, rather than merely grounding spatial objects with relationships.

  • Differences with PhysVLM-AVR

    • First, the two works differ fundamentally in their research motivation and problem definition. PhysVLM-AVR primarily focuses on improving multimodal models' performance on interactive visual question answering, particularly in synthetic object-centric scenes. Importantly, it does not establish a conceptual framework for spatial intelligence, identify its constituent spatial abilities, or systematically evaluate spatial concepts. In contrast, ESI-Bench takes embodied spatial intelligence itself as the central research question. Our motivation is to understand what fundamental spatial abilities enable intelligent agents to acquire and apply spatial knowledge through embodied interaction.

    • Second, the benchmarks differ substantially in task coverage and the spatial concepts being evaluated. PhysVLM-AVR introduces CLEVR-AVR, built on simplified CLEVR-style synthetic scenes, with questions centered on object attributes, counting, existence, comparison, and arithmetic under occlusion and stacking. These tasks evaluate whether agents can obtain sufficient visual evidence to answer questions, but do not systematically distinguish or assess different spatial intelligence capabilities. ESI-Bench, in contrast, explicitly organizes 10 categories and 29 subcategories around a broad taxonomy of spatial abilities. Built on OmniGibson and BEHAVIOR-1K assets, it covers diverse spatial and physical concepts in complex indoor and outdoor environments rather than resolving a restricted set of object-centric synthetic questions.

 

We thank the reviewer for this valuable suggestion. Note that ESPIRE was already discussed in our Related Works section, both in Table 1 and in the accompanying text. We have further revised the manuscript to expand our comparison with ESPIRE and add an explicit comparison with PhysVLM-AVR, including its citation. All revisions are highlighted in blue to clarify the distinctive contributions of ESI-Bench.

AuthorsOpenReview ↗

Response to Reviwer 7HHs (2/3)

W2: While some findings from ESI-BENCH are interesting (for example, imperfect 3D representations may be more harmful than 2D baselines, and models tend to commit prematurely with high confidence), it is unclear what actionable insights they offer for model development. For example, given the finding that poor actions lead to poor observations, which further result in worse actions, can a better perception-action mechanism mitigate this cycle, and is this a fundamental issue with pretrained VLMs/LLMs?

We thank the reviewer for this insightful question. Our findings reveal several key insights including:

  • Weak perception–action correlation: Current VLMs struggle to understand what actions lead to desired future perceptions.

  • Exploration–exploitation imbalance: Models often commit prematurely to answers without acquiring sufficient evidence, reflecting an inability to balance exploration and exploitation.

While our primary contribution is a benchmark rather than a new method, to address the reviewer's concern, we highlight two actionable directions for model development motivated by our findings:

  • World models that learn from experience to anticipate perceptual consequences : Learn from past interactions to better predict which actions lead to which future perceptions.

  • Action models that learn when to explore versus when to commit : Better balance exploration and exploitation.

To further investigate how these insights can translate into actionable improvements, inspired by Reflective Test-Time Planning [1] and T2-z [2], we explore a preliminary framework (co-TTT) that jointly adapts an action-conditioned world model and a high-level action policy through test-time training. Specifically, we optimize the Wan2.2-based world model using prediction errors from observed action–observation transitions, while updating the Qwen-based action policy through LoRA using retrospective reflection on action effectiveness, information gain, and stopping decisions. These two components form a mutually improving loop: better world predictions guide more effective exploration, while more informative interactions provide additional experience for refining both models. We focus on three ESI-Bench subcategories: Rigid Containment, Dimensional Size, and Action Sequencing, which closely correspond to the Fitting, Selection, and Preparation tasks studied in Reflective Test-Time Planning [1], respectively. We give the results below.

SubcategoryGPT-5GPT-6Gemini 3.1QwenCo-TTT
Rigid Containment42.560.067.547.565.0
Dimensional Size67.769.568.357.366.9
Action Sequencing41.663.654.539.757.4

The preliminary results suggest that joint test-time training can substantially mitigate cascading perception–action failures. These findings suggest that improving the perception–action mechanism through experience-driven adaptation is a promising direction.

Regarding whether these failures are fundamental to pretrained VLMs/LLMs, our supplementary experiments suggest that they are not necessarily intrinsic limitations of pretrained models, but reflect substantial weaknesses in their perception–action mechanisms. In Appendix K.2 (Table 18), providing more informative, task-relevant oracle viewpoints substantially improves performance without modifying the pretrained models, indicating that the models often possess the perceptual and reasoning capabilities needed to solve tasks but struggle to acquire the right observations. Furthermore, Appendix M.1 (Tables 19–20) identifies action selection as the largest individual source of failure, while also highlighting the importance of memory and belief updating. We believe the primary bottleneck lies on the action side: current pretrained VLMs have not sufficiently developed the action → perception → action mapping. We hypothesize that this limitation partly stems from the lack of experience-based learning through embodied interaction in current pretraining paradigms. Our preliminary Co-TTT results suggest that these failures can be mitigated through experience-based adaptation.

 

[1] Yining Hong, Huang Huang, Manling Li, Li Fei-Fei, Leonidas Guibas, Jiajun Wu, and Yejin Choi. “Learning from Trials and Errors: Reflective Test-Time Planning for Embodied LLMs.”

[2] Chenyuan Zhou, Yining Hong, Huang Huang, Yunfan Jiang, Wenlong Huang, David Held, Vitor Campagnolo Guizilini, Yue Wang, Li Fei-Fei, Jiajun Wu, Leonidas Guibas, and Yejin Choi. “T2-z: Test-Time Optimization of Environment Latents for Robotic World Models.”

AuthorsOpenReview ↗

Response to Reviwer 7HHs (3/3)

W3.1: The paper lacks real-world robotic evaluations, so it is unclear whether the findings from the simulated ESI-BENCH environment transfer to the real world.

We thank the reviewer for raising this important concern. We conducted additional real-world robotic experiments across seven ESI-Bench subcategories: Spatial Distance, Geometric Configuration, and Counting with Occlusion, Transparency, Physical Contact, Physical Containment and Inclined Plane (20 instances each). The experimental setup and qualitative demonstrations are available at https://anonymous.4open.science/w/esi-bench2-0/#real-world-experiments.

Real-World Experimental Setup. For Spatial Distance, Geometric Configuration, and Counting with Occlusion, Transparency, Physical Contact, we use the I2RT Linear Bot, a mobile robotic platform supporting omnidirectional movement, vertical adjustment, and manipulation. For Physical Containment and Inclined Plane, we use Franka arm. We adopt the same high-level action space as ESI-Bench, including movement (move_forward, move_backward, move_left, move_right, move_up, move_down, pick up, put down, pour, turn left, turn right, turn up, turn down). This allows us to evaluate the same embodied spatial reasoning capabilities in physical environments.

Quantitative & Qualitative Results. We report the performance across the seven real-world subcategories below, alongside their corresponding simulation results.

SubcategorySim. Passive SingleSim. ActiveSim. GT Multi-ViewReal Passive SingleReal ActiveReal GT Multi-View
Spatial Distance66.466.483.655.060.085.0
Geometric Configuration31.623.057.220.025.055.0
Counting with Occlusion3.320.056.70.010.040.0
Transparency43.164.291.725.050.0100.0
Rigid Containment52.560.095.045.055.0100.0
Inclined Plane59.082.091.855.085.0100.0
Physical Contact52.570.090.045.075.090.0

You can also find qualitative results here https://anonymous.4open.science/w/esi-bench2-0/#real-world-experiments.

Consistent Findings Across Simulation and the Real World. Quantitatively, our real-world experiments exhibit trends consistent with simulation. Passive Single-View and Active Exploration are generally more challenging in the real world due to increased visual complexity, environmental variability, and imperfect physical execution. Interestingly, GT Multi-View sometimes achieves higher accuracy in the real world, potentially because real-world observations avoid certain simulation rendering artifacts and provide clearer visual evidence. Nevertheless, the substantial performance gap between Active Exploration and GT Multi-View remains highly consistent across simulation and the real world. Our qualitative examples reveal failure patterns consistent with our simulation findings. Models frequently perform extensive exploration but fail to acquire sufficient spatial evidence, committing prematurely to incorrect answers with high confidence. These observations, together with the consistent accuracy trends, suggest that the fundamental bottleneck is not specific to simulated environments, but lies in the models' inability to effectively close the perception–action loop. Therefore, our simulation environment provides a reproducible and controlled testbed that also captures key challenges of real-world embodied spatial intelligence, enabling systematic evaluation and diagnosis of perception–action failures without the cost and variability of physical experiments.

We will expand our real-world evaluations to additional spatial categories and robotic platforms in future work.

W3.2: In the embodied domain, developing VLAs that unify reasoning and action has become a trend, but the paper lacks a comparison with such models.

We attach preliminary results on π₀.₅ below. We'll run it on full categories for the final version.

Subcategoryπ₀.₅
Stacking & Stability68.4
Physical Contact66.7
Traversable Passage57.3

 

We wish that our response has addressed your concerns and turns your assessment to the positive side. If so, would you kindly consider raising your score? If you have more questions, feel free to let us know during the rebuttal window. Thank you!

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

General Response to All Reviewers

We sincerely appreciate all reviewers’ time and efforts in reviewing our paper. In addition to the response to specific reviewers, here we would like to highlight our contributions and the new experiments that we add in the rebuttal.

[Our Contributions]

We are glad to find out that the reviewers generally acknowledge our contributions:

  • The paper introduces a comprehensive benchmark for embodied spatial intelligence, covering 10 categories, 29 subcategories, and 3,081 instances [7HHs, mGmn, bJyT]
  • The paper moves beyond passive spatial reasoning to systematically evaluate the perception–action loop through active exploration and manipulation [7HHs, mGmn, bJyT]
  • The benchmark provides valuable diagnostic insights into action selection, evidence acquisition, and spatial reasoning failures [7HHs, bJyT]
  • Experimental evaluation is extensive, including comparisons across passive, active, oracle, 2D/3D, and human settings [7HHs, mGmn, bJyT]

[New Experiments]

In this rebuttal, we have added more supporting experiments to address reviewers’ concerns.

  • Real-world robotic experiments across seven spatial subcategories (7HHs, bJyT)
  • Co-TTT experiments on joint adaptation of world models and action policies (7HHs)
  • VLA evaluation using π₀.₅ (7HHs)
  • Random guessing baselines and evidence-supported accuracy analysis (mGmn)
  • Human–model exploration efficiency and step-budget exhaustion analysis (mGmn)
  • Additional human audits on simulator–visual consistency, action outcomes, and evidence recoverability (bJyT)

We hope our responses convincingly address all reviewers’ concerns. We thank all reviewers for their time and efforts again!

AuthorsOpenReview ↗

Summary of Our Rebuttal and Discussion

Summary of Our Rebuttal and Discussion

We genuinely thank all reviewers and ACs for their efforts and time in reviewing our paper, as well as their constructive suggestions that contribute to the improvement of our paper! We sincerely appreciate the positive 3-3-3 (Originally 2-3-3) evaluation from reviewers 7HHs, mGmn, and bJyT, especially 7HHs's willingness to raise their rating from 2 to 3 following our rebuttal.

Contributions

We would like to emphasize the contributions of this paper:

  • We introduce ESI-Bench, a comprehensive benchmark for embodied spatial intelligence, covering 10 categories, 29 subcategories, and 3,081 instances.
  • We move beyond passive spatial reasoning to evaluate how agents actively acquire spatial knowledge through perception, locomotion, and manipulation.
  • We construct diverse interactive environments spanning 51 scenes, 300+ rooms, and 1,900+ object combinations, with rigorous simulation and human verification.
  • We conduct extensive evaluations across state-of-the-art multimodal models, comparing passive, active, oracle, 2D/3D, and human settings.
  • We reveal fundamental limitations in current models' action selection, evidence acquisition, and belief updating, highlighting the substantial gap between human and model exploration.

Additional Experiments

  • [Real-World Robotic Experiments] [7HHs, bJyT] We conducted experiments across seven spatial subcategories (140 instances) using real robots, demonstrating consistent findings between simulation and the real world.
  • [Joint Test-Time Training] [7HHs] We introduced Co-TTT experiments jointly adapting world models and action policies, significantly improving performance across three spatial reasoning tasks.
  • [VLA Evaluation] [7HHs] We added experiments with π₀.₅ across three embodied spatial tasks.
  • [Random Guessing Baselines] [mGmn] We introduced chance-level baselines across all 10 categories to distinguish spatial reasoning from guessing.
  • [Evidence-Supported Accuracy] [mGmn] We conducted trajectory-level evidence audits, showing that 81.4%–99.0% of correct predictions are supported by sufficient visual evidence.
  • [Human Exploration Efficiency] [mGmn] We added human–model comparisons of exploration steps and step-budget exhaustion across all categories.
  • [Human Verification] [bJyT] We performed additional audits of simulator–visual consistency, action outcomes, and evidence recoverability.
  • [Benchmark Diversity and Evaluation] [mGmn] We clarified environment diversity, observation budgets, answer normalization, and evaluation protocols.
  • [Qualitative Examples] We provide additional real-world demonstrations and exploration analyses on our project website.

We sincerely thank all reviewers for their insightful feedback, which has significantly improved our paper. All additional experiments and modifications will be reflected in the final version.

Best,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.