acceptodds
Withdrawn from ICLR 2027

HippoSpatial: Persistent Spatial Scene Graphs for Multi-View Spatial Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-view spatial reasoning requires integrating partial observations from multiple viewpoints into a coherent global understanding of object positions, relations, and perspectives, a task on which Vision-Language Models (VLMs) consistently perform near random chance. Existing training-free methods either estimate object positions without metric grounding or re-derive geometry from scratch at every reasoning step, leaving no persistent spatial representation that spans all available views. Inspired by two principles of hippocampal spatial memory: pairwise relational encodings between objects enabling inference about pairs never directly co-observed, and the strict separation between allocentric long-term memory and egocentric working memory combined only through an explicit transform at retrieval time, we propose HippoSpatial, a training-free framework for multi-view spatial reasoning. Given a question and its candidate answers, HippoSpatial uses perception tools to segment relevant objects, recover their metric 3D positions in a shared world frame, and encode pairwise spatial relations in a Spatial Scene Graph, while separately maintaining camera geometry for each input view. At query time, a VLM performs visual grounding to resolve object references to graph nodes, then dispatches deterministic geometric tools that combine allocentric object positions with egocentric viewpoint information, delegating all numerical computation away from LLM reasoning. Experiments on MindCube, MMSI-Bench, and SPAR-Bench demonstrate that HippoSpatial outperforms prior training-free methods in most backbone-benchmark settings, with gains of up to 7.0 points over the best prior method.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Spatial Reasoning, Multi-view Reasoning, Multi-Modal LLMs

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2DUiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HippoSpatial, a training-free framework for multi-view spatial reasoning. It uses segmentation and 3D reconstruction to build a metric scene graph, stores object relations separately from camera information, and answers spatial questions using deterministic geometric tools.

Critical strengths
  • The division between scene construction, language grounding, and geometric computation is intuitive.
  • The evaluation is extensive, covering three VLM backbones and three benchmarks.
Critical weaknesses
  • Table 1 does not show a substantial advantage over competing methods when using GPT-5-mini. The SPAR-Bench results also differ considerably between the Qwen and Gemma backbones.
  • The ablation settings are insufficiently specified. What exactly do “w/o Strict Separation” and “w/o Pairwise Relations” change, and what input representation is constructed for each variant?
  • The description of graph construction is inconsistent. Section 3.2 states that the scene graph is constructed before the question is read, whereas Section 3.3.1 uses the question and candidate answers to identify relevant objects. Please clarify whether graph construction is conditioned on the question.
  • The method appears to assume static objects, assigning a single canonical position to each object. It is unclear how the framework handles dynamic tasks, such as object-motion questions in MMSI-Bench.
Questions for authors

See weakness

1 comment

We thank the reviewer for the feedback and address each point below.

Advantage with GPT-5-mini and SPAR-Bench variance. With GPT-5-mini, HippoSpatial still leads on MindCube (70.9 vs 67.5) and MMSI-Bench (55.4 vs 51.8). The SPAR-Bench gap is narrowest (59.8 vs 60.4) because stronger backbones already handle simpler spatial queries well, leaving less room for graph-based gains, a pattern visible across all methods and not only ours. The larger SPAR-Bench variance between Qwen and Gemma reflects SPAR's 20-task breadth, where model capacity plays a bigger role than on MindCube's narrower distribution.

Ablation definitions. W/o Strict Separation removes the boundary between the allocentric object store and the egocentric View Registry by merging them into a single joint representation. The drop is largest on subtasks requiring cross-view integration (Among −5.3 pts, Around −3.9 pts) and smallest on Rotation (−1.8 pts), directly supporting the design claim. W/o Pairwise Relations removes precomputed edges from the graph; the agent recovers partial information by calling tools over raw positions, leading to a smaller overall drop. We will add explicit definitions to the ablation caption in the revision.

Graph construction conditioned on the question. Object discovery uses the question and candidate answers to determine which objects to segment, so graph construction is question-conditioned by design. The graph is nonetheless fully fixed once Phase 2 begins and is never modified during reasoning. The description in Section 3.2 conflated graph construction with the subsequent reasoning phase; we will revise it to draw this boundary clearly.

Static objects and dynamic tasks. The motion questions in MMSI-Bench ask about camera movement between views (translation direction and rotation angle), not about objects physically moving across the scene. The query_camera_motion tool handles these directly by reading camera pose pairs from the View Registry. No dynamic object tracking is needed, and none of the three evaluated benchmarks test scenarios where scene objects themselves move between frames.

Reviewer_8gGNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

HippoSpatial is a training-free pipeline for multi-view spatial QA. A VLM first lists the objects relevant to the question and its answer options. SAM3 then segments these objects in every view, and Depth Anything 3 places them in one metric frame. Observations with the same label are merged across views using a geometry-plus-CLIP score. The result is serialized as a scene graph: object nodes, pairwise distance and direction edges, and a separate list of camera poses. Finally, the VLM grounds the question to the graph and calls up to three of five deterministic geometry tools. On 1,000-question subsets of MindCube, MMSI-Bench and SPAR-Bench, using three backbones, the method beats pySpatial and SpatialClaw in 6 of 9 settings.

Critical strengths
  1. The method is simple and clearly documented. Perception is kept separate from reasoning, all geometry is done by five fixed tools, and the VLM only has to ground the question and pick a tool. With the tool definitions, prompts and worked examples in the appendices, the pipeline is easy to follow and should be easy to re-implement.
  2. The method is also cheap. Appendix G.1 reports 3.5 VLM calls per query, compared with 9.7 for SpatialClaw, and Figure 4 shows accuracy improving with a better reconstruction backbone without any retraining. If the accuracy numbers hold up, a small fixed tool set that matches a much more expensive code agent would be a useful result for this line of work.
Critical weaknesses
  1. The MindCube results do not fit the protocol described in the paper. According to Appendix A.1, 1,000 questions are sampled in proportion to the three settings. That gives roughly 51 Rotation and 88 Around questions, so per-setting accuracy can only move in steps of about 2 and 1.1 points. Most of the Rotation and Around values in Table 1 (for example 61.4 and 74.0) cannot be produced by that many questions. Since decoding is greedy and only one run is reported, either the protocol description is wrong or the numbers in Table 1 did not come from the stated evaluation, and the authors need to clarify which.
  2. In every backbone and every column of Table 1, the prompting baselines improve in the same order, Direct < CoT < View Interpolation < Cognitive Map, and each step is between 1.5 and 2.9 points; on a split of about 51 questions, a 1.6-point step is less than one question. pySpatial on MMSI-Bench jumps from 34.8 with Qwen3.5-9B to 54.2 with Gemma-4-31B, enough to beat HippoSpatial there, while the Direct baseline moves only from 31.1 to 37.2. The paper also does not explain how View Interpolation was run on MMSI-Bench and SPAR-Bench, which have no intermediate frames. Finally, all three sweeps in Table 4 peak exactly at the default values, and they were run on MindCube, which is also a test benchmark.
  3. The main claims about the representation are not supported by the experiments. The graph is called persistent and "built once", but object discovery depends on the question and its answer options (Section 3.3.1), and Appendix G.1 rebuilds the graph for every query, so reuse across questions is never tested. The pairwise edges are coordinate differences within one shared frame, which is consistent with the small drop when they are removed. The ablation that removes "strict separation" is never defined. In Example 3 the VLM still has to compare two magnitudes itself to choose between candidates, which contradicts the claim that no spatial arithmetic is left to the language model. As a result, I cannot tell which part of the design, if any, explains the gains beyond using SAM3 and Depth Anything 3.
Questions for authors

Q1. How many questions per MindCube setting were evaluated, and is any number averaged over runs? Please release the per-question predictions behind Table 1.

Q2. How were the prompting baselines run, in particular View Interpolation on MMSI-Bench and SPAR-Bench?

Q3. Why does pySpatial gain 19.4 points on MMSI-Bench from the backbone switch alone?

Q4. What does "w/o Strict Separation" actually change? How does a plain list of the same object coordinates, with no edges or tools, perform?

1 comment

We thank the reviewer for the careful reading and appreciate the specific questions raised. We address each point below.

View Interpolation on MMSI-Bench and SPAR-Bench. For scene groups without a source video, intermediate frames were synthesized using Stable Virtual Camera [1] from the available sparse views, consistent with the spirit of the original baseline. We will add this detail to the experimental setup.

pySpatial discrepancy. Our re-run replaced pySpatial's original VGGT backbone with DA3+SAM3 to hold the perception pipeline constant across all tool-augmented agents. The 33.2 figure corresponds to pySpatial running with VGGT under its original setup. We will add a footnote clarifying this.

MindCube category counts. We evaluated on the full MindCube evaluation split (1,081 Rotation, 18,204 Among, 1,869 Around), not a 1,000-question subsample. The 1,000-question cap in Appendix A.1 applies to MMSI-Bench and SPAR-Bench; MindCube was evaluated on the full split. With 1,081 Rotation questions the step size is 0.09 pts, which is why values such as 61.4 are not multiples of 1.96 pts. We will correct Appendix A.1 and release per-question predictions in the revision.

Ablation definitions. We clarify that w/o Strict Separation fuses the allocentric SSG object nodes and the egocentric View Registry into one joint representation, removing the boundary between the two memory systems. The resulting drop is concentrated on Among (−5.3 pts) and Around (−3.9 pts), the subtasks that require integrating evidence across viewpoints, while Rotation is barely affected (−1.8 pts), which is consistent with the claim that cross-view queries suffer most from coordinate-frame confusion. W/o Pairwise Relations removes the precomputed edges; the smaller drop (−1.5/−2.5/−1.7) reflects the fact that the agent can partially recover by calling tools over raw 3D positions. We will add explicit definitions of both variants to the ablation caption.

Graph built once vs. question-dependent grounding. The question and candidate answers are used to identify relevant objects during graph construction, but the resulting graph is fully fixed before Phase 2 reasoning begins and is never modified or extended during it. The phrase in Section 3.2 was intended to convey this boundary between construction and reasoning, not that construction is question-independent. We will clarify this in the revision.

[1] Zhou, Jensen, et al. "Stable virtual camera: Generative view synthesis with diffusion models." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.

Reviewer_5korOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper tackles multi-view spatial reasoning, where VLMs struggle to combine partial observations from several viewpoints. It proposes HippoSpatial, an agentic framework that first builds a metric scene graph from all views using off-the-shelf segmentation and 3D reconstruction, storing object positions and pairwise relations separately from camera poses. A VLM then links the question to graph entries and calls a small set of deterministic geometric tools to compute the answer. Experiments on three multi-view benchmarks with three VLM backbones report gains over prompting methods and prior tool-augmented agents in most settings.

Critical strengths
  • The design keeps the VLM's role small (grounding and tool selection) and moves geometry into an explicit graph and deterministic tools. This makes errors traceable. The failure analysis in Fig 3 attributes most errors to perception rather than grounding or tool calls.
  • On MindCube with Qwen3.5-9B and Gemma-4-31B, HippoSpatial is ahead of both agent baselines by 5-12 points while using fewer VLM calls and less time per query than SpatialClaw.
Critical weaknesses

W1. The gains over other tool-using agents are not convincing.

  • Same tools, mixed results. SpatialClaw uses the same perception models (Depth Anything 3 and SAM3; [1] App E.1), yet it beats HippoSpatial on SPAR-Bench with Qwen3.5-9B (55.4 vs 54.4) and GPT-5-mini (60.4 vs 59.8) (Tab 1). With Gemma-4-31B, SpatialClaw's own paper reports 51.3 on MMSI-Bench ([1] Tab 3), higher than the re-run here (49.9). It also reports 72.8 on MindCube, while the re-run gives 68.4 when Tab 1's per-setting scores are weighted by question count.

  • MindCube average. The MindCube Avg in Tab 1 is an unweighted mean of the three settings (App A.1), although Among makes up 86% of the questions and Rotation and Around only 5% and 9%. Weighting by question count, SpatialClaw is ahead of HippoSpatial with GPT-5-mini (73.4 vs 72.0), so the claim of the best MindCube result with all three backbones (Sec 4.2) depends on this choice. The MindCube paper and its official evaluation code report accuracy over all questions, as do pySpatial and SpatialClaw, so Tab 1's Avg departs from the standard metric and cannot be compared with published numbers.

  • pySpatial numbers do not match. With the same Gemma-4-31B backbone, Tab 1 gives pySpatial 54.2 on MMSI-Bench, while [1] Tab 3 reports 33.2 for its official code. The plain Gemma-4-31B scores in the two papers agree (37.2 here, 37.9 in [1] Tab 2), so the gap comes from how pySpatial was run. Sec 4.1 also does not say which perception models the re-run used (the original uses VGGT [2]).

  • Unclear source of the gain. No experiment gives the same five tools to an agent without the scene graph, so it is unclear whether the graph helps or the tools do, which are written for specific question types (e.g. the rotation tool for MindCube Rotation). The association thresholds were also picked on the MindCube test set (Tab 4), and changing them moves accuracy by about as much as the advantage over SpatialClaw.

  • Missing baselines. Think3D [3] is a training-free 3D-tool agent tested on MindCube. ConceptGraphs [4] and HOV-SG [5] already build 3D scene graphs from multiple views and query them with an LLM, which is the paper's core idea, and Coarse Correspondences [6] is a training-free multi-view method. None is cited or compared.

W2. The reported numbers do not match the stated setup or the released code, so the main results in Tab 1 cannot be verified.

  • MindCube counts. App A says the authors take 1,000 of MindCube's 21,154 questions in proportion to each setting, which gives 51 Rotation, 861 Among, and 88 Around questions (the released sampler gives the same). With 51 questions, a Rotation score must be a multiple of 1/51 (1.96 points). HippoSpatial's Rotation score with Qwen3.5-9B, 61.4, falls between them, and so do 74.0 and 71.8 for the other backbones. Overall, 21 of the 23 Rotation scores in Tab 1 are not possible with 51 questions, and 21 of the 23 Around scores are not possible with 88. The Rotation drops of 1.5 and 1.8 in Sec 4.3 are not possible either. It is unknown whether the scores come from fitting the full splits or averaging over several runs (but decoding is greedy as in App B so run variance should be low).

  • Coordinate frame. The rotation and camera-motion tools assume the z axis points up (App C, App H). The released code never aligns the reconstruction to gravity, and Depth Anything 3 returns poses in a camera frame where z points forward. When I ran the released tools on App H Example 3 in that frame, they put the correct answer on the wrong side and reported no turn for a 90 deg turn. The code also calls a function (model.infer) that the reconstruction libraries do not have. So the released code cannot have produced Tab 1, and any alignment step that was used is not described.

References

[1] SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning, Cho et al., NeurIPS 26

[2] pySpatial: Generating 3D Visual Programs for Zero-Shot Spatial Reasoning, Luo et al., ICLR 26

[3] Think3D: Thinking with Space for Spatial Reasoning, Zhang et al., arXiv 26

[4] ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning, Gu et al., ICRA 24

[5] Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation, Werby et al., RSS 24

[6] Coarse Correspondences Boost Spatial-Temporal Reasoning in Multimodal Language Model, Liu et al., CVPR 25

Questions for authors
  1. For the pySpatial and SpatialClaw re-runs, which reconstruction and segmentation models, code versions, etc. were used? In particular, please explain the pySpatial result on MMSI-Bench with Gemma-4-31B. Tab 1 reports 54.2, which is 17.0 points above the plain model (37.2) and above SpatialClaw (49.9). The SpatialClaw paper ran the official pySpatial code with the same backbone and got 33.2, below its plain-model score of 37.9 (its Tabs 2-3), and pySpatial's own paper reports a 7.0-point gain on MMSI-Bench with GPT-4o (30.3 to 37.3, its Tab A1). Since the plain-model scores in the two papers agree, what was changed in the pySpatial run, e.g., was it given Depth Anything 3 or SAM3 instead of VGGT?

  2. How many MindCube questions per setting were evaluated for Tab 1 and Fig 2a, and are the numbers from a single run or averaged? Please report per-setting counts and confidence intervals, and give the MindCube results in Tab 1 as accuracy over all questions, the standard MindCube metric, alongside the unweighted average.

  3. How is a z-up frame obtained from the Depth Anything 3 output (choice of reference view, gravity estimate, inversion of the world-to-camera extrinsics)? Please release the code that produced Tab 1. What is the Rotation accuracy of the released pipeline as is?

  4. Can you isolate the contribution of the graph, e.g., by giving the same five tools to an agent that computes them from the reconstruction on the fly without the graph, or by adding them to SpatialClaw? How much accuracy depends on the question-type-specific tools (e.g., without the hypothetical-rotation tool)?

  5. What are the results when the association hyperparameters are chosen on data outside the evaluated subset, such as other MindCube questions or a different benchmark?

  6. How does HippoSpatial compare with Think3D on MindCube-1K, with a ConceptGraphs-style scene graph built from the same Depth Anything 3 and SAM3 outputs and queried by the same VLM, and with Coarse Correspondences?

  7. Sec 3.2 says the graph is built before the question is read, but object discovery uses the question and the released runner rebuilds the graph for every question. For runtime, please provide the breakdown of runtime of each component, latency on MMSI-Bench and SPAR-Bench; how SpatialClaw's 9.7 calls were counted (planner and in-kernel VLM calls); and the amortized cost when reconstruction is reused across questions that share the same views.

  8. The released code scores SPAR-Bench by exact match (datasets/spar_bench.py uses the default letter or string match), while SpatialClaw's paper (its Tab 5) and code give numerical answers partial credit with mean relative accuracy, following SPAR's evaluation scripts. Four of the nine multi-view SPAR tasks (depth and distance prediction) have numerical answers, so with uniform sampling about 444 of the 999 sampled questions are numerical. Under exact match these would rarely score, which caps overall accuracy near 56% even with every multiple-choice question correct, yet 12 of the 23 SPAR scores in Tab 1 exceed that (e.g., 65.7 for HippoSpatial with Gemma-4-31B). How were numerical questions scored for Tab 1, and were all methods scored the same way? More broadly, the paper follows SpatialClaw's 1,000-sample cap, and SpatialClaw's evaluation code is public. Why not use its sampling and scoring, or the official benchmark evaluators? Please report Tab 1 under that protocol, so all methods can be compared with published numbers.

1 comment

We thank the reviewer for the detailed and constructive comments. We address the main concerns below.

Evaluation protocol and pySpatial discrepancy. Our re-run of pySpatial replaced its original VGGT backbone with DA3+SAM3 to hold perception constant across all tool-augmented agents, which accounts for the gap between 54.2 and 33.2 on MMSI-Bench. For SPAR-Bench, numerical tasks were scored using Mean Relative Accuracy and Violation-Corrected Interval following SpatialClaw's evaluation protocol, with all methods evaluated under the same scoring. We will add a footnote clarifying both the backbone substitution and the scoring protocol.

MindCube unweighted average. The unweighted mean was chosen to weight each question type equally rather than letting the dominant Among category (86%) drive the aggregate. We acknowledge this departs from the official metric and will report the question-count-weighted accuracy alongside in the revision.

MindCube category counts. We evaluated on the full MindCube evaluation split (1,081 Rotation, 18,204 Among, 1,869 Around; 21,154 total), not a 1,000-question subsample. The 1,000-question cap in Appendix A.1 applies to MMSI-Bench and SPAR-Bench; MindCube was evaluated on the full split. With 1,081 Rotation questions the step size is 0.09 pts, which is why per-category accuracies such as 61.4 are not multiples of 1/51 ≈ 1.96 pts. We will correct Appendix A.1 and release per-question predictions in the revision.

Missing baselines and hyperparameter sensitivity. Think3D, ConceptGraphs, and HOV-SG target different settings (dense RGB-D streams and robot navigation with continuous depth sensors) rather than sparse multi-view QA from casual images, which is why they were not directly compared. We will add a paragraph in Related Work clarifying this distinction. On hyperparameters, Table 4 shows smooth, monotonic degradation as each value moves away from the default rather than a sharp peak, indicating the method is not sensitive to precise values. The defaults were set from geometric intuition before evaluation began.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.