acceptodds
Under review as a conference paper at ICLR 2027

SpaceMCP: Vision Language Model Spatial Reasoning via Model Context Protocol

Abstract

Despite impressive advances in large language models (LLM), vision–language models (VLM) still exhibit limited spatial reasoning capabilities. Recent progress in feed-forward 3D reconstruction has enabled precise, real-time geometric understanding from visual inputs. In this work, we propose SpaceMCP, a framework that leverages these advances to introduce a Model Context Protocol (MCP) style paradigm for spatial reasoning. Our approach augments a VLM with structured function calls, allowing it to query a scene graph for spatial information such as distances, object positioning, and camera properties. The scene graph is constructed in real time using feed-forward 3D reconstruction to recover metric geometry, combined with open-vocabulary segmentation to identify and localize relevant objects within the scene. Paired with VLM's strong reasoning capabilities, the VLM is able to combine this primitive information to solve more complex spatial tasks. Across nine VLMs and without any training, this approach improves scores by an average of 16 points on VSI-Bench, 6 on MMSI-Bench and 14 on VSTI-Bench, and with matched backbones it outperforms the strongest training-free tool-using agent on VSI-Bench. Gains are largest on metric and camera-geometry questions, while questions that hinge on viewpoint conventions, temporal order or fine measurements remain hard. Given the rapid advances in feed-forward 3D computer vision, we propose this paradigm as an augmentation to VLM spatial reasoning rather than a replacement: the vision models contribute precise metric answers and the VLM context and nuance, and improving native VLM spatial reasoning remains necessary.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision Language Models, 3D geometry models, Spatial Reasoning, Training Free

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yL1nOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SpaceMCP is a training-free framework that exposes reconstructed 3D scenes through a reusable interface for spatial reasoning. It combines metric reconstruction, language-guided segmentation, and geometric queries, allowing a model to plan tool calls and answer using the returned information. The paper evaluates the framework across nine backbones and three benchmarks.

Critical strengths
  1. A useful engineering contribution. The framework brings together useful elements of existing spatial reasoning systems into a coherent interface, with reusable scene representations, cached perception features, on-demand object extraction, and composable geometric queries. Although the individual components are established, their integration offers a practical foundation for subsequent research and potential deployment. I particularly appreciate this effort to make the existing paradigm easier to reuse.

  2. Broad evaluation with informative failure analysis. Comparisons against the original backbones show substantial improvements on metric and camera-geometry questions. The paper also reports categories where performance declines, giving a useful account of the framework’s capabilities and limitations.

Critical weaknesses
  1. The methodological novelty is not huge. The contribution is primarily the organization and implementation of the interface. Controlled comparisons using the same perception outputs would help establish the benefits of this particular design.

  2. The evaluation protocol and efficiency evidence need clarification. The paper does not clearly specify evaluation sample counts or whether all benchmarks are evaluated in full. Comparisons with published results may involve different conditions. The reported perception timings also exclude model calls, leaving a gap between evaluation and actual use.

Questions for authors

Were all three benchmarks evaluated in full? If subsets were used, how were they selected, and were the directly rerun baselines evaluated on the same questions?

What are the measured end-to-end latency and average model-call costs of SpaceMCP and SpatialClaw under matched conditions, including preprocessing and its amortization across questions?

Reviewer_rhnzOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SpaceMCP, a training-free method that provides spatial agents with a metric 3D scene they can query. Using perception tools it creates stores a scene and builds a scene graph. The agent generates a plan using geometric tools and answers based on the returned values. SpaceMCP shows improvements on VSI-Bench, MMSI-Bench, and VSTI-Bench across multiple backbones.

Critical strengths
  1. The general idea is clear and makes sense. Figure 2 clearly shows the method builds a 3D scene and allows the agent to query it rather than estimating geometry directly from images.

  2. They evaluate the method across nine backbones and three benchmarks, showing improvements, especially on metric and camera-related questions.

Critical weaknesses
  1. The novelty of the work is not clear to me. What does MCP add beyond ordinary tool calling? What's new compare to SpatialClaw or GCA these papers logically first grounding objects and then reasoning about them. The paper should clarify what is novel and show which part accounts for the improvements.

  2. The baseline comparisons do not all use the same backbone/perception tool and visual inputs. The paper includes some matched-backbone comparisons with SpatialClaw, but since this is 3D perception tools can highly affect the results. Without controlled backbone/perception tool comparisons it's hard to assess the empirical contribution.

  3. While you claim a general MCP interface, the tools and instructions seem closely matched to the benchmark question types, which makes the generalization claim difficult to assess. Also, the contribution claims coverage "without per-task adaptation," while §4.1 describes a VSTI-specific left/right example revision. Can you compare your method with related work such as GCA and SpatialClaw, and show where your method generalizes better both qualitatively and quantitatively?

Questions for authors
  1. Was the VSTI left/right prompt revised based on test-set errors? Do the baselines have similar prompt optimization or similar examples?
  2. It wasn't clear if you evaluated the full MMSI-Bench or a subset. If a subset, are you comparing it with full-benchmark results of related works?
  3. What is the total runtime, including planning, answering, the target extractor when needed, and the base answer used for fallback? A comparison of runtime, model calls, and token usage with related works would help
  4. Can you provide an ablation study of the components beyond the fallback?
  5. Reusing the scene could also be useful, so I suggest reporting the cost of answering several questions on the same video or embodied robotic observations.
Reviewer_tS1bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SpaceMCP is a training-free system that exposes a metric scene graph to a language-model planner. Depth Anything 3 reconstructs geometry and camera poses; SAM 3 supplies floor and query-dependent object masks; geometric getters support dimensions, distances, bearings, counts and camera motion. A text-only planning turn composes calls, and a second text-only turn converts the outputs into an answer, with a separately obtained visual answer as fallback. The evaluation spans nine backbones and three benchmarks, using 49 questions per category according to the supplement. Gains are substantial on metric and camera-geometry tasks; viewpoint-sensitive, temporal and fine-measurement tasks remain challenging.

Critical strengths
  1. The system provides a coherent way to reuse a reconstructed scene across different questions. Separating static reconstruction from query-dependent segmentation and exposing composable geometric operations is practical and easy to understand. Across nine backbones, the results support the finding that external geometry substantially improves metric spatial answers without retraining: the eight stronger planners reach approximately 60-64 on the paper’s VSI aggregation and 82-85 on VSTI. These are meaningful gains over their own no-tool baselines.

  2. The paper gives a candid, useful account of failures, with per-category results, paired bootstrap intervals for VSI/MMSI, a fallback comparison, and explanations of viewpoint reversals, grounding errors and weak-planner failures. Supplementary code, sample IDs and experimental notes support auditing. The authors correctly warn about incompatible VSTI label versions and report that their same-backbone SpatialClaw rerun ties SpaceMCP on MMSI. The resulting account of which spatial tasks benefit from geometry is informative.

Critical weaknesses
  1. The superiority claim over prior tool agents is not established under a common protocol. Section 4.3 claims VSI gains of 9 and 17 points over SpatialClaw with matched Gemma backbones. SpaceMCP uses a stratified 490-question subset, while the supplementary RESULTS_9_MODELS.md says the quoted SpatialClaw results use different random subsets and per-question averaging. Table 2 reweights only SpaceMCP to an eight-task macro average (Table 9), retaining the published comparator scores; SpatialClaw Appendix B confirms per-sample aggregation. Matching the backbone therefore matches neither the questions nor the metric. This does not show that the advantage disappears, but the quoted margin is not a controlled method comparison. The subset protocol should also be explicit in Section 4.1. Furthermore, the supplement records that the VSTI left/right example was corrected after inspecting failures and rerun on the same 49 questions; a frozen-prompt held-out result is needed.

  2. The experiments do not isolate the proposed interface’s benefit from that of strong pretrained perception. The main comparison changes perception, visual sampling, available information and inference procedure together. Related approaches already expose 3D tools or provide 3D facts as text, yet no controlled comparison holds DA3/SAM 3 outputs and input frames fixed while changing the interface. Table 10 addresses fallback, but not this attribution question. Efficiency could be a valuable contribution, but timing an 8.34-second static stage and a 6.50-second four-prompt augmentation does not establish an end-to-end accuracy/latency advantage over SpatialClaw. The fallback variant also obtains a visual baseline answer beyond its two text turns. A matched comparison would clarify the benefits of the compact interface and caching design.

Questions for authors
  1. Can the authors compare SpaceMCP and SpatialClaw on identical VSI question IDs, backbone, input frames/resolution and scorer, reporting both per-question and official task aggregation with uncertainty? Please clarify how Table 2 reconciles SpatialClaw’s per-sample aggregation with Table 9’s eight-task mean.

  2. After freezing the corrected viewpoint example, how does one common prompt perform on questions not used to diagnose or revise it? Please clarify the separation between prompt-development and evaluation examples, and whether the correction is applied consistently across benchmarks.

  3. With identical reconstructed geometry, segmentation and frames, how does the proposed API compare with structured-text scene facts or the closest agent interface in accuracy and measured end-to-end latency? Please include the visual fallback call and distinguish uncached first-question cost from amortized cost across questions on one video.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.