acceptodds
Withdrawn from ICLR 2027

SpatialCrew: Multi-Agent Framework for Reliable Tool-Augmented Spatial Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Tool augmentation can improve the spatial reasoning of vision-language models (VLMs), but erroneous intermediate outputs can propagate through otherwise valid reasoning chains. Our empirical analysis identifies this error cascade as a major source of failure and shows that verification becomes less effective as reasoning spans more tools and domains. We propose SpatialCrew, a multi-agent framework that combines domain-specific Write–Execute–Verify–Adjust (WEVA) loops with coordinated reasoning across perception, geometry, and temporal specialists. This design keeps local verification focused while checking consistency between domains. On 20 benchmarks with Gemma4-31B, SpatialCrew reduces the error cascade rate (ECR) from the five tool-augmented baselines' mean of 78.0% to 23.4%, a 54.6-percentage-point reduction. Its overall benchmark macro-average is 62.6, exceeding the mean of all six baselines by 9.6 points and the strongest baseline by 2.7 points. Across all five evaluated backbones, SpatialCrew achieves a mean score of 62.9, an average gain of 9.1 points over the corresponding no-tool models, and a mean ECR of 24.7%. These results highlight intermediate-output quality control as a practical route to reliable tool-augmented spatial reasoning. Code: https://anonymous.4open.science/r/SpatialCrew-B650.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Spatial Reasoning, Vision Language Models, Tool-augmented Reasoning, Multi-agent Collaboration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_egFYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies error propagation in spatial reasoning tasks for visual-language models, with a particular focus on tool-augmented methods. It points out that the major issue in spatial reasoning is not due to the wrong tool selection, but rather to incorrect intermediate tool outputs. The paper proposed SpatialCrew, a multi-agent framework that involves a coordinator agent and three domain-specialized agents, including perception, geometry, and temporal. Each agent operates specific tools and has their own WEVA loop for verification and revision of the problematic intermediate output. The framework also performs the cross-agent output consistency check before providing final results. The author provides a detailed analysis of the existing tool-use spatial reasoning method, including one-shot VP, API-based reasoning, and Code-based reasoning, supporting their assumption that a large portion of failure came from incorrect intermediate error cascades. The experiment covers a broad range of benchmarks, showing that SpatialCrew improves task performance and substantially reduces the EC rate. The ablation shows interesting findings as well. The component analysis shows that the SpatialCrew with domain-specialized agents is better than SpatialClaw (single agent + all tool uses), indicating that fewer tools and shorter contexts to each specialist help the reasoning better and achieve lower ECR.

Critical strengths
  1. The paper identifies an interesting phenomenon and the practical failure mode in tool-augmented reasoning: the context reasoning window for the agent to solve complex spatial reasoning tasks. Thus, breaking down the agent into a domain-specialized subagent and a coordinator agent can improve reasoning.

  2. The error cascade analysis provides a meaningful perspective for understanding the main issue in existing tool-use methods.

  3. The overall framework design with tool-set configuration, intra-agent WEVA loop, and inter-agent communication is coherent. It provides a convincing way to reduce the error propagation in the long-context spatial reasoning chain.

  4. The experimental results are extensive. The paper evaluates on broad benchmarks, including single image, multi-view, 4D/video settings, and evaluates across four backbones. Error cascade analysis compares SpatialCrew with other methods, showing it largely reduces the EC rate, and also increase the Correction success rate.

Critical weaknesses
  1. Lack of explanation on the WEVA Loop revision mechanism: The write-execute-verify-adjust loop is the key thing in the whole method. However, the paper does not clearly explain how the "adjust" step is carried out in practice. It is unclear how the agent adjusts itself in response to the verifier's output. What is the specific signal that it takes to decide if it needs to revise the code or tool-calling prompt, reasoning strategy, etc? It would be great if the author could provide more details about the revision policy and also provide representative examples that show how different verification results will lead to different adjustments.

  2. The reliability of the verifier is not clearly demonstrated: The method seems to directly assume that the verifier can reliably determine if the intermediate output is correct or not. Sec. 4.3 mentioned that the verification criteria range from numerical checks to more qualitative checks, such as visual or physical plausibility. It is unclear how reliably the latter are judged by the same underlying VLM used for reasoning. For example, when a check fails, the temporal agent triggers revision of the temporal analysis before its results are returned. How does the revision work?

  3. Unclear contribution of the gain on domain specialization: The authors claim that SpatialCrews improve task performance since each domain-specialized operator only needs to operate in a smaller tool space with a narrower context window, and has clearer domain-specific verification criteria. The current ablation (Table 5) shows that the multi-agent design performs better w/ coordinator reasoning and inter-agent domain-specific checks, but does not clearly isolate whether the methods increase the efficiency of spatial reasoning by reducing tool-selection complexity, shortening context, or simply increasing test-time computation. It would be useful to directly report quantities such as total tokens per query, average context length per step for each domain-specialized agent, and the number of failure-revision trials per query.

  4. Efficiency evaluation is incomplete -- high inference cost of SpatialCrew: I believe the whole multi-agent framework has relatively high inference latency, on the order of roughly one minute per query on average. The author reports mainly the runtime of SpatialCrew rather than a comparison with strong baselines such as SpatialClaw. I think reporting relative latency, token usage model calls, would make the performance-cost trade-off much clearer.

  5. Provided method is built upon the existing method SpatialClaw: The method adds a WEVA loop and separates agents in a domain-specific manner. The novelty of the architecture is incremental, and there is incomplete causal evidence for the proposed explanation of why the system works.

Questions for authors
  1. Following up the aboved weakness, how reliable is the verification mechanism itself? Can the authors report verifier precision/recall against manually annotated intermediate errors, rather than only the correction rate conditioned on detected errors?

  2. For the experimental setup, are the single-agent and multi-agent ablations matched in total test-time compute? In particular, if each SpatialCrew specialist can take up to 30 reasoning steps, how does the total model/tool-call budget compare with the single-agent baseline?

  3. Can the authors quantify the claimed reduction of reasoning complexity and the efficiency by reporting average context length/per agent, average context length/per query, and tool calls per query for the single-agent and SpatialCrew variants?

Reviewer_B7n4OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper argues that tool-augmented spatial reasoning agents fail mostly through "error cascades": a wrong intermediate tool output (depth, mask, reconstruction) is not checked and goes on to corrupt the final answer. An analysis of 300 annotated questions found that, among incorrect answers, about 80% (mean ECR over four baseline methods) trace back to uncorrected intermediate tool errors. To combat this, SpatialCrew proposes a structure comprising a Coordinator and 3 independent specialist roles (Perception, Geometry, Temporal). Within this structure, each role operates a Write–Execute–Verify–Adjust process of tool interaction and checking. The Coordinator validates across specialists for consistent operations. Using 20 benchmarks and the Gemma4-31B model, the researchers note a drop in Error-Completion Rate (ECR) from a mean baseline of 78.0% to 23.4%, and an overall performance score of 62.6, compared to a best baseline of 59.9 by SpatialClaw.

Critical strengths
  1. Looking at Figure 3 is helpful for thinking about different types of tool use errors and has sparked my interest in thinking about errors that happen when an agent must reason to decide what to do next vs. what happens when a tool output needs to be processed to be used later in the agent's decision making process.

  2. Finding 3 shows that code-based correction rate drops as the number of tool invocations in the reasoning chain grows (more than 6). This motivates the authors' domain split, although the paper does not test the context-load explanation directly.

Critical weaknesses
  1. The methods described in the paper were not fully implemented. I read the entire released repository (about 3.3k lines) but did not execute it.
  • Although a vision component exists within the model, it is not used. A generate_vision function is included but it remains uncalled and only an image count is passed to the LLM.

  • In order to function, a kernel needs to have both the tools and the images injected. However, the existing build_init_code function only provides imports for the numpy, scipy, and matplotlib libraries. As a result, calls made to tools.SAM3 would fail, and the prompt tells agents to use a feedback.show() method that is not defined.

  • SAM3 has no implementation; rather, the SegmentationService simply provides a placeholder for text-based prompts. In such cases, the system returns a disk with a score of 0.85, irrespective of the input. There is no tools.DA3. Depth comes from Depth-Anything-V2 run on each frame separately, with fixed intrinsics and no extrinsics, so there is no multi-view reconstruction.

  • Confidence is hard-coded to 0.9 when an answer is returned and 0.0 otherwise, so it is not self-assessed. _cross_verify() only checks for low confidence and whether any warnings exist; the cross-domain consistency check described in Sec. 4.5 is not there. The domain checks are regexes on stdout (e.g. "distance" < 0), and the dependency context passed to specialists is never used.

  • Only SPBench is supported, and run.py reloads only the first subset before scoring. 19 of the 20 benchmarks, 3 of the 5 backbones, all baselines and the ECR/CSR annotation are missing. "MRA" is implemented as chance-corrected accuracy, which is not the relative-error MRA normally used for numerical answers. Taken together, I cannot see how this code produced the reported results.

  1. The key numbers are not well supported.
  • Table 9 is not measured data. It copies the Gemma4-31B gaps onto four other backbones: SpatialClaw is always −2.7 accuracy and +40.1 ECR, and SpaceTools is always −13.1 and +49.1. The Introduction nevertheless claims SpatialCrew outperforms all existing baselines across all five backbones, but baselines were only measured on Gemma4-31B.

  • Section 3 explains that ECR needs manual annotation of traces and that only 300 questions were annotated. However, Appendix B.1 states that the full-suite ECR/CSR in Table 2 are averages over all 20 benchmarks. There are no descriptions of any of the annotation methods, and there is no discussion of any inter-rater agreement for the results.

  • There are no reported variance estimates and no discussion of the baselines' computational performance. In fact, three of the tool baselines are worse than the no-tool baseline. These findings indicate that the authors' implementation of these baselines may be inaccurate.

Questions for authors

Questions:

  1. Is the released code the same as the one in the paper? If yes, how do the images get to the model and the tools get to the kernel, when neither of these things occur in the release? Please release the real implementation, including SAM3 and DA3.
  2. How were the ECR/CSR values for the full suite in Table 2 annotated? Please give the annotation process and the annotators and their agreement.
  3. How was the baseline recreated, and does it achieve the published results? What is the compute budget used?
  4. Which MRA was used for numerical solutions?

Suggestions: 5. Substitute Table 9 with actual runs or delete it, and state variances for different seeds.

Minor: Fan et al. 2025a and 2025b are the same reference (VLM-3R) listed twice.

Reviewer_1qQxOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This work studies the problem of tool-augmented spatial reasoning in a multi-agent scenario. The paper argues that errorneous intermediate outputs can propagate through the reasoning chains causing the final answer to be wrong. The paper introduces two measures of such failures: Error Cascade Rate (ECR) which is fraction of incorrect samples where final error is traced back to the intermediate tool errors, and Correction Rate (CR) which is a fraction of detected intermediate errors that the method successfully corrects. Further, they propose SpatialCrew which is a multi-agent framework that addresses this error propagation problem through self-refining specialists (such as Perception Agent, Geometry Agent, and Temporal Agent, each with an allowed a set of tools) coupled with a separate Coordinator agent that routes the input, results, and performs cross-agent consistency checks. The proposed method is empirically evaluated across 20 different benchmarks across 5 VLMs.

Critical strengths

S1. The paper does extensive empirical validation of the proposed method: across 20 benchmarks and with 5 different VLMs showing that each benchmark and VLM sees improvement with coupled with SpatialCrew. Further, the problem that paper attempts to study is important.

Critical weaknesses

W1. SpatialCrew combines already known/established components: a coordinator agent that handles the task/result routing, iterative specialists, and verification-based feedback loops. The Write-Execute-Verify-Adjust (WEVA) cycle is also conceptually similar to standard execute-verify-refine approaches. Further, the domain-specific WEVA-style loops with iterative feedback and verification has been explored previously [1,2,3]. The Related Work (Sec. 2) is extremely limited and the paper does not comment on how its contributes differ from these prior works, nor does it specify how it empirically compares to these methods since Tab. 3 only reports methods that do not rely on domain-specific execute-verify loops.

[1] Guo, Y., Hou, Y., Ma, W., Tang, M., & Yang, M. H. (2026, April). Pursuing minimal sufficiency in spatial reasoning. In International Conference on Learning Representations (Vol. 2026, pp. 121192-121222).

[2] Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y., Wang, J., ... & Schmidhuber, J. (2024, May). MetaGPT: Meta programming for a multi-agent collaborative framework. In International Conference on Learning Representations (Vol. 2024, pp. 23247-23275).

[3] Xu, A., & Tai, Y. W. (2026). Meta-Agent: From Task Descriptions to Verified Multi-Agent Systems. arXiv preprint arXiv:2605.25233.

W2. The paper's main claim is that SpatialCrew addresses errors through WEVA loops that originate from unreliable spatial tools. But the verification procedure primarily relies on heuristic checks, LLM-based inspection or repeatedly calling the same tools. In Sec. 4.3, these are depth positivity, segmentation thresholding, visual inspection with the LLM, etc. None of these necessarily correct the underlying prediction errors, particularly when incorrect outputs satisfy the heuristic constraints. Further, the evaluation criteria is accuracy, ECR and CR with tool correctness being an implicit measure. There is no explicit report on whether the individual outputs produced by these spatial tools become more accurate after WEVA corrections, i.e., depth error before and after verification, segmentation IoU, distance estimation errors, etc. Without these, the experiments do not explicitly show/quantify that SpatialCrew fixes the errors that motivated the work in the first place.

Questions for authors

Q1. I have a few questions about the ECR measure and the manual annotation. Because the metric relies on manually attributing incorrect final answers to cascading intermediate-output errors, distinguishing an error cascade from tool-orchestration failure can be ambiguous, consider a case where multiple failures contribute to an incorrect answer. Since the paper under-specifies the annotation criteria, how and what were the ambiguous cases and how were they resolved, what did the inter-annotator agreement look like, what was the annotation protocol or sample sizes, etc., the manuscript would benefit if these details were provided.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.