acceptodds
Withdrawn from ICLR 2027

Parse Before You Execute: Efficient Spatial Reasoning with Executable Primitives

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Spatial reasoning in Vision-Language Models (VLMs) has increasingly benefited from tool-integrated pipelines that ground computation in explicit geometry. However, existing systems often compile queries into relatively open-ended artifacts, such as code, task constraints, tool plans, or gathered evidence, leaving what must be resolved before execution under-specified. This can lead to unnecessary tool acquisition and iterative refinement, while still producing plausible reasoning traces from incomplete spatial commitments. We identify this failure mode as under-committed query compilation. To address it, we propose PEPS, a training-free agentic paradigm Parsing Executable Primitives for Spatial reasoning under a typed FESM schema over Frame, Entity, State, and Metric. PEPS follows a parse-before-execute principle: the Parser commits each query to a compact set of executable requirements; the Executor selectively grounds the required primitives through fixed geometric APIs and deterministic computation; and the Verifier checks execution sufficiency, enabling early termination or targeted repair. Extensive experiments across challenging spatial reasoning benchmarks show that PEPS consistently improves reasoning accuracy while substantially reducing refinement rounds, API cost, and end-to-end latency over strong tool-integrated baselines. These results demonstrate that explicit requirement commitment can make tool-integrated spatial reasoning not only more reliable, but also more execution-efficient.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Spatial reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qgrCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a parse-execute-verify loop for spatial reasoning. They argue that tool-based methods suffer from "under-committed query compilation" where the proposed plan does not specify what must be resolved before execution. They propose a three-step loop to address this problem. In their parser, an agent selects from the four FESM primitives. Then an Executor agent fills them using perception tools and code, and finally a third Verifier agent scores the answer and provides feedback. Their method shows improvement in 3D spatial reasoning benchmarks such as MMSI, ViewSpatial, and Omni3D. They also provide evidence on the efficiency of their method compared to MSSR and the generalizability of their method to other benchmarks such as MindCube.

Critical strengths
  1. The paper provides a detailed and clear explanation of the method and how it works.

  2. The paper proposes a structured output interface for the spatial reasoning task. This is a good idea and a step toward more transparent, inspectable spatial reasoning by making audits easier and evaluations more fine-grained.

  3. They provide a detailed analysis of the performance of the method across multiple backbones and datasets and show improvement over MSSR in iteration efficiency.

Critical weaknesses
  1. The method seems to be a combination of GCA’s task formalization (a structured reference-frame/objective constraint fixed before execution) with the MSSR-style sufficiency-driven loop. However, the paper only acknowledges GCA's geometric knowledge, while GCA already separates task formalization from execution. The main difference seems to be the more detailed, predefined FESM schema, since separating parsing from execution is already present in prior work. The paper should clarify these differences and isolate their effects.

  2. Baseline comparison.

    • 2.1 GCA: Based on my understanding of GCA, the perception backends seem to differ with PEPS's Table 1. In the GCA’s codebase configurations, they use the backbone VLM as the detector for the specific benchmarks in this paper, while Table 1 mentions GroundingDINO for PEPS. Which version did you use? Differences in detection may contribute to the reported gains.

    • 2.2 MSSR: PEPS (and GCA) uses MoGe-2 metric scale and Farnebäck optical flow, which are absent from MSSR’s toolset. This can directly affect PEPS's gains over MSSR. PEPS’s largest MMSI gains over MSSR are on Attribute (+5.5) and Motion (+8.0), which the two tools can help with, but in Positional Relationship (+1.9) and Multi-Step Reasoning (+0.4) the gains are smaller. Can you compare the methods using the same toolset to isolate the proposed framework's contribution?

    • 2.3 Small gains and missing statistical support. Not considering the problems above, the reported gains over MSSR are 3.0 percentage points on MMSI, 0.6 on ViewSpatial, and 0.8 on OMNI3D. These gains are small and do not include paired significance tests, so their statistical significance is unclear. Can you provide paired tests on the per-question outcomes?

    • 2.4 Generalization baselines: Table 8 reports results on MindCube-Tiny and OmniSpatial without tool-integrated baselines and Figure 4(b) compares PEPS only with the raw VLMs. Please include GCA, MSSR, and pySpatial under comparable settings.

    Along with the issues mentioned (see W3 as well), a fair comparison with the same backbone, tools, and access to benchmark examples would help isolate that contribution.

  3. Potential test-set contamination of the schema, prompts and examples.

    • 3.1 Schema: The FESM schema is induced from 300 questions sampled from MMSI-Bench, ViewSpatial-Bench, and OMNI3D-Bench, but these datasets are evaluation-only, and inducing based on those questions may lead to test-set contamination. Appendix A.4 describes these as “training/development questions.” Table 8 suggests some transfer, but it lacks tool-integrated baselines. Considering the small gain over MSSR, which operates zero-shot, this makes the comparison difficult to assess. Is PEPS as effective and efficient when its schema is generated without examples from the evaluated benchmarks? Access to these examples may also partly explain PEPS’s lower iteration count.
    • 3.2 Prompts: Worked examples in the schema appendix and in the Parser/Executor prompts correspond to specific MMSI-Bench test items. These examples should be removed when evaluating on that benchmark.
    • 3.3 Examples: The Example Library stores accepted evaluation traces. Do you clear the library between runs and benchmarks, and does retrieval exclude the current question? Otherwise, later runs may retrieve previously accepted answers to the same evaluation questions.
Questions for authors
  1. Were the 300 questions used to build the schema excluded from evaluation?
  2. Why does Table 2 show 48.1 for MSSR score but Figure 4(a) shows 49.5? Where does the 49.5 come from?
  3. What happens if FESM is replaced with GCA’s task constraints while keeping the rest of PEPS fixed?
  4. Can you provide a tool-matched fair comparison between GCA, MSSR, and PEPS? Which object detector was used for GCA and PEPS? 5. Do the gains over MSSR remain when both methods have the same metric-scale and optical-flow tools?
  5. How does the Verifier work with Free-form and Flat-schema? Can you provide an example?
  6. Are the gains statistically significant when comparing results on the same questions?
  7. Can you provide the accuracy of the verifier when it accepts or rejects an answer, along with score calibration? Is the threshold 0.7 tuned? Some details on calibration/ranking of the verifier scores would be helpful.
  8. In table 2, how is MSSR's ViewSpatial overall score calculated?
Reviewer_waxuOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces an agentic pipeline for spatial reasoning that encompasses 3 stages following "parse-before-execute" principle: 1) Parser that maps the query to requirements, 2) Executor that runs and evaluates these requirements in a deterministic manner, and 3) Verifier that checks the outcome and enables early termination or targeted fixes. Experiments of spatial reasoning benchmarks shows that the proposed scheme improves the performance of tool-integrated agents when using ChatGPT-4o while reducing the number of refinement rounds across several models.

Critical strengths
  • The paper tackles spatial reasoning, one of the most challenging tasks for LLMs.

  • The proposed strategy reduces API usage and refinement iterations effectively across different models.

Critical weaknesses
  1. The paper proposes a planning scheme that includes computing some primitives before starting the execution as well as a structured output FESM. However, planning and the use of structured output are well-known practices when designing agentic LLMs, and the paper does not provide any new insights that can push the field forward. Therefore, I find the contribution of the paper very thin.

  2. The paper uses ChatGPT-4o as a base VLMs which is considered obsolete now. To be able to judge if the proposed pipeline would continue to beneficial, evaluating stronger and more recent LLMs such as ChatGPT-5 or Gemini 3 where the gains can diminish. Moreover, open-source VLMs must be evaluated as baselines to understand if this pipeline can be beneficial to the open-source community as well.

  3. I find it concerning that several proprietary models (e.g. ChatGPT 5 and Gemini 3.6) were included in API cost overhead (Table 4 and 5), but were not evaluated as base LLMs in reasoning performance (Table 2 and 3). To be able to draw conclusions on the effectiveness of the proposed approach, the use of base model must be consistent throughout the experiments section.

  4. Color coding in Table 4 and 5 is confusing. For instance, GPU memory for PEPS is higher in Table 5, but it is shown in Green indicating that it is a positive trend. Explaining the color coding in the captions would help interpret these tables.

Minor:

  1. Figure 2 is very cluttered and needs some abstraction to make the idea easier to convey.

  2. The writing, in particular when explaining the proposed pipeline, is unnecessarily repetitive at several places including section 1 and 2. Making these part more concise and expanding the related work early in the paper can help position this work better.

Questions for authors

1- Would the proposed pipeline be beneficial to more recent LLMs beyond ChatGPT-4o?

Reviewer_xgygOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

this paper built an agentic framework for solving visual spatial reasoning tasks based on the idea of "parse before execute" to improve efficiency. The framework consists of 3 agents (1) parser agent to decompose questions into FESM primitives (2) executor agent to retrive spatial information using perception tools and python code (3) a verifier agent check the results and provide feedback for refinement. experiment results on multiple benchmarks show the imporved accuracy and efficiency over baselines

Critical strengths

This paper clearly identified the limitation of existing agentic spatial reasoning frameworks which may lead to unnecessary tool execution. And comprehensive experiment results show the proposed method outperform baselines in both accuracy and efficiency

Critical weaknesses

The difference with existing agentic spatial reasoning frameworks is not clear. The overall pipeline seems to largely follow existing frameworks. e.g. SpatialClaw already incorporates a planning stage that explicitly identifies reference frames, required geometric information, computation steps, and verification procedures before execution. (Pursuing Minimal Sufficiency in Spatial Reasoning) focuses on retriving minimal sufficient information for spatial reasoning which is similar to the author's idea. Therefore the main technical contribution is the FESM formulation which organize spatial requirements into predefined categories and then provide a more strutured interface for the agent. This is not substantial enough as the novelty of a paper and in the experiments it's not clear how much does it contributes to the final performance

Questions for authors

see weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.