- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a parse-execute-verify loop for spatial reasoning. They argue that tool-based methods suffer from "under-committed query compilation" where the proposed plan does not specify what must be resolved before execution. They propose a three-step loop to address this problem. In their parser, an agent selects from the four FESM primitives. Then an Executor agent fills them using perception tools and code, and finally a third Verifier agent scores the answer and provides feedback. Their method shows improvement in 3D spatial reasoning benchmarks such as MMSI, ViewSpatial, and Omni3D. They also provide evidence on the efficiency of their method compared to MSSR and the generalizability of their method to other benchmarks such as MindCube.
Critical strengths
-
The paper provides a detailed and clear explanation of the method and how it works.
-
The paper proposes a structured output interface for the spatial reasoning task. This is a good idea and a step toward more transparent, inspectable spatial reasoning by making audits easier and evaluations more fine-grained.
-
They provide a detailed analysis of the performance of the method across multiple backbones and datasets and show improvement over MSSR in iteration efficiency.
Critical weaknesses
-
The method seems to be a combination of GCA’s task formalization (a structured reference-frame/objective constraint fixed before execution) with the MSSR-style sufficiency-driven loop. However, the paper only acknowledges GCA's geometric knowledge, while GCA already separates task formalization from execution. The main difference seems to be the more detailed, predefined FESM schema, since separating parsing from execution is already present in prior work. The paper should clarify these differences and isolate their effects.
-
Baseline comparison.
-
2.1 GCA: Based on my understanding of GCA, the perception backends seem to differ with PEPS's Table 1. In the GCA’s codebase configurations, they use the backbone VLM as the detector for the specific benchmarks in this paper, while Table 1 mentions GroundingDINO for PEPS. Which version did you use? Differences in detection may contribute to the reported gains.
-
2.2 MSSR: PEPS (and GCA) uses MoGe-2 metric scale and Farnebäck optical flow, which are absent from MSSR’s toolset. This can directly affect PEPS's gains over MSSR. PEPS’s largest MMSI gains over MSSR are on Attribute (+5.5) and Motion (+8.0), which the two tools can help with, but in Positional Relationship (+1.9) and Multi-Step Reasoning (+0.4) the gains are smaller. Can you compare the methods using the same toolset to isolate the proposed framework's contribution?
-
2.3 Small gains and missing statistical support. Not considering the problems above, the reported gains over MSSR are 3.0 percentage points on MMSI, 0.6 on ViewSpatial, and 0.8 on OMNI3D. These gains are small and do not include paired significance tests, so their statistical significance is unclear. Can you provide paired tests on the per-question outcomes?
-
2.4 Generalization baselines: Table 8 reports results on MindCube-Tiny and OmniSpatial without tool-integrated baselines and Figure 4(b) compares PEPS only with the raw VLMs. Please include GCA, MSSR, and pySpatial under comparable settings.
Along with the issues mentioned (see W3 as well), a fair comparison with the same backbone, tools, and access to benchmark examples would help isolate that contribution.
-
-
Potential test-set contamination of the schema, prompts and examples.
- 3.1 Schema: The FESM schema is induced from 300 questions sampled from MMSI-Bench, ViewSpatial-Bench, and OMNI3D-Bench, but these datasets are evaluation-only, and inducing based on those questions may lead to test-set contamination. Appendix A.4 describes these as “training/development questions.” Table 8 suggests some transfer, but it lacks tool-integrated baselines. Considering the small gain over MSSR, which operates zero-shot, this makes the comparison difficult to assess. Is PEPS as effective and efficient when its schema is generated without examples from the evaluated benchmarks? Access to these examples may also partly explain PEPS’s lower iteration count.
- 3.2 Prompts: Worked examples in the schema appendix and in the Parser/Executor prompts correspond to specific MMSI-Bench test items. These examples should be removed when evaluating on that benchmark.
- 3.3 Examples: The Example Library stores accepted evaluation traces. Do you clear the library between runs and benchmarks, and does retrieval exclude the current question? Otherwise, later runs may retrieve previously accepted answers to the same evaluation questions.
Questions for authors
- Were the 300 questions used to build the schema excluded from evaluation?
- Why does Table 2 show 48.1 for MSSR score but Figure 4(a) shows 49.5? Where does the 49.5 come from?
- What happens if FESM is replaced with GCA’s task constraints while keeping the rest of PEPS fixed?
- Can you provide a tool-matched fair comparison between GCA, MSSR, and PEPS? Which object detector was used for GCA and PEPS? 5. Do the gains over MSSR remain when both methods have the same metric-scale and optical-flow tools?
- How does the Verifier work with Free-form and Flat-schema? Can you provide an example?
- Are the gains statistically significant when comparing results on the same questions?
- Can you provide the accuracy of the verifier when it accepts or rejects an answer, along with score calibration? Is the threshold 0.7 tuned? Some details on calibration/ranking of the verifier scores would be helpful.
- In table 2, how is MSSR's ViewSpatial overall score calculated?