acceptodds
Under review as a conference paper at ICLR 2027

Spatial Tool Forests for Structured Spatial Reasoning

Abstract

Existing tool-augmented spatial reasoning methods rely on predefined tools whose limited coverage restricts complex reasoning, while directly expanding the tool library increases selection ambiguity. We propose Spatial Tool Forest (STF), a hierarchical framework that organizes spatial tools into an operation-specific forest and uses constrained Monte Carlo Tree Search (MCTS) for tool-chain planning. Given an image sequence and a natural-language query, the framework analyzes the query to extract its spatial requirements and searches legal trunks from compatible tool-forest branches for a reliable, task-adaptive tool chain. When no existing candidate trunk fully covers the required spatial operations, the framework retains the candidate trunk that best matches the current task as a scaffold, and a constrained LLM-based module generates only the missing tool under explicit interface, spatial-semantic, and evidence constraints. The resulting chain is executed to produce traceable evidence and the final answer. We evaluate STF on MINDCUBE-1k, OMNI3D-BENCH, and SPAR-Bench, observing consistent improvements across datasets and task categories. On MINDCUBE-1k, it improves overall accuracy over pySpatial by 6.38 percentage points and increases accuracy in the challenging around category from 51.60% to 70.40%. These results demonstrate improvements in spatial reasoning accuracy, stability, and adaptability through hierarchical tool organization, constrained planning, and targeted tool generation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
spatial reasoning, tool-using, MCTS

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_o1LGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Spatial Tool Forest (STF) as a method to search for the optimal path of chaining and using tools, instead of expanding the set of predefined tools. On multiple benchmarks, the method shows improvement over baseline visual programming approaches.

Critical strengths

(1) The motivation that simply expanding the tool set would not necessarily be beneficial is insightful, and the approach of instead focusing on making better use of the tools is intuitive.

(2) The method of applying MCTS to tool-use programming is an interesting and well-grounded attempt in this direction.

Critical weaknesses

(1) Baselines in Tab. 1–3 are inconsistent: a lot of baseline models are evaluated in only one table (benchmark) but not in the others. Furthermore, at the core of STF is the attempt to program better with existing tools (or to generate new ones). Given that, a fair baseline nowadays would be to apply coding agents to directly refine the visual program, and then compare to see whether STF still makes a clear contribution to better visual programming.

(2) The performance of STF needs to be further validated with more state-of-the-art base models (rather than just GPT-4o from 2024), to see whether the designs are still useful given a more powerful backbone LLM.

Questions for authors

(1) Related to weakness (2), I encourage the authors to also experiment with more up-to-date models as the backbone of STF, and then ablate its components.

(2) The Limitations section is missing. The authors should discuss the limitations of this paper.

2 comments

We thank the reviewer for the insightful and positive comments! In the following, we provide our point-by-point response and hope our response helps address your concerns. We also look forward to the subsequent discussion which may further help to solve the current issues.

W1. Choice of commercial backbone models

  1. Backbone-matched comparison. We use GPT-4o so that STF and baselines such as pySpatial and VADAR share the same backbone. This separates STF's contribution from differences in the underlying LLM's capability.
  2. Generalization across backbones. Figure 4 also evaluates STF with Qwen3-VL-8B-Instruct, an open-weight vision-language model released in October 2025. The improvement with this backbone indicates that STF is not limited to GPT-4o or to commercial models.
  3. Newer commercial models. Following the reviewer's suggestion, we extended the evaluation to GPT-5.1 (2025) and GPT-5.6 Sol (2026) on all 1,050 MINDCUBE-1k questions. With the complete STF system, GPT-5.1 improves from 55.43% to 59.71% (+4.29 pp), and GPT-5.6 Sol improves from 60.19% to 61.52% (+1.33 pp).

Overall accuracy on MINDCUBE-1k (1,050 questions)

Backbone (release date)Without STFWith STFGain (pp)
GPT-4.1 mini (Apr. 14, 2025)44.67%57.05%+12.38
Qwen3-VL-8B-Instruct (Oct. 15, 2025)34.76%55.33%+20.57
GPT-5.1 (Nov. 13, 2025)55.43%59.71%+4.29
GPT-5.6 Sol (Jul. 9, 2026)60.19%61.52%+1.33

Together with the Qwen3-VL-8B-Instruct results, these gains support the applicability of STF beyond GPT-4o, including with open-weight and newer commercial backbones.

W2. Limitations

We discuss STF's limitations in Appendix A.4 (Failure Analysis and Limitations) and illustrate them with two representative failure cases.

  1. Dependence on 3D reconstruction quality. Limited viewpoints, occlusion, and missing depth can weaken the geometric evidence available for reasoning. In the motion-chain example, the reconstructed geometry after the camera transformation was too sparse to support a reliable distance comparison.
  2. Errors in translating language into spatial operations. STF translates spatial requirements into reference-frame updates, camera movements, and local relation checks. In one failure case, the program used the camera frame where the question required a sofa-centered frame.
  3. Error propagation along the tool chain. Errors in object grounding or reference-frame construction can affect subsequent operations. A program may execute successfully and still base its final answer on an incorrect spatial relation.

W3. Common baselines and coding-agent comparison

Common baselines across Tables 1–3. The revised tables include GPT-4o and Claude Sonnet 4.6 as common baseline models and ViperGPT, VADAR, and pySpatial as common visual programming baselines. These shared baselines make it easier to compare STF with existing methods across the three benchmarks. STF outperforms several listed baselines on each benchmark, supporting its effectiveness across different spatial reasoning tasks.

GroupMethodMINDCUBE-1kOMNI3D-BENCHSPAR-Bench
Baseline modelsGPT-4o42.29%42.9036.39
Claude Sonnet 4.644.19%28.9027.76
Visual programmingViperGPT36.95%26.7014.11
VADAR40.76%40.4014.66
pySpatial50.76%39.3036.10
STF (ours)57.14%42.8038.40

Comparison with a coding agent (GPT-4o). Following the reviewer’s suggestion, we evaluated a GPT-4o coding agent on the full MINDCUBE-1k split. The agent used public spatial tools to generate executable Python programs and revised them using tool outputs and error messages when its budget allowed. Most questions allowed a single model call.

Datasetcoding agent(GPT-4o)Full STFGain (pp)
MINDCUBE-1k38.00%57.14%+19.14

As backbone models improve, additional rounds of execution feedback may further improve coding agents. STF’s constraints on tool-call order, input and output interfaces, and execution evidence offer reusable methods for future coding agents to generate, revise, and verify visual programs.

Dear Reviewer nThV,

Thank you again for your time and constructive feedback. As the discussion period approaches its end, we would like to follow up on our point-by-point response.

We have clarified which scores remain fixed within a search, how visit counts affect exploration priorities, and how experience writeback updates historical priors across questions. We have also explained STF's differences from pySpatial, VADAR, and AlloSpatial/World2Mind, illustrated the role of its constraints with a concrete failure case, and provided a comparison with AlloSpatial on MINDCUBE-1k using the same GPT-4o controller and input images. For the ablations, we have specified the enabled components and controlled settings, provided comparisons with greedy selection and direct ranking using the same reward, and discussed the small accuracy difference in relation to inference variability. Finally, we have detailed the experience library's initialization, stored information, reuse across questions, and dependence on evaluation order.

We would greatly appreciate it if you could let us know whether our response has addressed your concerns or whether any further clarification would be helpful. If you find our response satisfactory, we would be grateful if you could consider updating your score.

Best regards, The Authors

Reviewer_nThVOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Spatial Tool Forest (STF), a framework for tool-augmented spatial reasoning from single- or multi-view images. Built on pySpatial, STF organizes spatial tools into operation-specific branches containing candidate tool chains with explicit dependencies, input requirements, and expected evidence. It analyzes each query to identify compatible branches and uses constrained MCTS to select a chain based on structural compatibility and historical performance. When required operations are unavailable, an LLM generates a missing tool under interface and evidence constraints. Execution results are scored and selectively stored to guide subsequent planning. Experiments on MINDCUBE-1k, OMNI3D-BENCH, and SPAR-Bench report improvements over pySpatial, including a 6.38-percentage-point overall gain on MINDCUBE-1k. Backbone comparisons and ablations further evaluate the framework’s components.

Critical strengths
  1. The framework addresses a concrete reliability problem in spatial tool use. STF explicitly represents tool-chain dependencies, input requirements, and expected evidence, and restricts missing-tool generation to a local operation within an existing chain. This provides a coherent approach to expanding tool coverage while controlling composition errors. Its emphasis on executable chains and inspectable intermediate results makes the design practically useful.
  2. The reported gains extend across benchmarks and controllers. STF improves over pySpatial on MINDCUBE-1k, OMNI3D-BENCH, and SPAR-Bench, with the largest improvement on MINDCUBE’s around category, from 51.60% to 70.40%. Improvements with multiple controllers provide additional evidence that the approach is useful beyond a single model configuration. These results support its empirical value as an extension of existing spatial-tool systems.
Critical weaknesses
  1. The independent contribution over closely related spatial-tool frameworks is insufficiently established. The broad paradigm of reconstructing scenes and exposing spatial knowledge through tools is already established by pySpatial and AlloSpatial/World2Mind. AlloSpatial additionally regulates tool invocation, evidence collection, and geometry–semantics arbitration, while VADAR already explores dynamically generated spatial-reasoning functions. STF’s potentially distinctive contribution therefore lies in its operation-specific tool-chain organization and constrained local completion. However, the paper neither discusses nor experimentally compares against AlloSpatial/World2Mind, leaving its advantages over an existing structured tool-use framework unclear. Evaluation on VSI-Bench, including its Tiny split used by AlloSpatial, would also help assess transfer to video-based spatial reasoning. These omissions make it difficult to establish whether STF offers a substantive methodological advance beyond extending and organizing the pySpatial tool library.
  2. The necessity of Forest-MCTS and the attribution of the reported gains remain unclear. In Section 3.3, planning precedes execution, and the search reward combines a pre-execution structural score with historical performance. As described, these quantities appear fixed for each candidate during a search, so repeatedly visiting a trunk does not obviously reveal new information. It is consequently unclear what Forest-MCTS contributes beyond directly ranking candidates using the same scores. A controlled comparison against greedy ranking and hierarchical retrieval, with identical tools, candidate chains, and execution budgets, is needed. The reported ablations do not resolve this concern: the authors explicitly acknowledge that the organization controls do not isolate organization, planning, completion, and evidence-aware selection. Moreover, Full STF scores 57.14%, versus 57.05% for the “Constrained LLM Tool Completion” configuration, without uncertainty estimates establishing whether this difference is meaningful. Thus, the overall improvement over pySpatial does not sufficiently substantiate the paper’s central planning contribution or justify its additional complexity.
Questions for authors
  1. Clarification—what is learned during Forest-MCTS? In Section 3.3, do the pre-execution score and historical prior change across visits to the same trunk within a search? If they remain fixed, what additional information does repeated exploration obtain, and how does the procedure differ from directly ranking candidates by the same reward?
  2. Clarification and comparative evidence—what distinguishes STF from closely related frameworks? How does STF advance beyond AlloSpatial/World2Mind’s structured spatial knowledge and tool-use harness, and VADAR’s dynamic function generation? In particular, what failure cases require STF’s forest-constrained planning or local completion? Can you provide evidence under matched controller and observation settings? VSI-Bench Tiny would provide a relevant shared evaluation setting.
  3. Clarification—what do the ablations isolate? Which components are enabled in each row of Table 4(a), particularly “MCTS-based Trunk Planning” and “Constrained LLM Tool Completion”? Are the tool library, candidate chains, generation budget, and evidence-selection mechanisms held constant? What evidence supports an independent benefit from MCTS over greedy selection, and is the 57.14% versus 57.05% difference robust to inference randomness?
  4. Clarification—how is experience writeback evaluated? How is the experience library initialized, and is it updated while processing evaluation questions? If so, what information is stored, can it transfer between questions sharing a scene, and are results sensitive to evaluation order? This is consequential because historical performance directly influences subsequent planning.
4 comments

We thank the reviewer for the insightful and positive comments! In the following, we provide our point-by-point response and hope our response helps address your concerns. We also look forward to the subsequent discussion which may further help to solve the current issues.

W1. What does Forest-MCTS learn?

  1. What does Forest-MCTS learn from previous questions? Forest-MCTS learns which tool chains have executed more reliably and provided stronger evidence on previous questions.

  2. Do the pre-execution score and historical prior change across visits to the same tool chain? Both remain fixed within a single search. Across questions, STF updates the historical prior through experience writeback. After each question, STF records execution results that meet the evidence requirements and updates the corresponding chain’s historical average score. STF uses the updated prior when it considers the same chain for a later question. For a chain with no history, the prior is 0.5.

  3. What is the purpose of repeated exploration in utc? Repeated exploration uses updated visit counts to allocate the search budget across candidate chains. After each visit, the selected chain’s visit count n(τ)n(\tau) and the total visit count NN increase. utc then recomputes the exploration bonus c,rpre(τ)log⁡Nn(τ)c\\,r_{\mathrm{pre}}(\tau)\sqrt{\frac{\log N}{n(\tau)}} using the fixed pre-execution score. This gives less-visited chains a chance to be explored and keeps the search from concentrating too early on a single chain.

  4. How does utc differ from direct ranking using the same reward? Direct Same-reward ranks candidates once using the initial reward. utc adjusts candidate priorities at each iteration, giving initially lower-ranked chains a chance to be executed and provide better evidence. In Table A2, Full MCTS Planning scores 0.29 percentage points higher overall than Direct Same-reward on MINDCUBE-1k.

    DatasetDirect Same-rewardFull MCTS Planning (utc)Gain (pp)
    MINDCUBE-1k56.76%57.05%+0.29

    With the same candidate chains and execution budget, utc changed which chains were executed on 319 questions. For example, when asked “From the viewpoint in Image 4, what is to the left of the black chair?”, both methods executed five of the six candidate chains. utc selected a multi-view relation chain and correctly answered A. Exit sign. Direct ranking selected a chain that flipped the left–right axis and answered C. Yellow wall.

We thank the reviewer for the insightful and positive comments! In the following, we provide our point-by-point response and hope our response helps address your concerns. We also look forward to the subsequent discussion which may further help to solve the current issues.

W2. What distinguishes STF, and where do its constraints help?

STF plans tool chains around the spatial operations required by each question, constraining their execution order and where missing operations are inserted. AlloSpatial/World2Mind builds a queryable global spatial map from local visual observations and uses geometric and semantic evidence to verify reasoning. STF focuses on translating the spatial requirements of a question into an executable tool chain. VADAR expands tool coverage by generating functions dynamically, but the generated program may call tools in an unsuitable order, and execution failures can trigger regeneration of the entire program. STF selects a compatible chain first and generates only the missing operation at its designated position. This also addresses a limitation of pySpatial’s predefined tools when its fixed API lacks a required reference-frame transformation or another spatial operation.

Forest constraints help preserve required steps and reference frames, and local completion fills gaps in the available tool set. In Figure 1, the question asks which object is nearest on the left when standing at the reference trash can and facing the same direction. One failing chain omits the viewpoint transformation, and the other uses the camera frame. Both select the brown leather ottoman. STF transforms the object positions into the requested observer frame before evaluating the left relation and correctly selects the green trash can. The component and tool organization comparisons in Table 4 show the following gains on MINDCUBE-1k.

ComparisonConfigurationOverall accuracyGain over previous row in the same panel (pp)
Table 4(a), component ablationExtended Spatial Tools54.29%—
+ Forest-constrained planning56.48%+2.19
+ Constrained local completion57.05%+0.57
Table 4(b), tool organizationUnconstrained LLM-Tool Forest54.00%—
Full STF57.14%+3.14

On MINDCUBE-1k, STF achieves 57.14% overall accuracy, 6.38 percentage points above our pySpatial reproduction and higher than both our AlloSpatial reproduction (40.86%) and the VADAR result reported by Luo et al. (40.76%). We evaluated STF, pySpatial, and AlloSpatial using the same GPT-4o controller and original images on all 1,050 questions. The reported VADAR result is from the same split.

MethodOverallRotationAmongAround
STF (ours)57.14%40.50%57.17%70.40%
pySpatial50.76%35.00%55.67%51.60%
AlloSpatial40.86%49.00%36.00%46.00%
VADAR40.76%33.50%40.67%46.80%

VSI-Bench Tiny is a relevant shared setting for evaluating video spatial reasoning. The matched comparison above uses MINDCUBE-1k. A matched evaluation on VSI-Bench Tiny remains open.

We thank the reviewer for the insightful and positive comments! In the following, we provide our point-by-point response and hope our response helps address your concerns. We also look forward to the subsequent discussion which may further help to solve the current issues.

W3. What do the ablations isolate?

  1. What does each set of ablations examine? Table 4(a) evaluates the incremental addition of STF components. Table 4(b) compares tool organization configurations. Table A2 compares chain selection strategies and examines the role of the historical prior.
  2. Which components are enabled in each row of Table 4(a)? The sequence of configurations reflects how we developed STF. We started with Extended Spatial Tools, which achieved 54.29% overall accuracy. MCTS-based Trunk Planning added search over compatible tool chains, raising accuracy to 56.48%. Constrained LLM Tool Completion retained MCTS selection and generated missing operations only at their designated positions, reaching 57.05%. Full STF then enabled answer selection based on execution evidence and experience writeback, reaching 57.14%.
  3. Which settings are held constant? All configurations in Table 4(a) share the same extended tool library. The executable chains and evidence-selection mechanism vary with the enabled components. In the follow-up comparison, we held the GPT-4o backbone, input images, tool library, initial candidate chains, local completion rules, and answer-selection mechanism fixed. Each strategy could execute up to five candidate chains per question. Local completion was triggered as needed, so the number of generation calls could vary. We evaluated the shared pySpatial baseline separately.
  4. What evidence shows that MCTS contributes beyond greedy selection? With the same candidate chains and execution budget, MCTS changed the set of executed chains on 319 questions and the final answer on 26 questions compared with Direct Same-reward. In Table A2, Full MCTS Planning achieved 57.05% overall accuracy, 1.05 percentage points above Greedy Pre-score Selection (56.00%) and 0.29 percentage points above Direct Same-reward (56.76%).
  5. Is the difference between 57.14% and 57.05% robust to inference randomness? Full STF answered one additional question correctly overall, which falls within the variation observed across repeated runs. Under the same settings, the number of correct answers varied by one or two questions. This variation may reflect differences in VQA responses or in the tools selected by the LLM during local completion.

W4. How is experience writeback evaluated?

  1. How does STF decide what to write back? After each question, STF computes a post-execution score rpostr_{\mathrm{post}} based on tool-chain execution, evidence strength, and the completeness of the required evidence. STF records trajectories scoring 0.70 or higher as successful experiences.
  2. How is the experience library initialized? Each question category starts with a separate, empty experience library. A tool chain receives a default prior of 0.5 when first encountered.
  3. Is the library updated during evaluation? Yes. After each question, STF records qualifying execution results for use on subsequent questions. In the follow-up experiment, 151 questions triggered writeback, and 515 questions encountered at least one candidate chain with a non-default prior.
  4. What information is stored? Each record contains the tool-chain ID, the post-execution score, and evidence diagnostics. During subsequent planning, STF uses the count and mean score of the recorded executions for each chain to update its historical prior.
  5. Can questions about the same scene share experience? Yes. A later question in the same category can reuse a chain’s historical execution statistics if it considers the same chain. The tool-chain ID determines which records are reused.
  6. Are the results sensitive to evaluation order? Question order determines which historical statistics are available to each question and can therefore affect the prior. In the matched follow-up comparison, enabling or disabling the historical prior produced identical final answers on all 1,050 questions.

Dear Reviewer nThV,

Thank you again for your time and constructive feedback. As the discussion period approaches its end, we would like to follow up on our point-by-point response.

We have clarified which scores remain fixed within a search, how visit counts affect exploration priorities, and how experience writeback updates historical priors across questions. We have also explained STF's differences from pySpatial, VADAR, and AlloSpatial/World2Mind, illustrated the role of its constraints with a concrete failure case, and provided a comparison with AlloSpatial on MINDCUBE-1k using the same GPT-4o controller and input images. For the ablations, we have specified the enabled components and controlled settings, provided comparisons with greedy selection and direct ranking using the same reward, and discussed the small accuracy difference in relation to inference variability. Finally, we have detailed the experience library's initialization, stored information, reuse across questions, and dependence on evaluation order.

We would greatly appreciate it if you could let us know whether our response has addressed your concerns or whether any further clarification would be helpful. If you find our response satisfactory, we would be grateful if you could consider updating your score.

Best regards, The Authors

Reviewer_dmzbOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a system called Spatial Tool Forest (SFT), which is a hierarchical framework that uses Monte Carlo Tree Search (MCTS) to more intelligently choose the tools to be used in a spatial reasoning task. The way the method is presented it can be seen as an extension to the pySpatial visual-programming system. The core concept of SFT is the categorization of tools into so-called branches, with each branch holding some hand-designed template programs denoted as trunks.

The method is evaluated on three spatial reasoning benchmark (MINDCUBE-1k, OMNI3D-BENCH, and SPAR-BENCH) in four different categories of existing baselines.

While the main claim of the paper is that it is 6.38 percentage points better than the original pySpatial on the MINDCUBE-1k dataset, it remains questionable why this gain drops to a negligible 0.41 percentage points when looking at Table A1 in the supplementary material.

Critical strengths
  1. The presented problem is interesting and real, since there is indeed a tension between tool coverage and selection ambiguity.

  2. The gains on MINDCUBE-1k as they are presented in the paper are large.

Critical weaknesses

*) The headline gain (6.38 pp on MINDCUBE-1k) almost completely disappears on the full split as shown in Table A1 in the supplementary material.

*) Comparisons to state-of-the-art commercial models are completely outdated as the method only compares to models that were introduced in 2024.

Some minor weaknesses:

*) The resolution of Figure 1 is far too low to allow the reader to properly read the query and the text in the output images. I would highly recommend to make this an e.g. Illustrator figure with everything that can be vector graphics (text, pipeline illustration) to be vector graphics.

*) In Figure 1, it is completely unclear which image is which, I would rather guess in the order it is shown it is right, front, left. Where is there even a black trash can? It looks rather blue in the front image.

Conclusion: In the current form, the paper is a clear Reject.

Questions for authors
  1. How come that there is +6.38 pp on MINDCUBE-1k but only +0.41 pp on the full MINDCUBE dataset when comparing Table 1 in the main paper and Table A1 in the supplementary material?

  2. Why does the paper use outdated commercial models from 2024 (e.g., GPT-4o)?

3 comments

We thank the reviewer for the insightful and positive comments! In the following, we provide our point-by-point response and hope our response helps address your concerns. We also look forward to the subsequent discussion which may further help to solve the current issues.

W1. Gains on MINDCUBE-1k and MINDCUBE-Full

The difference mainly reflects the category distributions of the two splits: Rotation and Around, where STF provides its strongest improvements, make up a much smaller share of MINDCUBE-Full.

Among, Rotation, and Around account for 86.05%, 5.11%, and 8.84% of MINDCUBE-Full, compared with 57.14%, 19.05%, and 23.81% of MINDCUBE-1k. On MINDCUBE-Full, STF improves Rotation and Around by 7.59 and 10.11 percentage points, while Among remains close to the baseline. Weighted by category size, these changes yield an overall gain of approximately 0.41 points.

Therefore, the overall gain depends on the category composition of each split: the 6.38-point gain applies to MINDCUBE-1k, while on MINDCUBE-Full STF still yields substantial improvements on Rotation and Around, whose contribution to the overall score is limited by their small share.

W2. Choice of commercial backbone models

  1. Backbone-matched comparison. We use GPT-4o so that STF and baselines such as pySpatial and VADAR share the same backbone. This separates STF's contribution from differences in the underlying LLM's capability.

  2. Generalization across backbones. Figure 4 also evaluates STF with Qwen3-VL-8B-Instruct, an open-weight vision-language model released in October 2025. The improvement with this backbone indicates that STF is not limited to GPT-4o or to commercial models.

  3. Newer commercial models. Following the reviewer's suggestion, we extended the evaluation to GPT-5.1 (2025) and GPT-5.6 Sol (2026) on all 1,050 MINDCUBE-1k questions. With the complete STF system, GPT-5.1 improves from 55.43% to 59.71% (+4.29 pp), and GPT-5.6 Sol improves from 60.19% to 61.52% (+1.33 pp).

    Overall accuracy on MINDCUBE-1k (1,050 questions)

    Backbone (release date)Without STFWith STFGain (pp)
    GPT-4.1 mini (Apr. 14, 2025)44.67%57.05%+12.38
    Qwen3-VL-8B-Instruct (Oct. 15, 2025)34.76%55.33%+20.57
    GPT-5.1 (Nov. 13, 2025)55.43%59.71%+4.29
    GPT-5.6 Sol (Jul. 9, 2026)60.19%61.52%+1.33

Together with the Qwen3-VL-8B-Instruct results, these gains support the applicability of STF beyond GPT-4o, including with open-weight and newer commercial backbones.

W3. Figure 1 resolution and annotations

Figure resolution. We have rebuilt Figure 1 with vector text, pipeline diagrams, arrows, and annotations, and exported it as PDF and SVG. The PNG version is 9448 × 5070 pixels (approximately 1333 DPI at a width of 180 mm).

Image and object identification. The inputs are now explicitly labeled as Image 1, Image 2, and Image 3, and blue outlines with consistent "Reference bin" labels identify the queried object. We also note that the "black trash can" in the original question has a black lid and a dark-blue body in the photographs.

Reviewer_dmzbOpenReview ↗

Some but not all concerns were fully addressed. Increase of score from 1 to 2.

I would like to thank the authors for the response and the additional conducted experiments.

*) Figure 1: The new version is much better. This concern is resolved.

*) MINDCUBE-1k vs. full MINDCUBE: The category distribution explains part of the difference, but as I see it not all of it. On the full split, the proposed STF is actually slightly worse than pySpatial on Among (55.86 vs. 56.87), which is 86% of the benchmark. Also, the gain on Around drops from 18.8 to 10.11. So, the large improvement seems to be at least partly specific to the 1k subset. I think the full-split results should be reported in the main paper and not only in the supplementary material.

*) Newer models: Thank you for adding GPT-5.1 and GPT-5.6 Sol. However, the gain gets much smaller with the newer and stronger models (+4.29 pp for GPT-5.1, +1.33 pp for GPT-5.6 Sol). Most interestingly, plain GPT-5.6 Sol beats STF with GPT-4o. I would be interested to see pySpatial with the same newer models, because a part of the gain comes from pySpatial itself.

Overall, I appreciate the effort the authors put into the reply to my concerns and I will raise my score from 1 to 2, but I am still leaning towards rejection.

We thank the reviewer for the insightful and positive comments! In the following, we provide our point-by-point response and hope our response helps address your concerns. We also look forward to the subsequent discussion which may further help to solve the current issues.

W1. MINDCUBE-1k and Full MINDCUBE

We will move the full MINDCUBE results into the main paper in the revised manuscript and add a category-level analysis.

On the full benchmark, STF and pySpatial perform similarly on Among, while STF’s main advantages lie in Rotation and Around. Their Among accuracies are 55.86% and 56.87%, respectively, showing that the gains vary across task categories.

Differences between the samples within Around may also affect the magnitude of the gain. Questions impose different demands on geometric reconstruction, object localization, and reference-frame transformation, which may influence the benefits of tool-chain planning and local completion. Although the gain decreases from 18.80 to 10.11 percentage points, a substantial improvement remains on the full evaluation set.

STF continues to achieve substantial improvements in both Rotation and Around on full MINDCUBE, indicating that its benefits extend to the full evaluation set.

W2. Newer Backbones and Comparison with pySpatial

We observed accuracy improvements from STF across four additional backbones, GPT-4.1 mini, Qwen3-VL-8B-Instruct, GPT-5.1, and GPT-5.6 Sol. The improvements were 12.38, 20.57, 4.29, and 1.33 percentage points, respectively. Although their magnitudes vary across backbones, these results support the effectiveness of STF with both open-weight and commercial models at different capability levels.

Following the reviewer’s suggestion, we additionally compared against pySpatial using the same newer backbones.

MethodOverallRotationAmongAround
pySpatial + GPT-5.156.10%38.00%56.33%70.00%
pySpatial + GPT-5.6 Sol60.29%41.00%60.00%76.40%

STF achieves higher overall accuracy than pySpatial with both newer commercial backbones. Together with the open-weight backbone results, this supports the applicability of tool-forest organization, constrained planning, and local completion across different types of backbone models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.