- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Spatial Tool Forest (STF) as a method to search for the optimal path of chaining and using tools, instead of expanding the set of predefined tools. On multiple benchmarks, the method shows improvement over baseline visual programming approaches.
Critical strengths
(1) The motivation that simply expanding the tool set would not necessarily be beneficial is insightful, and the approach of instead focusing on making better use of the tools is intuitive.
(2) The method of applying MCTS to tool-use programming is an interesting and well-grounded attempt in this direction.
Critical weaknesses
(1) Baselines in Tab. 1–3 are inconsistent: a lot of baseline models are evaluated in only one table (benchmark) but not in the others. Furthermore, at the core of STF is the attempt to program better with existing tools (or to generate new ones). Given that, a fair baseline nowadays would be to apply coding agents to directly refine the visual program, and then compare to see whether STF still makes a clear contribution to better visual programming.
(2) The performance of STF needs to be further validated with more state-of-the-art base models (rather than just GPT-4o from 2024), to see whether the designs are still useful given a more powerful backbone LLM.
Questions for authors
(1) Related to weakness (2), I encourage the authors to also experiment with more up-to-date models as the backbone of STF, and then ablate its components.
(2) The Limitations section is missing. The authors should discuss the limitations of this paper.
2 comments
Dear Reviewer nThV,
Thank you again for your time and constructive feedback. As the discussion period approaches its end, we would like to follow up on our point-by-point response.
We have clarified which scores remain fixed within a search, how visit counts affect exploration priorities, and how experience writeback updates historical priors across questions. We have also explained STF's differences from pySpatial, VADAR, and AlloSpatial/World2Mind, illustrated the role of its constraints with a concrete failure case, and provided a comparison with AlloSpatial on MINDCUBE-1k using the same GPT-4o controller and input images. For the ablations, we have specified the enabled components and controlled settings, provided comparisons with greedy selection and direct ranking using the same reward, and discussed the small accuracy difference in relation to inference variability. Finally, we have detailed the experience library's initialization, stored information, reuse across questions, and dependence on evaluation order.
We would greatly appreciate it if you could let us know whether our response has addressed your concerns or whether any further clarification would be helpful. If you find our response satisfactory, we would be grateful if you could consider updating your score.
Best regards, The Authors
We thank the reviewer for the insightful and positive comments! In the following, we provide our point-by-point response and hope our response helps address your concerns. We also look forward to the subsequent discussion which may further help to solve the current issues.
W1. Choice of commercial backbone models
Overall accuracy on MINDCUBE-1k (1,050 questions)
Together with the Qwen3-VL-8B-Instruct results, these gains support the applicability of STF beyond GPT-4o, including with open-weight and newer commercial backbones.
W2. Limitations
We discuss STF's limitations in Appendix A.4 (Failure Analysis and Limitations) and illustrate them with two representative failure cases.
W3. Common baselines and coding-agent comparison
Common baselines across Tables 1–3. The revised tables include GPT-4o and Claude Sonnet 4.6 as common baseline models and ViperGPT, VADAR, and pySpatial as common visual programming baselines. These shared baselines make it easier to compare STF with existing methods across the three benchmarks. STF outperforms several listed baselines on each benchmark, supporting its effectiveness across different spatial reasoning tasks.
Comparison with a coding agent (GPT-4o). Following the reviewer’s suggestion, we evaluated a GPT-4o coding agent on the full MINDCUBE-1k split. The agent used public spatial tools to generate executable Python programs and revised them using tool outputs and error messages when its budget allowed. Most questions allowed a single model call.
As backbone models improve, additional rounds of execution feedback may further improve coding agents. STF’s constraints on tool-call order, input and output interfaces, and execution evidence offer reusable methods for future coding agents to generate, revise, and verify visual programs.