- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this paper, the authors design the harness SPLIT to deal with spatial reasoning problems in a training-free manner. First, the scene is reconstructed into 3D space to support further inspect. Then, the VLM will call perception tools, write and execute python code to obtain necessary information about the scene to answer the question. The evaluation on 5 different benchmarks shows that SPLIT can improve the performance of a VLM without training and the performance will be higher if providing the ground truth result of tools.
Critical strengths
-
The authors mimic the procedure of current general agents that leverage different tools to enhance the capability of VLM in spatial reasoning by dividing perception ability into tool calls. This could be widely applied to different VLMs without training them for spatial reasoning.
-
The harness framework makes how the VLM calls the tools and leverages the information for decision clearer so that it is more explainable to analyze whether the problem lies in the perception or reasoning.
Critical weaknesses
-
The inference cost of the system is pretty, compared with a single turn of VLM inference. According to the experiments, SPLIT uses a median of 16 VLM calls and 5980 generated tokens per question, which can be costly and restrict application for just a spatial reasoning problem.
-
I think the experiments that compare SPLIT with SPLIT+GT do not clearly separate perception from reasoning. Since the GT information also provide task labels, target-object labels and task-specific procedures, they could also provide information which should be acquired from reasoning.
Questions for authors
Please refer to the weaknesses
Response to Reviewer zvdm (R3)
Thank you for your review and for recognizing that SPLIT applies broadly without training and explains its errors by separating perception from reasoning.
1. Is SPLIT's inference cost too high compared with a single VLM call?
No: SPLIT answers up to 1.4× faster than the base VLM. It also costs less, because it generates 1.6× fewer tokens: a median of 1,379 per question compared to the base VLM's 2,238. Our answer to Question 1 of the general response shows that SPLIT is faster, uses fewer tokens and costs less with both Gemini models (Table R1).
2. Do the ground-truth experiments separate perception from reasoning, given that they also received task labels, object labels and procedures?
Yes. Our answer to Question 3 of the general response shows that with SPLIT's own prompt, which has no task labels, object labels or procedures, ground-truth perception still reaches 91% on average, within 2% of the submission. So the gain comes from better perception, not from task information. The open-weight Gemma-4-31B and Qwen3.6-27B show the same pattern (Table R3).