acceptodds
Withdrawn from ICLR 2027

Perception, Not Reasoning, Limits Video Spatial Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Current vision-language models (VLMs) struggle with spatial reasoning across time, and aggregate benchmark scores obscure whether failures arise from perception or reasoning. We introduce SPLIT (Separating Perception from LLM Inference with Tools), a training-free harness for video spatial reasoning that builds a metric 3-D reconstruction of a video and provides a base VLM with tools to query it. SPLIT improves accuracy on four video benchmarks (VSIBench, VSTIBench, ReVSI, and DSI-Bench) and the multi-image MMSI-Bench by up to 20.6% over the base VLM answering without tools, with gains extending to open-weight base VLMs. SPLIT helps more when a question's objects are not co-visible in any frame: on VSIBench relation questions, its improvement over the base VLM alone nearly doubles when no frame shows all the objects a question names. Accurate perception closes most of the gap to a perfect score: when the tools return ground truth (SPLIT + GT), that gap shrinks 3.6-fold on subsets of VSIBench, VSTIBench, and ReVSI, and accuracy reaches an average of 92.9%; two open-weight base VLMs nearly match it, with Gemma-4-31B averaging 90.4% and Qwen3.6-27B 88.7%. We will release the harness and code.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video spatial understanding, agentic harness

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zvdmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors design the harness SPLIT to deal with spatial reasoning problems in a training-free manner. First, the scene is reconstructed into 3D space to support further inspect. Then, the VLM will call perception tools, write and execute python code to obtain necessary information about the scene to answer the question. The evaluation on 5 different benchmarks shows that SPLIT can improve the performance of a VLM without training and the performance will be higher if providing the ground truth result of tools.

Critical strengths
  1. The authors mimic the procedure of current general agents that leverage different tools to enhance the capability of VLM in spatial reasoning by dividing perception ability into tool calls. This could be widely applied to different VLMs without training them for spatial reasoning.

  2. The harness framework makes how the VLM calls the tools and leverages the information for decision clearer so that it is more explainable to analyze whether the problem lies in the perception or reasoning.

Critical weaknesses
  1. The inference cost of the system is pretty, compared with a single turn of VLM inference. According to the experiments, SPLIT uses a median of 16 VLM calls and 5980 generated tokens per question, which can be costly and restrict application for just a spatial reasoning problem.

  2. I think the experiments that compare SPLIT with SPLIT+GT do not clearly separate perception from reasoning. Since the GT information also provide task labels, target-object labels and task-specific procedures, they could also provide information which should be acquired from reasoning.

Questions for authors

Please refer to the weaknesses

1 comment
AuthorsOpenReview ↗

Response to Reviewer zvdm (R3)

Thank you for your review and for recognizing that SPLIT applies broadly without training and explains its errors by separating perception from reasoning.

1. Is SPLIT's inference cost too high compared with a single VLM call?

No: SPLIT answers up to 1.4× faster than the base VLM. It also costs less, because it generates 1.6× fewer tokens: a median of 1,379 per question compared to the base VLM's 2,238. Our answer to Question 1 of the general response shows that SPLIT is faster, uses fewer tokens and costs less with both Gemini models (Table R1).

2. Do the ground-truth experiments separate perception from reasoning, given that they also received task labels, object labels and procedures?

Yes. Our answer to Question 3 of the general response shows that with SPLIT's own prompt, which has no task labels, object labels or procedures, ground-truth perception still reaches 91% on average, within 2% of the submission. So the gain comes from better perception, not from task information. The open-weight Gemma-4-31B and Qwen3.6-27B show the same pattern (Table R3).

Reviewer_8ovaOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

SPLIT proposes a perception-inference separation framework for spatial understanding in videos that requires no additional training. The method constructs a shared metric 3D representation, extracts spatial information using object localization and geometric measurement tools, and then performs inference by invoking the VLM planning tool and combining it with Python operations. Main experiments on five benchmarks demonstrate that this method enhances the performance of base models, particularly for spatial relationship and camera motion tasks.

Critical strengths
  1. The problem is valuable, and the experimental results are clear. The main model outperformed its own baseline on all five benchmarks, with the VSTIBench score improving from 57.7 to 78.3, supporting the practical value of explicit geometry for spatial question-answering.

  2. The evidence is quite persuasive. On VSTIBench, while keeping the prompts unchanged, truth perception raised the analysis subset score from 81.5 to 95.7, indicating that there is significant room for improvement in the quality of perception.

Critical weaknesses
  1. The method’s innovativeness and distinctions from prior work are not yet sufficiently established. The approach of using language models to decompose tasks, invoking off-the-shelf vision modules, and programmatically combining the results has already been established by VISPROG [r1] and ViperGPT [r2]. Furthermore, SpatialPIN [r3] has enhanced the spatial reasoning capabilities of VLMs using external 3D foundation models, while VLM-Grounder [r4] and GeoProg3D [r5] have demonstrated tool-assisted multi-view 3D localization and compositional spatial reasoning, respectively.

  2. For spatial intelligence tasks, inference speed is also critical. The evaluation of inference efficiency is insufficient. The paper does not provide end-to-end inference latency or a breakdown of the time required for 3D reconstruction, perception tools, and language model calls.

  3. Since this method simultaneously incorporates task labels, object labels, and task-specific problem-solving steps into the reasoning process, its performance improvement cannot be attributed solely to perceptual enhancements. Structured measurement and procedural execution also alter the reasoning task itself. The authors need to conduct detailed ablation experiments to clarify the sources of the performance gains.

[r1] Gupta, Tanmay, and Aniruddha Kembhavi. "Visual programming: Compositional visual reasoning without training." 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023.

[r2] Surís, Dídac, Sachit Menon, and Carl Vondrick. "Vipergpt: Visual inference via python execution for reasoning." 2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2023.

[r3] Ma, Chenyang, et al. "Spatialpin: Enhancing spatial reasoning capabilities of vision-language models through prompting and interacting 3d priors." Advances in neural information processing systems 37 (2024): 68803-68832.

[r4] Xu, Runsen, et al. "Vlm-grounder: A vlm agent for zero-shot 3d visual grounding." arXiv preprint arXiv:2410.13860 (2024).

[r5] Yasuki, Shunsuke, et al. "Geoprog3d: Compositional visual reasoning for city-scale 3d language fields." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.

Questions for authors

See Weakness.

My main concern relates to the novelty of the work; at this point, I am having difficulty identifying the core differences between this paper and existing research.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 8ova (R2)

Thank you for your detailed review and for finding the problem valuable and the results clear. Your questions on related systems, latency and the sources of the gains helped us strengthen both the comparisons and the ground-truth analysis.

1. How does SPLIT differ from VISPROG, ViperGPT, SpatialPIN, VLM-Grounder and GeoProg3D?

Among these systems, only SPLIT natively analyzes video in 3D without per-scene training, and we use it to run extensive experiments showing that perception, not reasoning, limits VLMs on spatial questions. Four of the five systems do not work with videos: VISPROG [r1] works on single images or image pairs, SpatialPIN [r3] on a single image, VLM-Grounder [r4] needs depth and camera poses for every frame, and GeoProg3D [r5] trains a 3D Gaussian field for each scene, which takes 15 minutes to 6 hours. ViperGPT [r2] is the only one that can work with video, but its perception tools work on single images: its code can pick a frame, but nothing relates objects seen in different frames in space. SPLIT instead enables the VLM to reason in metric 3D space through Python functions. It builds one metric 3D reconstruction from the video in ~4 s, so the VLM can relate objects seen in different frames.

When comparing with ViperGPT where both use Gemini 3.1 Pro, SPLIT makes 2.8× fewer errors than ViperGPT on the static benchmarks (VSIBench 69% vs 29%, VSTIBench 82% vs 32%, ReVSI 73% vs 25%) and 1.5× fewer on the dynamic ones, where Table R2 of the general response shows SPLIT ahead on every benchmark.

Most importantly, SPLIT lets us test what limits VLMs: when its tools return ground-truth perception instead of estimates, SPLIT reaches 91% accuracy on average (Question 3 of the general response), so better perception, not reasoning, is what VLMs need. We will release the harness so others can build on it. We describe the differences to these models in the revised paper.

2. What is SPLIT's end-to-end latency, and how much time goes to 3D reconstruction, perception tools and VLM calls?

SPLIT answers up to 1.4× faster than the base VLM end-to-end. Our answer to Question 1 of the general response shows that SPLIT is faster, generates fewer tokens and costs less than the base VLM with both Gemini models (Table R1). In SPLIT's median end-to-end runtime, 3D reconstruction takes 4.2 s and runs alongside the first VLM call, SAM3 and the other tools take 2.3 s, VLM calls take 17.4 s, and the remaining 0.5 s passes between calls (Figure 5 of the revised paper shows the timeline).

3. Do SPLIT's gains come from perception, or from task labels, object labels and task-specific steps?

SPLIT's gains come from perception: SPLIT's prompt is the same for every question and contains no task labels, object labels or task-specific steps. In our submission, we included task information only in the ground-truth experiments for VSIBench, not in SPLIT's prompt. We provide the prompt in Listing 1 in the appendix of the revised paper for review. We have also rerun the VSIBench ground-truth experiments without task information (Question 3 of the general response): ground-truth perception still cuts Gemini 3.1 Pro's errors 3× (74% to 91%), within 2% of the 92.9% with task information, and Qwen3.6-27B scores even higher without it (Table R3). SPLIT's gains over the base VLM also point to perception: they nearly double when no frame shows all the objects a question names, where only the 3D reconstruction can relate them (Table R4).

Table R4. SPLIT's gain nearly doubles when no frame shows all the objects a question names. Accuracy (%) on VSIBench relation questions with Gemini 3.1 Pro; co-visible means at least one of the 32 frames shows all the named objects. This is Table 5 of the revised paper (Appendix Table 8 of the submission).

Named objectsQuestionsBase VLMSPLITGain
Co-visible23776.388.2+11.9
Never co-visible1,32262.584.6+22.1

Thank you to the author for their response. I still have some reservations regarding the innovativeness of the paper. Specifically, firstly, the fact that existing large models lack spatial perception capabilities is hardly a surprising finding; much recent work has been focused on enhancing their perceptual abilities by incorporating features from 3D foundation models such as VGGT. Clearly, by utilising current state-of-the-art models in this field, it is naturally possible to achieve better perceptual results. However, this does not address the inherent lack of spatial perception capabilities in large models themselves.

Secondly, utilising advanced model tools to enhance a model’s capabilities is also a well-established practice. Judging by the scope of this paper’s research, the orchestration and training of these tools—particularly their adaptation to video inputs—are not the main focus; rather, the paper draws upon the capabilities of existing foundation models and incorporates some heuristic designs.

Overall, I personally believe that the work presented in this paper is more of an engineering contribution, lacking sufficient new knowledge and innovation. For ICLR, I place greater value on the paper’s novel contributions to the field of spatial intelligence than on high scores achieved on a few specific datasets. Therefore, I maintain my current rejection score.

Reviewer_riaqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SPLIT, a training-free framework that separates perception from reasoning by equipping vision-language models with tools to query metric 3D reconstructions of videos. SPLIT improves accuracy across five video and multi-image spatial reasoning benchmarks by up to 20.6% over base models without tools. Its gains are particularly strong when the objects mentioned in a question never appear together in a single frame. With ground-truth perception, SPLIT achieves an average accuracy of 92.9% across several VLM spatial reasoning benchmarks.

Critical strengths
  1. SPLIT consistently improves performance across different VLMs and spatial reasoning benchmarks.
  2. SPLIT is training-free, requiring no additional training data or computational resources for model training, while preserving the base VLM’s generalization capabilities.
Critical weaknesses
  1. As acknowledged by the authors, SPLIT incurs substantial inference-time overhead, with a median of 16 base VLM calls and 5,980 generated tokens per question on the VSIBench analysis subset. This cost may limit its scalability and applicability in resource-constrained settings.
  2. The evaluation primarily covers static indoor scenes and a limited range of motion questions. Although the authors acknowledge this limitation, SPLIT’s effectiveness on unconstrained videos with dynamic scenes remains unclear. I suggest that the authors evaluate SPLIT on additional benchmarks, such as [1][2][3].

[1] Li, Yun, et al. Sti-bench: Are mllms ready for precise spatial-temporal world understanding?. ICCV 2025. [2] Wu, Mingrui, et al. From indoor to open world: Revealing the spatial reasoning gap in mllms. CVPR 2026. [3] Sun, Pengzhan, et al. ReSTI: A Source-Grounded Audit and Repair of STI-Bench. arXiv 2026.

Questions for authors

Please refer to the weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer riaq (R1)

Thank you for your careful review and for recognizing that SPLIT gives consistent gains across base VLMs and benchmarks without training while preserving the base VLM's generalization. We answer both of your concerns in the general response: efficiency in Question 1 and dynamic videos in Question 2.

1. Does SPLIT's inference overhead (16 VLM calls and 5,980 tokens per question) limit its use?

No: SPLIT answers up to 1.4× faster than the base VLM. It also costs less, because it generates 1.6× fewer tokens: a median of 1,379 per question compared to the base VLM's 2,238. Our answer to Question 1 of the general response shows that SPLIT is faster, uses fewer tokens and costs less with both Gemini models (Table R1).

2. Does SPLIT work on unconstrained videos with dynamic scenes?

Yes: SPLIT beats the base VLM with all three base VLMs on the three benchmarks you suggested and on DSI-Bench. Our answer to Question 2 of the general response shows the results (Table R2): SPLIT gains even more on dynamic videos (+12.6%) than on static benchmarks (+11.4%).

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (1/2)

We thank reviewers for their thoughtful and helpful comments. They agree that SPLIT addresses an important problem with clear, consistent gains across spatial benchmarks (R1, R2), applies broadly across base VLMs with zero training while preserving generalization (R1, R3), excels when queried objects never appear in the same frame (R1), and separates perception from reasoning errors and shows the room left for better perception (R2, R3). We answer the common questions here and each reviewer's remaining points in the individual replies. Throughout, SPLIT uses Gemini 3.1 Pro unless we name another base VLM, and the base VLM answers without tools in a single VLM call. The revised paper marks the new results in blue: Table 2 (ground truth), Table 3 (dynamic videos), and Table 4 with Figures 3 and 5 (efficiency).

1. Question: Efficiency - Is SPLIT’s inference speed, token use, and cost practical (R1, R2, R3)?

SPLIT is faster, more token-efficient, and cheaper than the base VLM.

SPLIT Pareto-dominates the base VLM with both Gemini models: it answers up to 1.4× faster end-to-end, generates up to 3.8× fewer tokens, costs less and makes up to 1.5× fewer errors (+13% accuracy) (Table R1 below). SPLIT is also more accurate than the base VLM at every thinking level (Figure 3 of the revised paper).

Why does the base VLM generate up to 3.8× more tokens than SPLIT? By analyzing its reasoning traces, we observe that the base VLM spends considerable thinking tokens trying to track objects across views, build a cognitive map across the video, and guess metric distances. Roughly 70% of its reasoning text goes into this mental 3D reconstruction. SPLIT instead offloads perception to fast vision models: G3T and UniK3D build a metric 3D point cloud and camera poses, and SAM3 segments the queried objects in parallel, so the VLM writes a few lines of Python geometry instead of reconstructing the 3D scene in text.

We reduce SPLIT's latency without changing its tools or models, cutting median generation from 5,980 tokens per question in the submission to 1,379. Independent tool calls such as SAM3 run concurrently, we remove a verification step that does not change accuracy, and 3D reconstruction runs in parallel with the VLM's first planning step, which ends after the reconstruction.

Table R1. SPLIT Pareto-dominates the base VLM across accuracy, latency, generated tokens, and cost. Accuracy is the average over the VSIBench, VSTIBench and ReVSI benchmarks; time, generated tokens and cost are medians per question. Cost sums API token prices and GPU time ($4/hr H100 for 3D reconstruction and SAM3).

ModelMethodAcc. (%)Time (s)Gen. TokensAPI ($)GPU ($)Total ($)
Gemini 3.1 ProBase VLM61.824.82,238$0.0955$0.0000$0.0955
SPLIT74.420.21,379$0.0850$0.0061$0.0911
Gemini 3.8 FlashBase VLM68.9185.918,992$0.0969$0.0000$0.0969
SPLIT78.1134.94,985$0.0911$0.0054$0.0966
AuthorsOpenReview ↗

General response (2/2)

2. Question: Generalization - Do SPLIT’s gains extend to dynamic, unconstrained indoor and outdoor videos (R1)?

SPLIT generalizes to dynamic, unconstrained indoor and outdoor videos.

We evaluate on the three dynamic benchmarks R1 requested (STI-Bench, ReSTI, and outdoor OSI-Bench) as well as DSI-Bench, spanning indoor rooms, tabletops, street driving, and outdoor walking. On dynamic videos, SPLIT uses a 4D reconstruction from Point4D. SPLIT beats the base VLM in all 12 comparisons across all three base VLMs (Gemini 3.1 Pro, Gemma-4-31B, and Qwen3.6-27B). We also run ViperGPT, which R2 asked about, with Gemini 3.1 Pro. On average, SPLIT gains even more on dynamic videos (+12.6%) than on the static benchmarks (VSIBench, VSTIBench and ReVSI; +11.4%).

Table R2. SPLIT beats the base VLM in every dynamic-video comparison. Accuracy (%) on 600 questions per benchmark and 1,000 on DSI-Bench. Bold marks the better score of each pair.

ModelMethodSTI-BenchReSTIOSI-BenchDSI-Bench
Gemini 3.1 ProBase VLM48.540.735.250.2
SPLIT51.257.040.877.2
ViperGPT48.030.716.739.3
Gemma-4-31BBase VLM50.835.233.045.0
SPLIT54.054.339.459.6
Qwen3.6-27BBase VLM42.735.336.448.8
SPLIT52.856.039.871.0

3. Question: Perception-reasoning separation - Do the ground-truth experiments cleanly isolate perceptual improvements from changes to prompting or reasoning (R2, R3)?

SPLIT's prompt contains no task information, and ground-truth perception still reaches 91% on average, within 2% of the submission's 92.9%.

In these experiments, SPLIT uses the same prompt for every question: it describes the tools and coordinate conventions and contains no task labels, object labels or procedures for solving a task. The submission's VSIBench ground-truth experiments added each question's task label, object labels and a per-task procedure. We reran the VSIBench ground-truth experiment without task labels, object labels or procedures. With Gemini 3.1 Pro, ground-truth perception reaches 91% on average across the three benchmarks, against 92.9% with the submission's task information, so the gain comes from better perception rather than task information. Gemma-4-31B and Qwen3.6-27B also reach 90% on average without task information, and Qwen3.6-27B scores even higher than in the submission (Table R3).

Table R3. With ground-truth perception, SPLIT scores about 90% accuracy with every base VLM, as in the submission. Accuracy (%) on VSIBench, VSTIBench and ReVSI. The “+ GT” rows use SPLIT’s prompt which contains no task information. (*) For VSIBench, in the original submission we ran the ground-truth experiments adding task information.

ModelMethodVSIBenchVSTIBenchReVSI
Gemini 3.1 ProSPLIT68.581.573.2
+ GT87.3 (91.9*)95.791.0
Gemma-4-31BSPLIT65.877.158.6
+ GT86.2 (86.8*)89.694.8
Qwen3.6-27BSPLIT65.469.564.4
+ GT87.5 (84.7*)88.892.6

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.