acceptodds
Under review as a conference paper at ICLR 2027

SpatialRoute-Bench: Benchmarking Long-Horizon Spatial Intelligence of Vision-Language Models in Outdoor Videos

Abstract

Vision-language models (VLMs) have made rapid progress in spatial understanding, yet existing benchmarks largely focus on local or scene-level spatial relations, leaving long-horizon spatial understanding in continuous outdoor videos underexplored. Such videos require models to integrate spatial evidence scattered across long time spans and large physical distances, as the observer's viewpoint and direction of travel change continuously along the route. To bridge this gap, we introduce SpatialRoute-Bench, a geometry-grounded benchmark for evaluating how VLMs perceive, reason over, and use spatial information along extended outdoor trajectories. SpatialRoute-Bench comprises 7,000 multiple-choice QA pairs across seven tasks, organized into a three-level hierarchy in which each level demands more extensive spatial integration than the one below: perception, reasoning, and decision. To enable scalable benchmark construction, we develop a fully automated pipeline that couples VLM-based semantic grounding with 3D reconstruction: semantic annotations determine which landmarks can be referred to, whereas reconstructed camera poses, trajectories, and scene geometry provide the spatial evidence from which answers are derived. Task-specific generators then convert this evidence into questions, retaining only those that pass geometric and semantic validity checks. Evaluations of representative VLMs reveal three critical limitations. First, there remains substantial room for improvement: even the strongest model reaches only 28.25% overall accuracy against a 25% random baseline. Second, camera-motion perception remains unreliable: even for a single motion event, average accuracy is only 26.2%. Third, spatial relations degrade over longer horizons: when the same relation must be recovered between landmarks observed minutes apart, average accuracy drops from 29.2% to 22.1%, below the random baseline. SpatialRoute-Bench exposes a clear gap between general-purpose video understanding and grounded spatial understanding of long outdoor traversals, and provides a diagnostic testbed for closing it.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, VLM, Spatial understanding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yDN5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors focus on long horizon spatial reasoning problem in outdoor scenes, which is neglected by the former methods and benchmarks. In this problem, the video illustrates an outdoor scene so that the distance scale and layout differs from indoor scenes a lot and the questions may require long horizon knowledge to answer. To investigate in this problem, the authors propose to construct a benchmark for evaluation. The authors propose an autonomous framework to construct both geometry and semantic information of the scene and generate questions spreading across three levels. The evaluation of current VLMs on this benchmark shows that they still perform badly in this problem.

Critical strengths
  1. The authors focus on a meaningful problem that the outdoor scenes have larger distance scale and more complicate layouts, while the former benchmarks or methods are targeted at indoor scenes. Moreover, videos in previous benchmarks are short horizon, which can limit their evaluation in reality.

  2. The three levels analysis of perception, reasoning and decision can support comprehensive analysis of the capability of one model.

  3. The evaluation of current models covers different kinds of VLMs, including propriety models, open-sourced models, spatial reasoning models, which can provide useful insights for further study.

Critical weaknesses
  1. I wonder how many videos are in the benchmarks and how long for each video. If the number of videos is too small, the diversity of the scene can be limited and if the duration of each video is not long enough, the analysis for long horizon reasoning can be not comprehensive enough.

  2. The videos in the benchmark mainly focus on the static outdoor scenes where the objects do not change their position. However, for most outdoor scenes, the objects are mobile and changing their positions continuously, which can limit the application of the proposed benchmark.

  3. The benchmark uses recorded routes rather than interactive navigation so I think the proposed destination tasks follow a unique recorded branch–destination correspondence rather than general shortest-path planning.

Questions for authors

Please refer to the weaknesses.

4 comments

Response (W1): We reconstructed and semantically annotated approximately 300 long outdoor videos. Three annotators then screened the reconstruction quality, spatial relations, and semantic labels to select 22 videos. Their durations range from 40.91 to 83.33 minutes, with a mean of 54.68 minutes.

  • Duration for long-horizon analysis: The mean duration of 54.68 minutes is the longest among the compared benchmarks (Table 1), e.g., 1.2 minutes for VSI-Bench and 38.62 minutes for VSI-Super-Wild. This duration is sufficient to expose long-horizon failures: with the same four-way relation, mean accuracy drops from 29.18% on P1 to 22.09% on R1, where landmarks are observed at least 180 s apart, and nine of the ten models degrade (Sec. 4.3). Fig. 3(b) further shows the temporal evidence spans of all tasks.
  • Diversity: Each video is a continuous long-distance traversal that passes through many distinct scenes, and the QA pairs span a wide range of spatial displacements and viewpoint changes (Fig. 3(c)). We acknowledge that the number of videos is limited. This results from our strict quality screening: many candidate videos would add scene diversity, such as night-time traversals or recordings with fast camera motion, but their 3D reconstruction is less reliable due to low illumination or motion blur, which would compromise the geometry-derived answers. The benchmark can be expanded with more diverse videos as reconstruction quality improves.

Response (W2): Thank you for raising this point. We acknowledge that this aspect was not sufficiently clarified in the original manuscript. To clarify, our videos are recorded in real-world outdoor environments and contain many moving objects, such as passing vehicles and walking pedestrians:

  • Moving objects in the input: moving objects remain in the video input as distractors and occluders, so models must recover spatial information from dynamic scenes rather than from clean, static views.
  • Observer motion as the dynamic component: P2 and R3 (Sec. 3.2) explicitly evaluate the camera's own motion, including turns near landmarks and consecutive turn sequences.
  • Static landmarks as spatial anchors: questions refer to static landmarks (Sec. 3.1) because long-horizon spatial reasoning requires references whose positions remain consistent over time; the position of a moving object observed minutes earlier cannot serve as a reliable reference.

Reasoning about the motion of other objects is complementary to our setting, and we will discuss it as a limitation and future direction in the revised manuscript.

Response (W3): Thank you for this accurate characterization. As stated in Sec. 3.2.3, our destination tasks evaluate destination-directed decisions grounded in the recorded routes, rather than general shortest-path planning:

  • Maneuver selection at revisited locations (D1, Sec. 3.2.3, Fig. 10): at a region the trajectory revisits, the model must select the maneuver (left, right, or straight) that leads toward a destination landmark. The answer is the maneuver of the unique recorded branch that reaches the destination.
  • Waypoint selection at revisited locations (D2, Sec. 3.2.3, Fig. 11): at a revisited region with multiple recorded branches, the model must select the waypoint on the branch that leads toward the destination. The answer follows the recorded correspondence between each branch and the destination it reaches.

Because we keep only cases in which exactly one recorded branch reaches the destination, each answer is uniquely determined by the observed routes, without assumptions about unobserved connectivity. Solving these tasks still requires recognizing the revisited location and linking each branch to what was later observed along it. We will clarify in the revised manuscript that SpatialRoute-Bench focuses on offline, long-horizon spatial understanding, while interactive navigation and shortest-path planning remain outside its current scope (Sec. 5).

Dear Reviewer,

We are grateful for your valuable comments. As the discussion period is nearing its end, with less than two days remaining, we want to kindly follow up and ensure that we have adequately addressed your concerns. If you have any remaining questions, please let us know. If you are satisfied with our responses, we would sincerely appreciate your consideration of a score increase, which would be a great encouragement to our team.

Thank you for your time and consideration!

The authors address part of my concerns. I keep my rating for weak acceptance.

Thank you very much for your feedback and for maintaining your positive score of our work. We sincerely appreciate your time and suggestions, which we will incorporate into the final version of the paper.

Thanks again for your support!

Reviewer_Q4XpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SpatialRoute-Bench, a geometry-grounded benchmark designed to evaluate long-horizon spatial understanding in continuous outdoor videos. The benchmark contains 7,000 multiple-choice questions across seven tasks, organized into three hierarchical levels: perception, reasoning, and decision. To construct the benchmark automatically, the authors develop a pipeline that combines VLM-based semantic grounding with 3D reconstruction to generate geometrically and semantically validated questions. Extensive evaluations reveal substantial limitations in current VLMs, particularly in camera-motion perception and long-range spatial reasoning, with even the strongest model performing only slightly above the random guessing baseline.

Critical strengths
  1. This work addresses a meaningful problem in long-horizon spatial understanding and constructs a large-scale benchmark with extensive data.
  2. Using LingBot-Map for streaming 3D reconstruction is an appropriate choice, providing reliable geometric information for benchmark construction.
  3. The evaluation covers not only open-source VLMs but also several state-of-the-art proprietary models, providing a broader assessment of current spatial understanding capabilities.
Critical weaknesses
  1. Several related works [1,2,3] have already investigated spatial understanding in outdoor scenarios. A more thorough discussion and comparison with these studies would help clarify the novelty and distinct contributions of SpatialRoute-Bench.
  2. The paper does not seem to clearly specify the number of input frames used for VLM evaluation, which may affect the reproducibility and fairness of the comparisons. Additionally, evaluating the impact of different numbers of sampled frames would be an important ablation study.
  3. Although current VLMs perform only slightly above the random guessing baseline, a blind evaluation (e.g., question-only without video) is still needed to assess whether their performance reflects genuine spatial understanding rather than linguistic biases or dataset artifacts.

[1] Li, Yun, et al. Sti-bench: Are mllms ready for precise spatial-temporal world understanding?. ICCV 2025. [2] Wu, Mingrui, et al. From indoor to open world: Revealing the spatial reasoning gap in mllms. CVPR 2026. [3] Sun, Pengzhan, et al. ReSTI: A Source-Grounded Audit and Repair of STI-Bench. arXiv 2026.

Questions for authors

Please refer to the weaknesses. Besides, I suggest the authors follow the benchmark design principles proposed in [4] to further examine whether SpatialRoute-Bench contains exploitable non-visual shortcuts or dataset biases, thereby ensuring its reliability in evaluating genuine spatial understanding.

[4] Brown, Ellis, et al. Benchmark designers should "train on the test set" to expose exploitable non-visual shortcuts. COLM 2026.

4 comments

Response (W1): Thank you for this helpful suggestion. We will update Table 1 to include STI-Bench, OSI-Bench, and ReSTI, as shown below.

Table 1. Comparison with representative video spatial understanding benchmarks.

DatasetVideo SettingAvg. Duration (min)#QAGeometry-GroundedMotion CompositionDestination Decision
VSI-BenchReal / Indoor1.25,130✓✗✓
VSI-SUPERConstructed / IndoorN/R700✗✗✗
LongSpace-BenchReal / Indoor21.44,073✗✗✗
OVO-S-BenchReal / Indoor & OutdoorN/R1,680△△✗
GST-BenchSynthetic / IndoorN/R2,762✓△✗
VSI-Super-WildReal / Indoor & Outdoor38.626,980△✗✗
STI-Bench [1]Real / Indoor & Outdoor0.612,064✓✓✗
OSI-Bench [2]Real / Indoor & OutdoorN/R8,736✓✗✗
ReSTI [3]Real / Indoor & Outdoor0.611,782✓✓✗
SpatialRoute-Bench (Ours)Real / Outdoor54.687,000✓✓✓

We will add the following comparison to the Related Work section:

STI-Bench [1] evaluates quantitative spatial-temporal understanding using source geometric annotations, including multi-turn trajectory descriptions. OSI-Bench [2] evaluates outdoor spatial reasoning using calibrated multi-sensor measurements. ReSTI [3] audits and repairs STI-Bench annotations while retaining its videos. SpatialRoute-Bench complements these efforts by reconstructing geometry from ordinary RGB videos, avoiding dependence on dataset-specific 3D annotations. Its long outdoor routes distribute spatial evidence across time, locations, and viewpoints, enabling us to test whether models can integrate local observations for route reasoning and destination-directed decisions.

Response (W2): We use 300 uniformly sampled frames per video as a shared budget across the API models, accounting for their differing input limits. We additionally evaluate all seven models with 150 frames and three models with 600 frames, as summarized below, "/" in the table indicates the input exceeds the context-length limit of either the model or OpenRouter platform.

Model150 frames300 frames600 frames
DeepSeek V4 Flash Vision25.0726.87/
Grok 4.622.3025.21/
Gemini 3.7 Flash22.3026.32/
Claude Sonnet 523.6824.2425.62
Gemini 3.8 Flash22.0227.15/
GPT-5.6 Terra26.0426.1827.84
GPT-5.6 Sol26.8728.2531.30

Response (W3 & Q1): We have already considered blind evaluation by conducting experiments with black videos, as reported in Table 3. Following the reviewer’s suggestion, we further conducted a question-only evaluation without any video input. We will update Table 3 to include these additional results, as shown below.

SettingP1P2R1R2R3D1D2Avg.
Original Video33.6423.8120.9140.9119.0528.4231.0328.25
Shuffled Video30.0018.1022.7318.1820.9529.4721.8422.99
Black Video23.6418.1020.9120.0018.1022.1120.6920.50
Shuffled Options35.4520.9521.8228.1822.8631.5829.8927.15
Question-only (no video)20.9118.1023.6422.7322.8623.1618.3921.47

Furthermore, following [4], we audit the questions with Qwen2-7B-Instruct using only question text and answer choices. We compare zero-shot performance with supervised text-only fine-tuning. For each of five folds, a fresh LoRA adapter is fitted using the other four folds and evaluated only on the held-out fold. We use both random and video-grouped splits; the latter keeps all questions from the same source-video group within one fold. The results are summarized below.

ExperimentAccuracy (%)Soft Score (%)
Text-only zero-shot25.7626.20
Text-only fine-tuning: random 5-fold CV28.3927.19
Text-only fine-tuning: video-grouped 5-fold CV28.6727.76

Soft score is the mean s(x)s(x) defined in [4], using its option-probability normalization and fallback rules.

Dear Reviewer,

We are grateful for your valuable comments. As the discussion period is nearing its end, with less than two days remaining, we want to kindly follow up and ensure that we have adequately addressed your concerns. If you have any remaining questions, please let us know. If you are satisfied with our responses, we would sincerely appreciate your consideration of a score increase, which would be a great encouragement to our team.

Thank you for your time and consideration!

My concerns have been resolved and I have rasied my rating to Weak Acceptance.

Thank you very much for your reply and for raising the score. We will incorporate the corresponding revisions into the final version of the paper.

Thanks again for helping us improve the paper.

Reviewer_SSrfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces SpatialRoute-Bench, a geometry-grounded benchmark comprising 7,000 questions across seven tasks for long-horizon spatial understanding in outdoor videos. Its strengths include substantial dataset construction, the integration of 3D reconstruction with semantic landmark grounding, and broad coverage of spatial perception, cross-temporal reasoning, and destination-directed decisions. Evaluating ten VLMs and diagnostic perturbations provides useful insights into current limitations in long-horizon spatial understanding. Further analysis of video-input settings and controls for viewpoint changes would strengthen the interpretation of these results.

Critical strengths

1.The benchmark combines geometry-derived answers with semantically grounded landmarks across seven tasks, using local/long-range task pairs and frame-order perturbations to probe spatial reasoning over extended trajectories.

Critical weaknesses

1.The benchmark insufficiently explores the integration of egocentric observations with global map representations, such as localizing observed landmarks on a map or selecting routes across revisited locations, limiting its assessment of navigation-oriented spatial understanding. 2.The evaluation could make fuller use of geometric and temporal information through tasks such as estimating route-distance ratios or, where scale calibration permits, travel speed and time to a destination.

Questions for authors

1.Given the benchmark’s navigation focus, could the authors consider tasks that connect egocentric observations to a global map, such as landmark localization or route selection? 2.Could the geometric and temporal information support questions about speed or travel time? It would be helpful to clarify what scale calibration these tasks would require. 3.How sensitive are the results to video input settings? Please report the sampling rates, frame budgets, and resolutions used for each model. 4.A basic navigation baseline would help contextualize the VLM results. Could the authors evaluate a navigation or end-to-end driving model with compatible inputs and outputs?

2 comments

Response (W1 & Q1): We acknowledge that this aspect was not sufficiently clarified in the original manuscript. To clarify, SpatialRoute-Bench already evaluates route selection at revisited locations and the integration of egocentric observations along the route:

  • Maneuver selection at revisited locations (D1, Sec. 3.2.3, Fig. 10): at a region the trajectory revisits, the model must select the maneuver (left, right, or straight) that leads toward a destination landmark. The answer is the maneuver of the unique recorded branch that reaches the destination.
  • Waypoint selection at revisited locations (D2, Sec. 3.2.3, Fig. 11): at a revisited region with multiple recorded branches, the model must select the waypoint on the branch that leads toward the destination. The answer follows the recorded correspondence between each branch and the destination it reaches.
  • Integration of egocentric observations (R1, Sec. 3.2.2, Fig. 7): the model must determine the relative direction between two landmarks observed at least 180 s apart, expressed in the camera frame of the earlier observation. This requires relating egocentric observations through the intervening camera motion.

Regarding landmark localization, our pipeline can directly support such tasks, as it already recovers the camera trajectory and the 3D positions of the landmarks, from which map-based questions can be generated with geometrically derived answers. We will add this task in the revised manuscript. We thank the reviewer for this valuable suggestion.

Response (W2 & Q2): We acknowledge that this aspect was also not sufficiently clarified in the original manuscript. Since our reconstruction is defined only up to a global scale (Sec. 3.1), our tasks use geometric and temporal information in scale-invariant forms:

  • Route distance (R2, Sec. 3.2.2, Fig. 8): the model must identify which landmark is farthest from an anchor landmark along the recorded route. The answer is computed from the cumulative distance traveled along the reconstructed trajectory, not from straight-line proximity.
  • Temporal order (R3, Sec. 3.2.2, Fig. 9): the model must identify two consecutive turns between two landmarks in the order they occurred (e.g., left then right). The answer is derived from the timeline of the reconstructed trajectory.
  • Temporal span (R1, Sec. 3.2.2, Fig. 7): the model must determine the relative direction between two landmarks observed at least 180 s apart. The answer is computed from their reconstructed 3D positions, and the minimum separation of 180 s explicitly tests long-horizon integration.

Route-distance ratios are also scale-invariant, and we have explored them within consistently scaled trajectory segments. Physical speed, however, requires a reliable metric scale, which current reconstruction cannot consistently maintain over long videos. Elapsed travel time can be read directly from video timestamps, whereas time to a destination would additionally require assumptions about future motion. We will clarify these distinctions in the revised manuscript.

Response (Q3): We evaluate the API models reported in the paper through OpenRouter using 300 uniformly sampled frames per video at 512 × 288 resolution. We chose this setting to account for the input limits of different models. We additionally evaluate all seven API models with 150 frames and three models with 600 frames, as shown below, "/" in the table indicates the input exceeds the context-length limit of either the model or OpenRouter platform.

Model150 frames300 frames600 frames
DeepSeek V4 Flash Vision25.0726.87/
Grok 4.622.3025.21/
Gemini 3.7 Flash22.3026.32/
Claude Sonnet 523.6824.2425.62
Gemini 3.8 Flash22.0227.15/
GPT-5.6 Terra26.0426.1827.84
GPT-5.6 Sol26.8728.2531.30

Response (Q4): Thank you for this insightful suggestion. We evaluate StreamVLN [R1] and DualVLN [R2], two representative navigation models published in 2026, on our D1 tasks (Sec. 3.2.3, Fig. 10) by mapping their outputs to the corresponding QA answer choices. The results are reported below.

ModelAccuracy (%)
StreamVLN42.11
DualVLN32.63
GPT-5.6 Sol28.42
Gemini 3.8 Flash22.11
Claude Sonnet 527.37
Grok 4.638.95
DeepSeek V4 Flash Vision31.58

[R1] Meng Wei et al. StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modeling. ICRA 2026.

[R2] Meng Wei et al. Ground Slow, Move Fast: A Dual-System Foundation Model for Generalizable Vision-Language Navigation. ICLR 2026.

Dear Reviewer,

We are grateful for your valuable comments. As the discussion period is nearing its end, with less than two days remaining, we want to kindly follow up and ensure that we have adequately addressed your concerns. If you have any remaining questions, please let us know. If you are satisfied with our responses, we would sincerely appreciate your consideration of a score increase, which would be a great encouragement to our team.

Thank you for your time and consideration!

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your time and effort. As the discussion deadline approaches, could you please respond to the authors’ rebuttal and share whether it addresses your concerns? Your continued participation in the subsequent discussions would also be greatly appreciated.

Best regards,
AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.