- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this paper, the authors focus on long horizon spatial reasoning problem in outdoor scenes, which is neglected by the former methods and benchmarks. In this problem, the video illustrates an outdoor scene so that the distance scale and layout differs from indoor scenes a lot and the questions may require long horizon knowledge to answer. To investigate in this problem, the authors propose to construct a benchmark for evaluation. The authors propose an autonomous framework to construct both geometry and semantic information of the scene and generate questions spreading across three levels. The evaluation of current VLMs on this benchmark shows that they still perform badly in this problem.
Critical strengths
-
The authors focus on a meaningful problem that the outdoor scenes have larger distance scale and more complicate layouts, while the former benchmarks or methods are targeted at indoor scenes. Moreover, videos in previous benchmarks are short horizon, which can limit their evaluation in reality.
-
The three levels analysis of perception, reasoning and decision can support comprehensive analysis of the capability of one model.
-
The evaluation of current models covers different kinds of VLMs, including propriety models, open-sourced models, spatial reasoning models, which can provide useful insights for further study.
Critical weaknesses
-
I wonder how many videos are in the benchmarks and how long for each video. If the number of videos is too small, the diversity of the scene can be limited and if the duration of each video is not long enough, the analysis for long horizon reasoning can be not comprehensive enough.
-
The videos in the benchmark mainly focus on the static outdoor scenes where the objects do not change their position. However, for most outdoor scenes, the objects are mobile and changing their positions continuously, which can limit the application of the proposed benchmark.
-
The benchmark uses recorded routes rather than interactive navigation so I think the proposed destination tasks follow a unique recorded branch–destination correspondence rather than general shortest-path planning.
Questions for authors
Please refer to the weaknesses.
4 comments
Dear Reviewer,
We are grateful for your valuable comments. As the discussion period is nearing its end, with less than two days remaining, we want to kindly follow up and ensure that we have adequately addressed your concerns. If you have any remaining questions, please let us know. If you are satisfied with our responses, we would sincerely appreciate your consideration of a score increase, which would be a great encouragement to our team.
Thank you for your time and consideration!
The authors address part of my concerns. I keep my rating for weak acceptance.
Thank you very much for your feedback and for maintaining your positive score of our work. We sincerely appreciate your time and suggestions, which we will incorporate into the final version of the paper.
Thanks again for your support!
Response (W1): We reconstructed and semantically annotated approximately 300 long outdoor videos. Three annotators then screened the reconstruction quality, spatial relations, and semantic labels to select 22 videos. Their durations range from 40.91 to 83.33 minutes, with a mean of 54.68 minutes.
Response (W2): Thank you for raising this point. We acknowledge that this aspect was not sufficiently clarified in the original manuscript. To clarify, our videos are recorded in real-world outdoor environments and contain many moving objects, such as passing vehicles and walking pedestrians:
Reasoning about the motion of other objects is complementary to our setting, and we will discuss it as a limitation and future direction in the revised manuscript.
Response (W3): Thank you for this accurate characterization. As stated in Sec. 3.2.3, our destination tasks evaluate destination-directed decisions grounded in the recorded routes, rather than general shortest-path planning:
Because we keep only cases in which exactly one recorded branch reaches the destination, each answer is uniquely determined by the observed routes, without assumptions about unobserved connectivity. Solving these tasks still requires recognizing the revisited location and linking each branch to what was later observed along it. We will clarify in the revised manuscript that SpatialRoute-Bench focuses on offline, long-horizon spatial understanding, while interactive navigation and shortest-path planning remain outside its current scope (Sec. 5).