- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces STRIDE, a benchmark for evaluating the spatiotemporal reasoning capabilities of vision-language models (VLMs) in challenging driving scenarios. Built on nuScenes and Waymo E2E, STRIDE contains 2,350 questions across six task families, covering spatial perception, spatial understanding, temporal memory, temporal extrapolation, trajectory prediction, and scene-context awareness. It combines multiple-choice questions with open-ended VQA to evaluate both fine-grained driving judgments and scene-level explanations. Experiments across 12 models show that current VLMs still struggle with spatiotemporal reasoning. The results also reveal notable gaps between positional recall and motion understanding, as well as between scene-description quality and safety-critical judgments.
Critical strengths
The benchmark focuses on complex and long-tail driving scenarios, making the evaluation more relevant to real-world driving safety. It also covers a range of spatial and temporal reasoning tasks, combining multiple-choice and open-ended questions to better assess VLM capabilities.
Critical weaknesses
- A closely related paper, STRIDE-QA: Visual Question Answering Dataset for Spatiotemporal Reasoning in Urban Driving Scenes (Ishihara et al., AAAI 2026), has already introduced a similar benchmark for spatiotemporal reasoning in driving scenarios, but it is not cited or discussed. Given the overlap in motivation, task design, and even naming, the authors should clarify what STRIDE adds beyond this work.
- Limited scientific insights. Most of the analysis focuses on comparing model scores across tasks, without explaining why models fail. More in-depth analysis would help reveal the actual limitations of current VLMs.
- Appendix A.6 reports that 239 out of 250 open-ended reference answers were rewritten, many with VLM assistance. This raises questions about label reliability, and further independent human validation is needed.
- Lack of ablation study. It is unclear whether the low scores come from poor spatiotemporal reasoning or other factors, such as visual perception and input quality. Ablations on temporal inputs and image resolution would help clarify this.
- Each question template contains only 50 samples, and some tasks are evaluated on only one dataset. More samples and cross-dataset evaluations would make the conclusions more convincing.
Questions for authors
None