acceptodds
Withdrawn from ICLR 2027

STRIDE: Evaluating Spatiotemporal Reasoning in Driving Edge Cases

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Evaluating vision-language models (VLMs) for autonomous driving requires assessing their ability to connect spatial observations of past events with future consequences. Existing benchmarks either assess driving edge cases through basic perceptual questions or evaluate temporal understanding under relatively simple scenarios, leaving the spatiotemporal reasoning required in complex driving scenarios insufficiently tested. To address this gap, we introduce STRIDE (SpatioTemporal Reasoning In Driving Edge Cases), an object-centric benchmark for evaluating spatiotemporal reasoning in long-tail and safety-critical scenarios. STRIDE defines six task families: spatial perception, spatial understanding, temporal memory, temporal extrapolation, trajectory prediction, and scene-context awareness. It pairs multiple-choice questions covering a broad range of spatiotemporal judgments with related free-form visual question answering (VQA) that examines the same capabilities through detailed scene explanations and justifications of driving decisions. We curate STRIDE by prioritizing scenes with unusual object behaviors and complex traffic interactions, constructing reference answers from structured dataset annotations through automated procedures, and applying limited language model assistance alongside human verification to support annotation quality. Evaluation of 12 proprietary, open-weight, and driving-specialized models reveals substantial limitations: even GPT-6-Astra achieves the highest average multiple-choice accuracy of 43.4%, compared with a random-choice baseline of approximately 20%. Our analysis further reveals that strong scene-description scores can mask weaknesses in safety-critical judgments, and models struggle more with reconstructing object motion than recalling past positions. These insights highlight STRIDE's diagnostic value for uncovering spatiotemporal driving limitations that aggregate VQA performance can conceal.

Primary area:
datasets and benchmarks
Keywords:
Vision Language Models, Autonomous Driving, Spatiotemporal Reasoning, Long-Tail Driving Scenarios, Visual Question Answering

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VqvrOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces STRIDE, a benchmark for evaluating the spatiotemporal reasoning capabilities of vision-language models (VLMs) in challenging driving scenarios. Built on nuScenes and Waymo E2E, STRIDE contains 2,350 questions across six task families, covering spatial perception, spatial understanding, temporal memory, temporal extrapolation, trajectory prediction, and scene-context awareness. It combines multiple-choice questions with open-ended VQA to evaluate both fine-grained driving judgments and scene-level explanations. Experiments across 12 models show that current VLMs still struggle with spatiotemporal reasoning. The results also reveal notable gaps between positional recall and motion understanding, as well as between scene-description quality and safety-critical judgments.

Critical strengths

The benchmark focuses on complex and long-tail driving scenarios, making the evaluation more relevant to real-world driving safety. It also covers a range of spatial and temporal reasoning tasks, combining multiple-choice and open-ended questions to better assess VLM capabilities.

Critical weaknesses
  1. A closely related paper, STRIDE-QA: Visual Question Answering Dataset for Spatiotemporal Reasoning in Urban Driving Scenes (Ishihara et al., AAAI 2026), has already introduced a similar benchmark for spatiotemporal reasoning in driving scenarios, but it is not cited or discussed. Given the overlap in motivation, task design, and even naming, the authors should clarify what STRIDE adds beyond this work.
  2. Limited scientific insights. Most of the analysis focuses on comparing model scores across tasks, without explaining why models fail. More in-depth analysis would help reveal the actual limitations of current VLMs.
  3. Appendix A.6 reports that 239 out of 250 open-ended reference answers were rewritten, many with VLM assistance. This raises questions about label reliability, and further independent human validation is needed.
  4. Lack of ablation study. It is unclear whether the low scores come from poor spatiotemporal reasoning or other factors, such as visual perception and input quality. Ablations on temporal inputs and image resolution would help clarify this.
  5. Each question template contains only 50 samples, and some tasks are evaluated on only one dataset. More samples and cross-dataset evaluations would make the conclusions more convincing.
Questions for authors

None

Reviewer_SgBYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces STRIDE, a benchmark for evaluating spatiotemporal reasoning in long-tail and safety-critical autonomous driving scenarios. Built upon nuScenes and Waymo E2E, STRIDE contains 2,350 questions across six task families, covering spatial perception, spatial understanding, temporal memory, temporal extrapolation, trajectory prediction, and scene-context awareness. The benchmark combines multiple-choice and open-ended questions to evaluate fine-grained driving judgments and scene-level explanations. Experiments on 12 vision-language models reveal substantial performance limitations, particularly in motion understanding, future-event prediction, and safety-critical decision-making.

Critical strengths

The paper addresses the limitations of existing vision-language models in understanding complex driving scenarios, particularly their ability to integrate spatial observations with temporal information. Evaluating such capabilities in long-tail and safety-critical situations is meaningful for autonomous driving research.

Critical weaknesses
  1. Although STRIDE integrates spatial reasoning, temporal understanding, and safety-critical scenarios, many of its task formulations and evaluation formats have already been explored in existing driving VQA benchmarks. The paper primarily emphasizes broader task coverage and increased difficulty, without sufficiently demonstrating which fundamentally new diagnostic capabilities STRIDE provides beyond previous benchmarks such as STSBench, DriveLM, and Reason2Drive. This weakens the novelty and significance of the benchmark contribution.

  2. Appendix A.6.1 reports that 15 of the initial 31 multiple-choice templates were rejected during quality review, raising concerns about the reliability of the automatic annotation pipeline. Although the authors perform subsequent filtering and verification, stronger evidence of final annotation accuracy is needed, particularly for questions involving estimated geometry and safety-related judgments. Moreover, the reported low accuracy may conflate failures in visual perception, temporal reasoning, input presentation, and output-format compliance. Without controlled experiments isolating these factors, the conclusions about models' spatiotemporal reasoning deficiencies remain insufficiently supported.

Questions for authors
  1. What specific diagnostic capabilities does STRIDE provide that cannot be captured by existing driving benchmarks? Could the authors offer more direct comparisons with STSBench and related benchmarks under controlled evaluation settings, rather than relying primarily on task-coverage comparisons?

  2. Given the substantial rejection rate of the initial question templates, how do the authors ensure the reliability of the final benchmark annotations? Can they provide quantitative human agreement or error-rate analysis for the retained questions, particularly those involving automatically estimated geometry and safety-critical decisions?

  3. Can the authors conduct controlled experiments using single-frame inputs, temporally shuffled frames, or different sampling intervals to verify that the benchmark specifically measures temporal reasoning? Such experiments would help distinguish genuine spatiotemporal reasoning failures from visual perception limitations and evaluation-format effects.

Reviewer_rmpNOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

STRIDE is a benchmark for spatiotemporal reasoning in driving edge cases, 2,350 questions over 1,452 clips drawn from nuScenes and WOD-E2E. Six task families, instantiated from 39 hand-written templates, cover spatial perception, spatial understanding, temporal memory, temporal extrapolation, trajectory prediction and scene-context awareness. Each item shows five consecutive 360-degree multi-camera frames at 2 Hz with the target object boxed, and pairs a multiple-choice question with a linked open-ended one. Ground truth comes from the datasets' structured annotations, with a language model used only as a verbalizer; human review kept 51.6% of the MCQ templates.

The authors evaluate 12 proprietary, open-weight and driving-specialized models. GPT-6-Astra leads at 45.5% MCQ accuracy on nuScenes and 41.2% on Waymo E2E, against roughly 20% for random guessing and a reported human ceiling of 79% and 69%. Two gaps carry the analysis. Models recall a previous object position at 70 to 72% but reconstruct recent motion at 14 to 22%, near the 16.7% chance level. Open-ended scene descriptions score 7 to 9 out of 10 while the linked safety-critical questions, about drivable region, risk source and preferred action, are answered at 22 to 44%. A Qwen3-VL-32B judge scores the open-ended answers and is validated against human raters in Table 4 at Spearman 0.845 with a mean absolute deviation of 0.90 on a 0-10 scale. Trajectory predictions are scored by ADE and FDE after JSON parsing, with unparseable output counted as failure.

Critical strengths
  • The dissociation between open-ended fluency and safety-critical accuracy is the finding worth having, and the paper quantifies it. GPT-5.5 scores 9.22/10 on scene description (SC-6, Table 9), then 22% on drivable region (SU-2) and on lane-change clearance (SU-4). GPT-6-Astra writes scene, risk and action explanations at 7.76/10 (SC-4, Table 10) while naming the main risk source at 36% (SC-2). The pattern repeats across models and both splits. That makes it an argument about aggregate VQA scores overstating driving competence, not a single-model anomaly.
  • The temporal-memory result isolates one failure a model developer can act on. Recall of a previous object position runs at 52 to 72%, reconstruction of recent motion at 14 to 22%, near the 16.7% chance baseline, and the gap holds for every model tested (Table 9, Table 10, Figure 4a). The labels come from the datasets' own 3D annotations, with the language model restricted to verbalizing them (Section 3.2, Appendix A.4). That makes the comparison more trustworthy than it would be in a benchmark whose answers are model-generated.
Critical weaknesses
  • The Waymo half of the benchmark rests on labels the paper derives rather than labels the dataset provides. Appendix A.1 states that object distance is not obtained from Waymo 3D labels; it comes from SAM3 detections, heuristic tracking and a flat-ground ray intersection that assumes the bottom-centre pixel touches the ground. That assumption fails on slopes, on raised crosswalks and wherever a box does not enclose the contact patch, and no error statistic is reported for any step in the chain. Quality review already discarded 48.4% of the MCQ templates (A.6.1, Table 5), which suggests the derived labels are fragile. The paper's central claim is that models are unreliable on safety-critical judgments, and at 41.2% on Waymo MCQ an unmeasured label-noise floor could account for part of that gap.
  • No result in the paper carries a variance estimate. Section 4.1 reports deterministic or near-deterministic decoding, which fixes the sampling but says nothing about how far apart two scores have to be before they differ. GPT-5.6-Sol at 38.5% and Gemini-3.1-Pro at 37.8% on nuScenes MCQ are 0.7 points apart over 800 items, inside one standard error of a binomial proportion, and the per-task comparisons in Table 9 are finer than that. A benchmark whose purpose is fine-grained diagnosis has to report the resolution of its own instrument, or the middle of every table is unreadable.
Questions for authors

Does the Qwen3-VL-32B judge score Qwen-family answers differently from GPT or Gemini answers on the same items? Four of the twelve evaluated models are Qwen. Table 4 validates the judge against human scores in aggregate, which does not rule out a per-family offset, and the claim in Section 5.2 that description quality and safety-critical accuracy come apart needs the open-ended scores to be comparable across families. A per-family check on a held-out subset would settle it.

What is the error of the derived Waymo labels? Appendix A.1 gives the geometry and no numbers. There is no comparison of ray-intersection distance against a measured range, no tracking accuracy, and no count of how many of the Waymo questions depend on a derived rather than an annotated label. The MCQ answer bins are narrow, 0-5 m and 5-15 m among them, so a systematic distance error of a few metres would move items between bins and show up as model failure.

When a model writes the correct hazard in the open-ended answer and then selects the wrong option, is that a knowledge gap or a format one? The masking finding in Section 5.2 reads differently depending on the answer, and an audit of twenty mismatched pairs for one model would separate the two.

On reproducibility, will the 2,350 questions, the SAM3 and tracking pipeline, and the judge prompts be released, and where? The appendix describes them and the paper names no repository.

One suggestion, not a question. Five frames at 2 Hz give 2.5 seconds of context, which is short for the temporal-extrapolation family, where the best accuracy is 40% on nuScenes and 36% on Waymo (TE-1). A longer-window split would separate "cannot integrate 2.5 s of motion" from "cannot predict beyond it".

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.