acceptodds
Withdrawn from ICLR 2027

LTDriveQA: Benchmarking Spatiotemporal Intelligence in Multi-View Long-Tail Driving Scenarios

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Driving visual question answering (VQA) provides a unified framework for assessing whether vision-language models (VLMs) and Vision-Language-Action (VLA) models can understand dynamic road scenes and reason about driving decisions. However, existing benchmarks often rely on generic templates, weakly probe temporal and cross-view reasoning, and underutilize long-tail driving events. We introduce LTDriveQA, an event-centered benchmark of scene-specific questions designed to probe temporal and cross-view reasoning. It spans Perception, Prediction, Planning, and Reasoning, supports both multiple-choice and open-ended evaluation. A scalable recursive benchmark construction pipeline refines questions through diagnostic feedback, evidence grounding, answer annotation, and restricted-input shortcut checks. Across 25 models, GPT-5.6-Sol leads with an overall score of 60.0; Qwen3-VL-8B and Qwen-Drive-1.0 score 40.5 and 45.6, respectively. Evidence ablations show aggregate visual gains across all open-source models, with temporal gains strongest in Perception and surround-view gains consistent across three models on multi-view questions. By grounding evaluation in challenging real-world driving, LTDriveQA probes the scene understanding and decision reasoning that driving VLMs and VLAs rely on, and provides a testbed for spatiotemporal intelligence in dynamic multi-view environments.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Driving VQA

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_P3DmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces LTDriveQA, an event-centered benchmark for evaluating spatiotemporal reasoning in long-tail driving scenarios. Built from WOD-E2E, it contains 2,889 human-reviewed questions over 719 multi-view video clips and covers four competencies: perception, prediction, planning, and reasoning. The questions are provided in both multiple-choice and open-ended formats and are constructed through a model-assisted iterative pipeline with evidence-grounding checks and restricted-input diagnostics. The authors evaluate 25 general-purpose and driving-oriented vision-language models and find substantial gaps from human performance, particularly on temporal and cross-view reasoning. Additional ablations examine models’ reliance on visual evidence, multiple views, and temporal ordering.

Critical strengths
  1. Clear and complete presentation. The paper is well organized and easy to follow, with the benchmark motivation, construction pipeline, evaluation protocol, and limitations presented clearly.

  2. Meaningful benchmark setting. Building on long-tail events from WOD-E2E makes the evaluation more relevant to safety-critical and uncommon driving scenarios than standard scene-level driving QA.

  3. Interesting spatiotemporal formulation. The use of synchronized multi-view videos enables questions that require temporal evolution and cross-camera evidence, extending beyond the predominantly single-frame settings of many existing driving VQA benchmarks.

  4. Comprehensive experimental evaluation. The paper evaluates a broad range of general-purpose and driving-specific models and includes human comparisons, restricted-input controls, multi-view and temporal ablations, and detailed failure analyses. These experiments provide a relatively complete characterization of both the benchmark and current model limitations.

Critical weaknesses
  1. Unclear external validity beyond driving QA. Although LTDriveQA is motivated as a benchmark of driving intelligence and includes a Planning category, all competencies are evaluated exclusively through language-answer correctness. The paper neither produces trajectories nor reports established planning metrics such as ADE/FDE, collision rate, or closed-loop performance. It is therefore unclear whether higher LTDriveQA scores correspond to safer or more accurate driving behavior. Correlating benchmark scores with trajectory-planning performance, or demonstrating that LTDriveQA supervision benefits an actual planner, would substantially strengthen its practical significance.

  2. Limited validation of genuine temporal reasoning. Although LTDriveQA uses multi-frame video, some results suggest that performance benefits primarily from access to multiple observations rather than understanding their temporal order. The final frame alone performs close to the full video overall and even outperforms it on Prediction. For Qwen3-VL-8B, shuffling the frames causes a relatively modest degradation compared with removing earlier frames entirely. Thus, the benchmark establishes multi-frame evidence dependence more convincingly than ordered temporal reasoning.

  3. Potential bias and uncertainty in model-assisted construction and scoring. Much of the question and reference-answer construction relies on VLM/LLM families that are also evaluated in the benchmark. Open-ended responses are further scored by a reference-based LLM judge rather than directly against the video. Although the authors provide useful human audits, the frame-grounded audit covers only 40 examples, and judge–human agreement on the paired video-grounded subset is 76.7%. More extensive independent human evaluation would strengthen confidence in the OpenVQA scores and model rankings.

Questions for authors

See weaknesses.

Reviewer_6j8EOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LTDriveQA, a driving VQA benchmark containing 2,889 questions over 719 event-centered clips extracted from Waymo E2E sequences. It covers four competencies(Perception, Prediction, Planning, and Reasoning) with both multiple-choice and open-ended questions, using synchronized observations from eight camera views. The authors propose a VLM-assisted pipeline for event selection, question generation and refinement, followed by human review. They evaluate 25 VLMs and analyze model performance under restricted visual inputs. The paper also reports human validation of the questions and agreement between the OpenVQA judge and human annotators.

Critical strengths
  1. The benchmark extracts event-centered clips from longer driving sequences.
    This preserves the temporal context of driving events and provides a useful basis for asking questions about how actors and situations change over time.

  2. The paper reports human validation and agreement measurements.
    The authors describe human review of the released questions and report agreement between the OpenVQA judge and human annotators. These details make the quality-control process more transparent.

Critical weaknesses
  1. The distinction from existing benchmarks and the definition of “long-tail” are insufficiently clear.
    The paper’s novelty claim rests on combining long-tail scenarios, temporal and multi-view reasoning, scene-specific questions, and evidence auditing. However, the comparison with prior work is primarily a feature checklist, without a direct comparison under matched evaluation conditions to show what new diagnostic value LTDriveQA provides. In addition, “long-tail” is not defined using a reproducible threshold or operational criterion. Since the source dataset is already curated for rare or challenging scenarios, it remains unclear how the authors distinguish long-tail events during their own VLM-based selection and how consistently those decisions can be reproduced.

  2. The benchmark’s task-level results and evidence for temporal reasoning may be unstable or inconclusive.
    Although the full dataset contains 2,889 questions, the OpenVQA subset contains only 595 questions, with 77 for Perception, 181 for Prediction, 124 for Planning, and 213 for Reasoning. The questions are also drawn from 192 source segments, so the effective sample size for task-level comparisons is smaller than the raw question count suggests. The paper does not report uncertainty estimates for all model and task-level results. The average MCQ difference between full video and the final frame is modest, and the final frame performs better on Prediction; gains for Planning and Reasoning are not clear. Thus, the results do not consistently establish that the benchmark measures temporal reasoning across competencies. The paper also does not quantitatively analyze residual MCQ biases, such as answer-option length or position effects, or potential bias from using a question-generation model family related to one of the evaluated families. These issues make it difficult to determine how robustly the reported scores reflect the intended capabilities.

Questions for authors
  1. What specific diagnostic value does LTDriveQA provide beyond the closest existing benchmarks? Can this distinction be demonstrated through a comparison using matched models and evaluation conditions?

  2. What operational criteria were used to select long-tail events, and how consistent are the VLM selection and subsequent human review under those criteria?

  3. Can the authors report source-segment-level confidence intervals for model and task-level scores, and clarify whether the full-video advantage persists on questions that explicitly require temporal evidence? Have residual MCQ option biases and potential question-generation-model-family biases been quantitatively assessed?

Reviewer_sKerOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LTDriveQA is an event-centered driving VQA benchmark containing 2,889 questions from 719 clips across 192 Waymo E2E segments. It covers perception, prediction, planning, and reasoning through multiple-choice and open-ended questions. Construction combines event localization, recursive question generation, grounded templates, restricted-input diagnostics, and human review. Prediction questions withhold the future interval used to establish their answers. The paper evaluates 25 models and analyzes their dependence on visual input, temporal observations, frame order, and camera coverage. Results show a substantial human–model gap on a matched MCQ subset, alongside uneven use of temporal and surround-view evidence. Extensive appendices document construction decisions, scoring reliability, input limitations, and statistical controls.

Critical strengths
  1. The evidence controls make the benchmark diagnostically useful. The paper does more than show that models obtain low scores. Question-only, final-frame, shuffled-frame, and camera-coverage tests help separate different sources of difficulty. I particularly appreciate the segment-clustered analysis in Appendix D.1, which accounts for questions sharing source footage. The authors also acknowledge that camera coverage changes per-view resolution and that additional video does not consistently help Prediction, keeping the interpretation appropriately specific.

  2. The construction process includes substantial human correction. Appendix A.7 reports that every released question was reviewed by at least two people, with about 44% edited. Prediction observation boundaries and the use of ego-state or calibrated camera evidence are described explicitly. The matched 100-question human baseline is also more informative than comparing human accuracy on one sample with model accuracy on the full benchmark. These details make the dataset contribution more credible than an automated question-generation pipeline alone.

Critical weaknesses
  1. OpenVQA ranking reliability is not yet established. Appendix B.2 reports useful validation, but its 88.7% human agreement measures reference-based judgments without video. The frame-grounded audit finds 8 disagreements in 40 responses, and changing judges shifts Qwen3-VL-8B's acceptance rate from 33.6% to 37.2%. That cross-judge test covers only one answering model. Since OpenVQA supplies half the cells in the overall score, model-dependent judging effects could change close rankings even if average reference quality is reasonable.
Questions for authors
  1. Could you re-score responses from all leading models with the alternative judge and report overall and competency-level rank stability, with segment-clustered confidence intervals? The existing single-model comparison cannot reveal whether judging differences systematically favor particular answer styles or model families.

  2. Could you expand the blinded, frame-grounded human audit across model families and competencies, especially ambiguous Planning and Reasoning answers? Please distinguish incorrect references from rubric disagreements and report how correcting or excluding affected questions changes the benchmark scores.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.