- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces LTDriveQA, an event-centered benchmark for evaluating spatiotemporal reasoning in long-tail driving scenarios. Built from WOD-E2E, it contains 2,889 human-reviewed questions over 719 multi-view video clips and covers four competencies: perception, prediction, planning, and reasoning. The questions are provided in both multiple-choice and open-ended formats and are constructed through a model-assisted iterative pipeline with evidence-grounding checks and restricted-input diagnostics. The authors evaluate 25 general-purpose and driving-oriented vision-language models and find substantial gaps from human performance, particularly on temporal and cross-view reasoning. Additional ablations examine models’ reliance on visual evidence, multiple views, and temporal ordering.
Critical strengths
-
Clear and complete presentation. The paper is well organized and easy to follow, with the benchmark motivation, construction pipeline, evaluation protocol, and limitations presented clearly.
-
Meaningful benchmark setting. Building on long-tail events from WOD-E2E makes the evaluation more relevant to safety-critical and uncommon driving scenarios than standard scene-level driving QA.
-
Interesting spatiotemporal formulation. The use of synchronized multi-view videos enables questions that require temporal evolution and cross-camera evidence, extending beyond the predominantly single-frame settings of many existing driving VQA benchmarks.
-
Comprehensive experimental evaluation. The paper evaluates a broad range of general-purpose and driving-specific models and includes human comparisons, restricted-input controls, multi-view and temporal ablations, and detailed failure analyses. These experiments provide a relatively complete characterization of both the benchmark and current model limitations.
Critical weaknesses
-
Unclear external validity beyond driving QA. Although LTDriveQA is motivated as a benchmark of driving intelligence and includes a Planning category, all competencies are evaluated exclusively through language-answer correctness. The paper neither produces trajectories nor reports established planning metrics such as ADE/FDE, collision rate, or closed-loop performance. It is therefore unclear whether higher LTDriveQA scores correspond to safer or more accurate driving behavior. Correlating benchmark scores with trajectory-planning performance, or demonstrating that LTDriveQA supervision benefits an actual planner, would substantially strengthen its practical significance.
-
Limited validation of genuine temporal reasoning. Although LTDriveQA uses multi-frame video, some results suggest that performance benefits primarily from access to multiple observations rather than understanding their temporal order. The final frame alone performs close to the full video overall and even outperforms it on Prediction. For Qwen3-VL-8B, shuffling the frames causes a relatively modest degradation compared with removing earlier frames entirely. Thus, the benchmark establishes multi-frame evidence dependence more convincingly than ordered temporal reasoning.
-
Potential bias and uncertainty in model-assisted construction and scoring. Much of the question and reference-answer construction relies on VLM/LLM families that are also evaluated in the benchmark. Open-ended responses are further scored by a reference-based LLM judge rather than directly against the video. Although the authors provide useful human audits, the frame-grounded audit covers only 40 examples, and judge–human agreement on the paired video-grounded subset is 76.7%. More extensive independent human evaluation would strengthen confidence in the OpenVQA scores and model rankings.
Questions for authors
See weaknesses.