- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces DriveAnswer, a driving VQA benchmark for evaluating whether vision-language models can determine when the available visual evidence is sufficient to answer a question. Built from nuScenes, it contains 20,151 grounded questions across seven spatial, temporal, cross-view, and decision-related families. For each question, the authors construct matched strong- and weak-evidence configurations that preserve the underlying scene fact and question while retaining or removing required camera views or temporal observations. Models are expected to answer under strong evidence and abstain under weak evidence, while also localizing the supporting visual regions. The paper evaluates five VLMs using answer accuracy, abstention, matched-pair success, and grounding metrics, and conducts staged adaptation experiments on Gemma and Qwen using answer–abstention, grounding, and matched-pair supervision. The results show that semantic correctness, evidence-sensitive abstention, and visual grounding can differ substantially, while direct supervision improves the latter two capabilities.
Critical strengths
-
The paper introduces an interesting setting that evaluates not only whether a model answers correctly, but also whether the available visual evidence is sufficient to support an answer.
-
The matched strong–weak evidence pairs and grounding annotations provide a useful benchmark for jointly evaluating answering, abstention, and evidence localization. The dataset construction also involves substantial effort.
Critical weaknesses
-
Limited task scope. The main benchmark focuses on retrospective perception and scene understanding. It does not address evidence sufficiency for future prediction or planning, where sufficient evidence is harder to define.
-
Unvalidated benefit to actual planning. The paper evaluates only answer/abstention and grounding performance. It does not test whether training with DriveAnswer improves the trajectory-planning ability of an autonomous driving VLA. For example, the authors could fine-tune the same driving model with and without the proposed QA supervision and compare ADE/FDE, collision rate, or closed-loop performance. Therefore, the practical value of evidence-aware QA for autonomous driving remains unclear.
-
Incremental contribution. Abstention under insufficient evidence has been widely studied in reliable VQA and selective prediction. This work mainly applies the idea to nuScenes through matched evidence pairs and grounding annotations, without demonstrating cross-dataset generalization or benefits to actual driving systems.
Questions for authors
See weaknesses.
2 comments
We include the reference for the papers we mentioned below.
References:
Sima, C., Renz, K., Chitta, K., Chen, L., Zhang, H., Xie, C., ... & Li, H. (2024, September). DriveLM: Driving with Graph Visual Question Answering. In European Conference on Computer Vision (pp. 256-274). Cham: Springer Nature Switzerland.
Xie, S., Kong, L., Dong, Y., Sima, C., Zhang, W., Chen, Q. A., ... & Pan, L. (2025, October). Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 6285-6297). IEEE.
Sohn, T. S., Dillitzer, M., Bach, J., Corso, J. J., Brühl, T., Schwager, R., ... & Sax, E. (2025, June). Drive4C: A Closed-Loop Benchmark on what Foundation Models Really Need to be Capable of for Language-Guided Autonomous Driving. In *2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) *(pp. 3859-3869). IEEE.
Whitehead, S., Petryk, S., Shakib, V., Gonzalez, J., Darrell, T., Rohrbach, A., & Rohrbach, M. (2022, October). Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectly. In European Conference on Computer Vision (pp. 148-166). Cham: Springer Nature Switzerland.
Guo, Y., Jiao, F., Shen, Z., Nie, L., & Kankanhalli, M. (2024). UNK-VQA: A Dataset and a Probe Into the Abstention Ability of Multi-Modal Large Models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12), 10284-10296.
We sincerely thank the reviewer for the thoughtful feedback. We appreciate the opportunity to elaborate on DriveAnswer's scope, its relevance to autonomous driving, and its relationship to prior work.
1. Scope and extension toward prediction and planning
We appreciate the suggestion to consider future prediction and planning, where uncertainty and evidence sufficiency become especially consequential. DriveAnswer takes a complementary first step by examining whether a driving VLM's conclusions are supported by the observations available to it. Even at the observational level, this requires reasoning across multiple camera views, temporal context, actor relationships, motion states, and map information. Through question-conditioned strong–weak pairs, DriveAnswer provides a controlled way to examine how models respond when evidence needed for a particular conclusion becomes unavailable.
This complements work such as DriveLM (Sima et al., ECCV 2024), which connects graph-structured visual question answering with perception, prediction, and planning. We see observational sufficiency as a foundation for further investigating evidence-aware prediction and planning, and will clarify this scope and future direction in the camera-ready version.
2. Practical significance for autonomous driving
We thank the reviewer's suggestion for evaluating downstream planning performance through ADE/FDE, collision rates, and closed-loop testing. We agree that demonstrating the benefits of evidence-aware supervision for planning is a valuable next step, and will clarify this direction in the revised manuscript.
We would also like to emphasize DriveAnswer's practical relevance through scene understanding and observational sufficiency as complementary foundations for reliable planning. DriveBench (Xie et al., ICCV 2025) identifies a critical reliability concern: driving VLMs can produce plausible responses based on general knowledge or textual cues without adequate visual grounding. Such unsupported interpretations could influence downstream planning decisions, even when conventional answer-accuracy metrics fail to reveal the underlying problem.
DriveAnswer directly addresses this evaluation gap. By distinguishing semantic correctness from observational support, its matched strong–weak protocol and grounding metrics, including Joint@t and JAGS, provide a diagnostic assessment of whether driving-scene conclusions are both correct and supported by the available evidence, an important diagnostic step before downstream planning. This perspective is also consistent with Drive4C (Sohn et al., CVPRW 2025), which emphasizes that aggregate driving scores do not necessarily reveal the underlying reasons for model failures and therefore evaluates scenario understanding separately from driving performance.
We therefore see DriveAnswer's diagnostic contribution as valuable independently, with future planning-oriented studies as a strong next step. Our experiments demonstrate that models can learn the benchmark's targeted capabilities; they are not intended as claims of improved trajectory planning. We will make this distinction explicit in the revision.
3. Relationship to selective prediction and existing driving benchmarks
We agree with the reviewer’s observation that abstention and selective prediction are well-established research directions. However, we see DriveAnswer as building on this foundation while addressing a complementary question: not only whether a model's answer is likely to be correct, but whether the observations provided actually support that answer.
For example, Reliable VQA (Whitehead et al., ECCV 2022) studies selective prediction by estimating the likelihood of answer correctness. UNK-VQA (Guo et al., TPAMI 2024) more directly investigates unanswerability through image and question perturbations. DriveAnswer builds on these ideas in the autonomous-driving setting by holding the question and underlying scene fact fixed while varying the evidence available to support the answer.
Through matched strong–weak pairs and observation-level grounding, we can distinguish a semantically correct but unsupported answer from one justified by the supplied observations. This evidence-centered distinction complements confidence-based selective prediction, particularly in multi-view and temporal driving scenes where the information required for an answer may be distributed across observations.
Our existing Table 1 compares DriveAnswer with representative driving benchmarks. We will expand the related-work discussion to clarify these connections and better communicate the contribution of question-conditioned observational sufficiency evaluation. We once again thank the reviewer for the feedback and look forward to future discussion.