acceptodds
Withdrawn from ICLR 2027

Seeing with Tools, Reasoning with Language: Recasting Vision as Structured Scene Text for Driving VQA

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Driving visual question answering (VQA) requires extracting relevant visual information from images or videos and reasoning over it. Current vision-language models (VLMs) couple these tasks, obscuring whether their errors arise from visual information extraction or downstream reasoning. We investigate this distinction with DriveTLM, a training-free framework that delegates visual information extraction to visual processing tools, combining off-the-shelf vision models with deterministic algorithms. Their outputs are organized into structured scene text, an explicit language interface that lets the answering model reason without visual tokens. The framework supports images and videos, Rule-based or Agentic tool selection, and both VLM and LLM answer backbones without parameter updates. Across image benchmark DriveBench and video benchmark LingoQA, DriveTLM improves over direct visual input for multiple model families and sizes, with particularly strong gains for smaller backbones and competitive end-to-end latency. Controlled studies examine these gains: most persist after removing rule-derived risk and answer-adjacent statements, expressing the same information as text is more effective than visual overlays, and selecting relevant information matters more than including every available detail. The model also responds selectively when individual facts are changed. These findings show the value of structured scene text as an interface for delivering tool-derived evidence to frozen answering models. The code will be released upon publication.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Mutlimodal, VQA, VLMs

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_4xvFOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper argues that current vision-language models (VLMs) couple both the extraction of information from visual inputs and reasoning over the extracted information for VQA tasks. The paper then proposes a tool-output-to-structure representation method, termed DriveTLM, which is a training-free framework that utilizes off-the-shelf vision tools and formats their outputs in a structured text format which the downstream VLM can digest instead of the visual image/video directly. The method is specific to the driving VQA task, and the paper reports performance on two benchmarks: DriveBench and LingoQA. The results show that across multiple VLM families and sizes, DriveTLM improves performance and observes stronger gains, particularly for sub-billion parameter models. The proposed structure is extensively ablated in terms of what components necessitate the observed gains.

Critical strengths

S1. The paper thoroughly ablates the proposed method DriveTLM in terms of its structure (Tab. 3), empirically verifies that each component is necessary (Tab. 5), and also compares with freeform serialization and visual overlays of text (Tab. 4), which are often the common ways of presenting visual information in textual form.

Critical weaknesses

W1. The paper's methodological contribution is the tool-output-to-structure representation framework, which essentially parses and formats the outputs from off-the-shelf vision tools (such as object detectors, etc.) into a text-based structure. However, it substantially overlaps with Perception Programs (P^2) introduced in [1], including the underlying motivation for using structured language-native representations of visual tools. Despite this direct relevance, the paper neither cites nor compares to [1]. Further, DriveTLM seems like a driving VQA instantiation of P^2 with a task-specific schema and additional heuristics. The authors should distinguish their contribution from [1] and discuss controlled comparisons using matched perception tools and answering backbones. It is unclear if the proposed structured scene text offers any distinct advantages over the existing work and if the gains stem from the proposed structure or task-specific adaptation.

[1] Janjua, M. K., Silva, H., Niu, D., & Rashidi, B. (2026, June). Don’t Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programs. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 5165–5174.

W2. The paper does not satisfactorily answer the central question it poses in Sec. 1 (lines 45-47): whether failures in (driving) VQA arise from inability to extract visual information or from limitations in downstream reasoning. While the empirical results show that providing structured text improves performance, they do not establish that visual information extraction is the primary source of failure. Since the experiments change how the information is extracted, represented, and presented to the model simulatenously, they do not disentangle extraction of information failures from reasoning failures. Further, the paper argues that tool-call and tool-use methods (Visual Sketchpad and V$^*$) are disadvantaged since they rely on manipulating the visual modality (e.g., through zoom, crop, etc.), but does not compare to them in Tabs. 1 and 2.

W3. The general-purpose VLM baselines in Tabs. 1 and 2 (GPT-4o and GPT-4V) are fairly dated (released in 2024 and 2023, respectively). Consequently, these comparisons do not show how DriveTLM performs relative to more recent general-purpose VLMs (e.g., GPT-5.5/5.6). This is particularly important given the central claim that explicit visual information extraction improves reasoning over direct visual inputs. The comparisons against recent VLMs should show whether reported advantages persist as visual perception and reasoning capabilities of general-purpose models improve.

Questions for authors

Q1. How are semantic/spatial relations and risk analysis derived in Sec. 3.3? Essentially, what are the deterministic rules and what tools are they based on?

1 comment

I thank the authors for providing additional experiments and details. The points I raised in W2 and W3 are now addressed. I still have concerns regarding W1, however. In particular, the substantial improvement from S2 language facts to rule-based DriveTLM on Perception (Tab. R1) suggests that the representation and associated processing are highly tailored to the downstream task since the task-specific rules contribute significantly to performance. I think this contradicts what the authors claim that improvements arise from representation rather than task-specific adaptation of the representation. Further, P^2 is not restricted to raw tool measurements, as pointed out, and can accommodate derived semantic information and relations. Did the experiments conducted make sure that information was supplied when the comparison analysis was done between the two?

Reviewer_RTQ3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce "DriveTLM", a training-free framework for driving VQA wherein external perception tools are used to convert images/videos into structured scene text, which is then provided to a frozen LLM-based answering model without visual tokens. The framework supports rule-based and question-conditioned agentic tool selection. Experiments on DriveBench and LingoQA show improvements across multiple backbones, particularly smaller models. Further analysis studies the contributions of scene evidence, hand-written task rules, textual representation, and temporal information.

Critical strengths
  1. The authors proposed method is straightforward, training-free and shows useful improvements on both image and video driving VQA over different LLMs. For Qwen2.5-VL-3B, agentic DriveTLM improves DriveBench Behavior accuracy from 38.5% to 53.0%, while the rule-based video pipeline improves LingoQA from 45.8 to 55.2. These results support method's main motivation of supplying explicit scene information to smaller models through relevant tools.

  2. The controlled experiments are useful in understanding the pipeline beyond its overall performance. The authors separately examine removing task-specific rules, presenting tool outputs as text versus visual annotations, selecting question-relevant information, and adding temporal evidence. Specifically, the matched-response-schema video experiment shows that combining snapshot and temporal tool text improves LingoQA from 47.8 to 51.2, even without the complete pipeline’s task-specific interpretations. The scene-disjoint development set and frozen configuration before benchmark evaluation are also appropriate controls.

Critical weaknesses
  1. The methodological novelty is relatively limited. The framework primarily combines existing perception models, deterministic rules for constructing scene text, and a frozen LLM. The agentic variant also uses straightforward one-shot tool selection. Further differentiation from related works such as AgentThink (EMNLP 2025) and Talk2BEV (ICRA 2024), and potentially stronger comparisons, would help establish the novelty.

  2. Concerns regarding evaluation reliability: as acknowledged in Appendix B.3, the Perception answer extractor can incorrectly interpret the article "a" as option A, and an always-A predictor already achieves 59% accuracy due to class imbalance. While Behavior uses a corrected extractor, Perception retains the original one. Re-evaluation with a consistent and validated extraction procedure would strengthen the reported results.

  3. (Relatively minor) The generalization of the hand-crafted rules and perception tools could be studied further. While the authors evaluate across DriveBench and LingoQA and provide some robustness analysis, it remains unclear how well the fixed rules and tool configurations generalize to different driving environments, road layouts, and unseen question types. Further cross-domain evaluation would strengthen the work's practical applicability.

Questions for authors

Please see weaknesses above.

Reviewer_v6S5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

To disentangle whether VLM failures in driving VQA stem from visual perception bottlenecks or downstream reasoning limitations, the authors introduce DriveTLM. DriveTLM uses a library of 8 frame-level and 3 video-temporal visual processing tools selected either via a Rule-based full pool or an agentic question-conditioned router. A deterministic textualizer converts tool outputs into a five-level hierarchy of structured scene text. The answering model receives only this structured text and the question. DriveTLM substantially improves frozen VLMs and enables a 0.6B text-only LLM to outperform a raw-image 3B VLM.

Critical strengths
  1. The experiment is convincing. The paper separates scene evidence from hand-crafted task rules, and the comparisons against the text interface and visual overlays are clear.
  2. The proposed approach empowers compact models and text-only LLMs across image and video modalities. DriveTLM boosts Qwen2.5-VL-3B on DriveBench from 28.5 to 50.2 and enables a tiny 0.6B text-only LLM to beat raw-image 3B/7B VLMs.
Critical weaknesses
  1. The algorithmic novelty is limited, and there exists a gap in open-ended generation metrics (e.g., ROUGE-L and CIDEr). Converting upstream perception outputs into structured text for LLM driving reasoning has been explored in previous approaches.
  2. DriveTLM completely discards the visual information. This may lead to overfitting on the tested benchmarks with prior visual information knowledge. However, in real-world driving scenarios, the visual inputs are complex and varying. Any visual information that does not fall within the predefined tools and fixed text template fields will be lost. This increases the risk of applying the proposed framework.
Questions for authors
  1. Can authors provide more ood results based on the provided tools and framework?
  2. In Table 4, why does adding raw images to selected tool facts as text lead to a severe score drop? If a question depends on an open-world visual detail not covered by your tools, how can DriveTLM recover without visual tokens?
Reviewer_ZUebOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DriveTLM, a training-free agentic framework that recasts visual inputs as structured scene text for driving VQA. Experiments demonstrate the efficacy of the proposed method.

Critical strengths
  1. The proposed method achieves a good improvement over its baselines.
  2. The figures are very elegant and clear, making it easy for readers to understand the core idea.
Critical weaknesses
  1. The main concern is about the novelty. The core idea is to transform the input image/video into a predefined format of structured scene text for LLMs. The performance is better than directly feeding the image/video or transforming it into a caption. However, the idea is similar to a scene graph but with an optimized format. The entire pipeline is prompting- and engineering-oriented, lacking scientific insights. The gain may be mainly from the manually designed input format.
  2. The experimental comparison may consider some other training-free methods in the literature.

Although this work achieves good results over baselines, it better matches venues with an application track. My rating at this stage is weak rejection. My final recommendation would depend on the rebuttal.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.