- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper argues that current vision-language models (VLMs) couple both the extraction of information from visual inputs and reasoning over the extracted information for VQA tasks. The paper then proposes a tool-output-to-structure representation method, termed DriveTLM, which is a training-free framework that utilizes off-the-shelf vision tools and formats their outputs in a structured text format which the downstream VLM can digest instead of the visual image/video directly. The method is specific to the driving VQA task, and the paper reports performance on two benchmarks: DriveBench and LingoQA. The results show that across multiple VLM families and sizes, DriveTLM improves performance and observes stronger gains, particularly for sub-billion parameter models. The proposed structure is extensively ablated in terms of what components necessitate the observed gains.
Critical strengths
S1. The paper thoroughly ablates the proposed method DriveTLM in terms of its structure (Tab. 3), empirically verifies that each component is necessary (Tab. 5), and also compares with freeform serialization and visual overlays of text (Tab. 4), which are often the common ways of presenting visual information in textual form.
Critical weaknesses
W1. The paper's methodological contribution is the tool-output-to-structure representation framework, which essentially parses and formats the outputs from off-the-shelf vision tools (such as object detectors, etc.) into a text-based structure. However, it substantially overlaps with Perception Programs (P^2) introduced in [1], including the underlying motivation for using structured language-native representations of visual tools. Despite this direct relevance, the paper neither cites nor compares to [1]. Further, DriveTLM seems like a driving VQA instantiation of P^2 with a task-specific schema and additional heuristics. The authors should distinguish their contribution from [1] and discuss controlled comparisons using matched perception tools and answering backbones. It is unclear if the proposed structured scene text offers any distinct advantages over the existing work and if the gains stem from the proposed structure or task-specific adaptation.
[1] Janjua, M. K., Silva, H., Niu, D., & Rashidi, B. (2026, June). Don’t Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programs. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 5165–5174.
W2. The paper does not satisfactorily answer the central question it poses in Sec. 1 (lines 45-47): whether failures in (driving) VQA arise from inability to extract visual information or from limitations in downstream reasoning. While the empirical results show that providing structured text improves performance, they do not establish that visual information extraction is the primary source of failure. Since the experiments change how the information is extracted, represented, and presented to the model simulatenously, they do not disentangle extraction of information failures from reasoning failures. Further, the paper argues that tool-call and tool-use methods (Visual Sketchpad and V$^*$) are disadvantaged since they rely on manipulating the visual modality (e.g., through zoom, crop, etc.), but does not compare to them in Tabs. 1 and 2.
W3. The general-purpose VLM baselines in Tabs. 1 and 2 (GPT-4o and GPT-4V) are fairly dated (released in 2024 and 2023, respectively). Consequently, these comparisons do not show how DriveTLM performs relative to more recent general-purpose VLMs (e.g., GPT-5.5/5.6). This is particularly important given the central claim that explicit visual information extraction improves reasoning over direct visual inputs. The comparisons against recent VLMs should show whether reported advantages persist as visual perception and reasoning capabilities of general-purpose models improve.
Questions for authors
Q1. How are semantic/spatial relations and risk analysis derived in Sec. 3.3? Essentially, what are the deterministic rules and what tools are they based on?
I thank the authors for providing additional experiments and details. The points I raised in W2 and W3 are now addressed. I still have concerns regarding W1, however. In particular, the substantial improvement from S2 language facts to rule-based DriveTLM on Perception (Tab. R1) suggests that the representation and associated processing are highly tailored to the downstream task since the task-specific rules contribute significantly to performance. I think this contradicts what the authors claim that improvements arise from representation rather than task-specific adaptation of the representation. Further, P^2 is not restricted to raw tool measurements, as pointed out, and can accommodate derived semantic information and relations. Did the experiments conducted make sure that information was supplied when the comparison analysis was done between the two?