acceptodds
Under review as a conference paper at ICLR 2027

Look to Understand: Small-Large Model Collaboration for Traffic Scene Understanding

Abstract

Vision-Language Models (VLMs) have demonstrated remarkable capabilities in traffic scene understanding and visual question answering. However, large VLMs still face several challenges in complex traffic scenes, including redundant visual information, insufficient utilization of spatial information, and unclear correspondence between question semantics and relevant visual regions. To address these challenges, we propose Look to Understand, a Small-Large Model Collaborative Framework that leverages small models for visual perception and large VLMs for scene semantic understanding and visual question answering. First, we design a Small Model-Guided Perception Chain for structured visual perception. Given a traffic scene image, a lightweight vision model extracts object categories and spatial bounding boxes. The detection results are encoded as structured perception tokens, providing the large VLM with explicit object category and spatial location information. Furthermore, we propose Small Model-Guided Question-Region Contrastive Learning, which leverages the object regions provided by the lightweight vision model to model their relevance to the question semantics and aggregate their features. The question and visual region representations are then projected into a shared latent space, where bidirectional contrastive learning is employed to constrain their semantic correspondence. Experiments on RoadSceneVQA and DRIVINGVQA demonstrate the effectiveness of the proposed framework.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Model (VLM), Traffic Scene Understanding, Model Collaboration

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_V5UEOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Look to Understand, a Small Model-Guided Collaboration (SGC) framework for traffic scene VQA. It combines a lightweight object detector that provides structured object categories and bounding boxes to a large VLM with Question-Region Contrastive Learning (QRCL), which aligns question representations with relevant visual region features. The framework is jointly trained using answer-generation and contrastive losses and evaluated on RoadSceneVQA and DRIVINGVQA, including ablation studies.

Critical strengths
  1. Small-Large Model Collaboration: Leveraging lightweight models to provide additional visual and spatial information to large VLMs is a promising direction for improving scene understanding.

  2. Socially Relevant Application: The focus on traffic scene understanding is meaningful, with potential applications in road safety and intelligent transportation, aligning well with AI for social good.

  3. Evaluation Across Multiple Datasets: The authors evaluate their framework on two traffic scene VQA datasets and provide ablation studies to examine the contributions of the proposed components.

Critical weaknesses

1. Limited Novelty and Insufficient Comparison with Prior Work. The proposed framework primarily combines object detection, structured bounding-box representations, and conventional contrastive learning. Existing works, such as Ego3D-VLM (ICLR 26), have already explored providing explicit spatial information to VLMs for reasoning. Although the tasks differ, the paper does not sufficiently establish its methodological novelty relative to existing spatially grounded VLM frameworks.

2. Unclear Motivation. The introduction does not clearly establish why existing VLMs cannot adequately address the targeted traffic scene understanding tasks or why the proposed combination of object detection and contrastive learning is necessary. The motivation remains broad, without identifying a sufficiently specific technical limitation that the proposed method resolves.

3. Insufficiently Supported Claims. Several broad claims in the introduction concerning limitations of existing VLMs, small-large model collaboration, and fine-grained visual understanding lack sufficiently specific supporting citations. The discussion would benefit from stronger grounding in relevant literature.

4. Missing Implementation Details. The paper does not specify the lightweight object detection model, its architecture, training data, supported object categories, or detection configurations. Furthermore, the process of extracting region features from the detected bounding boxes is insufficiently described. These omissions significantly affect reproducibility.

5. Unclear Handling of Multiple Object Instances. The framework does not explain how it reliably distinguishes between multiple instances of the same object category, particularly when questions involve specific objects or spatial relationships. Although question-guided attention is employed, there is no explicit mechanism or evaluation demonstrating correct instance-level identification.

6. Architectural Ambiguity. Figure 1 identifies the language model as Qwen2.5 rather than Qwen2.5-VL, while the experimental section specifies Qwen2.5-VL-7B. It is unclear whether the authors use the pretrained VLM architecture or independently integrate a vision encoder and projector with a language model. Additionally, the interaction between the proposed contrastive learning module and the answer-generation pathway is insufficiently explained.

7. Weak Question-Region Grounding Formulation. The proposed contrastive objective considers matching question-image samples as positive pairs rather than using explicit question-to-object correspondences. Moreover, contrastive learning is performed using aggregated region representations instead of individual target regions. Consequently, the objective does not directly enforce the fine-grained question-region grounding claimed by the authors.

8. Lack of Grounding Evaluation. Despite emphasizing fine-grained question-region correspondence as a central contribution, the paper provides no quantitative evaluation of region selection, localization accuracy, or question-object alignment. The reported VQA metrics alone are insufficient to establish whether the model actually learns to identify the relevant visual regions.

9. Limited and Inadequate Baseline Comparisons. The experimental evaluation includes only a small selection of VLMs, omitting numerous recent and stronger alternatives. More importantly, there are no comparisons against existing region-aware, grounding-enhanced, or structured-perception VQA methods. This substantially limits the evidence supporting the competitiveness and novelty of the framework.

10. Inconsistent Experimental Reporting. Table 1 reports an Exact Match score of 0.152 for Qwen2.5-VL, whereas Table 3 reports a baseline score of 0.188 without clearly explaining the difference in experimental configurations. Additionally, InternVL3-8B is mentioned among the experimental models, but its results are not reported in the main comparison tables.

11. Poor Figure Quality and Presentation. The figures suffer from low visual resolution and readability, particularly Figure 1, where several components and connections are difficult to interpret. The architectural presentation also lacks sufficient clarity regarding the data flow and integration of the proposed modules, making the methodology unnecessarily difficult to follow.

Questions for authors

Please refer to the concerns raised in the Weaknesses section.

Reviewer_5ygnOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents a framework for using a small vision model to first spot important objects and their locations in traffic images, then passing that information to a larger vision-language model to answer questions. It also teaches the large model to focus on question-relevant regions, aiming to improve traffic-scene understanding and question answering.

Critical strengths

The only point from the paper that the reviewer can back is the simplicity and interpretability of the proposed detector-assisted VQA pipeline. Explicit object categories, bounding boxes, and question-conditioned region weighting provide a transparent mechanism for injecting spatially grounded information into an MLLM, and the approach yields consistent (though modest) improvements across.

Critical weaknesses

(The reviewer is intentionally not commenting further on rest of the paper beyond the method section as it doesn’t warrant further review due to lack of ANY new insights as per ICLR standards.)

  • The reviewer sees the method as a combination of “perception chain” i.e. run a lightweight detector, convert object classes + bounding boxes into structured tokens, and feed them alongside the image to the MLLM, and a version of “compute question-conditioned attention” which is over detected region features where we aggregate them, and align the resulting representation with the question using a standard bidirectional contrastive objective.

    • Neither of these are fundamentally new. Detector-based region features, bounding-box grounding, question-conditioned region attention, and image-text/region-text contrastive learning all have substantial prior precedent [1-3]. What the paper mainly does is combine these existing ideas for traffic VQA.

[1] Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering, CVPR 2018
[2] RegionCLIP: Region-Based Language-Image Pretraining, CVPR 2022
[3] Ferret: Refer and Ground Anything Anywhere at Any Granularity, ICLR 2024

Questions for authors

None

Reviewer_uQa3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Look to Understand, a small-large model collaboration framework for traffic-scene visual question answering. A lightweight object detector first extracts object categories and bounding boxes, which are converted into structured tokens and provided to a large VLM together with the original image. The paper further introduces a question-region contrastive objective to align question representations with detector-proposed visual regions. Experiments on RoadSceneVQA and DRIVINGVQA evaluate the method using exact match and text-generation metrics, reporting modest improvements over fine-tuned VLM baselines.

Critical strengths
  1. The motivation of providing explicit object and spatial cues to VLMs is intuitive and relevant to understanding cluttered traffic scenes.
  2. The framework is modular and conceptually simple, and the paper evaluates it on two traffic-scene VQA datasets with component-wise ablations.
Critical weaknesses
  1. The evaluation does not validate the central grounding claim or broader driving relevance. The paper reports only answer-level language metrics such as EM, BLEU, ROUGE, CIDEr, METEOR, and SPICE. No grounding or region-retrieval metric is provided, despite question-region alignment being a central contribution. The experiments also do not show whether the method benefits any downstream perception or driving task. Therefore, the results establish only modest improvements on traffic VQA.

  2. The “small model” component is insufficiently specified. The paper does not identify the detector architecture, checkpoint, training data, detection accuracy, or computational cost. It also inconsistently refers to a lightweight vision model and an object detection model without clarifying whether these are the same component. Moreover, the small model is simply frozen and used to provide object labels, bounding boxes, and region features, making the framework closer to detector-assisted VQA than genuine small-large model collaboration. These omissions substantially limit reproducibility.

  3. The technical contribution and empirical gains are limited. Detector-proposed regions with question-guided attention have long been studied in VQA, while modern region-aware VLMs already encode bounding boxes or RoI features into language models. The proposed framework mainly combines these established components with a standard question-region contrastive loss and reframes them as “small-large collaboration,” without introducing a substantially new collaboration mechanism. The empirical improvements are also small: EM increases from 0.188 to 0.201 on RoadSceneVQA and from 0.456 to 0.465 on DRIVINGVQA, without multiple runs or significance analysis. Moreover, although the paper claims to evaluate a QRCL-only configuration, Table 3 does not report it.

  4. The efficiency motivation is not validated. The paper reports no latency, FLOPs, memory usage, or parameter overhead. Since SGC still sends the original image through the full VLM while additionally running a detector and adding structured tokens, it may increase total computation rather than reduce it.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.