acceptodds
Withdrawn from ICLR 2027

COVLM-BENCH: A REAL-WORLD BENCHMARK FOR COOPERATIVE DRIVING QUESTION ANSWERING AND PLANNING

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models (VLMs) have made substantial progress in autonomous driving, but their success has primarily been studied in ego-centric scenes. Infrastructure-side observations provide views beyond the ego vehicle's field of view, yet conventional cooperative-driving systems typically transform them into geometric representations for downstream perception and planning. Directly incorporating these views into VLMs offers an opportunity to improve cooperative scene understanding and trajectory planning. However, question answering and trajectory planning have not been jointly evaluated on the same real-world vehicle-infrastructure scenes. We present CoVLM-Bench, a benchmark for cooperative driving question answering (CDQA) and cooperative planning (CP) on vehicle-infrastructure paired scenes. CoVLM-Bench provides scene-grounded CDQA annotations, three-part rationales as auxiliary supervision, and future trajectory targets derived from recorded ego motion. It contains 2,196 paired frames with 35,136 CDQA annotations, while CP predicts six waypoints over a three-second horizon. The annotations combine model-assisted drafting, record-based computation, and human verification. Built upon CoVLM-Bench, we introduce CoVLM-Drive, a unified VLM baseline that directly uses paired views for both CDQA and CP. Experiments show that CDQA adaptation improves answer accuracy and that CoVLM-Drive reaches a lower FDE than the compared V2X planners; QA initialization and rationale supervision each reduce planning error. Together, CoVLM-Bench and CoVLM-Drive support the training and comparison of VLMs for cooperative scene understanding and planning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Cooperative Autonomous Driving, Vision–Language Models, Vehicle–Infrastructure Cooperation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nRR7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces COVLM-Bench, a dataset for testing whether driving VLMs can use both the car’s camera and a roadside camera. It asks the model to answer driving questions and predict the car’s future path. The authors also build CoVLM-Drive, a baseline model trained on both tasks. The benchmark is based on real-world synchronized vehicle-infrastructure scenes overall.

Critical strengths
  • The reviewer thinks (and likes) that the authors provide a controlled benchmark for measuring whether multimodal models truly use complementary infrastructure-side visual evidence. The proposed COVLM-Bench repurposes an existing real-world V2I dataset (V2X-Seq-SPD) into a multimodal reasoning-and-planning benchmark by adding source-aware QA, rationale, and trajectory supervision aligned on the same synchronized vehicle-infrastructure scenes.

  • Another interesting point is that the evaluation (described in Sec. 5.5) provides evidence that the proposed cooperative QA tasks genuinely require matched infrastructure-side visual information rather than being solvable from ego-view or language biases alone.

Critical weaknesses
  • COVLM-Bench is basically a re-annotation of an existing benchmark so that the VLM can be tested for, with the same scene, “What is happening?” and “What should the car do next?”. In the reviewer's opinion, even the model contribution is much weaker: COVLM-Drive mainly feeds the two images into an existing VLM, uses LoRA, and adds structured command/trajectory heads.

  • Also, the benchmark labels some objects as “roadside-only” based on annotation matching, not direct visual confirmation that the ego camera cannot see them. Because missing an ego-side label does not necessarily mean an object is invisible, the reviewer is concerned that some QA ground truth may reflect annotation gaps rather than genuine cross-view visibility in practice.

Questions for authors

So one question: As per Table 6, if cooperative vision is genuinely helping planning, a mismatched roadside image should hurt much more, yet the shuffled condition is only 0.065 m worse in FDE than the correctly matched view (4.976 vs. 4.911 m), and Appendix F shows the matched view wins on just 51.4% of examples. Can the authors comment on this?

Reviewer_jY8bOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CoVLM-Bench, a benchmark for cooperative driving question answering (CDQA) and cooperative planning (CP) using synchronized vehicle and infrastructure camera views. Built on V2X-Seq-SPD, it contains 2,196 paired frames, 35,136 QA annotations, and 2,129 examples with future-trajectory targets. The authors use Qwen3-VL-32B to draft scene descriptions, QA annotations, and rationales; some answers and planning targets are computed from the source object and motion records, and the annotations are then human-verified. The paper also proposes CoVLM-Drive, a VLM-based baseline that generates QA answers and rationales and uses structured prediction heads for driving commands and waypoints. CDQA adaptation increases Qwen3-VL-8B’s QA accuracy from 24.7% to 39.6%, while the standard CP configuration reports an FDE of 4.911 m on 654 evaluation examples. The paper also compares matched roadside input with ego-only, blank, and shuffled roadside inputs.

Critical strengths
  1. The benchmark evaluates cooperative QA and planning on the same real-world scenes.
    Connecting the two tasks to the same vehicle–infrastructure views provides a useful basis for studying scene understanding and trajectory prediction together.

  2. The paper includes input controls to examine the effect of infrastructure imagery.
    It compares ego-only, blank roadside, shuffled roadside, and matched paired-view inputs, and evaluates multiple VLMs and V2X planning baselines. The authors also report that generated annotations were checked by human reviewers.

Critical weaknesses
  1. The practical planning benefit of infrastructure-camera input is not clearly established.
    The motivation that roadside cameras can reveal objects outside the ego vehicle’s field of view is reasonable, but the paper does not clearly show when this information is necessary for trajectory planning. The FDE is 4.911 m with matched roadside input, 5.776 m with ego-only input, 5.085 m with a blank roadside image, and 4.976 m with a shuffled roadside image. Although the paired input improves over the ego-only condition, the gain over a shuffled roadside view is only 0.065 m. The evaluation also compares a single ego view against an ego–infrastructure pair, without a multi-camera ego-view baseline. Thus, the results suggest that paired input can help, but do not establish that the infrastructure viewpoint is necessary for planning or more useful than additional vehicle-mounted camera views. The benefit seen in QA does not, by itself, demonstrate a meaningful benefit for trajectory planning.

  2. The methodological novelty and the value of combining QA with planning remain unclear.
    CoVLM-Drive uses a VLM backbone to generate language outputs and structured planning heads to predict driving commands and waypoints. The main contribution therefore seems to be the benchmark’s combination of QA and planning, rather than a new modeling approach. However, the comparison with prior cooperative QA and V2X planning work is largely presented as a feature checklist; the paper does not directly establish what new insight this joint evaluation provides over evaluating the tasks separately. The experiments also do not clearly show that a model supporting both tasks is better than task-specific models. QA initialization and rationale supervision each improve planning results, but the reported results do not establish a consistent combined benefit. Finally, the 35,136 QA annotations come from 16 fixed templates per frame; the reported QA evaluation uses 2,861 items, and planning is evaluated on 654 examples. The paper does not report repeated-run variability, making it difficult to assess the stability and generality of the results.

Questions for authors
  1. In what driving situations is infrastructure-camera information necessary for planning? Given the small FDE difference between matched and shuffled roadside inputs, can you provide a scenario-level analysis showing when the infrastructure view meaningfully changes or improves the predicted trajectory? Do you have results comparing against multiple ego-mounted cameras?

  2. What is the main methodological contribution of CoVLM-Drive, and what new findings does evaluating QA and planning on the same benchmark enable compared with evaluating them separately? Can you compare directly with the closest cooperative QA and V2X planning approaches under a shared protocol?

  3. Is CoVLM-Drive intended to perform QA and planning in one model, or does it provide task-specific adapted models? What evidence shows that combining the tasks is beneficial, given that the reported improvements from QA initialization and rationale supervision do not clearly demonstrate an additional benefit from joint training?

Reviewer_eqXSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a benchmark for both cooperative driving question answering (CDQA) and cooperative planning (CP). Based on V2X-Seq-SPD, they construct CoVLM-Bench for the joint evaluation of collaborative perception and planning, by organizing annotations into four layers: "object evidence," "scene descriptions", "question-answer annotations", and "three-part rationales", while also providing ground-truth future trajectories for each scene. The authors further propose CoVLM-Drive, a VLM baseline that directly ingests images from both the ego-vehicle and roadside infrastructure. It utilizes language decoder for question answering and explanation generation, and structured prediction heads for trajectory prediction, while also analyzing the impact of QA initialization, rationale supervision, and roadside-input conditions on performance.

Critical strengths

The paper establishes a clear set of controlled experiments to examine the role of roadside information.

Critical weaknesses
  1. The boundaries of innovation relative to existing work are not sufficiently clear. Since V2V-LLM/V2V-QA already cover collaborative question answering and trajectory planning in real-world scenarios, this paper needs to further clarify what substantive research questions or evaluation capabilities it introduces beyond RGB inputs and data construction.
  2. The evidence for the benefits of accurate roadside information for planning is weak. The difference in Final Displacement Error (FDE) between scenarios with correctly matched versus mismatched roadside imagery is only about 0.065 meters. Furthermore, the lack of repeated experiments and statistical uncertainty analysis makes it difficult to conclusively demonstrate that the model effectively leverages scene-relevant collaborative information.
  3. The connection between linguistic reasoning and actual actions is not sufficiently validated. The reasoning text serves only as auxiliary supervision and does not directly inform the action prediction head; moreover, its annotation is derived with reference to the ground-truth future trajectory.
Questions for authors
  1. V2V-LLM/V2V-QA has already evaluated both comprehension-based QA and continuous trajectory planning within real-world collaborative scenarios; thus, the task composition in this paper overlaps significantly with that prior work. Given the differences in input modalities and collaborative configurations, please clarify: does the core contribution of this paper primarily stem from new perception challenges, a novel annotation scheme, or a distinct evaluation protocol? Specifically, what research capabilities are introduced or substantially strengthened by these design choices? It is recommended to include a direct comparison with the aforementioned work in the core comparison table to demonstrate that this paper’s contributions extend beyond mere changes to data sources or input interfaces.
  2. Since the reference answers in Maneuver QA represent the actual actions taken, while the questions ask for the safest course of action, a model might be marked incorrect if it provides a more conservative answer that differs from the recorded behavior. How do the authors address this issue "the inconsistency in labels"?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.