- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces COVLM-Bench, a dataset for testing whether driving VLMs can use both the car’s camera and a roadside camera. It asks the model to answer driving questions and predict the car’s future path. The authors also build CoVLM-Drive, a baseline model trained on both tasks. The benchmark is based on real-world synchronized vehicle-infrastructure scenes overall.
Critical strengths
-
The reviewer thinks (and likes) that the authors provide a controlled benchmark for measuring whether multimodal models truly use complementary infrastructure-side visual evidence. The proposed COVLM-Bench repurposes an existing real-world V2I dataset (V2X-Seq-SPD) into a multimodal reasoning-and-planning benchmark by adding source-aware QA, rationale, and trajectory supervision aligned on the same synchronized vehicle-infrastructure scenes.
-
Another interesting point is that the evaluation (described in Sec. 5.5) provides evidence that the proposed cooperative QA tasks genuinely require matched infrastructure-side visual information rather than being solvable from ego-view or language biases alone.
Critical weaknesses
-
COVLM-Bench is basically a re-annotation of an existing benchmark so that the VLM can be tested for, with the same scene, “What is happening?” and “What should the car do next?”. In the reviewer's opinion, even the model contribution is much weaker: COVLM-Drive mainly feeds the two images into an existing VLM, uses LoRA, and adds structured command/trajectory heads.
-
Also, the benchmark labels some objects as “roadside-only” based on annotation matching, not direct visual confirmation that the ego camera cannot see them. Because missing an ego-side label does not necessarily mean an object is invisible, the reviewer is concerned that some QA ground truth may reflect annotation gaps rather than genuine cross-view visibility in practice.
Questions for authors
So one question: As per Table 6, if cooperative vision is genuinely helping planning, a mismatched roadside image should hurt much more, yet the shuffled condition is only 0.065 m worse in FDE than the correctly matched view (4.976 vs. 4.911 m), and Appendix F shows the matched view wins on just 51.4% of examples. Can the authors comment on this?