- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a multi-task benchmark for VLMs on fixed-camera Infrastructure AI footage across warehouse, transportation, and smart-space domains. It comprises 8 tasks over 4 reasoning pillars on 3346 media assets, annotated by trained professionals. The authors evaluate 18 open-weight and proprietary VLMs in a zero-shot setting and identify an Infrastructure AI Gap: the performance deficit relative to consumer-centric benchmarks is concentrated in event verification and temporal grounding.
Critical strengths
VANTAGE-Bench covers eight tasks across semantic, spatial, temporal, and spatio-temporal reasoning, and five of the eight tasks require structured outputs such as bounding boxes, temporal boundaries, or trajectories. This makes the benchmark more representative of realistic infrastructure-AI use cases. The paper goes beyond leaderboard reporting with diagnostic analyses that decompose semantic and temporal performance and examine tracking behavior across different temporal horizons and failure modes.
Critical weaknesses
The central claim of the Infrastructure AI Gap is not cleanly identified. Table 2 subtracts scores on different benchmarks, so the distribution shift is mixed up with differences in task difficulty and baseline performance. Moreover, the benchmark does not consistently use fixed-infrastructure imagery despite this being central to the paper's motivation: 2D Spatial Pointing is built from vehicle-mounted KITTI-style footage, and Referring Expressions uses drone imagery.
Questions for authors
-
The related-work coverage seems incomplete, which makes the novelty claims difficult to assess. Closely related surveillance and infrastructure video benchmarks, including UCA (CVPR 2024), TUMTraf VideoQA (ICML 2025), and WTS (ECCV 2024), are not discussed. Could the authors add a comparison table covering relevant prior work along dimensions such as domain coverage, camera setting, task and output format, and annotations? This would make clear which aspects of VANTAGE-Bench are novel and which are shared with prior work.
-
Could the authors report duration statistics (e.g., mean, median, and range) for the videos in each video-based task, as well as the duration statistics of the ground-truth temporal segments for Temporal Localization and Dense Video Captioning? This would help characterize the difficulty of the benchmark.
-
Thinking is disabled for open-weight models, while proprietary APIs, which likely reason internally, are uncontrolled. How do the open-weight models perform with thinking enabled, particularly on EV and Temp Loc?