acceptodds
Withdrawn from ICLR 2027

VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

As Vision-Language Models (VLMs) rapidly advance toward physical deployment, the predominant focus has remained on action-oriented Embodied AI evaluated on subject-centric consumer video. This trajectory overlooks a pervasive class of Physical AI: Infrastructure AI, which relies on fixed-infrastructure cameras for open-loop insights like safety monitoring and operational logging. How well VLMs process dense, fixed-viewpoint visual data remains insufficiently explored. We introduce VANTAGE-Bench, an evaluation benchmark that measures this "Infrastructure AI Gap," distinguished by four features: 1) Domain Relevance, spanning three operational environments (Warehouse, Transportation, and Smart Spaces); 2) Modality Breadth, unifying image and video evaluation to probe semantic, spatial, temporal, and spatio-temporal capabilities; 3) Task Format Diversity, moving beyond multiple-choice to eight task formulations spanning discriminative QA, generative dense captioning, and spatio-temporal grounding; and 4) Evaluation Novelty, a single-pass trajectory-generation protocol for Single Object Tracking and, to our knowledge, the first such evaluation on fixed-camera infrastructure video, scored against specialist trackers. Annotation spans three regimes over 3,346 expert-annotated media assets: video tasks (3,342 annotations over 854 videos), image grounding (4,281 over 1,864 images), and dense detection (27,404 boxes over 628 images). Evaluating 18 models zero-shot, we find the shortfall relative to consumer-centric benchmarks is concentrated rather than general. Event verification and temporal localization fall 11.7 to 23.5 points at all three scales we can match, while video question answering stays within 5.3 points of VideoMME and 2D spatial pointing is flat or better against BLINK. The temporal pillar is the weakest: GPT-6 Astra is the only model of the eighteen to clear 55.71 mIoU on temporal localization or 37.28 SODA on dense video captioning, leading the field by 9.4 mIoU on the former and 0.9 SODA on the latter. On tracking, the strongest frontier model matches purpose-built specialist trackers at every horizon we test, while weaker open-weight models fall to a static-box baseline. Open-weight models lead 2D object localization outright, so neither parameter count nor proprietary access accounts for the pattern. The dataset, evaluation harness, and public leaderboard are publicly released; the harness is available at an anonymized repository, and the dataset and leaderboard URLs are withheld for double-blind review.

Primary area:
datasets and benchmarks
Keywords:
vision-language models, benchmark, evaluation, physical AI, infrastructure cameras, fixed-camera video, video understanding, spatio-temporal grounding, single object tracking, temporal localization, dense video captioning, referring expressions

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_fHvFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a multi-task benchmark for VLMs on fixed-camera Infrastructure AI footage across warehouse, transportation, and smart-space domains. It comprises 8 tasks over 4 reasoning pillars on 3346 media assets, annotated by trained professionals. The authors evaluate 18 open-weight and proprietary VLMs in a zero-shot setting and identify an Infrastructure AI Gap: the performance deficit relative to consumer-centric benchmarks is concentrated in event verification and temporal grounding.

Critical strengths

VANTAGE-Bench covers eight tasks across semantic, spatial, temporal, and spatio-temporal reasoning, and five of the eight tasks require structured outputs such as bounding boxes, temporal boundaries, or trajectories. This makes the benchmark more representative of realistic infrastructure-AI use cases. The paper goes beyond leaderboard reporting with diagnostic analyses that decompose semantic and temporal performance and examine tracking behavior across different temporal horizons and failure modes.

Critical weaknesses

The central claim of the Infrastructure AI Gap is not cleanly identified. Table 2 subtracts scores on different benchmarks, so the distribution shift is mixed up with differences in task difficulty and baseline performance. Moreover, the benchmark does not consistently use fixed-infrastructure imagery despite this being central to the paper's motivation: 2D Spatial Pointing is built from vehicle-mounted KITTI-style footage, and Referring Expressions uses drone imagery.

Questions for authors
  1. The related-work coverage seems incomplete, which makes the novelty claims difficult to assess. Closely related surveillance and infrastructure video benchmarks, including UCA (CVPR 2024), TUMTraf VideoQA (ICML 2025), and WTS (ECCV 2024), are not discussed. Could the authors add a comparison table covering relevant prior work along dimensions such as domain coverage, camera setting, task and output format, and annotations? This would make clear which aspects of VANTAGE-Bench are novel and which are shared with prior work.

  2. Could the authors report duration statistics (e.g., mean, median, and range) for the videos in each video-based task, as well as the duration statistics of the ground-truth temporal segments for Temporal Localization and Dense Video Captioning? This would help characterize the difficulty of the benchmark.

  3. Thinking is disabled for open-weight models, while proprietary APIs, which likely reason internally, are uncontrolled. How do the open-weight models perform with thinking enabled, particularly on EV and Temp Loc?

Reviewer_F2BMOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces VANTAGE-Bench, a benchmark for evaluating vision-language models in infrastructure monitoring scenarios, including warehouses, transportation, and smart spaces. It consists of eight tasks covering semantic, spatial, temporal, and spatio-temporal understanding, with 3,346 media assets. The authors evaluate 18 VLMs and report that temporal understanding, particularly temporal localization and dense video captioning, remains challenging.

Critical strengths

The paper provides a reasonably broad evaluation of 18 VLMs across eight tasks, covering both open-weight and proprietary models. The inclusion of open-ended localization and tracking tasks beyond conventional multiple-choice QA is useful.

Critical weaknesses

this problem has already been studied in prior work. For example, Towards Surveillance Video-and-Language Understanding: New Dataset, Baselines, and Challenges (CVPR 2024) introduces a surveillance video-language benchmark with multiple tasks and temporally grounded annotations. More recent benchmarks, such as SurveillanceVQA-589K and UCVL, also explicitly evaluate multimodal models on surveillance videos, including spatial, temporal, and causal understanding. The paper does not adequately discuss or compare against these closely related benchmarks. As a result, the novelty of VANTAGE-Bench is unclear. Combining existing visual understanding tasks into a single evaluation suite is not a concrete contribution

The paper repeatedly emphasizes fixed-infrastructure cameras as the defining characteristic of the proposed benchmark. However, several of its evaluation tracks do not follow this setting. The 2D Referring Expressions task uses aerial imagery from RefDrone, 2D Spatial Pointing uses vehicle-mounted imagery, and Single Object Tracking relies entirely on synthetic warehouse sequences.

Most tasks in the benchmark, including video question answering, object localization, referring expression grounding, temporal localization, dense captioning, and object tracking, are established vision-language or computer vision tasks. The main findings, such as weaker temporal grounding and different performance patterns across model families, are useful observations but do not clearly establish fundamentally new insights about VLM capabilities.

Questions for authors

How does VANTAGE-Bench differ substantially from existing surveillance video-language benchmarks, particularly UCA (CVPR 2024), SurveillanceVQA-589K, and UCVL? Why are these closely related benchmarks not discussed or systematically compared in the paper?

Reviewer_vjJXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed a VANTAGE-Bench, to measures “Infrastructure AI Gap,” which relies on fixed-infrastructure cameras for open-loop insights like safety monitoring and operational logging and is insufficiently explored.

This bench has four features: 1) Domain Relevance, 2) Modality Breadth, 3) Task Format Diversity, 4) Evaluation Novelty.

Evaluating 18 models zero-shot, the authors find the shortfall relative to consumer-centric benchmarks is concentrated rather than general. Three application domains are evaluated: Warehouse, Transportation, and Smart Spaces.

Four evaluation tasks are:

  1. Temporal: Dense Video Captioning, Temporal Localization
  2. Spatio-Temporal: Single Object Tracking
  3. Spatial: 2D Spatial Pointing, 2D Object Localization, 2D Referring Expressions
  4. Semantic: Event Verification, Video Question Answering
Critical strengths

S1: The logics and writing is clear

Critical weaknesses

W1: How different and relative novel this benchmark is, compared with existing similar benchmark, in terms of task difficulty, data complexity, number of cameras? The authors can better position their differentiations and challenges compared with other existing benchmarks, e.g., with a comparision table, and more importantly, how technically difficult it becomes when we are dealing with infra-/fixed cameras and what is the gap / why the existing VLM bench cannot handle those difficulties.

W2. Some important related work is missing, e.g., [R1], which is also fixed cameras. [R1] GTR-Bench: Evaluating Geo-Temporal Reasoning in Vision-Language Models

Questions for authors

Q1: Is it easier to deal with and understand the videos from the fixed cameras.

Reviewer_4fWiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a new evaluation setting for VLMs referred to as Infrastructure AI, a setting consisting of videos recorded from monitoring cameras. They introduce VANTAGE-Bench to measure capabilities in this setting, and evaluate a large number of existing VLMs on their benchmark. The benchmark contains eight tasks organized into four pillars (semantic, spatial, temporal, and spatio-temporal) across three domains (warehouse, transportation, and smart spaces). Their key motivation for exploring so many tasks is to move beyond the MCQ format that dominates existing video benchmarks, since a candidate set lets a model select an answer it could not produce on its own. The authors evaluate 18 open-weight and proprietary VLMs zero-shot and find that, relative to consumer-video benchmarks, performance drops most on event verification and temporal localization, while video question answering and spatial pointing are largely unaffected.

Critical strengths

The setting explored is important, many existing VLM benchmarks leverage web videos and do not measure domain-specific performance in infrastructure settings

Moving beyond MCQ to tasks that require generating boxes, timestamps, and trajectories is a meaningful improvement. It removes the implicit answer prior that candidate sets provide

Critical weaknesses

W1: A key idea of the paper is that it targets this Infrastructure AI setting, which is primarily identified as a setting consisting of fixed-viewpoint monitoring cameras. However, evaluation settings consisting of fixed-viewpoint cameras have been explored in activities-of-daily-living (ADL) datasets such as NTU RGB+D 60/120 [1,2], Toyota Smarthome [3], and ETRI-Activity3D [4], which are not discussed. If the intended distinction is not camera viewpoint the paper should define Infrastructure AI in those terms and explain why ADL settings, particularly given the overlap with the Smart Spaces domain, fall outside it.

W2: The 2D Referring Expression and 2D Spatial Positioning tasks are derived from RefDrone and “vehicle-mounted footage” respectively. The reviewer believes that videos from these sources capture an embodied perspective, which seems to conflict with the “fixed-infrastructure perspective” the authors aim to target in this work.

W3: The MCQ critique on existing benchmarks, used to motivate the format-diverse evaluation, is not specific to Infrastructure AI and applies equally to consumer-video benchmarks, many of which already include generative grounding tasks. The paper should either justify why format diversity matters more in this setting or present it as a separate contribution from the domain claim.

W4: A key finding is that “The Temporal Pillar Is Where Capability Is Still Being Acquired”, but does not establish why, or whether the deficit is specific to Infrastructure AI. Weak temporal grounding is a well-documented limitation of VLMs on consumer video as well (e.g., TOMATO [5]). The proposed explanations (training-data asymmetry, event sparsity, egocentric-to-fixed viewpoint mismatch) are offered as hypotheses without supporting experiments. Some analysis here would help determine whether the deficit truly reflects the Infrastructure AI setting or whether it is a general limitation of current VLMs.

W5: The paper does not convincingly establish that the gap exists, or that it is caused by the fixed-viewpoint domain. Table 2 is the only direct evidence, but the references here differ from VANTAGE-Bench in more than just viewpoint. MLVU is a general long-video benchmark rather than a binary event verification task. For temporal localization, the paper deliberately constructs videos with long quiescent periods and brief, sparse events (Section 2.3), which makes the task harder regardless of camera placement. The Charades-STA control holds prompt and metric fixed, but not video length or event density. The observed drops could therefore reflect harder task construction rather than domain shift to the Infrastructure AI setting.

[1] Shahroudy et al., “NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis”, CVPR 2016 [2] Liu et al., “NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding”, IEEE TPAMI 2020 [3] Das et al., “Toyota Smarthome: Real-World Activities of Daily Living”, ICCV 2019 [4] Jang et al., “ETRI-Activity3D: A Large-Scale RGB-D Dataset for Robots to Recognize Daily Activities of the Elderly”, IROS 2020 [5] Shangguan et al., “TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models”, ICLR 2025

Questions for authors

(Corresponding to the weaknesses discussed above)

Q1: What differentiates Infrastructure AI from fixed camera settings like activities of daily living, which seem to have overlap with the Smart Spaces domain the authors explore?

Q2: Since two of the tasks of VANTAGE-Bench come from the embodied perspective, don't these two tasks violate the assumptions made about the Infrastructure AI setting (i.e., that the camera must be fixed)?

Q3: Why is format diversity particularly important for the Infrastructure AI setting?

Q4: Could the hypotheses proposed by the authors to explain why the temporal gap exists in this Infrastructure AI setting be validated? The reviewer is concerned this gap might exist in any domain-specific benchmark and is not specific to Infrastructure AI

Q5: It seems there are a few variables other than camera viewpoint that are uncontrolled in the Table 2 analysis (e.g., video length/event density). Could the authors explain why these variables would not explain the performance gap?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.