acceptodds
Withdrawn from ICLR 2027

AirRelComp: A Diagnostic Benchmark for Video-Text Compositional Understanding in Airport Surveillance

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Understanding continuous multi-entity airport activities requires resolving temporal order, spatial configuration, and entity participation beyond merely recognizing entities and actions, supporting operational analysis and safety assessment. Broad video-text matching can conceal errors in these relations. We introduce AirRelComp, a diagnostic benchmark for fine-grained compositional understanding in airport surface videos. AirRelComp organizes Temporal, Spatial, and Entity Relations under a common event schema. We manually annotate positive video-text pairs from surveillance videos and construct four types of controlled negatives using predefined rules. Each negative changes one designated relation while preserving the remaining event structure. AirRelComp contains nearly 4K controlled comparisons and supports similarity comparison for video-text models and binary choice for large multi-modal models (LMMs). Evaluations reveal limited zero-shot transfer in video-text models and uneven discrimination across disruption types in LMMs. We therefore develop a learning framework that integrates supervision from positive–negative correspondence and relation identity with representative visual patch selection. With the same backbone, the framework raises Overall Accuracy from 63.45% under positive-only fine-tuning to 78.98%. Ablations show that the structured use of controlled negatives outperforms generic negative augmentation. LoRA fine-tuning on AirRelComp further substantially improves LMM binary-choice accuracy. These results show that AirRelComp diagnoses compositional weaknesses and provides effective supervision for improving relation discrimination.

Primary area:
datasets and benchmarks
Keywords:
airport video surveillance, compositional benchmarks, negative sample learning, video-text alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_2MZzOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces AirRelComp, a diagnostic benchmark for video-text compositional understanding in airport surveillance videos. The benchmark organizes positive video-text pairs along Temporal, Spatial, and Entity Relations, and constructs four types of negatives (Loc-Replace, Dir-Reverse, Stage-Reorder, Entity-Swap) by altering one relation while preserving the rest.

Critical strengths

The annotation protocol and quality control pipeline of the proposed dataset are described in detail, i.e., LLM checker + human review on 25% of flagged pairs.

Critical weaknesses

Constructing a dedicated dataset for the airport surveillance scenario can only serve as a benchmark for video-text compositional understanding. It should not be claimed that Video-Text Compositional Understanding in Airport Surveillance is itself a novel contribution of this paper.

The main contribution of this paper is the construction of a new dataset, which, in my view, is not well aligned with the research scope and standards of ICLR. Moreover, the paper does not clearly explain why this dataset is necessary or what fundamental limitations of existing datasets it is intended to address.

The proposed AirRelComp is built from only 118 videos (15.19 hours) recorded at a single airport from two fixed viewpoints, yielding 1,544 positive pairs and 3,855 controlled comparisons. The benchmark is too small and too narrow to serve as a general benchmark.

For LMM binary choice (Table 3), the closed-source models (Gemini 3.5 Flash, ChatGPT series) are evaluated zero-shot, but no fine-tuned LMM baseline from prior compositional work is included.

The paper's core pipeline, controlled negative construction from existing positive pairs, a pairwise preference loss that ranks positives above relation-specific negatives, and evaluation of both video-text models and LMMs, is substantially the same as VideoComp [1].

[1] Videocomp: Advancing fine-grained compositional and temporal alignment in video-text models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 29060–29070, 2025.

No comparison to VideoComp's training objective [1] on AirRelComp, despite it being the closest method.

Questions for authors

The paper is not written to a sufficiently rigorous academic standard, and the proposed method does not demonstrate meaningful technical novelty. I strongly recommend that the authors rely less on GPT-generated writing and instead provide clearer technical insights, stronger justification, and more original analysis.

Reviewer_K1o5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work present a new benchmark and a new method to solve the task of Video-Text Compositional Understanding in Airport Surveillance.

Critical strengths

This benchmark provide data to train and test the finer level video-text understanding. And the method significantly improves the Overall ACC from 63.45% to 78.98%.

Critical weaknesses

The amount of the videos, texts and the variety of the scenes of this benchmark is limited.

Questions for authors
  • How does this work inspire the filed of representation learning or other works that may be related?
  • Why do you choose airport videos rather than more general videos to do the task of discribing the timeline, composition and movements of more kinds of entities?
Reviewer_bUm7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents AirRelComp, a benchmark for fine-grained video-text compositional understanding in airport surveillance, covering temporal, spatial, and entity relations through four types of controlled negative captions. The authors further propose a framework combining RelCompLoss and SpaSelection to improve relation discrimination. Experiments on video-text models and multimodal LLMs show that existing models struggle with these relations, while the proposed framework improves over standard fine-tuning; ablation studies support the effectiveness of its components.

Critical strengths
  1. The controlled negative construction is a strong design choice. By altering one relation at a time while largely preserving the rest of the event description, the benchmark helps isolate which types of relational understanding models fail to capture, making the evaluation more informative than overall video-text matching accuracy alone.

  2. The experimental results are convincing. The authors evaluate a range of video-text models and multimodal LLMs, and the proposed framework shows clear improvements over the baselines. The ablation studies comparing different uses of controlled negatives are particularly useful, as they help clarify the source of the performance gains.

Critical weaknesses

1.The dataset uses a single airport with only two fixed viewpoints. While understandable, this limits generalization: models may learn scene-specific patterns rather than general relational reasoning, weakening the broader applicability of the conclusions.

  1. Despite efforts to reduce textual shortcuts, text-only performance remains strong, especially for temporal relations—e.g., Qwen3.5-VL-9B reaches 79.87% on Stage-Reorder without video. This raises concerns that performance may partly reflect learned airport-activity priors rather than visual understanding, an important limitation for a diagnostic benchmark.
Questions for authors
  1. Regarding language bias: The text-only results for Stage-Reorder are quite high. Could the authors clarify how much this bias might affect the interpretation of the benchmark results? In particular, would the relative model rankings remain similar after accounting for text-only performance?
  2. Regarding generalization: Since the dataset is collected from only two camera viewpoints at one airport, do the authors have any evidence that the learned relational representations generalize to different scenes or camera configurations?
  3. Regarding SpaSelection: The results show that SpaSelection improves performance, especially on Entity-Swap. Could the authors provide more direct evidence that the selected patches actually cover small or spatially dispersed entities better than standard Top-K selection?
Reviewer_E1VbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AirRelComp, a video-text benchmark for evaluating video understanding performance in airport surveillance. Specifically, the positive pairs are constructed through human annotation with LLM standardization. The negative pairs are constructed using four predefined relation-change rules, specifically targeting spatial, temporal, and entity relations. Based on the constructed benchmark, the authors introduce RelCompLoss and SpaSelection to exploit the supervision provided by AirRelComp. Experiments reveal that the existing MLLMs show some weakness in fine-grained relation discrimination, while the proposed learning framework achieves higher video–text retrieval performance.

Critical strengths
  1. The overall paper is well drafted and easy to follow.
  2. The motivation is clear that existing evaluations fall short in determining whether a model truly understands the temporal, spatial, and entity relationships within an event.
  3. The construction process of the benchmark is compact.
Critical weaknesses
  1. The motivation of this work is that existing evaluations fall short in determining whether a model truly understands the temporal, spatial, and entity relationships within an event. However, the proposed framework uses exact same four negative construction rules during training and testing. It is unclear whether the performance gains come from the model having a better understanding of surveillance videos or just overfitting to the four rules. Besides, are these four rules for negative pairs are generalized enough to the real complex scenarios?
  2. The proposed learning framework demonstrated limited novelty. The RelCompLoss is based on InfoNCE, negative preference learning, and relation-specific representation learning. Table 4 shows that the main gains come from the negative preference learning, while the gain from relation-specific representation learning is limited. The authors should justify more about the exact contribution of relation-specific representation learning.
  3. The constructed benchmark seems to have evident bias towards text. The experiments on Qwen3.5 with text only can still achieve relatively good performance. This makes the reviewer suspect the effectiveness of the evaluation protocol and the proposed benchmark. The authors can justify more on this.
Questions for authors

Please see the weakness for justification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.