acceptodds
Under review as a conference paper at ICLR 2027

Multi-modal Open-vocabulary Video Visual Relationship Detection

Abstract

Open-vocabulary Video Visual Relationship Detection (OV-VidVRD) aims to detect and localize subject-predicate-object relationship triplets in videos, even for relationships unseen during training. While pivotal for comprehensive video understanding, existing methods are largely confined to the visual modality. This uni-modal focus neglects rich contextual cues from other modalities, which are often vital for disambiguating complex relationships in real-world videos. To bridge this gap, we introduce the new task of Multi-modal Open-vocabulary Video Visual Relationship Detection (MM-OV-VidVRD) and present the first-of-its-kind benchmark VaM-VidVRD to catalyze research in this area. Our benchmark features numerous videos where each instance pairs the visual stream with a heterogeneous set of auxiliary modalities, such as audio, textual descriptions, and 3D data. Importantly, the benchmark encompasses many predicate categories that are visually indistinguishable yet semantically distinct—for instance, sing_to and speak_to share nearly identical visual appearances but are readily separable given audio cues—making auxiliary modalities essential rather than merely supplementary. This design mirrors the diverse and source-dependent nature of real-world multi-modal data. To tackle this new challenge, we propose a novel and flexible framework that effectively processes arbitrary combinations of modalities. The core of this framework is a Multi-modal Synergistic Prompting (MMSP) mechanism, which learns to dynamically align and fuse features from diverse sources into a unified and relationship-aware representation. Extensive experiments on our benchmark show that our method significantly outperforms vision-only approaches.

Primary area:
datasets and benchmarks
Keywords:
Open-Vocabulary Video Visual Relationship Detection, Multimodal Video Understanding, Multimodal Learning, Multimodal Prompting

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3sgfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Multi-modal Open-vocabulary Video Visual Relationship Detection (MM-OV-VidVRD), extending standard OV-VidVRD beyond the visual stream by incorporating audio, textual descriptions, and 3D data. The authors additionally release VaM-VidVRD, a heterogeneous benchmark aggregating multiple existing sources (VidVRD, VidOR, MammAlps, VggSound, KITTI, nuScenes, ActivityNet) with manually validated trajectories and predicate annotations. The core method is Multi-modal Synergistic Prompting (MMSP), which encodes available auxiliary modalities, fuses them with visual features via cross-attention over learnable fusion queries, and generates visual/textual prompts injected into a frozen CLIP backbone. The framework is evaluated under SGDet, SGCls, and PredCls, with base/novel splits. Experiments show clear gains over vision-only baselines (RePro, OV-MMP, EOV-MMP) and zero-shot MLLMs (Qwen3-VL, Gemini 2.5 Pro).

Critical strengths
  1. The limitation of uni-modal OV-VidVRD is clearly argued. The examples sing vs. speak_to (audio) and above / in_front_of (3D) are compelling cases where visual features alone are insufficient.

  2. Comparison with EOV-MMP plus MLLM zero-shot baselines gives a useful picture of where current general MLLMs fail on structured spatio-temporal relation prediction.

  3. VaM-VidVRD is, to my knowledge, the first VidVRD benchmark that systematically supports V+A+T+3D with source-dependent modality availability.

Critical weaknesses
  1. The method's novelty is limited. Cross-attention multimodal fusion and V/L prompt tuning are both well-established. The incremental step is “apply this to arbitrary-modality VidVRD,” but the architectural contribution is not very deep. The paper would benefit from a sharper claim: is the contribution mainly the benchmark, or a nontrivial fusion mechanism? Right now it sits between the two.

  2. The two-stage pipeline limits the “open-vocabulary” claim. Object trajectories come from a frozen pre-trained localizer; only object/predicate classifiers are trained. This inherits the standard two-stage bottleneck: if the localizer misses a novel object, the relation can never be recovered. Recent OV-VidVRD works have moved toward unified or end-to-end detection+relation frameworks, which are not compared here.

  3. Missing comparison in the ablation study. Table 4 only compared V plus one auxiliary modality. Using two of the three is missing.

  4. Limited analysis of failure cases. Qualitative cases show one microphone active → only one sing_to is predicted. This is interesting, but there is no systematic error breakdown: do audio predicates fail under background noise? Do 3D predicates help only for spatial predicates? A predicate-type stratified result would be valuable.

  5. Is it fair to compare the proposed method with SOTA using the text descriptions? I see that the auxiliary text modality contains rich semantic information, even containing the GT relationship, which requires the model to predict. If the text modality is added to existing vision-only relationship detection methods by simply fusing strategies, how will the SOTA comparison results change in Table 3?

  6. Missing comparison with recent OV-VidVRD methods. The related work mentions OV-MMP/EOV-MMP but not newer unified frameworks such as METOR or semantic-teacher-style approaches. Since the method still runs on VidVRD-style pipelines, excluding these reduces empirical significance.

  7. Clarity / presentation issues: Table 3 mixes blank CLIP SGDet cells. CLIP is not a full SGDet baseline, which is fine but should be noted. “Full Modalities” is not truly full per video. It is full per source. This should be stated explicitly in the main results section, not only in dataset construction.

Questions for authors

My main concerns are the method's novelty and the experimental settings and results, as I noted in the "Weaknesses" section.

1 comment

Response to W1: We agree that cross-attention and visual/language prompt tuning are established primitives. Our primary contribution is MM-OV-VidVRD and VaM-VidVRD: the first benchmark for open-vocabulary VidVRD with source-dependent, incomplete audio, text, and 3D evidence. MMSP is positioned as a unified and effective baseline for this setting, rather than as a claim of fundamentally new fusion primitives. It converts available auxiliary evidence into visual and category-text prompts for both object and predicate recognition.

Response to W2: We agree that the frozen trajectory localizer is a bottleneck: a missed novel object cannot be recovered at the relation stage. Our method does not claim to solve open-vocabulary trajectory localization. To isolate the impact of auxiliary evidence, all compared methods use the same trajectory localizer; the gains therefore reflect improved multimodal object and predicate recognition. We will clarify this scope and identify end-to-end multimodal trajectory-and-relation prediction as future work.

Response to W3: We add the only naturally occurring two-auxiliary configuration, V+A+T. V+A+3D and V+T+3D do not occur in VaM-VidVRD because modality availability is source-dependent.

Modality configurationNovel mAPAll mAP
V-Only5.7925.40
V+A+T9.3627.19
Full available modalities10.3328.16

Here, V+A+T enables audio and text when available while disabling 3D; “Full available modalities” enables every modality provided by each source. The results show complementary gains beyond single-modality settings.

Response to W4: We have added Appendix G (Failure Case Analysis) and Figure 10, which present three diagnostic failure modes: (i) incorrect assignment of a global barking cue among multiple trajectories, (ii) background barking that induces a false-positive bark_at prediction, and (iii) short-lived, unannotated surrounding vehicles in LiDAR that interfere with fine-grained left/right reasoning.

Predicate typeV-Only mAPV+3D mAP
Spatial29.6834.17
Non-spatial20.1922.06

3D yields a substantially larger gain for spatial predicates (+4.49 mAP) than non-spatial predicates (+1.87 mAP), consistent with its intended geometric role. Since VaM-VidVRD has no clip-level noise labels, we present background-audio failures as qualitative observations rather than claiming a high-/low-noise stratified evaluation.

Response to W5: We agree that vision-only comparisons alone do not separate the benefit of text from that of fusion. We therefore add EOV-MMP + Direct Cross-Attention, which receives the same visual backbone, trajectory localizer, textual descriptions, and evaluation protocol as MMSP.

MethodInput modalitiesNovel mAPAll mAP
EOV-MMPV-Only4.8524.98
EOV-MMP + Direct Cross-AttentionV+T6.7725.53
MMSPV+T7.9826.35
MMSPFull available modalities10.3328.16

Adding text improves EOV-MMP by 1.92/0.55 Novel/All mAP. Under the identical V+T setting, MMSP further improves by 1.21/0.82 mAP, showing that its gains cannot be attributed solely to access to textual descriptions.

Response to W6: We have expanded related work to discuss METOR and SAGE. A direct comparison on VaM-VidVRD is currently not reliable: METOR’s official repository does not provide a runnable implementation, and we could not locate an official released implementation for SAGE. A fair comparison requires retraining it under the VaM-VidVRD protocol. We avoid reporting unverified reimplementations or non-comparable published results, and will add these comparisons when reproducible implementations become available.

Response to W7: We have stated in the Table 3 caption that standalone CLIP lacks a trajectory localizer and is therefore not a complete SGDet baseline. We have explicitly defined “Full Modalities” in the main results section as all modalities available for each video’s source, not audio, text, and 3D for every individual video.

Reviewer_Kxa9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces multi-modal open-vocabulary video visual relationship detection and VaM-VidVRD, a benchmark of 3,600 videos from seven sources with manually validated trajectories and relationships. Its MMSP framework fuses available auxiliary modalities through masked cross-attention and generates prompts for both CLIP branches. Experiments report improvements over vision-only methods, with ablations on modality contributions and fusion designs.

Critical strengths
  1. The benchmark addresses a useful gap in video relationship detection by incorporating source-dependent audio, text, and 3D cues. The unified annotations across diverse sources provide a valuable resource for studying relationships that are difficult to distinguish visually.
  2. The auxiliary-modality gains are supported by useful controls. Novel SGDet mAP improves from 5.79 to 10.33 over the same framework using vision alone (Table 4). Semantic masking retains substantial gains, while same-source shuffling reduces performance to 5.77 (Appendix D), supporting the importance of aligned auxiliary evidence.
Critical weaknesses

The distinctive value of MMSP over existing dynamic-prompt approaches remains unclear. SyP (Zhang et al., ICCV 2025) already conditions prompts on available inputs to handle missing modalities. Table 6 establishes gains over concatenation and generic cross-attention, but does not compare against a dynamic-prompt approach adapted to the same backbone and auxiliary inputs.

Questions for authors

Under the same backbone, auxiliary inputs, and training protocol, how does MMSP compare with an adapted dynamic-prompt baseline such as SyP?

1 comment

We thank the reviewer for suggesting this comparison. We add SyP as a dynamic-prompt baseline under the same backbone, frozen encoders, auxiliary inputs, and training/evaluation protocol.

SyP was originally proposed for two-modality visual recognition with missing image/text inputs. We faithfully reimplemented and adapted it to our setting: encoded visual and available auxiliary features are concatenated, missing auxiliary features are zero-filled, and SyP's dynamic adapter generates scaling factors to modulate the visual and textual base prompts. The adapted SyP prompt module replaces MMSP, while all other components remain unchanged.

MethodNovel mAPAll mAP
SyP-adapted9.6727.34
MMSP10.3328.16

MMSP improves over SyP-adapted by 0.66 Novel mAP and 0.82 All mAP. SyP adapts base prompts through input-dependent scaling factors, whereas MMSP uses learnable fusion queries and cross-attention to aggregate the available auxiliary evidence into sample-specific prompts. This direct aggregation is better suited to our source-dependent setting, which includes optional audio, text, and 3D inputs rather than a fixed image-text pair.

Reviewer_tQsMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces VaM-VidVRD, a video relationship benchmark with source-dependent audio, text, and 3D inputs. Its MMSP module fuses available modalities into visual and textual prompts for frozen encoders. Evaluation covers both novel and familiar object and predicate categories.

Critical strengths
  1. The benchmark addresses a useful problem: some relationships are difficult to distinguish visually, and available modalities vary across sources. The 3,600 videos, refined trajectories, and explicit category splits provide a substantial basis for studying this setting.
  2. The auxiliary-modality gains are supported by meaningful controls. Novel SGDet mAP improves from 5.79 to 10.33, much of the gain survives text masking, and same-source feature shuffling removes it. These results support the value of aligned evidence.
Critical weaknesses
  1. Auxiliary evidence is compressed before it is explicitly associated with the target relation. Appendix A.1 pools the 3D spatial map into one vector, and MMSP pools its fusion queries into a single prompt. The subject/object visual streams provide conditioning, but the design does not explicitly bind points or sound sources to the relevant trajectories. This leaves an important ambiguity when several entities could explain the same auxiliary cue.
  2. Availability masking does not itself provide the noise robustness claimed in Appendix E. An available but misleading modality remains eligible for attention, and the fused signal conditions both visual features and category text. Learned attention may suppress such evidence, but the method supplies no explicit reliability criterion for conflicting inputs. Training on naturally noisy, source-dependent combinations does not establish robustness to different corruption patterns or unseen combinations.
  3. Benchmark validation still needs clarification. The paper does not quantify annotation agreement for ambiguous predicates, explain recording-level train/test separation, or fully specify how novel-category instances in training footage are handled. These details are especially important because benchmark construction is a central contribution.
Questions for authors
  1. How are auxiliary cues associated with a particular subject–object pair after pooling, especially when multiple entities share the same time interval?
  2. What guides MMSP to reject a present but conflicting modality, and how does this differ from simply masking an absent one?
  3. What annotation agreement and recording-level separation were obtained, and how were novel-category instances excluded from training supervision?
3 comments

Response to Q1. We thank the reviewer for this important question. We agree that MMSP does not explicitly bind points or sound sources to trajectories. For each candidate pair, MMSP is applied separately to subject, object, and global visual streams; their visual features enter cross-attention with auxiliary evidence to generate pair-conditioned prompts. This implicit association can remain ambiguous when multiple entities coexist.

To quantify its practical effect, we group test clips by the number of temporally co-active trajectories:

Scene groupConcurrent entitiesV-Only Novel mAPMMSP Novel mAPV-Only All mAPMMSP All mAP
Two-entity scenes26.9811.5326.6429.59
Multi-entity scenes>25.179.4324.8127.60

MMSP improves Novel/All mAP by 4.55/2.95 points in two-entity scenes and 4.26/2.79 in multi-entity scenes. Thus, auxiliary cues remain beneficial, although grounding ambiguity persists. Appendix G/Fig. 10 presents cue-to-pair, background-audio, and LiDAR-clutter failures.

Response to Q2. We clarify that availability masking is used only to remove absent modalities from cross-attention. The noise tolerance of MMSP comes from the overall fusion mechanism: content-dependent cross-attention weights auxiliary evidence according to its compatibility with visual context, while training exposes the model to naturally imperfect text alignment and real-world background sounds.

To directly evaluate present but conflicting evidence, we perform same-source feature replacement at inference time. For randomly selected test instances, available auxiliary features are replaced with those from another video of the same source, while their availability masks remain unchanged.

Test inputCorruption ratioNovel mAPAll mAP
Aligned auxiliary features0%10.3328.16
Same-source replaced features25%9.9827.88
Same-source replaced features50%8.6426.92
V-Only—5.7925.40

Performance decreases gradually with replacement, but remains above V-Only even at 50% corruption, by 2.85 Novel mAP and 1.52 All mAP. This supports tolerance to partially conflicting, yet available, auxiliary evidence.

We further test unseen modality combinations. During training, each instance is randomly provided with either V+A or V+T, such that V+A+T is not observed. The model nevertheless generalizes to V+A+T at test time:

Training modalitiesTest modalitiesNovel mAPAll mAP
Randomly sampled V+A or V+TV+A12.4827.76
Randomly sampled V+A or V+TV+T12.1527.85
Randomly sampled V+A or V+TV+A+T12.8328.37
V+A+TV+A+T13.7229.06

All results in this table are evaluated on the V+A+T-capable subset.

Response to Q3. We add an independent re-annotation audit on 100 stratified randomly sampled 30-frame intervals. Audit annotators do not access the released annotations or the original adjudication decisions; their annotations are compared with the final benchmark labels.

Annotation componentEvaluation protocolAgreement metricResult
Object trajectoriesFrame-level comparison of matched trajectoriesMean bounding-box IoU96.14%
Relation tripletsPredicate comparison given matched subject-object trajectoriesMicro-F197.48%

The few relation disagreements mainly arise from occasional omissions of next_to in crowded scenes, rather than contradictory predicate assignments. During the original annotation process, annotations are cross-validated, and disagreements are resolved by one designated adjudicator who reviews the video, trajectories, synchronized auxiliary inputs, and annotation guidelines.

The dataset is partitioned at the source-video level: all clips from the same source video are assigned to the same split, preventing recording overlap between training and testing. The base/novel split is defined from post-annotation category frequencies. Once the split is fixed, novel-category labels appearing in training videos are excluded from training supervision and are never exposed as category labels during training.

Thank you for the detailed rebuttal. The independent annotation audit and the clarification of the train/test split address my main concerns about the benchmark. The gains in multi-entity scenes are encouraging, and the new failure cases help clarify where cue-to-trajectory association remains ambiguous.

I have one remaining question about robustness. On the same set of examples selected for feature replacement, could you compare mAP with aligned, replaced, and masked auxiliary inputs? This would show whether the model handles misleading cues better than simply dropping them. My overall rating remains 3

We thank the reviewer for this constructive suggestion. We rerun the evaluation, keeping the candidate pairs, visual features, proposals, class set, and thresholds fixed. We compare three conditions: (i) aligned auxiliary features, (ii) same-source replaced features with the availability mask unchanged, and (iii) the same auxiliary inputs marked unavailable and removed from cross-attention.

Test inputCorruption ratioNovel mAPAll mAP
Aligned auxiliary features0%10.3328.16
Same-source replaced features25%9.9827.88
Masked same auxiliary inputs25%9.2227.03
Same-source replaced features50%8.6426.92
Masked same auxiliary inputs50%8.3526.49

As shown in the table, replaced inputs consistently outperform masked inputs at both corruption levels. At 25% corruption, replaced inputs achieve 9.98 Novel mAP and 27.88 All mAP, while masked inputs achieve 9.22 and 27.03; at 50%, replaced inputs achieve 8.64 Novel mAP and 26.92 All mAP, while masked inputs achieve 8.35 and 26.49. These results show that even when auxiliary features are mismatched, the model still extracts complementary information and performs better than simply discarding the auxiliary inputs. We attribute this advantage to the cross-attention over learnable fusion queries, which conditions the auxiliary representations on the visual context and can down-weight evidence inconsistent with that context, whereas masking removes all auxiliary cues, including those that remain useful.

Comments on the paper

1 comment

Dear Reviewers,

A friendly reminder to check the authors’ rebuttals and engage in the discussion, as the deadline is approaching. Authors still have time to address any further questions or concerns you may have, so your timely feedback would be greatly appreciated.

Thank you for your time and effort!

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.