- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Multi-modal Open-vocabulary Video Visual Relationship Detection (MM-OV-VidVRD), extending standard OV-VidVRD beyond the visual stream by incorporating audio, textual descriptions, and 3D data. The authors additionally release VaM-VidVRD, a heterogeneous benchmark aggregating multiple existing sources (VidVRD, VidOR, MammAlps, VggSound, KITTI, nuScenes, ActivityNet) with manually validated trajectories and predicate annotations. The core method is Multi-modal Synergistic Prompting (MMSP), which encodes available auxiliary modalities, fuses them with visual features via cross-attention over learnable fusion queries, and generates visual/textual prompts injected into a frozen CLIP backbone. The framework is evaluated under SGDet, SGCls, and PredCls, with base/novel splits. Experiments show clear gains over vision-only baselines (RePro, OV-MMP, EOV-MMP) and zero-shot MLLMs (Qwen3-VL, Gemini 2.5 Pro).
Critical strengths
-
The limitation of uni-modal OV-VidVRD is clearly argued. The examples sing vs. speak_to (audio) and above / in_front_of (3D) are compelling cases where visual features alone are insufficient.
-
Comparison with EOV-MMP plus MLLM zero-shot baselines gives a useful picture of where current general MLLMs fail on structured spatio-temporal relation prediction.
-
VaM-VidVRD is, to my knowledge, the first VidVRD benchmark that systematically supports V+A+T+3D with source-dependent modality availability.
Critical weaknesses
-
The method's novelty is limited. Cross-attention multimodal fusion and V/L prompt tuning are both well-established. The incremental step is “apply this to arbitrary-modality VidVRD,” but the architectural contribution is not very deep. The paper would benefit from a sharper claim: is the contribution mainly the benchmark, or a nontrivial fusion mechanism? Right now it sits between the two.
-
The two-stage pipeline limits the “open-vocabulary” claim. Object trajectories come from a frozen pre-trained localizer; only object/predicate classifiers are trained. This inherits the standard two-stage bottleneck: if the localizer misses a novel object, the relation can never be recovered. Recent OV-VidVRD works have moved toward unified or end-to-end detection+relation frameworks, which are not compared here.
-
Missing comparison in the ablation study. Table 4 only compared V plus one auxiliary modality. Using two of the three is missing.
-
Limited analysis of failure cases. Qualitative cases show one microphone active → only one sing_to is predicted. This is interesting, but there is no systematic error breakdown: do audio predicates fail under background noise? Do 3D predicates help only for spatial predicates? A predicate-type stratified result would be valuable.
-
Is it fair to compare the proposed method with SOTA using the text descriptions? I see that the auxiliary text modality contains rich semantic information, even containing the GT relationship, which requires the model to predict. If the text modality is added to existing vision-only relationship detection methods by simply fusing strategies, how will the SOTA comparison results change in Table 3?
-
Missing comparison with recent OV-VidVRD methods. The related work mentions OV-MMP/EOV-MMP but not newer unified frameworks such as METOR or semantic-teacher-style approaches. Since the method still runs on VidVRD-style pipelines, excluding these reduces empirical significance.
-
Clarity / presentation issues: Table 3 mixes blank CLIP SGDet cells. CLIP is not a full SGDet baseline, which is fine but should be noted. “Full Modalities” is not truly full per video. It is full per source. This should be stated explicitly in the main results section, not only in dataset construction.
Questions for authors
My main concerns are the method's novelty and the experimental settings and results, as I noted in the "Weaknesses" section.
Response to W1: We agree that cross-attention and visual/language prompt tuning are established primitives. Our primary contribution is MM-OV-VidVRD and VaM-VidVRD: the first benchmark for open-vocabulary VidVRD with source-dependent, incomplete audio, text, and 3D evidence. MMSP is positioned as a unified and effective baseline for this setting, rather than as a claim of fundamentally new fusion primitives. It converts available auxiliary evidence into visual and category-text prompts for both object and predicate recognition.
Response to W2: We agree that the frozen trajectory localizer is a bottleneck: a missed novel object cannot be recovered at the relation stage. Our method does not claim to solve open-vocabulary trajectory localization. To isolate the impact of auxiliary evidence, all compared methods use the same trajectory localizer; the gains therefore reflect improved multimodal object and predicate recognition. We will clarify this scope and identify end-to-end multimodal trajectory-and-relation prediction as future work.
Response to W3: We add the only naturally occurring two-auxiliary configuration, V+A+T. V+A+3D and V+T+3D do not occur in VaM-VidVRD because modality availability is source-dependent.
Here, V+A+T enables audio and text when available while disabling 3D; “Full available modalities” enables every modality provided by each source. The results show complementary gains beyond single-modality settings.
Response to W4: We have added Appendix G (Failure Case Analysis) and Figure 10, which present three diagnostic failure modes: (i) incorrect assignment of a global barking cue among multiple trajectories, (ii) background barking that induces a false-positive
bark_atprediction, and (iii) short-lived, unannotated surrounding vehicles in LiDAR that interfere with fine-grained left/right reasoning.3D yields a substantially larger gain for spatial predicates (+4.49 mAP) than non-spatial predicates (+1.87 mAP), consistent with its intended geometric role. Since VaM-VidVRD has no clip-level noise labels, we present background-audio failures as qualitative observations rather than claiming a high-/low-noise stratified evaluation.
Response to W5: We agree that vision-only comparisons alone do not separate the benefit of text from that of fusion. We therefore add EOV-MMP + Direct Cross-Attention, which receives the same visual backbone, trajectory localizer, textual descriptions, and evaluation protocol as MMSP.
Adding text improves EOV-MMP by 1.92/0.55 Novel/All mAP. Under the identical V+T setting, MMSP further improves by 1.21/0.82 mAP, showing that its gains cannot be attributed solely to access to textual descriptions.
Response to W6: We have expanded related work to discuss METOR and SAGE. A direct comparison on VaM-VidVRD is currently not reliable: METOR’s official repository does not provide a runnable implementation, and we could not locate an official released implementation for SAGE. A fair comparison requires retraining it under the VaM-VidVRD protocol. We avoid reporting unverified reimplementations or non-comparable published results, and will add these comparisons when reproducible implementations become available.
Response to W7: We have stated in the Table 3 caption that standalone CLIP lacks a trajectory localizer and is therefore not a complete SGDet baseline. We have explicitly defined “Full Modalities” in the main results section as all modalities available for each video’s source, not audio, text, and 3D for every individual video.