acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Visual Evidence Extraction for Open-Vocabulary Video Visual Relationship Detection

Abstract

Open-vocabulary video visual relationship detection (Open-VidVRD) aims to recognize objects and their relationships in videos beyond a predefined vocabulary of object and relationship categories. Relationship classification typically relies on representations of trajectory pairs constructed from sampled frames, yet existing pipelines commonly divide videos or candidate relationship intervals into fixed-length temporal segments and sample the same number of frames from each segment. This uniform sampling strategy is suboptimal because relationships exhibit distinct temporal dynamics and require different numbers of frames to capture sufficient visual evidence. Stable relationships may require only a few frames, whereas dynamic ones require more. To address this issue, we propose an adaptive visual evidence extraction architecture for Open-VidVRD. It first partitions videos into temporal segments according to changes in optical flow and bounding box layouts, and then predicts the required number of sampled frames for each segment using aggregated appearance and motion cues. Subsequently, visual evidence is extracted from the selected frames and fused with contextual information from the entire segment for relationship classification. Due to the absence of annotations regarding the optimal number of sampled frames for each segment of a trajectory pair, we construct pseudo supervision by combining relationship-level temporal priors obtained from a multimodal large language model (MLLM) with annotated relationship durations and the temporal coverage of relationship instances within each segment. The MLLM is used only to derive offline supervision and is not invoked during inference. Extensive experiments on the VidVRD and VidOR datasets demonstrate that our method achieves state-of-the-art performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Open-Vocabulary Video Visual Relationship Detection, Adaptive Visual Evidence Extraction, Temporal Segmentation, Adaptive Frame Sampling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_rTmoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes adaptive evidence extraction for open-vocabulary video visual relationship detection. It combines motion-aware segmentation, prediction of observation counts for trajectory pairs, and quality/diversity-based frame selection. Offline MLLM-derived priors provide weak supervision for count prediction. Results show improved relationship detection with fewer selected frames.

Critical strengths
  1. The method addresses a concrete limitation of fixed observation budgets and offers a coherent mechanism for adapting visual evidence to temporal dynamics.
  2. The evaluation provides useful support for the proposed design and its accuracy–computation trade-off.
Critical weaknesses

The baseline Novel mAP is 14.45 in Table 3 but 16.45 in Table 7, while All mAP is 25.22 in both. The main text does not explain a setting difference.

Questions for authors

Do Tables 3 and 7 use the same baseline checkpoint and Novel SGDet evaluation setting? If so, please identify and correct the erroneous value. Otherwise, please explain the setting difference and ensure the component comparison uses a consistent baseline.

1 comment

Response to Q1: Thank you for identifying this inconsistency. Tables 3 and 7 use the same EOV-MMP-based baseline checkpoint and the same Novel SGDet evaluation setting. The Novel mAP value of 16.45 in Table 7 is a typographical error; the correct value is 14.45, consistent with Table 3. The All mAP value (25.22), FLOPs, runtime, and all results for our method remain unchanged. We have corrected the relevant entries in the revised manuscript.

Reviewer_vNM9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes adaptive visual evidence extraction for open-vocabulary video visual relationship detection. Existing methods typically use fixed temporal segments and uniform frame sampling, overlooking differences in relationship dynamics. The proposed framework introduces motion-aware segmentation, predicts observation counts for each trajectory-pair segment, and selects representative frames based on quality and diversity. The count predictor is supervised using pseudo labels combining offline MLLM-derived temporal priors with annotated relationship extents and visual dynamics. Experiments on VidVRD and VidOR demonstrate improved detection performance with fewer selected frames.

Critical strengths
  • The paper addresses a meaningful limitation of uniform sampling by adapting both temporal segmentation and observation budgets to visual dynamics.

  • The pseudo-supervision design provides a practical learning signal without requiring frame-count annotations. The MLLM is used only offline, avoiding additional inference calls.

  • Experiments cover both novel and all-category settings on two benchmarks. Ablations examine supervision, segmentation, and frame selection, while efficiency measurements demonstrate savings in downstream processing.

Critical weaknesses
  • The reliability of MLLM-derived observation counts remains insufficiently established. A category-level estimate may depend on clip selection, prompting, and the recognition model itself.

  • The efficiency evaluation excludes trajectory detection and feature extraction, although the framework requires video-wide appearance and optical-flow information before sampling. Consequently, fewer selected frames do not directly establish equivalent savings for the complete pipeline

  • The methodological novelty appears incremental relative to prior adaptive frame-selection methods. The main distinction lies in pair–segment-level budget prediction and MLLM-derived pseudo supervision, while the remaining components largely combine established techniques

Questions for authors

Please see the weakness part

1 comment

Response to W1: We treat MLLM counts as coarse weak priors, not model-independent optimal budgets. The first three tables vary clip sampling, the prior-generating MLLM, and prompt wording, one factor at a time. Across these settings, Novel/All mAP ranges from 17.82–18.33/28.46–28.93, with 2.96–3.16 selected frames on average, indicating limited sensitivity.

The fourth table uses the same offline category priors with two relationship classifiers. Task-loss gradients improve both classifiers' accuracy while reducing selected frames. These results support task-refinable weak supervision: dynamics-aware, capacity-constrained allocation constructs pair–segment targets, and relationship-classification feedback adapts budget prediction to the recognizer.

Clip-sampling seedNovel mAPAll mAPAvg. Frames
Seed 1 (ours)18.0828.933.12
Seed 217.8428.753.07
Seed 318.1128.693.16
Prior-generating MLLMNovel mAPAll mAPAvg. Frames
GPT-5 (ours)18.0828.933.12
DeepSeek-V4.1-Flash17.8228.503.03
Gemini-3.8-Flash17.9128.463.07
Prompt configurationNovel mAPAll mAPAvg. Frames
Original prompt (ours)18.0828.933.12
Equivalent rephrasing 118.3328.583.15
Equivalent rephrasing 217.9928.772.96
Relationship classifierTask-loss gradient to count predictorNovel mAPAll mAPAvg. Frames
OV-MMP✗14.6525.183.56
OV-MMP✓15.7625.833.30
Ours✗17.9527.853.56
Ours✓18.0828.933.12

Response to W2: We exclude only trajectory detection and frame-level CLIP feature extraction, which are identical shared costs for our method and the baseline. All method-specific overhead—including optical-flow extraction, motion-feature computation, segmentation, count prediction, and adaptive frame selection—is included in the reported FLOPs and runtime.

After accounting for this overhead, our method reduces computation by 19.1% and runtime by 16.8% while improving detection performance. These are measured savings, not estimates inferred from fewer frames. We have clarified in Section 4.3 that these percentages apply to the pipeline excluding shared preprocessing, not full end-to-end video processing.

Response to W3: We acknowledge that motion cues and quality/diversity selection are established techniques. Our contribution is task-specific evidence-budget learning for Open-VidVRD: trajectory pairs within the same segment can exhibit different interaction dynamics and require different observation budgets. We predict pair–segment budgets, construct weak targets through dynamics-aware, capacity-constrained allocation of offline category priors, and refine predictions through the relationship-classification loss. Component ablations and comparisons support this formulation. In Table 5, our method achieves 18.08/28.93 Novel/All mAP with 3.12 frames on average, versus 14.49/27.04 for A.I.R., 15.38/27.26 for ReaSon, and 14.12/26.46 for AKS, using 4, 8, and 4 frames, respectively. These results support the utility of task-specific budget learning beyond directly adopting existing selection strategies.

Reviewer_pHwVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors present a framework to perform open-vocabulary video visual relationship detection (Open-VidVRD). Specifically, they propose an adaptive visual evidence extraction architecture where temporal segments are variable in length (based on optical flow) and the number of frames extracted to be used in relationship classification is also adaptive. Top results are achieved on the VidVRD and VidOR datasets.

Critical strengths

S1: The proposed method achieves state-of-the-art results on the two datasets that have been tested.

S2: Dividing a video into adaptive temporal segments and extracting a variable number of frames based on the content of the video seems intuitively a sensible thing to do.

Critical weaknesses

W1: The proposed method utilizes a large number of (tuning) parameter values which are stated in the Implementation Details paragraph on page 7, but there is no explanation of justification given for their specific values. Have these values been heuristically obtained? Have other values been tested?

W2: "For all compared methods, the baseline results reported in Tables 1 and 2 are directly taken from the evaluation reported by EOV-MMP, rather than from the original papers or independent reruns in this work." The results are not from the original papers and not from re-running the experiments. So they were obtained how?

Note: In Section 4.3 the authors state: "The costs of trajectory detection and feature extraction are excluded because they are shared by all methods." If I understand correctly then all the compared methods use the same detectors (for objects, trajectories, optical flows, etc.). This is very important because it means that the input information is the same for all methods. This fact should be mentioned earlier in Section 4.

Questions for authors

Q1: How were the comparison results obtained (see W2)?

Q2: The methods and detectors used in the video preprocessing use a number of parameter settings. How were those settings obtained or decided?

1 comment

Response to Q1: Thank you for the question. The comparison results in Tables 1 and 2 are neither copied from the original papers nor independently re-run by us. Instead, we adopt the re-evaluated results reported by EOV-MMP under its Open-VidVRD protocol.

Several previous methods use a closed-set pretrained trajectory detector whose original training data include object categories designated as novel in Open-VidVRD. Using their originally reported results would therefore introduce extra novel-category supervision and would not constitute a fair comparison. EOV-MMP addressed this issue by retraining the trajectory detector after excluding training data associated with novel object categories, then re-evaluating the relevant methods with this detector. We use these results to ensure a consistent novel-category exclusion protocol for all compared methods.

Response to Q2: Thank you for raising this important point. The trajectory detector and shared preprocessing settings follow the EOV-MMP protocol, ensuring a fair comparison. For the method-specific hyperparameters examined below, we used validation-based selection. The remaining settings were fixed heuristically or inherited from existing implementations, as detailed below.

Specifically, we split the original VidVRD training set into 720 training videos and 80 validation videos. We selected hyperparameters using this split, then retrained the final model on all 800 training videos with the selected configuration. All results below are measured on the held-out 80-video validation set.

Allocation weight α_allocNovel mAPAll mAPAvg. Frames
0.320.1130.644.25
0.6 (Ours)21.1732.234.16
0.920.5531.074.02
Redundancy penalty λ_redNovel mAPAll mAPAvg. Frames
0.2520.8531.664.16
0.50 (Ours)21.1732.234.16
0.7520.6831.444.09
Count-supervision weight λ_cNovel mAPAll mAPAvg. Frames
0.2520.1631.244.42
0.50 (Ours)21.1732.234.16
0.7520.8531.354.06
Object-score fusion weight λ_objNovel mAP_oAll object mAP_oNovel mAPAll mAP
0.239.5954.6220.1130.05
0.441.5258.0920.6531.31
0.6 (Ours)43.5859.2421.1732.23
0.842.3858.4520.6831.52

For the quality-score weights, we vary the complementary spatial-separation and sharpness weights:

ηc\eta_cηs\eta_sNovel mAPAll mAPAvg. Frames
0.01.019.9430.364.37
0.20.820.5131.204.19
0.40.621.1732.234.16
0.60.420.6631.424.11
0.80.220.3130.954.03
1.00.020.2330.884.03

The selected values achieve the highest validation mAP among the tested settings, with comparable average frame counts. We select Kmax⁡=24K_{\max}=24 exclusively on the held-out validation split. Table~9 reports test-set sensitivity results for fixed Kmax⁡K_{\max} values; these results are not used for hyperparameter selection. The remaining parameters were fixed without fine-tuning. Specifically, we use αg=αm=αl=1/3\alpha_g=\alpha_m=\alpha_l=1/3 because the three change measures are independently min–max normalized; equal weighting avoids imposing an a priori preference for any cue. The segmentation settings wloc=3w_{\mathrm{loc}}=3, wnms=10w_{\mathrm{nms}}=10, and Lmin⁡=10L_{\min}=10 are heuristic choices based on the 30-FPS setting and the minimum segment-length constraint. For the straight-through mask, $ \gamma=1.0 $ fixes the relaxation temperature, while δ=0.5\delta=0.5 aligns the soft-mask threshold with the rounded observation count. Architecture and training settings follow CLIP/EOV-MMP where applicable. The same hyperparameters selected on VidVRD are used for VidOR.

Comments on the paper

1 comment

Dear Reviewers,

A friendly reminder to check the authors’ rebuttals and engage in the discussion, as the deadline is approaching. Authors still have time to address any further questions or concerns you may have, so your timely feedback would be greatly appreciated.

Thank you for your time and effort!

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.