acceptodds
Under review as a conference paper at ICLR 2027

PACT: Pair-Aware Contextual Tokens for Video Visual Relation Detection

Abstract

Video Visual Relation Detection (VidVRD) focuses on detecting ⟨subject, predicate, object⟩ relations together with their temporal extents in videos. Most existing methods construct relation representations from features extracted from detected object regions. Consequently, contextual information from the surrounding scene is largely overlooked, and the representation remains dependent on the quality of the object detector. However, many predicates are determined by how two objects interact within the broader scene. We propose Pair-Aware Contextual Tokens (PACT), a relation representation that preserves global scene context while being aware of the target pair. PACT represents each frame using all patch tokens from a frozen image encoder. Since the patch tokens are shared across different pairs, PACT injects information about the target pair into them. Specifically, each token encodes its spatial overlap with the subject and object regions, together with the semantic embeddings of the two object categories. Experiments on ImageNet-VidVRD and VidOR show that PACT achieves state-of-the-art mAP performance on both benchmarks. These results show that preserving the global scene context while making the representation aware of the target pair is an effective way to learn relation representations.

Primary area:
learning on time series and dynamical systems
Keywords:
Video Understanding, Video Visual Relation Detection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3TeaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PACT for video visual relation detection. It conditions full-frame visual tokens on a subject–object pair using spatial overlaps and category text embeddings, followed by spatial–temporal attention and query-based relation prediction. Experiments support the usefulness of the proposed method.

Critical strengths
  1. The pair-conditioning mechanism provides a simple way to identify the target pair within shared full-frame features while retaining scene information.
  2. The evaluation covers two benchmarks and includes useful internal comparisons. The ablation results support the effectiveness of the design.
Critical weaknesses

None.

Questions for authors

Section 4.4 describes the mask supervision and inference, but Appendix A.4 does not explain this head. Please specify the temporal-mask prediction head and how its outputs are mapped from the resampled sequence to the original frame indices.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 3Tea: Additional Experiments and Clarifications

We sincerely thank you for your thoughtful review and positive assessment, and appreciate the opportunity to clarify the implementation.

The temporal-mask head combines each decoded relation query with frame features from the Spatio-Temporal Transformer to predict a probability at each resampled position. Thresholding yields intervals in which the relation holds. Their boundaries are mapped first to the dataset temporal grid and then to original video frames, using indices retained during resampling.

To make this explicit, we will add the following descriptions to Appendix A.4, including implementation details of the Spatio-Temporal Transformer and Relation Decoder. Location key: M = Main paper Sec.; S = Suppl. Sec.

We will include these additions in the revised paper and supplementary material and add a cross-reference in Section 4.4.

Q1. Temporal-mask head and mapping to original frames (M 4.4; S A.3.2, A.4)

Temporal-mask head. At each resampled time ℓ\ell, we take a weighted average of the Spatio-Temporal Transformer's spatial tokens to obtain a frame feature fℓf_\ell. Tokens in the subject/object regions receive weight 1.1 and background tokens receive weight 0.1, retaining scene context. Each decoded relation query q^n\hat q_n is combined with this feature and its change over time:

σ^n,ℓ=sigmoid⁡(WoLN⁡[(Wqq^n)⊙(Wtfℓ+WdΔfℓ)]+bo).\hat{\sigma}_{n,\ell}=\operatorname{sigmoid}\left(W_o\operatorname{LN}\left[(W_q\hat{q}_n)\odot(W_t f_\ell+W_d\Delta f_\ell)\right]+b_o\right).

Here, Δfℓ=fℓ−fℓ−1\Delta f_\ell=f_\ell-f_{\ell-1} for ℓ>0\ell>0, with Δf0=0\Delta f_0=0. The WW terms are learned projections, and ⊙\odot denotes element-wise multiplication. The value before sigmoid is the temporal logit; σ^n,ℓ\hat{\sigma}_{n,\ell} is the relation probability. Hungarian-matched queries are trained with weighted binary cross-entropy on logits against ground-truth masks sampled at the same stored indices.

Mapping to original frames. Features are linearly resampled over the pair's coexistence interval. At each resampled position, we retain the lower temporal-grid index and the mapping from that grid to original video frames.

For each predicate, we use the mask of the query with the highest probability for that predicate. We threshold the probabilities, merge short gaps and remove short intervals on the resampled grid. We then map the interval boundaries through the two stored index mappings, clip them to the pair's coexistence interval, and interpolate the object trajectories for evaluation. Temporal indices are zero-based and interval ends are exclusive.

If no valid mask interval remains, decoding falls back to the pair's coexistence interval.

The evaluator uses a default probability threshold of 0.35. The maximum gap for merging intervals and the minimum interval length are both two resampled time steps. If no position exceeds the threshold, we use 0.6 times the peak probability as a relative threshold. If no valid interval remains, we use the pair’s coexistence interval. We will document these decoding rules and the two-stage index mapping in the appendix.

Additional Architecture Details

Spatio-Temporal Transformer. The default configuration uses two spatial encoder layers followed by two temporal encoder layers. Each layer has four attention heads, a hidden dimension of 256, a feed-forward dimension of 1,024, dropout of 0.3, and pre-LayerNorm.

Relation Decoder. The default decoder has three layers, each with query self-attention, cross-attention to the encoded memory, and a feed-forward network. It uses the same head count, dimensions, dropout, and pre-LayerNorm as the encoders, with a final LayerNorm before prediction.

We hope these clarifications address your question and resolve your concern. If so, we would be very grateful if you would consider raising your score.

AuthorsOpenReview ↗

Follow-up on Our Response to Your Review

Dear Reviewer,

Thank you again for your valuable feedback and constructive comments. We have provided detailed responses to your questions and concerns, and we would greatly appreciate it if you could take a moment to review them.

If you have any further questions or concerns, we would be more than happy to provide additional clarifications and engage in further discussion.

Thank you once again for your time and consideration.

Reviewer_3LL8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes PACT, a relation representation for Video Visual Relation Detection. Instead of building relation features only from subject/object regions, PACT uses full-frame visual tokens from a frozen DINOv2 encoder, injects subject/object category text embeddings into tokens according to spatial overlap, processes the pair-specific tokens with a spatio-temporal transformer, and decodes predicate plus temporal span with a DETR-style relation decoder. Experiments are conducted on ImageNet-VidVRD and VidOR, with ablations on representation design and model components.

Critical strengths
  1. Object-localized relation features can miss scene context and be sensitive to detection/tracking errors. The motivation is clear.
  2. The pair-aware token injection design is simple and supported by a useful ablation on semantic text embeddings versus role/category tokens. The method achieves best mAP on two standard VidVRD benchmarks.
Critical weaknesses
  1. Improvements in mAP are modest, and PACT loses on several recall/precision metrics. This weakens the SOTA claim.
  2. It is not very clear whether all baselines use the same frozen detector, tracker, visual backbone, and tubelets. Without this control, the reported gains may reflect stronger frozen features rather than the proposed representation. Full-frame tokens plus text injection plus transformer decoding is incremental. Processing full-frame tokens for every pair may be expensive, but no runtime/memory comparison is provided.
Questions for authors
  1. Do all compared baselines use the same setting, such as Grounding DINO detections, ByteTrack tubelets, and frozen visual features? If not, can you provide a controlled comparison under identical inputs?
  2. Can you report statistical significance or variance across runs for the main results, especially given the small mAP gains?
  3. What are the runtime and memory costs per video/pair compared with region-based and single-stage baselines?
  4. Why does PACT underperform on several R@50/P@5 metrics despite better mAP, and can this be improved?
  5. Can you provide full ablations on VidOR as well, not only ImageNet-VidVRD?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 3LL8: Additional Experiments and Clarifications

We thank the reviewer for the careful review and constructive suggestions. We have incorporated additional results and clarifications into the revised manuscript, with changes marked in red. Scores are percentages. Location key: M = Main paper Sec.; S = Suppl. Sec.

Q1. Detectors, trackers and visual backbones (M 5.1-5.2; S A.3.1)

PACT uses Grounding DINO, ByteTrack and DINOv2. We retrained VidVRD-II with these components, keeping its relation architecture and training protocol fixed.

VidVRD-II configurationmAPR@50R@100P@1P@5
Original components, retrained19.127.538.4862.0041.40
Detector/tracker replaced21.6110.3412.1264.5046.30
Visual backbone replaced20.558.509.3572.5053.00
Both replaced23.1711.3512.6668.0053.10

Detector/tracker replacement improves recall more, while backbone replacement improves tagging precision more. Combining both gives the highest VidVRD-II mAP and recall.

These component swaps quantify complementary upstream contributions. PACT integrates pretrained visual and semantic features with pair locations and tracks in full-frame tokens, preserving scene context for pair-specific relation modeling.

Q2. Variance and significance (M 5.2)

We re-evaluated two existing PACT runs on ImageNet-VidVRD at epoch 56, with matched recorded training settings and the same 200-video SGDet protocol (alpha=1.1). Mean +/- sample SD (n=2): mAP 31.17+/-1.03, R@50 19.92+/-0.15, R@100 22.95+/-0.66, P@1 75.25+/-1.77, P@5 56.69+/-0.58. This run-level summary places the mean mAP 0.75 percentage points above the highest reported baseline mAP (VSTRD, 30.42).

Q3. Runtime and GPU memory (M 6)

Frozen visual features are shared across pairs. We measured PACT SGDet (alpha=1.1) on one H200, FP32, video batch size 1: five warm-up videos, then three passes over 200 test videos.

PACT measurementValue
Cached relation pipeline0.3192 s/video
Model forward0.04994 s/video
Amortized forward / active ordered pair2.02 ms
Peak allocated / reserved19.36 / 41.18 GiB

Sharing features avoids repeated image encoding while retaining pair-specific modeling. Pipeline time includes cache loading, preparation/transfers and decoding/NMS; detector/tracker/backbone extraction, model loading and GT scoring are excluded. Latency depends on cache state. Memory is the PyTorch allocator peak including weights.

Q4. Higher mAP but lower R@50/P@5 (M 5.2; S B.1)

mAP averages per-video detection AP across confidence thresholds, recall pools GT instances at a fixed output budget, and tagging evaluates distinct triplets without localization or time. PACT's leading mAP and R@100 on VidVRD support detection quality across thresholds and recovery within a broader output budget.

We analyzed unchanged alpha=1.1 predictions on 200 videos (mAP 31.90, a different run from the reported 32.11).

K10203050100
RelDet R@K11.4415.0817.1720.0223.41

Of 1,132 top-100 matches, 968 (85.51%) already occur within top-50; ranks 51-100 recover 164 additional valid detections. Thus, most recovered relations rank within the smaller budget, while a broader budget captures further relations.

Tag K1510
Existing P@K76.5057.1046.26
Fixed-K diagnostic76.5055.7040.55

P@1 of 76.50 reflects strong top-ranked triplet predictions. The two rows distinguish evaluation conventions: existing precision uses min(K,M), where M counts unique triplets in raw top-200 (empty=0); the diagnostic uses K. These tagging results complement the detection analysis by measuring label selection separately from localization and time.

Q5. VidOR ablations (M 5.4; S_soon_)

Each attention variant was trained on 7,000 VidOR videos for 50 epochs and evaluated on 835 validation videos. We selected the best validation mAP among epochs 45-50; both completed variants peaked at epoch 47.

Removed componentmAPR@50R@100P@1P@5
Spatial attention10.1410.5012.3759.9847.88
Temporal attention10.3310.7512.7260.3448.44

These completed experiments extend the architectural analysis to a second benchmark. A matched full-model control remains the final step for quantifying each component's contribution. We will continue to expand the VidOR experiments and upload the results as soon as each experiment is completed.

AuthorsOpenReview ↗

Follow-up on Our Response to Your Review

Dear Reviewer,

Thank you again for your valuable feedback and constructive comments. We have provided detailed responses to your questions and concerns, and we would greatly appreciate it if you could take a moment to review them.

If you have any further questions or concerns, we would be more than happy to provide additional clarifications and engage in further discussion.

Thank you once again for your time and consideration.

Thank you for the detailed response and the additional experiments. The responses of controlled component-swap comparison experiment and runtime and memory addressed some of the concerns. However, the reported variance and significance are somewhat weak, based on only n=2 runs. Retraining VidVRD-II is helpful, but the original concern was whether all compared baselines use the same detector, tracker, backbone, and tubelets. Please state explicitly which baselines were retrained or reused under identical inputs, and which were not. The statistical evidence for the modest mAP gain and the VidOR ablation remain insufficient for me to change my assessment. I would keep the original rating for now.

AuthorsOpenReview ↗

Additional Response to Reviewer 3LL8

Additional Response to Reviewer 3LL8

Thank you for recognizing the component-swap and efficiency experiments. To address your remaining concerns, we clarify the baseline comparisons and provide additional training runs and VidOR ablations. We hope this expanded evidence helps clarify the experimental support for PACT. Scores are percentages.

Scope of the baseline comparisons. The main-table baseline results—VIDVRD, 3DRN, VidVRD-II, VRD-STGC, VRDFormer, RelTR-v2, and VSTRD, where available—are quoted from the original papers and compare complete systems under their reported settings.

For the additional component-swap experiment, only VidVRD-II was retrained. We retained its relation architecture and training protocol and evaluated four conditions: original components, detector/tracker replacement, backbone replacement, and both replacements. The replacements use PACT's corresponding pretrained upstream components. Thus, this additional study examines component effects within VidVRD-II, whereas the main table compares published complete-system performance. We will state these two scopes explicitly in the revised table caption and experimental description.

Additional VidOR ablations. We have extended the component analysis to VidOR. The spatial/temporal attention and loss variants were trained for 50 epochs on 7,000 videos. All five ablation variants were evaluated on 835 validation videos. The table includes the published full-model result as a reference.

ConfigurationmAPR@50R@100P@1P@5
PACT (published reference)10.8711.0512.9661.4151.24
w/o spatial attention10.1410.5012.3759.9847.88
w/o temporal attention10.3310.7512.7260.3448.44
w/o classification loss *10.2510.5912.4359.8649.37
w/o box loss *9.9710.4912.3859.7448.62
w/o ST Transformer *10.2410.5412.6959.1347.76

* Newly added in this update: classification-loss, box-loss, and ST Transformer ablation results.

The full-model reference is higher than every evaluated variant across all five metrics. Among the ablated variants, removing the box loss gives the lowest RelDet mAP (9.97), while removing the ST Transformer gives the lowest RelTag P@1/P@5 (59.13/47.76). Spatial-attention removal also scores below temporal-attention removal on all five metrics. These patterns are consistent with complementary roles for spatial/temporal modeling and the classification and box-regression losses, extending the component-level evidence to VidOR.

Training variation (four completed runs). We now report four independently trained VidVRD checkpoints: two existing runs and two additional runs with seeds 42/43. All four use the same current evaluation on 200 test videos, alpha=1.1, and fixed stored epoch 56, with matching annotations, vocabulary, evaluation code, and decoding settings.

RunmAPR@50R@100P@1P@5
Existing run A31.9020.0223.4176.5057.10
Existing run B30.4419.8122.4874.0056.28
Additional run (seed 42)29.9419.2822.6374.0055.06
Additional run (seed 43)30.6119.3022.3674.0057.00
Mean±sample SD (n=4)30.72±0.8319.60±0.3722.72±0.4774.63±1.2556.36±0.94

The four-run mAP is 30.72±0.83 (mean±sample SD; ddof=1). SD is in percentage points. This expands the previously reported two-run evaluation; further independent training runs are ongoing.

Comparison with the paper results. Compared with the paper's VidVRD results (32.11/20.14/23.60/77.50/57.40 for mAP/R@50/R@100/P@1/P@5), the four-run means are lower by 1.39/0.54/0.88/2.88/1.04 percentage points, respectively (calculated before rounding). The highest mAP among these four runs is 31.90, 0.21 points below the paper's 32.11. Nevertheless, the mean RelDet mAP of 30.72 exceeds the highest published baseline mAP in Table 1 (VSTRD, 30.42) by 0.30 points. PACT thus retains SOTA-level mAP within the reported comparison when assessed by its four-run mean.

Further experiments and revision. The visual representation ablations are ongoing, and we sincerely apologize for the delay due to their training time. We will complete the remaining ablations and strengthen the revised paper with the full results, experimental details, and analysis. Your feedback has helped us expand the evaluation and clarify the experimental comparisons, improving the quality of our work. We sincerely thank you for your careful and constructive review.

Reviewer_o2uvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Pair-Aware Contextual Tokens (PACT) for Video Visual Relation Detection. Existing object-centric representations may underutilize surrounding scene context. PACT instead uses full-frame visual tokens and injects subject–object category embeddings according to their spatial overlap with image patches. A spatio-temporal transformer and relation decoder then predict predicates and their temporal intervals. Experiments on ImageNet-VidVRD and VidOR achieve the highest reported mAP among the compared methods, while ablations support the effectiveness of pair-specific conditioning and temporal modeling.

Critical strengths
  • The paper addresses a meaningful limitation of object-centric representations by retaining scene context while explicitly identifying the target subject–object pair.

  • The spatially weighted semantic injection is simple and well-motivated. Supplementary ablations comparing text embeddings with category and role tokens provide useful evidence for its design.

  • The method achieves strong mAP on two benchmarks using a frozen image encoder. Component ablations and qualitative examples help explain the contributions of temporal modeling and auxiliary supervision.

Critical weaknesses
  • The evidence for the central benefit of global context remains limited. In Table 2, full-frame tokens improve mAP only from 31.68 to 32.11 over object-region tokens. Controlled context masking and separate spatial-only and semantic-only ablations would better establish where the gains originate.

  • The comparison with prior methods does not clearly control for differences in detectors, trackers, and visual backbones. Matched comparisons would strengthen attribution to PACT. Moreover, the VidOR mAP gain over VSTRD is only 0.14 points.

  • The paper lacks comparisons with general-purpose VLMs, which could potentially predict relation triplets and temporal intervals when prompted with the target entity pair. Evaluating a VLM baseline using the same detected tubelets and predicate vocabulary would help establish the benefit of the proposed specialized relation representation.

  • The framework relies on a multi-stage pipeline involving detection, tracking, and pair-specific relation modeling, raising concerns about error propagation and generalization. Evaluation is limited to ImageNet-VidVRD and VidOR, two old benchmarks. Experiments on newer benchmarks or under cross-dataset and crowded-scene settings would better establish its robustness and broader applicability.

Questions for authors

Please see the weakness part

2 comments
AuthorsOpenReview ↗

Response to Reviewer o2uv: Additional Experiments and Clarifications

We thank the reviewer for the careful review and constructive suggestions. We have incorporated the additional results and clarifications into the revised manuscript, with changes marked in red. Scores are percentages. Location key: M = Main paper Sec.; S = Suppl. Sec.

W1. Context and semantic injection (M 5.4, Tab. 2; S A.5, Tab. 4)

The object-region control in Tab. 2 (M 5.4) uses semantic injection. Below we combine Tab. 2 (M 5.4), Tab. 4 (S A.5) and a newly trained control to separate context and semantic contributions.

Visual tokensAdded pair informationmAPR@50
Full frameNone25.2314.81
Object regionsNone25.8114.97
Object regionsCategory semantics31.6819.81
Full frameSubject/object role markers31.1219.21
Full frame (PACT)Category semantics32.1120.14

Semantic injection raises mAP by 6.88 points for full-frame tokens and 5.87 for object-region tokens. With semantics in both settings, full-frame tokens further improve mAP from 31.68 to 32.11, supporting complementary benefits from semantics and scene context. Tab. 4 (S A.5) fixes the injection location: PACT also outperforms role-only markers, showing the value of category semantics beyond location and role cues.

W2. Detectors, trackers and visual backbones (M 5.1-5.2; S A.3.1)

We use Grounding DINO, ByteTrack and DINOv2 as the detector, tracker and visual backbone, respectively. We retrained VidVRD-II with PACT's components while keeping its relation architecture and training protocol fixed.

VidVRD-II configurationmAPR@50R@100P@1P@5
Original components, retrained19.127.538.4862.0041.40
Detector/tracker replaced21.6110.3412.1264.5046.30
Visual backbone replaced20.558.509.3572.5053.00
Both replaced23.1711.3512.6668.0053.10

Detector/tracker replacement improves recall more; backbone replacement improves tagging precision more. Their combination gives the highest mAP and recall, suggesting complementary roles.

PACT combines pretrained visual and semantic features with pair locations and tracks to retain scene context, turning these complementary inputs into pair-specific relation representations.

W3. General-purpose VLM baseline (M 5.2)

On all VidVRD test videos, we compared frozen Qwen3-VL-32B-Instruct and PACT using the same annotated tracks, categories, relation vocabulary and evaluator. Relation labels and intervals were withheld at inference. The VLM saw sampled full frames with pair boxes and timestamps.

MethodmAPR@50R@100P@1P@5
VLM: generated intervals7.383.994.0538.0029.26
VLM: object-coexistence intervals7.323.813.9338.0029.26
PACT43.7330.6336.3088.5067.50

PACT achieves 43.73 mAP versus 7.38 for the frozen VLM under shared object inputs and evaluation. The VLM misses many relation labels even without interval matching, while changing intervals has little effect. Together, these results support the value of learning pair-specific visual and temporal relations for this task.

W4. Error propagation (M 6)

Detected inputs can contain location, tracking and category errors. With annotated tracks, object boxes and identities are given but categories are predicted; the last setting also gives category labels. No relation labels or intervals are provided.

Object informationmAPR@50R@100P@1P@5
Detected tracks, predicted categories32.1120.1423.6077.5057.40
Annotated tracks, predicted categories40.7628.7534.1583.5063.70
Annotated tracks and categories43.7330.6336.3088.5067.50

Annotated tracks raise mAP by 8.65 points, and annotated categories add 2.97. PACT effectively uses more accurate object inputs, with track quality and candidate coverage providing the larger improvement opportunity.

W5. Crowded scenes (M 6)

We grouped the same VidVRD test videos by the 90th percentile of per-frame annotated object counts, using thresholds fixed on training data. Values are mAP.

CrowdingVideosDetected tracksAnnotated tracks+ Categories
Low (<=2)12739.1045.0648.53
Medium (>2, <=3)2921.3129.7531.89
High (>3)4418.0835.6337.65

In the high-crowding group, annotated tracks raise mAP from 18.08 to 35.63 (+17.55 points), while category annotations add 2.02. This demonstrates substantial relation-recovery potential in crowded scenes when reliable tracks are available, identifying track quality and candidate-pair coverage as concrete improvement targets.

AuthorsOpenReview ↗

Follow-up on Our Response to Your Review

Dear Reviewer,

Thank you again for your valuable feedback and constructive comments. We have provided detailed responses to your questions and concerns, and we would greatly appreciate it if you could take a moment to review them.

If you have any further questions or concerns, we would be more than happy to provide additional clarifications and engage in further discussion.

Thank you once again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.