- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes PACT for video visual relation detection. It conditions full-frame visual tokens on a subject–object pair using spatial overlaps and category text embeddings, followed by spatial–temporal attention and query-based relation prediction. Experiments support the usefulness of the proposed method.
Critical strengths
- The pair-conditioning mechanism provides a simple way to identify the target pair within shared full-frame features while retaining scene information.
- The evaluation covers two benchmarks and includes useful internal comparisons. The ablation results support the effectiveness of the design.
Critical weaknesses
None.
Questions for authors
Section 4.4 describes the mask supervision and inference, but Appendix A.4 does not explain this head. Please specify the temporal-mask prediction head and how its outputs are mapped from the resampled sequence to the original frame indices.
2 comments
Follow-up on Our Response to Your Review
Dear Reviewer,
Thank you again for your valuable feedback and constructive comments. We have provided detailed responses to your questions and concerns, and we would greatly appreciate it if you could take a moment to review them.
If you have any further questions or concerns, we would be more than happy to provide additional clarifications and engage in further discussion.
Thank you once again for your time and consideration.
Response to Reviewer 3Tea: Additional Experiments and Clarifications
We sincerely thank you for your thoughtful review and positive assessment, and appreciate the opportunity to clarify the implementation.
The temporal-mask head combines each decoded relation query with frame features from the Spatio-Temporal Transformer to predict a probability at each resampled position. Thresholding yields intervals in which the relation holds. Their boundaries are mapped first to the dataset temporal grid and then to original video frames, using indices retained during resampling.
To make this explicit, we will add the following descriptions to Appendix A.4, including implementation details of the Spatio-Temporal Transformer and Relation Decoder. Location key: M = Main paper Sec.; S = Suppl. Sec.
We will include these additions in the revised paper and supplementary material and add a cross-reference in Section 4.4.
Q1. Temporal-mask head and mapping to original frames (M 4.4; S A.3.2, A.4)
Temporal-mask head. At each resampled time ℓ, we take a weighted average of the Spatio-Temporal Transformer's spatial tokens to obtain a frame feature fℓ. Tokens in the subject/object regions receive weight 1.1 and background tokens receive weight 0.1, retaining scene context. Each decoded relation query q^n is combined with this feature and its change over time:
σ^n,ℓ=sigmoid(WoLN[(Wqq^n)⊙(Wtfℓ+WdΔfℓ)]+bo).
Here, Δfℓ=fℓ−fℓ−1 for ℓ>0, with Δf0=0. The W terms are learned projections, and ⊙ denotes element-wise multiplication. The value before sigmoid is the temporal logit; σ^n,ℓ is the relation probability. Hungarian-matched queries are trained with weighted binary cross-entropy on logits against ground-truth masks sampled at the same stored indices.
Mapping to original frames. Features are linearly resampled over the pair's coexistence interval. At each resampled position, we retain the lower temporal-grid index and the mapping from that grid to original video frames.
For each predicate, we use the mask of the query with the highest probability for that predicate. We threshold the probabilities, merge short gaps and remove short intervals on the resampled grid. We then map the interval boundaries through the two stored index mappings, clip them to the pair's coexistence interval, and interpolate the object trajectories for evaluation. Temporal indices are zero-based and interval ends are exclusive.
If no valid mask interval remains, decoding falls back to the pair's coexistence interval.
The evaluator uses a default probability threshold of 0.35. The maximum gap for merging intervals and the minimum interval length are both two resampled time steps. If no position exceeds the threshold, we use 0.6 times the peak probability as a relative threshold. If no valid interval remains, we use the pair’s coexistence interval. We will document these decoding rules and the two-stage index mapping in the appendix.
Additional Architecture Details
Spatio-Temporal Transformer. The default configuration uses two spatial encoder layers followed by two temporal encoder layers. Each layer has four attention heads, a hidden dimension of 256, a feed-forward dimension of 1,024, dropout of 0.3, and pre-LayerNorm.
Relation Decoder. The default decoder has three layers, each with query self-attention, cross-attention to the encoded memory, and a feed-forward network. It uses the same head count, dimensions, dropout, and pre-LayerNorm as the encoders, with a final LayerNorm before prediction.
We hope these clarifications address your question and resolve your concern. If so, we would be very grateful if you would consider raising your score.