- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes frame-level audio-visual temporal embeddings (FATE). Instead of pooling each modality into one vector, the model keeps frame-level tokens. Audio is the temporal anchor; the visual sequence is downsampled to the audio token count by nearest-neighbor alignment; and cross-modal similarity is the mean inner product over time-aligned frame pairs. Training combines a cross-video semantic contrastive loss with an intra-video temporal soft contrastive loss whose raised-cosine targets put the half-height at the human-perceivable asynchrony threshold. Overall, the claims and the evidence in this paper largely align, but some claims are not sufficiently supported.
Critical strengths
The core motivation is clean. Global pooling often destroys temporal information. The method keeps frame-level tokens and aligns them on physical time is a straightfoward and reasonable method.
The diagnostics are well controlled. The w/o frame-level variant changes only the pooling, holding data, backbone, LoRA configuration, and training budget fixed, which splits representation design from training objective.
The cross-task evidence is convincing. Retrieval, zero-shot event localization, and synchronization scoring of generated videos are quite different tasks, and FATE is best or near-best on all of them without task-specific tuning or extra heads.
Critical weaknesses
-
The recipe's backbone generality is unverified. Nothing in the recipe is specific to PE-AV. Dropping the pooling head, aligning on time, and averaging inner products over aligned frames would work with any dual encoder, and that is the most attractive part of the paper. But every experiment uses PE-AV-small on VGGSound, and with LoRA touching 2.3% of the weights, much of the result may come from PE-AV's pretraining. Two design choices also depend on the backbone: the alignment assumes video has more tokens per second than audio, and the temporal resolution is set by the audio encoder's frame rate. This is my main concern. If the recipe only works on PE-AV, the paper is an improvement to PE-AV, not a general method.
-
Temporal training only contrasts a clip with shifted copies of itself. All 17 candidates in the temporal loss come from the same video, so the model never sees the harder case of plausible content at the wrong time, for example barking from another dog video laid over this one. Semantics and timing therefore stay mixed in the score, which may be why the generation correlations look driven by semantics. The paper's main claim is that the embedding captures when a sound happens, not just what it is. Without mismatched pairs from other videos, in training or at least in testing, it is hard to tell whether the score tracks timing or content.
Questions for authors
- In the AVE evaluation, how is the audio query chosen? If it is taken from the annotated event segment, the task is easier than standard AVEL.
- FATE is evaluated with your own AVE protocol, while DCCA, AVDLN, and DAM are reported under the standard protocol. How should the reader compare the two, given they are different tasks? Is there a reason the supervised baselines stop at DAM (2019)?
- Can the recipe be reproduced on a second dual-encoder backbone?
- Could you add mismatched pairs from different videos of the same event class, at least as a test set, to check whether the score follows timing rather than content?
- Two controls would show whether the gain comes from the frame-level design or from fine-tuning on VGGSound: scoring frozen PE-AV frame features with Eq. 3, and training the pooled model with the same recipe but only the semantic loss.
- Eq. 3 averages over frames, which dilutes the signal when most frames contain no event. How do max or top-k aggregation compare?
3 comments
Official Response to Reviewer cSZz [2/2]
Q5. Frozen features and semantic-only training.
We scored frozen PE-AV-small features with Eq. 3 and trained pooled and frame-level models using only the semantic loss. Each cell reports intra-video R@3 (%) as V2A / A2V.
| Setting | AVSync-15 pooled | AVSync-15 frame-level | VGG-Sync pooled | VGG-Sync frame-level |
|---|---|---|---|---|
| Frozen PE-AV-small | 18.00 / 17.33 | 14.89 / 17.78 | 20.28 / 17.93 | 18.21 / 18.38 |
| Semantic-only training | 19.33 / 24.44 | 18.89 / 24.89 | 21.12 / 20.28 | 21.51 / 19.66 |
Neither control shows a consistent advantage for frame-level scoring. Isolating the contribution of temporal supervision would require a joint-loss run with the same data and training schedule.
Q6. Mean, max, and top-k aggregation
We report the original mean-aggregation results from Table 1 for reference. In the supplementary evaluation, we compared mean, max, and the mean of the top 25% of aligned frame scores using the same candidate windows. All values are R@3 (%).
| Dataset | Aggregation | Intra V2A | Intra A2V | Inter V2A | Inter A2V |
|---|---|---|---|---|---|
| AVSync-15 | Mean | 55.74 | 56.22 | 34.13 | 30.44 |
| AVSync-15 | Top 25% mean | 51.56 | 48.67 | 30.89 | 24.67 |
| AVSync-15 | Max | 46.44 | 36.22 | 26.67 | 20.00 |
| VGG-Sync | Mean | 48.37 | 44.83 | 37.76 | 31.54 |
| VGG-Sync | Top 25% mean | 43.64 | 42.75 | 36.47 | 28.07 |
| VGG-Sync | Max | 41.03 | 37.31 | 29.30 | 21.34 |
Mean aggregation performed best in all eight R@3 comparisons. Averaging rewards agreement across the window, while max can give a mismatched clip a high score based on a single frame.
Please let us know if you have any further questions. We are happy to discuss. :) Best regards
Thank you very much for you rebuttal, my concerns have been addressed. Since my confidence to this paper is not very high, I will keep my original positive assessment and will not consider raising to the highest score.
Official Response to Reviewer cSZz [1/2]
Thank you very much for taking the time to review and for your support. We try our best to address your questions as follows.
W1: Backbone generality
Thank you for this suggestion. We extended FATE to PE-AV Base, a larger variant within the same family, and ImageBind, a different dual-encoder backbone. Both use frame-level alignment and joint semantic–temporal training. The table reports R@3 (%) before → after fine-tuning, using frame-level scoring in both cases.
Both additional backbones improve across all reported retrieval settings. We will include these results in the revised manuscript. We apologize that GPUs are currently unavailable and plan to evaluate more backbone variants when resources become available.
W2 / Q4. Same-class audio mismatches.
The temporal loss contrasts 17 windows from the same video. The semantic loss adds negatives from other videos sampled without conditioning on event class. Following your suggestion, we tested 28 AVSync-15 queries across 14 event classes. The table reports mean similarity scores.
FATE assigns the highest mean similarity to synchronized audio, scoring it 0.041 above the same-video shift and 0.149 above audio from a different video of the same class. PE-AV scores the shifted audio slightly higher than the synchronized audio. These results suggest that FATE is sensitive to temporal alignment, while the cross-video comparison also reflects differences between individual recordings.
Q1 & Q2: AVE query selection and baselines
Thank you for raising this. Our AVE evaluation follows the cross-modality localization (CML) task introduced by Tian et al. [1]. The annotated event segment serves as the query in one modality, and the model locates the matching interval in the other modality. Table 2 uses the published CML results for DCCA [1, 2], AVDLN [1], and DAM [3]. These baselines are trained on AVE, while FATE is evaluated zero-shot. We will name the CML task explicitly and clarify the query selection in Sec. 5.2.
We will also extend Table 2 beyond DAM to include M2N [4], DCMR [5], and CCLN [6], which report average CML accuracies of 52.4%, 56.1%, and 63.9%, respectively. FATE’s zero-shot accuracy (48.3%) is close to DAM’s (47.8%) and below those of these later supervised methods.
[1] Tian et al., “Audio-Visual Event Localization in Unconstrained Videos”, ECCV 2018.
[2] Andrew et al., “Deep Canonical Correlation Analysis”, ICML 2013.
[3] Wu et al., “Dual Attention Matching for Audio-Visual Event Localization”, ICCV 2019.
[4] Wang et al., “Multi-Modulation Network for Audio-Visual Event Localization”, arXiv 2021.
[5] Liu et al., “Dense Modality Interaction Network for Audio-Visual Event Localization”, IEEE TMM 2023.
[6] Sun et al., “Listen With Seeing: Cross-Modal Contrastive Learning for Audio-Visual Event Localization”, IEEE TMM 2025.