acceptodds
Under review as a conference paper at ICLR 2027

FATE: Frame-Level Audio-Visual Temporal Embedding

Abstract

When a dog opens its mouth and barks, humans naturally recognize what the sound is and when it occurs. Building audio-visual models with this same ability requires representations that capture both semantic and temporal alignment. Current approaches fall short on one side or the other: embedding models capture semantics but lose temporal information; synchronization models capture temporal offsets but lack semantic understanding. To bridge this gap, we propose FATE, Frame-level Audio-visual Temporal Embedding. Unlike prior embedding models that pool each modality into a single embedding and discard temporal information, FATE retains frame-level sequences, aligns them on the physical timeline, and computes similarity over strictly aligned frame pairs. Unlike synchronization models that output only an offset prediction, FATE encodes synchronization in a reusable embedding space, trained with a joint objective combining cross-video semantic and within-video temporal contrastive learning to capture both what sounds and when it occurs. Across three tasks, FATE surpasses the strongest baseline on temporal and semantic retrieval by a large margin, matches fully supervised methods on event localization in a zero-shot setting, and achieves the best correlation with human judgments as a generation evaluation metric. The source code can be found at https://anonymous.4open.science/r/FATE-ICLR.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Audio-Visual, synchronization, embedding model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cSZzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes frame-level audio-visual temporal embeddings (FATE). Instead of pooling each modality into one vector, the model keeps frame-level tokens. Audio is the temporal anchor; the visual sequence is downsampled to the audio token count by nearest-neighbor alignment; and cross-modal similarity is the mean inner product over time-aligned frame pairs. Training combines a cross-video semantic contrastive loss with an intra-video temporal soft contrastive loss whose raised-cosine targets put the half-height at the human-perceivable asynchrony threshold. Overall, the claims and the evidence in this paper largely align, but some claims are not sufficiently supported.

Critical strengths

The core motivation is clean. Global pooling often destroys temporal information. The method keeps frame-level tokens and aligns them on physical time is a straightfoward and reasonable method.

The diagnostics are well controlled. The w/o frame-level variant changes only the pooling, holding data, backbone, LoRA configuration, and training budget fixed, which splits representation design from training objective.

The cross-task evidence is convincing. Retrieval, zero-shot event localization, and synchronization scoring of generated videos are quite different tasks, and FATE is best or near-best on all of them without task-specific tuning or extra heads.

Critical weaknesses
  1. The recipe's backbone generality is unverified. Nothing in the recipe is specific to PE-AV. Dropping the pooling head, aligning on time, and averaging inner products over aligned frames would work with any dual encoder, and that is the most attractive part of the paper. But every experiment uses PE-AV-small on VGGSound, and with LoRA touching 2.3% of the weights, much of the result may come from PE-AV's pretraining. Two design choices also depend on the backbone: the alignment assumes video has more tokens per second than audio, and the temporal resolution is set by the audio encoder's frame rate. This is my main concern. If the recipe only works on PE-AV, the paper is an improvement to PE-AV, not a general method.

  2. Temporal training only contrasts a clip with shifted copies of itself. All 17 candidates in the temporal loss come from the same video, so the model never sees the harder case of plausible content at the wrong time, for example barking from another dog video laid over this one. Semantics and timing therefore stay mixed in the score, which may be why the generation correlations look driven by semantics. The paper's main claim is that the embedding captures when a sound happens, not just what it is. Without mismatched pairs from other videos, in training or at least in testing, it is hard to tell whether the score tracks timing or content.

Questions for authors
  1. In the AVE evaluation, how is the audio query chosen? If it is taken from the annotated event segment, the task is easier than standard AVEL.
  2. FATE is evaluated with your own AVE protocol, while DCCA, AVDLN, and DAM are reported under the standard protocol. How should the reader compare the two, given they are different tasks? Is there a reason the supervised baselines stop at DAM (2019)?
  3. Can the recipe be reproduced on a second dual-encoder backbone?
  4. Could you add mismatched pairs from different videos of the same event class, at least as a test set, to check whether the score follows timing rather than content?
  5. Two controls would show whether the gain comes from the frame-level design or from fine-tuning on VGGSound: scoring frozen PE-AV frame features with Eq. 3, and training the pooled model with the same recipe but only the semantic loss.
  6. Eq. 3 averages over frames, which dilutes the signal when most frames contain no event. How do max or top-k aggregation compare?
3 comments
AuthorsOpenReview ↗

Official Response to Reviewer cSZz [1/2]

Thank you very much for taking the time to review and for your support. We try our best to address your questions as follows.

W1: Backbone generality

Thank you for this suggestion. We extended FATE to PE-AV Base, a larger variant within the same family, and ImageBind, a different dual-encoder backbone. Both use frame-level alignment and joint semantic–temporal training. The table reports R@3 (%) before → after fine-tuning, using frame-level scoring in both cases.

DatasetBackboneIntra V2AIntra A2VInter V2AInter A2V
AVSync-15PE-AV-small18.00 → 55.7417.78 → 56.227.56 → 34.138.67 → 30.44
AVSync-15PE-AV Base20.22 → 56.6719.62 → 58.1112.00 → 38.339.33 → 36.10
AVSync-15Frame-level ImageBind16.44 → 45.7814.12 → 44.340.67 → 33.110.44 → 28.23
VGG-SyncPE-AV-small20.28 → 48.3717.93 → 44.8310.70 → 37.767.11 → 31.54
VGG-SyncPE-AV Base21.59 → 48.4618.32 → 46.0512.90 → 39.337.68 → 34.25
VGG-SyncFrame-level ImageBind19.19 → 43.5320.76 → 43.587.73 → 30.287.90 → 33.28

Both additional backbones improve across all reported retrieval settings. We will include these results in the revised manuscript. We apologize that GPUs are currently unavailable and plan to evaluate more backbone variants when resources become available.

W2 / Q4. Same-class audio mismatches.

The temporal loss contrasts 17 windows from the same video. The semantic loss adds negatives from other videos sampled without conditioning on event class. Following your suggestion, we tested 28 AVSync-15 queries across 14 event classes. The table reports mean similarity scores.

ModelSynchronized audioSame video (shifted by 2 s)Different video (same class)
PE-AV0.07470.08340.0548
FATE0.2440.2030.095

FATE assigns the highest mean similarity to synchronized audio, scoring it 0.041 above the same-video shift and 0.149 above audio from a different video of the same class. PE-AV scores the shifted audio slightly higher than the synchronized audio. These results suggest that FATE is sensitive to temporal alignment, while the cross-video comparison also reflects differences between individual recordings.

Q1 & Q2: AVE query selection and baselines

Thank you for raising this. Our AVE evaluation follows the cross-modality localization (CML) task introduced by Tian et al. [1]. The annotated event segment serves as the query in one modality, and the model locates the matching interval in the other modality. Table 2 uses the published CML results for DCCA [1, 2], AVDLN [1], and DAM [3]. These baselines are trained on AVE, while FATE is evaluated zero-shot. We will name the CML task explicitly and clarify the query selection in Sec. 5.2.

We will also extend Table 2 beyond DAM to include M2N [4], DCMR [5], and CCLN [6], which report average CML accuracies of 52.4%, 56.1%, and 63.9%, respectively. FATE’s zero-shot accuracy (48.3%) is close to DAM’s (47.8%) and below those of these later supervised methods.

[1] Tian et al., “Audio-Visual Event Localization in Unconstrained Videos”, ECCV 2018.

[2] Andrew et al., “Deep Canonical Correlation Analysis”, ICML 2013.

[3] Wu et al., “Dual Attention Matching for Audio-Visual Event Localization”, ICCV 2019.

[4] Wang et al., “Multi-Modulation Network for Audio-Visual Event Localization”, arXiv 2021.

[5] Liu et al., “Dense Modality Interaction Network for Audio-Visual Event Localization”, IEEE TMM 2023.

[6] Sun et al., “Listen With Seeing: Cross-Modal Contrastive Learning for Audio-Visual Event Localization”, IEEE TMM 2025.

AuthorsOpenReview ↗

Official Response to Reviewer cSZz [2/2]

Q5. Frozen features and semantic-only training.

We scored frozen PE-AV-small features with Eq. 3 and trained pooled and frame-level models using only the semantic loss. Each cell reports intra-video R@3 (%) as V2A / A2V.

SettingAVSync-15 pooledAVSync-15 frame-levelVGG-Sync pooledVGG-Sync frame-level
Frozen PE-AV-small18.00 / 17.3314.89 / 17.7820.28 / 17.9318.21 / 18.38
Semantic-only training19.33 / 24.4418.89 / 24.8921.12 / 20.2821.51 / 19.66

Neither control shows a consistent advantage for frame-level scoring. Isolating the contribution of temporal supervision would require a joint-loss run with the same data and training schedule.

Q6. Mean, max, and top-k aggregation

We report the original mean-aggregation results from Table 1 for reference. In the supplementary evaluation, we compared mean, max, and the mean of the top 25% of aligned frame scores using the same candidate windows. All values are R@3 (%).

DatasetAggregationIntra V2AIntra A2VInter V2AInter A2V
AVSync-15Mean55.7456.2234.1330.44
AVSync-15Top 25% mean51.5648.6730.8924.67
AVSync-15Max46.4436.2226.6720.00
VGG-SyncMean48.3744.8337.7631.54
VGG-SyncTop 25% mean43.6442.7536.4728.07
VGG-SyncMax41.0337.3129.3021.34

Mean aggregation performed best in all eight R@3 comparisons. Averaging rewards agreement across the window, while max can give a mismatched clip a high score based on a single frame.

Please let us know if you have any further questions. We are happy to discuss. :) Best regards

Thank you very much for you rebuttal, my concerns have been addressed. Since my confidence to this paper is not very high, I will keep my original positive assessment and will not consider raising to the highest score.

Reviewer_gMzhOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces FATE, an audio-visual representation method that captures both semantic correspondence and temporal synchronization. Built on PE-AV, FATE retains frame-level features, aligns audio and video sequences through nearest-neighbor resampling, and combines cross-video semantic contrastive learning with within-video soft-label supervision based on temporal offsets. The authors evaluate the method on temporal retrieval, zero-shot event localization, and generated audio-video evaluation, reporting substantial retrieval gains and demonstrating the benefit of retaining temporal structure through ablations using the same backbone.

Critical strengths
  1. The retrieval gains are substantial and supported by controlled comparisons using the same backbone. In Table 1, FATE improves V2A R@3 on AVSync-15 from Synchformer’s 38.89 to 55.74 for intra-video retrieval and from 13.56 to 34.13 for inter-video retrieval. More convincingly, the global-pooling variant achieves 29.11 and 19.33 under the same backbone, training data, LoRA configuration, and training budget. This comparison supports the benefit of retaining temporal structure under the evaluated settings, rather than attributing the gains solely to a stronger pretrained model.
  2. The evaluation covers several applications and includes a sizeable human study. The study evaluates 750 generated videos with ten ratings per video, alongside localization transfer experiments and loss ablations. This provides a broader assessment than retrieval results alone. Although parts of the evaluation protocol need clarification, the range of experiments and the human judgments remain valuable.
Critical weaknesses
  1. Dependence on the backbone configuration remains untested. All experiments use PE-AV-small. The paper does not evaluate whether the proposed frame-level alignment and joint training strategy also benefit other model sizes or variants within the same family.
  2. The generation evaluation protocol is inconsistent across sections. Section 4.2 defines model-ranking correlation by averaging scores over the 150 videos from each generator and then computing a single Spearman correlation across the five generator-level averages. Section 5.3 and Table 4 instead compute a correlation across the five outputs for each prompt and average these correlations over prompts. These statistics answer different questions and are not interchangeable. Appendix A.3 also describes individual 1–5 ratings for each video, which are averaged to obtain MOS, whereas Section 5.3 states that annotators rank the five outputs within each prompt and reports a “cross-prompt relative-rank correlation.” If MOS values are converted to within-prompt ranks, the paper should specify this conversion, the treatment of ties, and the subsequent correlation calculation. As written, it is unclear whether the reported value of 0.1724 reflects agreement with video-level MOS or with relative rankings within prompts.
  3. The temporal evaluation is limited to a single window and stride configuration. Training and the main retrieval experiments both use 2-second windows with a 0.5-second stride. These experiments demonstrate discrimination between clips on a half-second grid, but do not establish synchronization accuracy at finer temporal resolutions. The average peak in Figure 3 also does not rule out the possibility that the model relies primarily on changes in window content or temporal overlap.
  4. The methodological contribution appears incremental. The individual components are relatively straightforward, and fine-grained audio-visual alignment has been explored in prior work. The controlled backbone ablations support the effectiveness of the proposed combination, but the paper does not sufficiently establish what it contributes beyond existing temporal representation methods.
Questions for authors
  1. Does FATE also improve performance on other PE-AV sizes or variants? How sensitive are the gains to the backbone’s temporal token resolution?
  2. What distinguishes FATE from prior fine-grained audio-visual alignment methods, particularly CAV-MAE Sync? Which experiments isolate this contribution beyond the choice of pretrained backbone?
5 comments
AuthorsOpenReview ↗

Official Response to Reviewer gMzh [1/2]

Thank you very much for taking the time to review and for your support. We try our best to address your questions as follows.

W1: Dependence on backbone configuration

Thank you for this suggestion. We extended FATE to PE-AV Base, a larger variant within the same family, and ImageBind, a different dual-encoder backbone. Both use frame-level alignment and joint semantic–temporal training. The table reports R@3 (%) before → after fine-tuning, using frame-level scoring in both cases.

DatasetBackboneIntra V2AIntra A2VInter V2AInter A2V
AVSync-15PE-AV-small18.00 → 55.7417.78 → 56.227.56 → 34.138.67 → 30.44
AVSync-15PE-AV Base20.22 → 56.6719.62 → 58.1112.00 → 38.339.33 → 36.10
AVSync-15Frame-level ImageBind16.44 → 45.7814.12 → 44.340.67 → 33.110.44 → 28.23
VGG-SyncPE-AV-small20.28 → 48.3717.93 → 44.8310.70 → 37.767.11 → 31.54
VGG-SyncPE-AV Base21.59 → 48.4618.32 → 46.0512.90 → 39.337.68 → 34.25
VGG-SyncFrame-level ImageBind19.19 → 43.5320.76 → 43.587.73 → 30.287.90 → 33.28

Both additional backbones improve across all reported retrieval settings. We will include these results in the revised manuscript. We apologize that GPUs are currently unavailable and plan to evaluate more backbone variants when resources become available.

W2: Clarification of the generation-evaluation protocol

We apologize for the inconsistent descriptions. Table 4 uses human relative rankings, not absolute MOS.

For each prompt, annotators ranked the five generated videos from 1 (worst) to 5 (best) in audio-visual synchronization, without ties. We averaged the available ranks from nine retained annotators for each video. The final analysis contains 150 valid prompts.

Both statistics use Spearman correlation between raw metric scores and negated mean human ranks, so higher values consistently indicate better synchronization:

  • Within-prompt (0.4441): compute the correlation across the five outputs of each prompt, then average over the 150 prompts.
  • Cross-prompt (0.1724): compute the correlation across the 150 outputs of each generator, then average over the five generators.

Ties in the aggregated human ranks receive average ranks when computing Spearman correlation. No MOS-to-rank conversion is involved. The value 0.1724 therefore measures cross-prompt agreement with within-prompt human rankings. The correlation across five generator-level averages described in Sec. 4.2 was not used for Table 4. We will correct Secs. 4.2–5.3 and Appendix A.3 to consistently describe this protocol.

AuthorsOpenReview ↗

Official Response to Reviewer gMzh [2/2]

W3. Finer temporal resolution and window-content effects

We ran an additional test on 16 AVSync-15 videos using 2-second windows and offset grids spaced at 0.25 s and 0.1 s. Sliding the audio window changes which samples it contains. Circular shifts keep the same samples but move them in time. The table shows how often the model ranked zero offset first on the 0.1-s grid (Hit@1, %).

Audio constructionFATEPooledChance
Sliding window7.0319.539.09
Circular shift44.3825.009.09

FATE outperformed the pooled baseline under circular shifts, where the wraparound boundary may provide an extra cue. With sliding windows, FATE fell below chance on this subset. We will report both conditions and clarify that exact localization on the 0.1-s grid remains difficult for FATE.

W4 & Q2. FATE’s contribution beyond prior work

CAV-MAE Sync already uses diagonal token matching at retrieval (its Eq. 5, our Eq. 3). We will credit this explicitly in Related Work and narrow the Abstract’s claim. During training, CAV-MAE Sync uses cross-video contrast with one frame and a 4-second audio window per video. CAVP uses within-video contrast after temporal pooling. FATE combines frame-level matching with within-video contrast on 2-second windows sampled every 0.5 seconds. Raised-cosine targets reflect the overlap between windows. LoRA adaptation preserves independent audio and video indexing, while Synchformer scores each pair jointly.

Backbone-controlled ablation. All variants use PE-AV-small. Adapted variants share the same VGGSound split and LoRA budget.

VariantScoringCross-video lossWithin-video targetsAVSync-15 inter-video R@3 (V2A / A2V)AVE Avg.
Frozen PE-AVPooled––10.54 / 10.1231.8
Pooled joint contrastPooled✓Soft19.33 / 20.4435.6
Cross-video contrast onlyFrame-level✓–19.78 / 11.46–
Joint contrast, hard targetsFrame-level✓Hard28.50 / 24.35–
FATEFrame-level✓Soft34.13 / 30.4448.3

With frame-level scoring fixed, adding within-video contrast improves retrieval in both directions. Soft targets add a further 5.63 / 6.09 R@3 points over hard targets. With soft targets fixed, replacing pooling with frame-level scoring raises AVE from 35.6 to 48.3.

Please let us know if you have any further questions. We are happy to discuss. :) Best regards.

Thank you for the additional experiments and clarifications. The PE-AV Base and ImageBind results substantially address my backbone-generalization concern, although sensitivity to temporal token resolution remains unclear. The controlled ablations also support the effectiveness of the proposed combination.

Two concerns remain. First, the response defines human ranks as 1 = worst and 5 = best, but then correlates metric scores with negated mean ranks. For FATE, whose higher scores indicate better alignment, these conventions appear inconsistent. Please clarify the actual implementation and the exclusion of one annotator.

Second, the 0.1-second sliding-window results do not establish reliable fine-grained synchronization, while circular shifts may introduce boundary cues. Please narrow the claims accordingly and clarify whether the CAV-MAE Sync baseline uses its native fine-grained retrieval scoring.

I currently retain my rating of 2. If the authors can provide further clarification on the remaining issues, I would be willing to raise my score.

Reviewer_NsSMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FATE, a frame-level audio-visual embedding framework designed to jointly model semantic correspondence and temporal synchronization between audio and video. The method aligns audio and video features along the physical timeline and combines semantic contrastive learning with temporal soft-contrastive learning to capture fine-grained cross-modal relationships. Experiments on cross-modal retrieval, zero-shot event localization, and audio-visual generation evaluation demonstrate the effectiveness of the proposed approach. Overall, the paper addresses a relevant research problem with a reasonable methodological design, although several aspects could benefit from further clarification.

Critical strengths

1.The paper addresses the difficulty of existing audio-visual embedding methods in simultaneously capturing semantic correspondence and fine-grained temporal synchronization, making the research problem relevant and well motivated. 2.The framework preserves frame-level features and combines semantic contrastive learning with temporal soft-contrastive learning to model fine-grained audio-visual relationships.

Critical weaknesses

1.Reporting statistical significance or confidence intervals for key results would help clarify whether the observed performance differences between methods are robust. 2.Further analysis of the nearest-neighbor temporal alignment strategy under different video frame rates and temporal resolutions would strengthen the evaluation. 3.The experiments cover a relatively small number of generative models, and further discussion of the method's generalizability across different types of generative models would help clarify the scope of the conclusions. 4.Further discussion of the metric's performance across different synchronization quality levels and its potential discrepancies with human judgments would help clarify its suitability and limitations as an automatic evaluation metric.

Questions for authors

See Weaknesses

1 comment
AuthorsOpenReview ↗

Official Response to Reviewer NsSM

Thank you very much for taking the time to review and for your support. We try our best to address your questions as follows.

W1: Statistical significance / confidence intervals

Thank you for this suggestion. We computed 95% bootstrap confidence intervals with 10,000 resamples, using source videos as the resampling unit for retrieval and prompts for generation evaluation. For generation evaluation, FATE achieves a within-prompt correlation of 0.444 [0.379, 0.506], with a statistically significant improvement of 0.194 [0.096, 0.291] over AV-Align. Its cross-prompt correlation is 0.172 [0.112, 0.231]. We will include these confidence intervals in the revised manuscript.

W2: Temporal alignment under varying frame rates and token resolutions

Thank you for this suggestion. We conducted an additional evaluation of nearest-neighbor temporal alignment by varying the video input frame rate and alignment-grid resolution while keeping the checkpoint fixed. Results are intra-video R@3 (%) on all 150 AVSync-15 queries.

Video inputAlignment grid (tokens/s)V2A R@3A2V R@3
Native frame rate25.055.7456.22
15 FPS25.049.5643.11
8 FPS25.035.7834.00
Native frame rate12.554.4443.78
Native frame rate6.050.1241.73

Halving the alignment grid from 25 to 12.5 tokens/s largely preserves performance, whereas reducing the video input to 8 FPS substantially lowers retrieval accuracy. These results highlight the importance of sufficiently dense visual sampling and show that a moderately coarser alignment grid remains effective.

W3: Generalizability across generative models

Thank you for this suggestion. Our evaluation covers five joint audio-video generators: BridgeDiT, JavisDiT, Ovi, LTX-2, and JointDiT. FATE evaluates the generated audio-video pairs without requiring access to the generator’s architecture or internal features. We will expand the discussion of differences among these generators and their synchronization errors, and clarify the scope of our conclusions. Evaluation across additional generator families is an important direction for future work.

W4: Synchronization quality levels and agreement with human judgments

Thank you for this suggestion. In the controlled temporal-shift analysis (Fig. 3), FATE’s score peaks at zero offset and decreases as the offset grows, demonstrating a graded response to temporal misalignment. For generated videos, FATE agrees more closely with human judgments within prompts (ρ = 0.444) than across prompts (ρ = 0.172), supporting its use for comparing outputs under the same prompt. Potential discrepancies arise when brief local errors are perceptually salient: averaging similarities across frames can reduce their influence on the overall score. We will expand this discussion to explain how temporal resolution and score aggregation affect sensitivity to local synchronization errors, and position FATE as an automatic comparison metric that complements human evaluation.

Please let us know if you have any further questions. We are happy to discuss. :) Best regards,

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.