acceptodds
Under review as a conference paper at ICLR 2027

Ambiguity-Aware Temporal Transport for Audio-Visual Deepfake Detection

Abstract

Learning audio-visual consistency from authentic videos reduces dependence on specific manipulation patterns in deepfake detection. However, sustained or recurring visual states may produce similar matching responses across neighboring audio segments, leaving local correspondences ambiguous. Under such uncertainty, local matching strength may become less reliable as detection evidence. This motivates evaluating both correspondence informativeness and the temporal coherence of matching evidence. We propose Ambiguity-Aware Temporal Transport (AATT), a detection framework that jointly models these two properties. AATT estimates correspondence informativeness from the normalized entropy of local matching distributions to weight evidence for temporal inference. The weighted evidence is integrated across time through marginalization over monotone paths, yielding soft correspondences between the two modalities. These correspondences guide the aggregation of audio features onto the visual timeline for an informativeness-weighted comparison with visual features. The detection score combines the resulting transport compatibility and mean correspondence informativeness with path surprisal, which measures the cost of imposing temporal constraints. The framework learns exclusively from authentic talking videos through a ranking objective that distinguishes matched audio-visual pairs from cross-video mismatches. Experiments demonstrate substantial improvements on talking benchmarks, a pronounced advantage in the more challenging singing setting, and robustness to different perturbations.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Audio-visual deepfake detection, Correspondence ambiguity, Temporal alignment, Audio-visual consistency

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xe7eOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes the Fuzzy Aware Temporal Transport (AATT) framework for deepfake detection in audio and video, which aims to address the problem of fuzzy local correspondence in real audio and video and the inaccurate reflection of cross-modal consistency by traditional matching scores. Specifically, the authors utilize normalized entropy to estimate the amount of locally corresponding information and achieve global temporal alignment by probabilistically marginalizing monotonic temporal paths. Furthermore, the method combines transport compatibility, path unpredictability, and average correspondence information content for forgery detection. The method is trained by using only real video samples to build both matched and unmatched samples, eliminating the need for fake samples. Experiments on four datasets—AVLips, FakeAVCeleb, THB, and SHDF—show that the proposed method achieves excellent performance in terms of detection accuracy, scene generalization ability, and robustness.

Critical strengths

This paper addresses a specific problem: the ambiguity of local correspondences in real-world audio and video reduces the reliability of traditional matching methods. This paper combines correspondence information with temporal consistency modeling to construct a relatively complete detection framework. Experimental results are remarkable: on multiple audio and video forgery detection datasets, the proposed method significantly outperforms existing methods, especially demonstrating good generalization ability when transitioning from speech scenes to singing scenes. Ablation experiments and related robustness analyses further validate the effectiveness of the proposed module.

Critical weaknesses
  • The proposed method offers limited innovation and lacks sufficient comparison with existing time alignment methods. This paper primarily combines existing techniques such as entropy-based uncertainty estimation, monotonic dynamic path planning, and cross-feature aggregation. While the combination of these components offers some advantages, the substantial innovation of its core algorithm compared to methods like Soft-DTW, existing dynamic programming, and attention matching remains unclear. Existing exclusion experiments are limited to verifying the role of each module and are insufficient to demonstrate that the proposed time transfer mechanism outperforms other existing time alignment methods.

  • The lack of validation in simulated scenarios at key time points casts doubt on the effectiveness of the main mechanism. Although this paper focuses on the temporal correspondence between audio and video, robustness experiments primarily concentrate on spatial perturbations such as image scaling, noise, compression, and blurring, without a systematic analysis of audio delay, local temporal errors, and non-uniform temporal variations. Furthermore, the proposed monotonic time transfer scheme allows for correction of actual asynchronous audio and video behavior through reordering within a specific time compensation range, but this may weaken the ability to detect communication fraud. These potential issues have not been adequately discussed or validated.

Questions for authors
  • What are the main advantages of AATT compared to standard Soft-DTW, monotonic dynamic programming, or attention-based temporal alignment methods? Is it possible to provide a more direct comparison showing that, under the same feature extractor and training environment, the improvement in performance is not primarily due to overall temporal alignment?

  • How does AATT handle fixed audio latency, local time shift, and time scaling? Could a transmission mechanism that allows for local time shift reduce detection sensitivity by reordering initially asynchronous audio and video?

  • In this paper, we calculate the correspondence information using the softmax entropy of the standard matching score. Does query-specific standardization lead to a loss of absolute confidence information from the original matching score? How can we show that this metric can distinguish between true correspondence ambiguities and highly concentrated but false matches?

  • In the training phase, negative samples are generated using audio between videos, which may simultaneously contain differences in the speaker, speech content, and recording environment. How do the authors show that this model primarily learns temporal consistency between audio and video rather than differences between other sensory information? Can similar detection performance be maintained even when using temporally out-of-time portions of the same video or different audio clips from the same speaker as hard negative samples?

4 comments
AuthorsOpenReview ↗

Response to Reviewer xe7e (1/2)

We sincerely thank you for your careful review and for recognizing the importance of the research problem, the strong detection performance, and the cross-scenario generalization of AATT. We address your comments point by point below. In the revised draft, we mark our major revisions in blue.


[Q1, W1] Advantages over existing alignment methods and controlled comparisons of gains beyond temporal alignment.

AATT incorporates correspondence ambiguity into both temporal inference and deepfake detection. Compared with Soft-DTW, AATT explicitly downweights ambiguous matching evidence and uses path surprisal to assess how well that evidence supports temporally admissible paths. Compared with the monotonic dynamic-programming baseline, AATT marginalizes over admissible paths rather than selecting the highest-scoring path. Compared with independent local attention, AATT infers correspondences jointly under temporal constraints.

For a direct comparison, we replace the alignment mechanism within AATT under the same feature extractor and training settings. All three controls retain the original computation of SS and all three detection descriptors. AATT outperforms all three alternatives on SHDF and THB, as shown in the table below. Moreover, Sec. 4.3 and Table 4 show that removing informativeness weighting causes a marked performance drop even when structured temporal alignment is retained. This demonstrates that temporal alignment alone cannot account for the performance of AATT, highlighting the contribution of ambiguity-aware evidence weighting.

Alignment mechanismSHDF AP (%)SHDF AUC (%)THB AP (%)THB AUC (%)
Soft-DTW91.886.697.598.1
Monotonic dynamic programming91.084.897.598.4
Independent local softmax attention91.585.597.397.8
AATT93.288.198.999.3

[Q2, W2] Fixed audio latency, local shifts, time scaling, and detection sensitivity after soft alignment.

For fixed audio latency, we evaluate positive and negative audio-visual offsets on SHDF. As shown below, AATT outperforms both baselines under both conditions, supporting its robustness to the fixed audio latency.

OffsetAVH-Align AP / AUC (%)T-AVFD AP / AUC (%)AATT AP / AUC (%)
−80 ms57.1 / 41.586.4 / 81.293.0 / 87.8
+80 ms54.8 / 36.384.6 / 78.593.5 / 88.6

For local time shifts and time scaling, we temporally resample extracted audio features while keeping the visual sequence unchanged. Gradually varying offsets simulate relative time scaling, while abrupt changes introduce local temporal discontinuities. AATT maintains strong detection performance under both conditions on SHDF, indicating that these time-varying misalignments do not substantially compromise detection performance.

Regarding detection sensitivity, soft alignment in AATT is constrained by local windows and monotonic paths. Its detection score also includes path surprisal SS, which evaluates how well the pre-transport matching evidence supports temporal constraints (Eq. 9). Thus, detection does not rely solely on feature similarity after alignment. Abrupt offsets increase mean anomaly scores for both authentic and manipulated videos, showing that temporal discontinuities remain reflected in the final detection scores after soft alignment (see below).

Scores are mean D×100D\times100. All three conditions use the same temporal crop.

ConditionAuthentic scoreManipulated scoreAP / AUC (%)
Matched reference328.4393.493.2 / 88.1
Gradual temporal drift330.8394.193.1 / 88.0
Abrupt temporal offsets342.1396.592.7 / 87.4
AuthorsOpenReview ↗

Response to Reviewer xe7e (2/2)

[Q3-1] Effect of query-wise standardization on absolute matching information.

Query-wise standardization removes the absolute score level and largely removes score-scale differences from the computation of rr by design (Eqs. 2-3). This allows rr to characterize relative preference concentration rather than absolute matching strength. Feature similarity remains available through transport compatibility CC, which directly compares visual and transported audio features without query-wise score standardization (Eq. 10). Under the same training protocol, query-wise normalized entropy outperforms fixed-scale entropy on both datasets (see below), supporting the benefit of reducing query-dependent score-scale effects when estimating correspondence informativeness.

Weighting schemeSHDF AP (%)SHDF AUC (%)THB AP (%)THB AUC (%)
Fixed-scale entropy91.886.098.398.9
AATT93.288.198.999.3

[Q3-2] Distinguishing correspondence ambiguity from concentrated false matches.

The entropy-based informativeness rr distinguishes diffuse from concentrated preferences, but does not by itself establish whether a match is correct. A highly concentrated false match can therefore receive high rr. AATT assesses consistency jointly through CC, SS, and II, rather than treating high informativeness as sufficient evidence of correct correspondence. In particular, SS evaluates whether local preferences jointly support temporally admissible paths (Eq. 9), providing structural evidence beyond local concentration.

We examine this additional temporal evidence through a paired control on matching responses from authentic AVLips and SHDF videos. We rearrange candidate matching responses while keeping the score distribution of each query and the path constraints unchanged. With local entropy and informativeness unchanged, temporal rearrangement increases mean SS on both datasets, as shown below. This demonstrates that SS provides additional temporal evidence beyond local preference concentration.

DatasetOriginal S×100S\times100Rearranged S×100S\times100
AVLips158.8235.6
SHDF215.7230.7

[Q4-1] Evidence that detection uses audio-visual temporal consistency.

AATT builds on frozen AV-HuBERT representations whose pretraining captures speech-lip correlations and temporal dependencies, providing a basis for learning audio-visual temporal consistency. In the abrupt temporal-offset experiment in [Q2, W2], shifting audio features within each video increases mean anomaly scores while keeping the recording source unchanged. This provides supporting evidence that the original model uses temporal correspondence for detection.


[Q4-2] Detection performance with same-recording hard negatives.

We further examine this temporal correspondence during training by introducing temporally separated, non-overlapping audio segments from the same recording as hard negatives. Under the same feature extractor and training protocol, the retrained model maintains comparable detection performance across all four benchmarks, as shown below. Together, the within-video temporal intervention and the hard-negative training results support audio-visual temporal consistency as an important basis for detection by AATT.

DatasetOriginal AATT AP / AUC (%)AATT with same-recording hard negatives AP / AUC (%)
AVLips98.9 / 98.898.5 / 99.1
FKAV98.8 / 98.298.8 / 98.1
THB98.9 / 99.398.4 / 98.8
SHDF93.2 / 88.192.9 / 87.6

Thank you for the explanation. I believe the basic questions have been answered quite clearly, and the article is of good writing quality. Considering the content of the article and the scores from other reviewers, I will maintain my original score and support the article's acceptance. Good luck!

AuthorsOpenReview ↗

Response to Reviewer xe7e

Dear Reviewer xe7e,

We are glad that our responses have addressed your concerns. We sincerely appreciate your positive assessment and support for our work. If you have any further questions or suggestions, we would be happy to address them.

Best regards,

Authors

Reviewer_RitCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

To address the ambiguity in local correspondences during deep pseudo-detection in audio and video, this paper proposes the Ambiguity-Aware Temporal Transmission Framework (AATT). AATT incorporates the information content and temporal coherence of correspondences into audio-video consistency modeling and combines cross-modal transmission compatibility, path surprise, and average correspondence information content to calculate the final detection score. This framework is trained using only real speaking videos and its effectiveness has been validated on multiple speaking and singing video benchmarks.

Critical strengths
  1. The motivation is reasonable and the methodology is clearly designed. It addresses local audio-visual correspondence ambiguity by jointly modeling the corresponding information content and temporal coherence, demonstrating clear targeting.

  2. It exhibits strong detection performance and cross-scene generalization. Trained using only real speaking videos, it achieves significant improvements on multiple speaking and singing video benchmarks.

  3. It balances robustness and computational efficiency. It maintains good performance under various visual perturbations and significantly reduces computational cost and inference latency compared to T-AVFD.

Critical weaknesses
  1. Insufficient integration between ambiguity measurement and motivation analysis. Motivation analysis uses a fixed scale, while the method employs query-by-query standardization; it has not yet been verified whether both identify the same corresponding ambiguities.

  2. Motivation validation mainly focuses on relevance. Highly ambiguous samples have larger detection errors, failing to sufficiently demonstrate that AATT's improvement stems from ambiguity handling; targeted controlled experiments are lacking.

  3. The role of path surprise is unclear. This metric is affected by both local distribution concentration and path constraints; experiments are lacking to verify its independent characterization of temporal discontinuity.

  4. Ablation does not adequately isolate component contributions. Some experiments simultaneously remove multiple designs, and there is a lack of pairwise combinations of descriptors, making it difficult to distinguish the independent contributions of information weighting, path inference, and each detection descriptor.

  5. Lack of validation for behavior under temporal mismatches. Current perturbation experiments primarily target visual degradation; it is unclear whether soft alignment corrects anomalous mismatches, thereby weakening detection evidence.

Questions for authors

Please refer to the weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer RitC (1/2)

We sincerely thank you for your careful review and for recognizing the clear motivation and design, strong cross-scenario detection performance, as well as the robustness and computational efficiency of AATT. We address your comments point by point below. In the revised draft, we mark our major revisions in blue.


[Q1] Agreement between fixed-scale ambiguity measurement and query-wise normalized entropy.

We compute Spearman correlations across the same videos, comparing their rankings under the fixed-scale ambiguity measure in Fig. 1 and the query-wise normalized entropy used by AATT. The resulting correlations (ρ=0.79\rho=0.79 on AVLips and 0.640.64 on SHDF) indicate that videos with higher ambiguity in Fig. 1 also tend to have higher AATT entropy.

Keeping the original Fig. 1 groups unchanged, the table below further shows higher AATT entropy in the higher-ambiguity group on both datasets. Thus, both the video-level correlations and the group-level differences support agreement between the ambiguity identified in the motivation analysis and that measured by AATT.

DatasetAATT entropy ×100: Lower groupAATT entropy ×100: Higher group
AVLips60.269.2
SHDF84.288.1

[Q2] Controlled evaluation of the contribution of ambiguity handling.

Informativeness weighting reduces the influence of ambiguous local evidence. To verify its contribution, we compare AATT with a variant without informativeness weighting while retaining the other components. Using the original ambiguity groups and ranking error metric from Fig. 1, we find that removing weighting increases ranking error more substantially in higher-ambiguity groups on both datasets. As shown in the table below, these results support the contribution of informativeness weighting to handling correspondence ambiguity, beyond merely showing an association between ambiguity and detection difficulty.

DatasetFig. 1 ambiguity groupRanking error w/o weighting (%)Ranking error AATT (%)
AVLipsLower0.00.0
AVLipsHigher6.52.4
SHDFLower9.08.2
SHDFHigher34.716.6

[Q3] Role of path surprisal in capturing temporal discontinuity beyond local matching concentration.

We conduct a paired control experiment on AVLips and SHDF videos, rearranging candidate matching responses while keeping local score distributions and path constraints unchanged. As shown in the table below, temporal rearrangement increases mean path surprisal SS, reflecting weaker joint support for temporally admissible paths (Eq. 9). This demonstrates that SS captures temporal discontinuity beyond local matching concentration.

DatasetOriginal S×100S\times100Rearranged S×100S\times100
AVLips158.8235.6
SHDF215.7230.7

[Q4] Isolating the contributions of information weighting, path inference, and detection descriptors.

Sec. 4.3 and Table 4 show that removing information weighting leads to a marked performance drop even when II and structured matching are retained, demonstrating the contribution of weighting itself.

We replace path-based structured matching with independent matching while retaining the original computation of path surprisal SS and all three detection descriptors. Structured matching (AATT) achieves better performance on both datasets, supporting the contribution of path-based inference.

Keeping information weighting and structured matching unchanged, we evaluate all three pairwise descriptor combinations. In the table below, the full combination (AATT) outperforms every pairwise combination on both datasets, showing that each descriptor provides additional value when the other two are present.

VariantSHDF AP (%)SHDF AUC (%)THB AP (%)THB AUC (%)
Independent matching, retaining SS91.585.597.397.8
C&S91.386.598.198.3
C&I90.384.097.798.3
S&I76.269.594.997.1
AATT93.288.198.999.3
AuthorsOpenReview ↗

Response to Reviewer RitC (2/2)

[Q5] Detection performance and sensitivity to temporal mismatches after soft alignment.

To complement the visual perturbation experiments, we evaluate audio-visual offsets on SHDF. The table below shows that AATT maintains strong detection performance and outperforms both baselines under both offsets, demonstrating its robustness to temporal misalignments.

OffsetAVH-Align AP / AUC (%)T-AVFD AP / AUC (%)AATT AP / AUC (%)
−80 ms57.1 / 41.586.4 / 81.293.0 / 87.8
+80 ms54.8 / 36.384.6 / 78.593.5 / 88.6

We further introduce feature-level temporal discontinuities on SHDF by switching the audio-feature offset from −8 to +8 steps at the clip midpoint. AATT allows small temporal variations, but soft alignment is constrained by local windows and monotonic paths. Moreover, path surprisal SS evaluates temporal compatibility before feature transport, so alignment does not necessarily eliminate temporal mismatch evidence. Abrupt offsets increase mean anomaly scores for both authentic and manipulated videos, while detection performance remains relatively stable (see below). These results indicate that AATT remains sensitive to anomalous temporal mismatches after soft alignment, while largely preserving its detection performance.

Scores are mean D×100D\times100. Both conditions use the same temporal crop.

ConditionAuthentic scoreManipulated scoreAP / AUC (%)
Matched reference328.4393.493.2 / 88.1
Abrupt offsets342.1396.592.7 / 87.4
AuthorsOpenReview ↗

Follow-up on Author Responses

Dear Reviewer RitC,

Thank you very much for your time and constructive feedback. As the discussion period draws to a close, we hope our responses and revisions have addressed your concerns.

Thank you again for your careful consideration.

Best regards,

Authors

Reviewer_18acOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

AATT detects deepfakes by learning how audio and video match in authentic recordings. The authors keep AV-HuBERT frozen and train a shared linear adapter and a small classifier. At each video timestep, the model compares nearby audio features. It gives more weight to matches with lower uncertainty, then combines possible alignments under local timing constraints. The classifier uses three scores: how well the aligned features match, how informative the local matches are, and how hard it is to satisfy the timing constraints. During training, audio and video from the same recording should score above pairs from different recordings.

Critical strengths
  1. The authors retrain AVH-Align and T-AVFD on the same authentic talking videos. AATT does better on all four benchmarks. The 7.9-point AUC gain over T-AVFD on SHDF is especially interesting because the training set contains no singing. AATT also does better when the authors reverse the transfer direction. I found these comparisons convincing evidence of better transfer on the tested datasets.

  2. The authors test several parts of the method separately. Without the uncertainty weights, SHDF AUC falls from 88.1 to 78.4. Independent matching reaches 82.6, and a frozen adapter gives 75.5. These are large differences. The authors train only 1.05M parameters. Table 3 reports 61.35 ms per clip, compared with 724.81 for T-AVFD and 33.32 for AVH-Align. The reported runtime is much lower than T-AVFD's, although AVH-Align is faster.

Critical weaknesses
  1. I would like a more direct check of the uncertainty weight. Equation 2 divides each query's scores by their own standard deviation. Unless that deviation is close to epsilon, scaling the differences between scores leaves the normalized scores almost unchanged. A weak preference can therefore get a similar weight to a much stronger preference with the same pattern. Figure 1 uses a fixed scale instead (Appendix A.2.2), so it does not directly test the weight used during detection. The drop in AUC without the weights shows that they help, but I am not sure why. A comparison with fixed-scale entropy or weights based on the score margin or variance would help. The independent-matching experiment is also hard to interpret because it removes both structured alignment and the path-surprisal score.

  2. The robustness tests cover image quality but leave out several changes to timing and audio. Table 6 tests resizing, pixel noise, JPEG compression, and blur. What happens with a small audio-video offset, a change in frame rate, dropped frames, or corrupted audio? Those changes can occur in genuine recordings too. AVH-Align also reports a shortcut involving leading silence in FakeAVCeleb, so I would want to see results after trimming that silence. Results for each forgery method and each changed modality would help show where AATT works well, especially when fake audio and video remain synchronized. The identity splits and the data used to choose hyperparameters also need more detail. These questions matter for the broader claim of generalization. AVH-Align's shortcut analysis.

Questions for authors
  1. Does the entropy after normalization actually track alignment errors or ambiguous matches? Have you compared it with fixed-scale entropy, or with weights based on the score margin or variance? I would like to understand what the normalization adds.

  2. Could you give more detail on the video and identity splits, encoder checkpoint, training duration, and how you chose the final checkpoint? How was the temporal radius chosen, and did that use any SHDF test labels? For the runtime comparison, what were the clip length and hardware, and was preprocessing included?

  3. Does AATT still do better than the baselines after trimming leading silence or adding small timing offsets and frame-rate changes? I would also be interested in results for each forgery method, including cases where the fake audio and video are well synchronized.

  4. Could you compare independent and structured matching while keeping the same path-surprisal score in both? The current experiment changes both at once, so I could not tell how much of the gain comes from the alignment itself.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 18ac (1/2)

We sincerely thank you for your careful review and for recognizing the strong cross-dataset and cross-scenario generalization, contributions of the key components, and computational efficiency of AATT. We address your comments point by point below. In the revised draft, we mark our major revisions in blue.


[Q1-1, W1] Meaning of query-wise normalized entropy and its relation to correspondence ambiguity.

The query-wise normalization in Eq. (2) allows the entropy-based correspondence informativeness rr (Eq. (3)) to characterize relative matching preferences rather than absolute matching strength. With a fixed softmax scale, increasing score differences reduces entropy even when the relative score pattern remains unchanged. Query-wise standardization intentionally makes the weights approximately invariant to positive score rescaling when the query standard deviation is large relative to ϵ\epsilon. Similar weights indicate similar concentration of relative preferences, not equal probabilities of correct alignment. Diffuse preferences receive lower weights in Eqs. (4) and (10), while transport compatibility CC and path surprisal SS provide complementary consistency evidence.

Since Fig. 1 uses fixed-scale T-AVFD responses, we additionally examine the actual normalized entropy used by AATT. We compute the video-averaged normalized entropy 100(1−I)100(1-I) while retaining the original T-AVFD-based ambiguity groups. The normalized entropy reflects correspondence ambiguity, with higher values in the original higher-ambiguity groups on both datasets, as shown below. Compared with the variant without informativeness weighting, AATT reduces ranking error by 4.1 and 18.1 in the higher-ambiguity groups of AVLips and SHDF, versus 0.0 and 0.8 in the lower-ambiguity groups. These results support the association between the actual informativeness measure and correspondence ambiguity. The larger detection gains in higher-ambiguity groups further support the benefit of downweighting ambiguous matching evidence.

DatasetAmbiguity groupNormalized entropy ×100Ranking error w/o weighting (%)Ranking error AATT (%)
AVLipsLower60.20.00.0
AVLipsHigher69.26.52.4
SHDFLower84.29.08.2
SHDFHigher88.134.716.6

[Q1-2, W1] Comparison with fixed-scale entropy, score-margin weighting, and variance-based weighting.

We independently train the three weighting variants under the same architecture and training protocol, changing only the computation of rr. Query-wise normalized entropy (AATT) outperforms all three alternatives on both datasets, as shown below. In particular, the comparison with fixed-scale entropy supports the benefit of query-wise normalization for estimating correspondence informativeness. Together with the ambiguity-group analysis, these results support the interpretation that informativeness weighting improves detection by reducing the influence of ambiguous matching evidence.

Weighting schemeSHDF AP (%)SHDF AUC (%)THB AP (%)THB AUC (%)
Fixed-scale entropy91.886.098.398.9
Score-margin weighting92.486.898.499.0
Variance-based weighting85.676.597.398.1
AATT93.288.198.999.3

[Q2-1, W2] Video and identity splits.

As described in Sec. 4.1 and Appendix A.2.3, AVLips is divided into training, validation, and test sets at a 6:1:3 ratio in the main experiments. We randomly split AVLips at the video level without overlap, rather than by identity. AATT, AVH-Align, and T-AVFD use the same authentic training videos and identical test sets for fair comparison. FKAV and THB are used to evaluate cross-dataset generalization, while SHDF evaluates talking-to-singing transfer. These external test datasets draw identities from sources different from those used for AVLips training.


[Q2-2] Encoder, training duration, and checkpoint selection.

We use the publicly available AV-HuBERT Large checkpoint (self_large_vox_433h.pt) and keep the encoders frozen. Training runs for 30 epochs on an NVIDIA A100 80GB GPU, taking approximately 40 minutes. The final checkpoint is selected based on the lowest ranking loss on authentic validation videos, without using test-set AP/AUC.


[Q2-3, W2] Choice of temporal radius and use of SHDF test labels.

We adopt τ=15\tau=15 following the default setting in the official AVH-Align implementation. Neither SHDF test labels nor test performance metrics were used to select τ\tau. Appendix A.3.2 and Table 12 report a subsequent sensitivity analysis, showing stable performance at neighboring radii.


[Q2-4] Runtime measurement and preprocessing.

All methods in Table 3 are evaluated under the same timing protocol using 2-second clips at 25 fps on an NVIDIA A100 80GB. Timing includes neural feature extraction and detection but excludes video decoding and face/mouth ROI preprocessing.

AuthorsOpenReview ↗

Response to Reviewer 18ac (2/2)

[Q3-1, W2] Detection performance after leading silence trimming and under temporal and audio perturbations.

All three methods maintain stable performance after removing leading silence from FKAV, with AATT achieving the best AP and AUC both before and after trimming. As shown below, these results indicate that the detection advantage of AATT persists after removing leading silence.

MethodBefore trimming AP / AUC (%)After trimming AP / AUC (%)
AVH-Align93.5 / 93.093.6 / 92.9
T-AVFD95.6 / 95.695.6 / 95.5
AATT98.8 / 98.298.8 / 98.2

AATT maintains stable performance under the audio-visual offsets, frame-rate changes, dropped frames, and audio noise, achieving the best performance across all conditions on SHDF and THB, as shown below. These results complement the image-quality perturbation experiments in the paper and further support the robustness of AATT to temporal and audio perturbations.

Clean denotes the standard evaluation. Offset conditions use matched temporal crops across methods.

DatasetConditionAVH-Align AUC (%)T-AVFD AUC (%)AATT AUC (%)
SHDFClean37.480.288.1
SHDFFrame rate: 25 → 20 → 25 fps39.478.988.1
SHDF5% frame drop-and-hold38.579.688.1
SHDFGaussian audio noise39.481.387.2
SHDFAudio-visual offset: −80 ms41.581.287.8
SHDFAudio-visual offset: +80 ms36.378.588.6
THBClean82.393.099.3
THBFrame rate: 25 → 20 → 25 fps64.992.099.0
THB5% frame drop-and-hold83.193.099.3
THBGaussian audio noise81.493.298.9
THBAudio-visual offset: −80 ms33.592.899.3
THBAudio-visual offset: +80 ms22.690.599.1

[Q3-2, W2] Performance across forgery methods and manipulated modalities, including well-synchronized forgeries.

As shown in the table below, AATT achieves the best performance across all three SHDF generators and three manipulation categories in our FKAV evaluation subset, demonstrating its effectiveness across different forgery types. Notably, the original SHDF study reports good lip synchronization for MEMO-generated videos, on which AATT maintains its detection advantage, supporting its effectiveness on well-synchronized forgeries.

DatasetGenerator / manipulation categoryAVH-Align AUC (%)T-AVFD AUC (%)AATT AUC (%)
SHDFEchoMimic32.380.796.2
SHDFHallo221.878.690.8
SHDFMEMO42.980.284.5
FKAVFake video + fake audio94.396.099.0
FKAVFake video + real audio90.894.997.0
FKAVReal video + fake audio95.694.996.9

[Q4, W1] Comparison of independent and structured matching with the same path-surprisal computation.

To isolate the contribution of structured alignment, we retrain an independent-matching variant while retaining the original computation of path surprisal SS. As shown below, structured matching achieves better performance in this controlled comparison, demonstrating its additional contribution beyond the path-surprisal score.

MatchingSHDF AP (%)SHDF AUC (%)THB AP (%)THB AUC (%)
Independent matching, retaining SS91.585.597.397.8
Structured matching (AATT)93.288.198.999.3
AuthorsOpenReview ↗

Follow-up on Author Responses

Dear Reviewer 18ac,

Thank you very much for your time and constructive feedback. As the discussion period draws to a close, we hope our responses and revisions have addressed your concerns.

Thank you again for your careful consideration.

Best regards,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Reviewers and AC,

We sincerely appreciate your valuable time and effort spent reviewing our manuscript.

We propose AATT, which jointly models correspondence informativeness and temporal coherence for audio-visual deepfake detection using only authentic training videos. We appreciate your recognition of our strong detection performance and cross-scenario generalization (18ac, RitC, xe7e), motivation to address local correspondence ambiguity (RitC, xe7e), and computational efficiency (18ac, RitC).

In response to the comments, we have carefully revised and enhanced the manuscript as follows:

  • Clarified query-wise normalization and added mechanism analyses of informativeness weighting and path surprisal (Sections 3.1, 3.3, and 4.4).
  • Added controlled alignment comparisons, alternative weighting schemes, and pairwise descriptor ablations (Section 4.3 and Appendix A.3.1).
  • Extended temporal and audio robustness evaluations, including leading-silence removal, and added generator- and modality-specific results (Appendices A.3.4-A.3.5).
  • Added training experiments with same-recording hard negatives (Appendix A.3.6).
  • Clarified data splits, training and checkpoint selection, temporal-radius selection, and runtime measurement (Appendices A.2.1, A.2.3, and A.3.2).

In the revised manuscript, these updates are highlighted in blue for ease of reference. We sincerely hope that these revisions help convey the benefits of AATT more clearly to the ICLR community.

Thank you again for your time and consideration.

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.