- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes the Fuzzy Aware Temporal Transport (AATT) framework for deepfake detection in audio and video, which aims to address the problem of fuzzy local correspondence in real audio and video and the inaccurate reflection of cross-modal consistency by traditional matching scores. Specifically, the authors utilize normalized entropy to estimate the amount of locally corresponding information and achieve global temporal alignment by probabilistically marginalizing monotonic temporal paths. Furthermore, the method combines transport compatibility, path unpredictability, and average correspondence information content for forgery detection. The method is trained by using only real video samples to build both matched and unmatched samples, eliminating the need for fake samples. Experiments on four datasets—AVLips, FakeAVCeleb, THB, and SHDF—show that the proposed method achieves excellent performance in terms of detection accuracy, scene generalization ability, and robustness.
Critical strengths
This paper addresses a specific problem: the ambiguity of local correspondences in real-world audio and video reduces the reliability of traditional matching methods. This paper combines correspondence information with temporal consistency modeling to construct a relatively complete detection framework. Experimental results are remarkable: on multiple audio and video forgery detection datasets, the proposed method significantly outperforms existing methods, especially demonstrating good generalization ability when transitioning from speech scenes to singing scenes. Ablation experiments and related robustness analyses further validate the effectiveness of the proposed module.
Critical weaknesses
-
The proposed method offers limited innovation and lacks sufficient comparison with existing time alignment methods. This paper primarily combines existing techniques such as entropy-based uncertainty estimation, monotonic dynamic path planning, and cross-feature aggregation. While the combination of these components offers some advantages, the substantial innovation of its core algorithm compared to methods like Soft-DTW, existing dynamic programming, and attention matching remains unclear. Existing exclusion experiments are limited to verifying the role of each module and are insufficient to demonstrate that the proposed time transfer mechanism outperforms other existing time alignment methods.
-
The lack of validation in simulated scenarios at key time points casts doubt on the effectiveness of the main mechanism. Although this paper focuses on the temporal correspondence between audio and video, robustness experiments primarily concentrate on spatial perturbations such as image scaling, noise, compression, and blurring, without a systematic analysis of audio delay, local temporal errors, and non-uniform temporal variations. Furthermore, the proposed monotonic time transfer scheme allows for correction of actual asynchronous audio and video behavior through reordering within a specific time compensation range, but this may weaken the ability to detect communication fraud. These potential issues have not been adequately discussed or validated.
Questions for authors
-
What are the main advantages of AATT compared to standard Soft-DTW, monotonic dynamic programming, or attention-based temporal alignment methods? Is it possible to provide a more direct comparison showing that, under the same feature extractor and training environment, the improvement in performance is not primarily due to overall temporal alignment?
-
How does AATT handle fixed audio latency, local time shift, and time scaling? Could a transmission mechanism that allows for local time shift reduce detection sensitivity by reordering initially asynchronous audio and video?
-
In this paper, we calculate the correspondence information using the softmax entropy of the standard matching score. Does query-specific standardization lead to a loss of absolute confidence information from the original matching score? How can we show that this metric can distinguish between true correspondence ambiguities and highly concentrated but false matches?
-
In the training phase, negative samples are generated using audio between videos, which may simultaneously contain differences in the speaker, speech content, and recording environment. How do the authors show that this model primarily learns temporal consistency between audio and video rather than differences between other sensory information? Can similar detection performance be maintained even when using temporally out-of-time portions of the same video or different audio clips from the same speaker as hard negative samples?
4 comments
Response to Reviewer xe7e (2/2)
[Q3-1] Effect of query-wise standardization on absolute matching information.
Query-wise standardization removes the absolute score level and largely removes score-scale differences from the computation of by design (Eqs. 2-3). This allows to characterize relative preference concentration rather than absolute matching strength. Feature similarity remains available through transport compatibility , which directly compares visual and transported audio features without query-wise score standardization (Eq. 10). Under the same training protocol, query-wise normalized entropy outperforms fixed-scale entropy on both datasets (see below), supporting the benefit of reducing query-dependent score-scale effects when estimating correspondence informativeness.
| Weighting scheme | SHDF AP (%) | SHDF AUC (%) | THB AP (%) | THB AUC (%) |
|---|---|---|---|---|
| Fixed-scale entropy | 91.8 | 86.0 | 98.3 | 98.9 |
| AATT | 93.2 | 88.1 | 98.9 | 99.3 |
[Q3-2] Distinguishing correspondence ambiguity from concentrated false matches.
The entropy-based informativeness distinguishes diffuse from concentrated preferences, but does not by itself establish whether a match is correct. A highly concentrated false match can therefore receive high . AATT assesses consistency jointly through , , and , rather than treating high informativeness as sufficient evidence of correct correspondence. In particular, evaluates whether local preferences jointly support temporally admissible paths (Eq. 9), providing structural evidence beyond local concentration.
We examine this additional temporal evidence through a paired control on matching responses from authentic AVLips and SHDF videos. We rearrange candidate matching responses while keeping the score distribution of each query and the path constraints unchanged. With local entropy and informativeness unchanged, temporal rearrangement increases mean on both datasets, as shown below. This demonstrates that provides additional temporal evidence beyond local preference concentration.
| Dataset | Original | Rearranged |
|---|---|---|
| AVLips | 158.8 | 235.6 |
| SHDF | 215.7 | 230.7 |
[Q4-1] Evidence that detection uses audio-visual temporal consistency.
AATT builds on frozen AV-HuBERT representations whose pretraining captures speech-lip correlations and temporal dependencies, providing a basis for learning audio-visual temporal consistency. In the abrupt temporal-offset experiment in [Q2, W2], shifting audio features within each video increases mean anomaly scores while keeping the recording source unchanged. This provides supporting evidence that the original model uses temporal correspondence for detection.
[Q4-2] Detection performance with same-recording hard negatives.
We further examine this temporal correspondence during training by introducing temporally separated, non-overlapping audio segments from the same recording as hard negatives. Under the same feature extractor and training protocol, the retrained model maintains comparable detection performance across all four benchmarks, as shown below. Together, the within-video temporal intervention and the hard-negative training results support audio-visual temporal consistency as an important basis for detection by AATT.
| Dataset | Original AATT AP / AUC (%) | AATT with same-recording hard negatives AP / AUC (%) |
|---|---|---|
| AVLips | 98.9 / 98.8 | 98.5 / 99.1 |
| FKAV | 98.8 / 98.2 | 98.8 / 98.1 |
| THB | 98.9 / 99.3 | 98.4 / 98.8 |
| SHDF | 93.2 / 88.1 | 92.9 / 87.6 |
Thank you for the explanation. I believe the basic questions have been answered quite clearly, and the article is of good writing quality. Considering the content of the article and the scores from other reviewers, I will maintain my original score and support the article's acceptance. Good luck!
Response to Reviewer xe7e
Dear Reviewer xe7e,
We are glad that our responses have addressed your concerns. We sincerely appreciate your positive assessment and support for our work. If you have any further questions or suggestions, we would be happy to address them.
Best regards,
Authors
Response to Reviewer xe7e (1/2)
We sincerely thank you for your careful review and for recognizing the importance of the research problem, the strong detection performance, and the cross-scenario generalization of AATT. We address your comments point by point below. In the revised draft, we mark our major revisions in blue.
[Q1, W1] Advantages over existing alignment methods and controlled comparisons of gains beyond temporal alignment.
AATT incorporates correspondence ambiguity into both temporal inference and deepfake detection. Compared with Soft-DTW, AATT explicitly downweights ambiguous matching evidence and uses path surprisal to assess how well that evidence supports temporally admissible paths. Compared with the monotonic dynamic-programming baseline, AATT marginalizes over admissible paths rather than selecting the highest-scoring path. Compared with independent local attention, AATT infers correspondences jointly under temporal constraints.
For a direct comparison, we replace the alignment mechanism within AATT under the same feature extractor and training settings. All three controls retain the original computation of S and all three detection descriptors. AATT outperforms all three alternatives on SHDF and THB, as shown in the table below. Moreover, Sec. 4.3 and Table 4 show that removing informativeness weighting causes a marked performance drop even when structured temporal alignment is retained. This demonstrates that temporal alignment alone cannot account for the performance of AATT, highlighting the contribution of ambiguity-aware evidence weighting.
[Q2, W2] Fixed audio latency, local shifts, time scaling, and detection sensitivity after soft alignment.
For fixed audio latency, we evaluate positive and negative audio-visual offsets on SHDF. As shown below, AATT outperforms both baselines under both conditions, supporting its robustness to the fixed audio latency.
For local time shifts and time scaling, we temporally resample extracted audio features while keeping the visual sequence unchanged. Gradually varying offsets simulate relative time scaling, while abrupt changes introduce local temporal discontinuities. AATT maintains strong detection performance under both conditions on SHDF, indicating that these time-varying misalignments do not substantially compromise detection performance.
Regarding detection sensitivity, soft alignment in AATT is constrained by local windows and monotonic paths. Its detection score also includes path surprisal S, which evaluates how well the pre-transport matching evidence supports temporal constraints (Eq. 9). Thus, detection does not rely solely on feature similarity after alignment. Abrupt offsets increase mean anomaly scores for both authentic and manipulated videos, showing that temporal discontinuities remain reflected in the final detection scores after soft alignment (see below).
Scores are mean D×100. All three conditions use the same temporal crop.