- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes a modality-conditional self-supervised framework for audio-visual deepfake detection that combines contrastive synchronization modeling with independent visual and audio JEPA branches. By routing inputs according to modality availability and filtering semantically inactive segments, the method improves robustness to missing modalities, natural silence, and cross-dataset distribution shifts.
Critical strengths
-
The paper identifies practically important failure modes in existing audio-visual deepfake detectors. In particular, it highlights modality collapse under missing inputs and false-positive drift during semantic silence, both of which are highly relevant to real-world deployment.
-
The paper’s analysis of partial video manipulations and the corresponding solution are interesting.
Critical weaknesses
-
Limited novelty. The idea of jointly modeling unimodal internal anomalies and cross-modal audio-visual inconsistencies is not entirely new, with related efforts appearing as early as ICCV 2021[1] and more recently at ICCV 2025[2]. The main contribution of this work appears to lie more in the conditional routing strategy and the particular combination of existing modules, rather than in introducing a fundamentally new detection paradigm.
-
The feature extraction strategy for the Visual JEPA raises concerns. The visual branch of AV-HuBERT was originally pretrained on lip-region ROIs, whereas this work directly applies its visual front-end to full-face inputs, which introduces a clear distribution shift. The paper lacks sufficient experiments to demonstrate that these features can reliably capture full-face dynamics such as head pose and eye gaze.
-
The comparison may not be fully fair due to differences in training data. The proposed method is trained on real videos from AV-Deepfake1M++ and is then evaluated on the same dataset, while baselines such as SpeechForensics were not trained using the same real videos from AV-Deepfake1M++. Therefore, part of the performance gap may come from differences in training data rather than from the method itself. In addition, the paper lacks systematic evaluation on other independent datasets, such as FakeAVCeleb and KoDF, making its claimed generalization advantage less conclusive.
-
The baseline setup for the missing-modality experiments is insufficient. The proposed method explicitly introduces Audio/Visual JEPA modules as fallback branches for missing modalities, whereas AVH-Align and SpeechForensics simply replace the missing input with a neutral mean tensor without any dedicated missing-modality adaptation. Therefore, these experiments mainly demonstrate that the original alignment-based models suffer from modality collapse, but they do not sufficiently show that the proposed unimodal solution is better than, or at least comparable to, dedicated unimodal or missing-modality methods.
[1] Joint Audio-Visual Deepfake Detection. ICCV 2021
[2] Intra-modal and Cross-modal Synchronization for Audio-visual Deepfake Detection and Temporal Localization. ICCV 2025
Questions for authors
-
I believe the authors should provide additional experiments addressing the above weaknesses, for example by evaluating the method on datasets such as FakeAVCeleb and KoDF, and by comparing it with dedicated unimodal methods such as RealForensics[3].
-
How robust is the proposed method to common real-world perturbations, such as video compression, audio noise, and other input corruptions?
[3] Leveraging real talking faces via self-supervision for robust forgery detection. CVPR 2022.
7 comments
W2. Full-face visual representations from AV-HuBERT
We thank the reviewer for raising this important concern regarding the transfer of a lip-pretrained visual representation to full-face inputs. Our choice is motivated by transferable spatiotemporal representation learning rather than the direct reuse of a lip-reading objective.
First, the visual input used by AV-HuBERT is not restricted to isolated lip pixels. Its crop includes the surrounding perioral region, retaining contextual facial information around the mouth along with articulatory movements. Thus, the pretrained representation captures a broader local facial region and its speech-related temporal dynamics, rather than only the lip boundary. AV-HuBERT's visual front-end is a 3D convolutional encoder pretrained on authentic talking-face recordings, including VoxCeleb2, which also provides source recordings for AV-Deepfake1M++.
In our framework, we use the frozen AV-HuBERT visual CNN as a feature extractor on full-face bounding boxes, rather than treating it as a full-face lip-reading or forgery-detection model. We then train an independent dual-target Visual JEPA on authentic full-face sequences. By predicting both masked representations and their temporal differences, this downstream objective learns temporal consistency directly from full-face feature sequences, without requiring manipulation labels.
Therefore, the pretrained speech-related representation provides a useful initialization, while the forensic predictive objective is learned specifically from authentic full-face recordings.
The manipulation-specific results provide empirical support for this design. The visual pathway achieves 0.672 AUC on visual-only forgeries and 0.650 on jointly manipulated recordings, compared with 0.530 when only audio is manipulated, and the visual stream remains authentic. This indicates that the extracted representations retain useful information for detecting visual manipulations, consistent with the intended role of the independent visual pathway.
We will refine the manuscript to distinguish the pretrained speech-related visual representation from the downstream full-face temporal predictive learning objective. We do not claim that the pretrained front-end was originally optimized to capture all global facial attributes; rather, our results demonstrate its empirical usefulness as a feature extractor for authentic-only full-face temporal prediction.
W3. Training-data fairness and cross-dataset generalization
Identical training data and evaluation protocol. We thank the reviewer for raising this important point. Our method and AVH-Align are trained on the exact same 45,000 authentic AV-Deepfake1M recordings originating from VoxCeleb2, using identical training split metadata. Both methods are evaluated on the same 8,930 AV-Deepfake1M++ test videos, with AVH-Align using its native preprocessing. Thus, differences in authentic training-data selection cannot explain the observed improvement: our method achieves 0.8525 AUC versus 0.8274 for AVH-Align.
Additional control with SpeechForensics. SpeechForensics is a training-free detector that uses pretrained AV-HuBERT representations without training an additional forgery classifier. To examine the effect of authentic-domain adaptation, we apply CCA using 5,000 authentic AV-Deepfake1M recordings, which improves its AUC from 0.663 to 0.683. We then apply our standard-deviation aggregation () to its temporal scores, yielding an AUC of 0.764. This additional analysis suggests that temporal-variability aggregation can also improve detection with a separate pretrained representation, beyond our proposed synchronization architecture. Independent cross-dataset evaluation. We additionally evaluate on FakeAVCeleb and InDeepFake without training on their manipulated recordings.
| Dataset | Ours AUC | AVH-Align AUC |
|---|---|---|
| FakeAVCeleb (additional two-sided read-out) | 0.977 | 0.888 |
| FakeAVCeleb () | 0.730 | 0.888 |
| InDeepFake () | 0.764 | 0.560 |
These results provide complementary evidence that our synchronization representations retain discriminative information across datasets and manipulation methods. Together, the identical-training-data comparison, authentic-only adaptation control, and independent evaluations strengthen the evidence that our improvements arise from the proposed modeling and aggregation choices rather than additional training data.
Additional evaluation on KoDF.
Following the reviewer's suggestion, we have already submitted a request for access to the KoDF dataset. However, we have not yet received access. The dataset is also substantial in size and involves considerable storage, preprocessing, and computational requirements. Nevertheless, we will make our best effort to include the additional evaluation if access is granted within the available timeline and the necessary computational resources are available. We agree that this would provide a useful additional assessment of cross-dataset generalization.
W1. Novelty and distinction from prior intra/cross-modal approaches
We appreciate the recognition that conditional routing is a central contribution of our work. We agree that intra-modal and cross-modal representation learning are established directions in audio-visual learning. Our novelty does not rest on claiming that these individual concepts have not been explored previously. Rather, we introduce a specific formulation of authentic-only audio-visual forensic detection that combines conditional modality handling with temporal heterogeneity of synchronization as a detection signal.
The distinctions from the cited approaches are summarized below.
Beyond these architectural distinctions, our central contribution is to investigate what temporal synchronization scores reveal about partially manipulated videos and why their temporal variability can be more informative than their average value. We supported this with several complementary analyses.
We summarize our contribution and novelty as follows: (i) This work brings attention to a normalized assumption about constant audio-video synchronization in videos, which is not always indicative of real-world footage, as silence and silence cues exist in normal speech. It brings an effective approach to this matter by creating a conditional architecture depending on the strength of each signal. (ii) Strong motivation: Modality collapse and false-positive drift are real pain points. and (iii) Simple and effective method: Complete decoupling + parameterless routing gate.
(i) Relationship between manipulated duration and synchronization variability.
Using ground-truth manipulation segments in AV1M++, we evaluate how (S_{\mathrm{std}}) changes with the proportion of manipulated content.
The consistent increase across all three manipulation types shows that synchronization variability carries information about the extent of manipulation.
(ii) Controlled temporal-splicing experiment.
To examine this relationship under controlled conditions, we construct paired real/fake samples from FakeAVCeleb using the same source clip. We progressively replace a fraction of the authentic sequence with the corresponding fake sequence.
The results show that when authentic and manipulated segments coexist, synchronization variability becomes highly discriminative, reaching 0.944–0.995 AUC for manipulated fractions between 10% and 60% across the three settings. Interestingly, (Sstd) follows an inverted-U pattern as the manipulated fraction increases towards complete synthesis. When the entire clip is synthetic, synchronization agreement becomes consistently low rather than temporally heterogeneous. In contrast, the mean synchronization score decreases monotonically from 8.38 to 1.07 in the jointly manipulated setting. This provides a useful interpretation: Mean synchronization measures the overall level of audio-visual agreement. (Sstd) measures the temporal heterogeneity of that agreement. Consequently, partially manipulated videos and fully synthesized videos can exhibit different synchronization patterns. Our analysis explains why temporal variability is particularly useful for the localized and partial manipulations that motivate our work.