acceptodds
Under review as a conference paper at ICLR 2027

Sync When It Counts: A Modality-Conditional Framework for Robust Audio-Visual Deepfake Detection

Abstract

Current audio-visual forgery detectors primarily exploit cross-modal alignment, assuming a continuous synchronization between facial movements and the acoustic speech signal. While effective on curated benchmarks, this rigid paradigm catastrophically fails under real-world conditions. When a single modality is missing or degraded, state-of-the-art alignment models (e.g., AVH-Align) collapse to near-random performance. This reveals the vulnerability that enforcing cross-modal correlation effectively blinds the model to the rich, independent forensic signals within each surviving modality. To address this, we propose a modality-conditional, self-supervised framework that completely decouples unimodal representation learning from cross-modal synchronization. Our architecture pairs a contrastive audio-visual alignment module with independent visual and audio Joint-Embedding Predictive Architecture (JEPA) streams trained entirely without manipulation labels. During inference, a deterministic routing gate evaluates the input across two tiers. To prevent modality collapse, it routes inputs with missing modalities directly to the independent JEPA streams. For standard audio-visual inputs, it halts false-positive drift by explicitly dropping natural silences (e.g., authentic pauses or nods) from the synchronization calculation. Our method achieves state-of-the-art in-domain performance on AV-Deepfake1M++ (0.8525 AUC, 0.9434 AP). Crucially, the unimodal JEPA pathways retain robust detection capabilities under severe modality degradation (0.739 audio-only AUC and 0.673 visual-only AUC), whereas prior baselines fail entirely. Furthermore, in zero-shot cross-dataset evaluation on InDeepFake, our approach achieves 0.764 AUC, outperforming several supervised in-domain models. These findings demonstrate that substituting rigid alignment with conditional, context-aware routing is crucial for resilient deepfake detection in real-world scenarios.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
audio-visual deepfakes, joint embedding predictive architecture, contrastive learning, self-supervised

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tzbyOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a modality-conditional self-supervised framework for audio-visual deepfake detection that combines contrastive synchronization modeling with independent visual and audio JEPA branches. By routing inputs according to modality availability and filtering semantically inactive segments, the method improves robustness to missing modalities, natural silence, and cross-dataset distribution shifts.

Critical strengths
  1. The paper identifies practically important failure modes in existing audio-visual deepfake detectors. In particular, it highlights modality collapse under missing inputs and false-positive drift during semantic silence, both of which are highly relevant to real-world deployment.

  2. The paper’s analysis of partial video manipulations and the corresponding solution are interesting.

Critical weaknesses
  1. Limited novelty. The idea of jointly modeling unimodal internal anomalies and cross-modal audio-visual inconsistencies is not entirely new, with related efforts appearing as early as ICCV 2021[1] and more recently at ICCV 2025[2]. The main contribution of this work appears to lie more in the conditional routing strategy and the particular combination of existing modules, rather than in introducing a fundamentally new detection paradigm.

  2. The feature extraction strategy for the Visual JEPA raises concerns. The visual branch of AV-HuBERT was originally pretrained on lip-region ROIs, whereas this work directly applies its visual front-end to full-face inputs, which introduces a clear distribution shift. The paper lacks sufficient experiments to demonstrate that these features can reliably capture full-face dynamics such as head pose and eye gaze.

  3. The comparison may not be fully fair due to differences in training data. The proposed method is trained on real videos from AV-Deepfake1M++ and is then evaluated on the same dataset, while baselines such as SpeechForensics were not trained using the same real videos from AV-Deepfake1M++. Therefore, part of the performance gap may come from differences in training data rather than from the method itself. In addition, the paper lacks systematic evaluation on other independent datasets, such as FakeAVCeleb and KoDF, making its claimed generalization advantage less conclusive.

  4. The baseline setup for the missing-modality experiments is insufficient. The proposed method explicitly introduces Audio/Visual JEPA modules as fallback branches for missing modalities, whereas AVH-Align and SpeechForensics simply replace the missing input with a neutral mean tensor without any dedicated missing-modality adaptation. Therefore, these experiments mainly demonstrate that the original alignment-based models suffer from modality collapse, but they do not sufficiently show that the proposed unimodal solution is better than, or at least comparable to, dedicated unimodal or missing-modality methods.

[1] Joint Audio-Visual Deepfake Detection. ICCV 2021

[2] Intra-modal and Cross-modal Synchronization for Audio-visual Deepfake Detection and Temporal Localization. ICCV 2025

Questions for authors
  1. I believe the authors should provide additional experiments addressing the above weaknesses, for example by evaluating the method on datasets such as FakeAVCeleb and KoDF, and by comparing it with dedicated unimodal methods such as RealForensics[3].

  2. How robust is the proposed method to common real-world perturbations, such as video compression, audio noise, and other input corruptions?

[3] Leveraging real talking faces via self-supervision for robust forgery detection. CVPR 2022.

7 comments

W1. Novelty and distinction from prior intra/cross-modal approaches

We appreciate the recognition that conditional routing is a central contribution of our work\color{red}{\text{We appreciate the recognition that conditional routing is a central contribution of our work}}. We agree that intra-modal and cross-modal representation learning are established directions in audio-visual learning. Our novelty does not rest on claiming that these individual concepts have not been explored previously. Rather, we introduce a specific formulation of\color{blue}{\text{Our novelty does not rest on claiming that these individual concepts have not been explored previously. Rather, we introduce a specific formulation of}} authentic-only audio-visual forensic detection that combines conditional modality handling with temporal heterogeneity of synchronization as a detection signal.\color{blue}{\text{authentic-only audio-visual forensic detection that combines conditional modality handling with temporal heterogeneity of synchronization as a detection signal.}}

The distinctions from the cited approaches are summarized below.

Aspect(ICCV 2021)(ICCV 2025)Our method
Training supervisionReal/fake labelsReal-video synchronization pretraining followed by manipulation-supervised detection/localizationAuthentic videos only; no manipulation labels
Modality handlingJoint audio-visual modelingJoint audio-visual modelingConditional selection of synchronization or dedicated unimodal fallback pathways
Missing-modality handlingNo dedicated JEPA fallbackNo dedicated JEPA fallbackIndependent Audio and Visual JEPA modules
Video-level forensic evidenceLearned classifierLearned classifierTemporal heterogeneity of synchronization scores (SstdS_{\mathrm{std}})

Beyond these architectural distinctions, our central contribution is to investigate what temporal synchronization scores reveal about partially manipulated\color{red}{\text{Beyond these architectural distinctions, our central contribution is to investigate what temporal synchronization scores reveal about partially manipulated}} videos and why their temporal variability can be more informative than their average value. We supported this with several complementary analyses.\color{red}{\text{videos and why their temporal variability can be more informative than their average value. We supported this with several complementary analyses.}}

We summarize our contribution and novelty as follows\color{green}{\text{We summarize our contribution and novelty as follows}}: (i) This work brings attention to a normalized assumption about constant audio-video synchronization in videos, which is not always indicative of real-world footage, as silence and silence cues exist in normal speech. It brings an effective approach to this matter by creating a conditional architecture depending on the strength of each signal. (ii) Strong motivation: Modality collapse and false-positive drift are real pain points. and (iii) Simple and effective method: Complete decoupling + parameterless routing gate.

(i) Relationship between manipulated duration and synchronization variability.

Using ground-truth manipulation segments in AV1M++, we evaluate how (S_{\mathrm{std}}) changes with the proportion of manipulated content.

Manipulation typeAUC: Lowest to Highest Manipulated-Fraction QuartileSpearman Correlation
Audio-only0.834 → 0.907 → 0.933 → 0.9520.44, p < 1e-100
Visual-only0.717 → 0.746 → 0.802 → 0.8450.21, p < 1e-21
Both0.815 → 0.869 → 0.890 → 0.9160.29, p < 1e-44

The consistent increase across all three manipulation types shows that synchronization variability carries information about the extent of manipulation.

(ii) Controlled temporal-splicing experiment.

To examine this relationship under controlled conditions, we construct paired real/fake samples from FakeAVCeleb using the same source clip. We progressively replace a fraction of the authentic sequence with the corresponding fake sequence.

Fraction replaced00.10.20.40.60.81.0
Fake video + fake audio0.4670.9740.9860.9900.9790.8100.191
Fake video + real audio0.5070.9440.9640.9710.9650.9020.334
Real video + fake audio0.5000.9810.9930.9950.9940.9820.932

The results show that when authentic and manipulated segments coexist, synchronization variability becomes highly discriminative, reaching 0.944–0.995 AUC for manipulated fractions between 10% and 60% across the three settings. Interestingly, (SstdS_{\mathrm{std}}) follows an inverted-U pattern as the manipulated fraction increases towards complete synthesis. When the entire clip is synthetic, synchronization agreement becomes consistently low rather than temporally heterogeneous. In contrast, the mean synchronization score decreases monotonically from 8.38 to 1.07 in the jointly manipulated setting. This provides a useful interpretation: Mean synchronization measures the overall level of audio-visual agreement. (SstdS_{\mathrm{std}}) measures the temporal heterogeneity of that agreement. Consequently, partially manipulated videos and fully synthesized videos can exhibit different synchronization patterns. Our analysis explains why temporal variability is particularly useful for the localized and partial manipulations that motivate our work.

W2. Full-face visual representations from AV-HuBERT

We thank the reviewer for raising this important concern regarding the transfer of a lip-pretrained visual representation to full-face inputs. Our choice is motivated by transferable spatiotemporal representation learning rather than the direct reuse of a lip-reading objective.

First, the visual input used by AV-HuBERT is not restricted to isolated lip pixels. Its crop includes the surrounding perioral region, retaining contextual facial information around the mouth along with articulatory movements. Thus, the pretrained representation captures a broader local facial region and its speech-related temporal dynamics, rather than only the lip boundary. AV-HuBERT's visual front-end is a 3D convolutional encoder pretrained on authentic talking-face recordings, including VoxCeleb2, which also provides source recordings for AV-Deepfake1M++.

In our framework, we use the frozen AV-HuBERT visual CNN as a feature extractor on full-face bounding boxes, rather than treating it as a full-face lip-reading or forgery-detection model. We then train an independent dual-target Visual JEPA on authentic full-face sequences. By predicting both masked representations and their temporal differences, this downstream objective learns temporal consistency directly from full-face feature sequences, without requiring manipulation labels.

Therefore, the pretrained speech-related representation provides a useful initialization, while the forensic predictive objective is learned specifically from authentic full-face recordings.

The manipulation-specific results provide empirical support for this design. The visual pathway achieves 0.672 AUC on visual-only forgeries and 0.650 on jointly manipulated recordings, compared with 0.530 when only audio is manipulated, and the visual stream remains authentic. This indicates that the extracted representations retain useful information for detecting visual manipulations, consistent with the intended role of the independent visual pathway.

We will refine the manuscript to distinguish the pretrained speech-related visual representation from the downstream full-face temporal predictive learning objective. We do not claim that the pretrained front-end was originally optimized to capture all global facial attributes; rather, our results demonstrate its empirical usefulness as a feature extractor for authentic-only full-face temporal prediction.

W3. Training-data fairness and cross-dataset generalization

Identical training data and evaluation protocol. We thank the reviewer for raising this important point. Our method and AVH-Align are trained on the exact same 45,000 authentic AV-Deepfake1M recordings originating from VoxCeleb2, using identical training split metadata. Both methods are evaluated on the same 8,930 AV-Deepfake1M++ test videos, with AVH-Align using its native preprocessing. Thus, differences in authentic training-data selection cannot explain the observed improvement: our method achieves 0.8525 AUC versus 0.8274 for AVH-Align.

Additional control with SpeechForensics. SpeechForensics is a training-free detector that uses pretrained AV-HuBERT representations without training an additional forgery classifier. To examine the effect of authentic-domain adaptation, we apply CCA using 5,000 authentic AV-Deepfake1M recordings, which improves its AUC from 0.663 to 0.683. We then apply our standard-deviation aggregation (SstdS_{\mathrm{std}}) to its temporal scores, yielding an AUC of 0.764. This additional analysis suggests that temporal-variability aggregation can also improve detection with a separate pretrained representation, beyond our proposed synchronization architecture. Independent cross-dataset evaluation. We additionally evaluate on FakeAVCeleb and InDeepFake without training on their manipulated recordings.

DatasetOurs AUCAVH-Align AUC
FakeAVCeleb (additional two-sided read-out)0.9770.888
FakeAVCeleb (SstdS_{\mathrm{std}})0.7300.888
InDeepFake (SstdS_{\mathrm{std}})0.7640.560

These results provide complementary evidence that our synchronization representations retain discriminative information across datasets and manipulation methods. Together, the identical-training-data comparison, authentic-only adaptation control, and independent evaluations strengthen the evidence that our improvements arise from the proposed modeling and aggregation choices rather than additional training data.

Additional evaluation on KoDF.

Following the reviewer's suggestion, we have already submitted a request for access to the KoDF dataset. However, we have not yet received access. The dataset is also substantial in size and involves considerable storage, preprocessing, and computational requirements. Nevertheless, we will make our best effort to include the additional evaluation if access is granted within the available timeline and the necessary computational resources are available. We agree that this would provide a useful additional assessment of cross-dataset generalization.

Although we believe our extensive generalization experiments demonstrate the effectiveness of the approach, merely adding one more dataset might\color{blue}{\text{Although we believe our extensive generalization experiments demonstrate the effectiveness of the approach, merely adding one more dataset might}} not provide any new drastic information.\color{blue}{\text{not provide any new drastic information.}}

Reviewer_AyuROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a self-supervised method for audio-visual deepfake detection. It uses audio-visual synchronization when both modalities are available, and separate predictors when one is missing. The method also filters out less useful windows and uses temporal score variation for detection. Experiments show some improvements on different datasets, including missing-modality settings.

Critical strengths

(a) Using separate unimodal predictors is a reasonable idea, because it allows the model to work when audio or video is missing, which is useful in practice.

(b) The experiments show that temporal score variation works better than average alignment scores, and the cross-dataset threshold transfer experiment is also useful for real applications.

Critical weaknesses

(a) What happens if all windows are filtered out? Also, the silent-video experiment uses visual JEPA scores, but the method seems to use synchronization when both modalities exist. The authors should clarify the actual decision process. (b) AUC and AP cannot directly show the false positive rate at a given threshold. Also, there is no direct comparison with and without the activity gate using the same aggregation method. So it is hard to tell where the improvement comes from. (c) The authors replace missing inputs with neutral tensors for methods that need both modalities. These methods are expected to perform worse in this case. A comparison with a unimodal model trained on the same real data would be more convincing.

Questions for authors

Please refer to the weaknesses.

3 comments

We thank the reviewer for the thoughtful and actionable assessment. We have conducted controlled experiments specifically addressing the three concerns: the exact routing rule, the independent contribution of activity filtering, and comparisons with dedicated unimodal detectors trained under matched conditions.

(a) What happens if all windows are filtered? How are silent videos handled?

We clarify that stream availability and synchronization eligibility are two distinct conditions in our framework. The inference procedure has three stages:

  1. Modality availability: If both streams exist, the synchronization pathway is selected. If audio or video is absent, the corresponding Audio or Visual JEPA pathway is selected.

  2. Activity filtering: Synchronization windows below the 1st-percentile activity threshold estimated from authentic validation videos are removed.

  3. All windows filtered: If fewer than two windows survive, the system uses the unimodal fallback:

S_{\mathrm{fallback}}= \max\left( \sigma(z_{\mathrm{audio}}), \sigma(z_{\mathrm{visual}}) \right). ]$$ The branch scores are normalized using statistics from 1,000 authentic validation videos. This condition occurs in 10 of the 8,930 test videos, including three authentic videos, and all such videos receive a defined final score. **Silent videos.** We verify that all 398 silent clips in AV1M++ contain no audio stream using ffprobe. Consequently, they are routed directly to the Visual JEPA pathway rather than the synchronization pathway. The synchronization score is therefore not computed for these clips, and the corresponding AUC is reported as N/A rather than zero.

(b) False-positive rates at fixed thresholds and contribution of activity filtering

We agree that AUC and AP alone do not establish false-positive behavior at a particular operating threshold. We therefore evaluate our method using thresholds selected exclusively on 1,000 authentic validation videos.

Evaluation MetricWithout FilteringWith Filtering
Overall AUC (controlled rerun)0.85360.8525
Average Precision0.94340.9460
TPR at 1% matched test FPR23.8%29.7%

Filtering improves TPR at 1% FPR by 5.9 percentage points (95% CI [+1.3,+15.8]) without a statistically established change in overall AUC (difference CI [-0.003,+0.002]). This distinction helps clarify the mechanism: standard-deviation aggregation primarily improves global discrimination, whereas activity filtering selectively benefits strict low-FPR operation. The aggregation ablation independently demonstrates an increase from 0.7379 AUC (mean) to 0.8525 ((Sstd)(S_{\mathrm{std}}), Table 9). At thresholds calibrated exclusively on authentic validation data, the gated detector achieves:

Target Validation FPRTest FPRTest TPR
1%1.2%32.7%
5%5.2%52.5%
10%10.9%64.3%
These validation-fixed results are distinct from the matched-test-FPR comparison above. At 1% matched test FPR, AVH-Align achieves 14.0% TPR, compared with 29.7% for our gated rerun.
Silent-video evaluation. Since silent webcam recordings belong to a different capture domain, we additionally calibrate the visual pathway using approximately 100 authentic recordings from this domain. With two-fold cross-fitting, the method achieves 5.5% FPR at a 5% target, demonstrating effective false-positive control with a small, manipulation-label-free calibration set.

(c) Comparison with unimodal detectors trained on the same authentic data

The central distinction of our framework is that it explicitly introduces independent Audio and Visual JEPA modules as fallback branches for missing modalities. AVH-Align and SpeechForensics rely on cross-modal representations and, in our missing-input evaluation, replace the unavailable stream with a neutral placeholder. They do not provide the dedicated modality-specific fallback adaptation introduced by our framework.

To further examine the role of predictive modeling, we build on the unimodal evaluations already reported in Tables 3–5 and extend the comparison to dedicated one-class detectors trained exclusively on authentic recordings. We evaluated dedicated one-class detectors trained exclusively on authentic recordings, including CA-SOADD (ICML 2026\color{red}{\text{ICML 2026}}) and matched kNN and feature-norm baselines. The latter use the same authentic training videos and feature representations as our JEPA pathways. This extended comparison reports manipulation-specific AUCs, showing how each detector performs when the manipulation affects its available modality. Our proposed approach significantly outperforms CA-SOADD (0.731 of proposed vs 0.546 of CA-SOADD\color{red}{\text{0.731 of proposed vs 0.546 of CA-SOADD}}).

Available Stream / DetectorAudio-Only FakeVisual-Only FakeBoth Fake
Audio / AVH-Align (visual input zeroed)0.4780.4980.469
Audio / CA-SOADD (one-class)0.5280.4830.546
Audio / kNN0.7060.4930.706
Audio / Feature Norm0.4830.4940.487
Audio / Our JEPA0.7390.4790.731
Visual / AVH-Align (audio input zeroed)0.4670.4570.460
Visual / kNN0.5750.6620.657
Visual / Feature Norm0.5740.6660.648
Visual / Our JEPA0.5300.6720.650

Our Audio JEPA achieves the strongest performance among the compared audio detectors on audio-only and both-stream manipulations. An interesting contrast appears with the Audio Feature Norm baseline, which performs near chance on audio fakes (AUC 0.483). We report this score as obtained, since the one-class scoring direction is fixed in advance rather than selected using test labels. Further analysis helps explain this result: generated speech is often smoother than real speech. The local feature-norm variability inside the manipulated segment is approximately 10% lower than in the authentic remainder of the clip (p=5×10−22)(p=5\times10^{-22}), opposite to the direction expected by this statistic. Moreover, since the manipulated segment occupies only about 4% of the clip, this local difference is largely averaged out in the whole-video score (0.363 for fake videos vs. 0.365 for real videos).

In contrast, predictive modeling can expose this localized inconsistency. For the same audio fakes, the Audio JEPA prediction error is 1.84 times higher inside the manipulated segment than outside it in 99% of videos (p=3×10−88)(p=3\times10^{-88}). By predicting masked frames from their surrounding context, the model captures local deviations that global variability statistics can miss. This provides evidence for the benefit of predictive modeling as an audio fallback mechanism. The visual comparison provides a complementary finding. Matched kNN and feature-norm baselines achieve competitive performance, while our Visual JEPA obtains the highest AUC for visual-only manipulations (0.672). This suggests that authentic-only visual representations support independent anomaly detection, while also showing that the relative benefit of predictive modeling varies across modalities.

Importantly, manipulation types are defined with respect to the original audio-visual recordings. A unimodal detector cannot observe a manipulation confined\color{red}{\text{Importantly, manipulation types are defined with respect to the original audio-visual recordings. A unimodal detector cannot observe a manipulation confined}} entirely to the unavailable stream; near-chance performance in such cases is therefore expected.\color{red}{\text{entirely to the unavailable stream; near-chance performance in such cases is therefore expected.}} Overall, these results extend the unimodal evaluations in Tables 3–5 by comparing different authentic-only predictors under the same modality-specific conditions. They support our central design: independent, manipulation-label-free inference when synchronization information is unavailable, without relying on synthetic replacement of the missing modality.

Reviewer_LUudOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Sync is a self-supervised audio-visual deepfake detector trained on authentic videos. It uses synchronization scores when both modalities are present and independent audio or visual JEPA predictors when one is absent. Inactive windows are filtered before synchronization scores are aggregated by standard deviation. The visual branch predicts both features and temporal changes. Experiments cover paired inputs, missing modalities, silent videos, and cross-dataset transfer. The main evaluation reports 0.8525 AUC versus 0.8274 for AVH-Align, with 0.764 AUC on InDeepFake.

Critical strengths
  1. The method provides a way to handle missing modalities without requiring manipulation labels for training. Separate audio and visual predictors allow it to use the remaining stream when synchronization cannot be computed.
  2. The aggregation results support temporal variability in synchronization as a useful detection cue. At the same context length, standard deviation raises AUC from 0.7379 with mean pooling to 0.8525.
Critical weaknesses
  1. The experiments leave the advantage of the conditional design over simpler alternatives unclear. On paired inputs, the JEPA heads are bypassed, and no ablation isolates activity filtering with standard-deviation aggregation fixed. The Max-Fusion comparison leaves score normalization unspecified. On missing inputs, outperforming paired detectors given mean-tensor substitutes does not establish an advantage over dedicated unimodal detectors trained under matched conditions.
  2. The handling of fully silent videos is not specified consistently. Visual-JEPA performance is reported, but the routing rule covers absent modalities without defining the fallback when both streams exist and every window is filtered out. The synchronization AUC is described as undefined yet listed as zero. The reported AUC and AP also do not establish the claimed reduction in false positives on authentic silent videos.
Questions for authors
  1. Keeping the trained synchronization model and standard-deviation aggregation fixed, how much does activity filtering help? How are branch scores normalized in the Max-Fusion comparison?
  2. How do the fallback predictors compare with unimodal anomaly detectors using matched features and authentic training data? Please report the missing-input results by manipulation type, since the removed stream may contain all manipulated content.
  3. What final score and fallback rule are used when both streams exist but all windows are filtered out? At thresholds fixed on authentic validation data, how does this handling affect false positives on authentic silent videos?
6 comments

We thank the reviewer for the precise and constructive questions. Our additional experiments distinguish the contribution of temporal aggregation, activity filtering, and unimodal fallback under controlled conditions.

Q1. Activity filtering and Max-Fusion normalization

Controlled gate ablation. We keep the trained synchronization model, features, and standard-deviation aggregation $ (S_{\mathrm{std}}) $ fixed, varying only activity filtering. At the 1% FPR operating point on the test ROC curve, filtering increases TPR from 23.8\textcolor{blue}{\text{23.8}}% to 29.7\textcolor{blue}{\text{29.7}}% (+5.9 percentage points; 95% CI of the improvement: [+1.3,+15.8]). AP improves from 0.9434 to 0.9460, while AUC remains statistically comparable (0.8536 without filtering vs. 0.8525 with filtering; 95% CI of the difference: [-0.003, +0.002]). These results show that the two components serve complementary purposes: (Sstd)(S_{\mathrm{std}}) provides the main improvement in overall discrimination (AUC 0.7379 with mean pooling vs. 0.8525 with (Sstd)(S_{\mathrm{std}}), Table 9), while conservative activity filtering improves sensitivity at a stringent false-positive operating point without materially changing AUC.

Max-Fusion. Each branch score is independently z-normalized using its mean and standard deviation estimated from the same 1,000 authentic validation videos. The normalized scores are passed through a sigmoid, and the maximum is taken. The respective (μ,σ)(\mu, \sigma) values are: sync (1.349, 0.372), audio JEPA (8.56e-5, 1.30e-5), and visual JEPA (2.58e-4, 2.19e-5). No manipulation labels or test data enter normalization.

Forgery SubtypePure Sync AUCMax-Fusion AUC95% CI of (Fusion - Sync)
Visual-Only Forged0.77740.7150[-0.075, -0.050]
Audio-Only Forged0.90600.8610[-0.054, -0.036]
Both Forged0.87310.8364[-0.047, -0.026]
Overall Dataset0.85250.8046[-0.057, -0.039]
Thus, unconditional Max-Fusion reduces discrimination across all three manipulation categories. This supports conditional selection over this particular fusion strategy, without claiming that every possible learned fusion scheme would perform worse.

Q2. Comparison with matched unimodal detectors

The central distinction of our framework is that it explicitly introduces independent Audio and Visual JEPA modules as fallback branches for missing modalities. AVH-Align and SpeechForensics rely on cross-modal representations and, in our missing-input evaluation, replace the unavailable stream with a neutral placeholder. They do not provide the dedicated modality-specific fallback adaptation introduced by our framework. To further examine the role of predictive modeling, we build on the unimodal evaluations already reported in Tables 3–5 and extend the comparison to dedicated one-class detectors trained exclusively on authentic recordings. We evaluated dedicated one-class detectors trained exclusively on authentic recordings, including CA-SOADD (ICML 2026\color{red}{\text{ICML 2026}}) and matched kNN and feature-norm baselines. The latter use the same authentic training videos and feature representations as our JEPA pathways. This extended comparison reports manipulation-specific AUCs, showing how each detector performs when the manipulation affects its available modality. Our proposed approach significantly outperforms CA-SOADD (0.731 of proposed vs 0.546 of CA-SOADD\color{red}{\text{0.731 of proposed vs 0.546 of CA-SOADD}}).

Available Stream / DetectorAudio-Only FakeVisual-Only FakeBoth Fake
Audio / AVH-Align (visual input zeroed)0.4780.4980.469
Audio / CA-SOADD (one-class)0.5280.4830.546
Audio / kNN0.7060.4930.706
Audio / Feature Norm0.4830.4940.487
Audio / Our JEPA0.7390.4790.731
Visual / AVH-Align (audio input zeroed)0.4670.4570.460
Visual / kNN0.5750.6620.657
Visual / Feature Norm0.5740.6660.648
Visual / Our JEPA0.5300.6720.650

Our Audio JEPA achieves the strongest performance among the compared audio detectors on audio-only and both-stream manipulations. An interesting contrast appears with the Audio Feature Norm baseline, which performs near chance on audio fakes (AUC 0.483). We report this score as obtained, since the one-class scoring direction is fixed in advance rather than selected using test labels. Further analysis helps explain this result: generated speech is often smoother than real speech. The local feature-norm variability inside the manipulated segment is approximately 10% lower than in the authentic remainder of the clip (p=5×10−22)(p=5\times10^{-22}), opposite to the direction expected by this statistic. Moreover, since the manipulated segment occupies only about 4% of the clip, this local difference is largely averaged out in the whole-video score (0.363 for fake videos vs. 0.365 for real videos).

In contrast, predictive modeling can expose this localized inconsistency. For the same audio fakes, the Audio JEPA prediction error is 1.84 times higher inside the manipulated segment than outside it in 99% of videos (p=3×10−88)(p=3\times10^{-88}). By predicting masked frames from their surrounding context, the model captures local deviations that global variability statistics can miss. This provides evidence for the benefit of predictive modeling as an audio fallback mechanism.

The visual comparison provides a complementary finding. Matched kNN and feature-norm baselines achieve competitive performance, while our Visual JEPA obtains the highest AUC for visual-only manipulations (0.672). This suggests that authentic-only visual representations support independent anomaly detection, while also showing that the relative benefit of predictive modeling varies across modalities.

Importantly, manipulation types are defined with respect to the original audio-visual recordings. A unimodal detector cannot observe a manipulation confined entirely to the unavailable stream; near-chance performance in such cases is therefore expected. Overall, these results extend the unimodal evaluations in Tables 3–5 by comparing different authentic-only predictors under the same modality-specific conditions. They support our central design: independent, manipulation-label-free inference when synchronization information is unavailable, without relying on synthetic replacement of the missing modality.

Q3. Handling fully filtered windows and silent videos

Our framework provides a well-defined inference pathway for every input condition: Stream availability: When both audio and video streams are available, the synchronization pathway is used. If one stream is absent, the corresponding unimodal JEPA pathway is selected. Activity filtering: Windows below the 1st-percentile activity threshold estimated from authentic validation videos are removed before synchronization aggregation. All windows filtered: If fewer than two windows survive despite both streams being available, the system uses the unimodal fallback:

S_{\mathrm{fallback}}= \max\left( \sigma(z_{\mathrm{audio}}), \sigma(z_{\mathrm{visual}}) \right). ]$$ Each branch is normalized using its corresponding statistics from the 1,000 authentic validation videos. The third case occurs in only 10 of the 8,930 test videos, including three authentic videos, and is included in all reported results. We explicitly incorporate this branch into Algorithm 1. Silent-video evaluation. We further clarify that all 398 silent clips in AV1M++ contain no audio stream, as verified using ffprobe. Therefore, these videos are routed directly to the Visual JEPA pathway, and no synchronization score is computed. Accordingly, the previously reported numerical zero for synchronization AUC is to be replaced with N/A because the synchronization score is undefined in this setting. False-positive control. We additionally evaluated the visual pathway at thresholds calibrated using authentic recordings from the silent-video domain. With approximately 100 genuine calibration recordings and two-fold cross-fitting, the method achieves 5.5% FPR at a 5% target, showing that reliable false-positive control can be obtained with a small, manipulation-label-free calibration set. $\color{red}{\text{These experiments establish that conditional routing enables seamless inference across available modalities, while authentic-only calibration provides a}}$ $\color{red}{\text{practical mechanism for controlling false alarms across recording conditions.}}$ We will clarify both aspects in the revised manuscript.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Discussion and Acknowledgement

We sincerely thank the reviewers for providing constructive comments even with these hectic deadline schedules.

But as we are aware that we’re about halfway through this critical discussion period, it would be great if you can revisit our response and provide us positive acknowledgement or provide us the timely opportunity to resolve any new/remaining comments.

We are actively looking forward for a constructive discussion with the reviewers. As we have addressed all the concerns thoroughly and deadline for discussion is approaching, we appreciate the reviewers engagement in this critical phase and positive assessment to our paper

Thanks again for all the constructive feedback.

We sincerely appreciate it if the reviewers can participate in this phase as without this we are unable to respond to any concern left. We have thoroughly addressed each concern raised by all the reviewers.

We are approaching towards the end of this critical discussion period, it would be great if you can revisit our response and provide us positive acknowledgement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.