acceptodds
Under review as a conference paper at ICLR 2027

Detecting Audio-Visual Deepfakes Outside the Generator Objective

Abstract

A generator optimized for an audio–visual synchronization objective may match that objective at an observed sample while retaining a different local articulatory response. We propose Outside the Objective (OtO)}, an authentic-only test of how acoustic features inferred by a visual-to-acoustic predictor change under local mouth-tangent queries. Our central construction maps declared generator objectives into the acoustic response space and tests the complementary response conditioned on the observed audio. A canonical detector-space cosine gives an exact local plane; declared differentiable losses contribute Riesz representatives. Conditional learning, objective projection, and probe selection use one consistent residual. In a controlled synchronization-weight sweep, the zero-order cosine detector falls from 88.9 to 66.9 AUROC while the projected first-order test remains between 90.2 and 90.7. Under a common cross-domain protocol,OtO reaches 92.0% AUROC on MMDF and 84.1% on HiFi-AVDF, with 3.6 and 4.8 points attributable to objective-direction removal beyond audio conditioning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
audio-visual deepfake detection, active probing, counterfactual intervention, multimodal learning, cross-domain generalization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_9UxAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an audiovisual deepfake detector trained only on authentic data. It perturbs tracked mouth trajectories and measures how the predictions of a frozen visual-to-acoustic model change. These responses are compared with what is expected for authentic speech, conditioned on the observed audio. The method then removes response directions associated with specified generator objectives before computing an anomaly score. The motivation is that generators may learn to satisfy the same audiovisual consistency objectives that detectors rely on. The paper evaluates this through controlled changes to generator objectives, several ablations, and experiments on four datasets.

Critical strengths

I find the main idea interesting. If a generator is explicitly trained to satisfy a particular consistency measure, relying on that measure for detection may become less effective. Looking at local predictor responses, and removing directions related to these objectives, is a useful alternative to investigate.

There are several relevant experiments supporting the approach. In particular, the comparison between matched, mismatched, random and energy-matched removal is useful for understanding whether the choice of removed directions is important. The ablations suggest that removing these directions helps even when the model already uses audio conditioning. The reported projection gains of 3.6 and 4.8 AUROC points on MMDF and HiFi-AVDF are good, especially given that the method does not use fake examples for training.

Critical weaknesses

I found the paper difficult to follow, and some important implementation details remain unclear after reading the supplement. The distinction between the predictor’s response, the generator-loss directions and the conditional reference could be explained more directly before introducing the maths. More importantly, I could not find the architectures of the controlled generators or the exact scalar losses used for the SyncNet, AVSyncNet and AV-HuBERT experiments. Appendix B also says that the tracker version and confidence thresholds were fixed, but does not specify them. These details are important because the controlled experiments are a central part of the insights for the method.

There are enough datasets to make the evaluation meaningful, but I am less clear about the range of generators covered. Since the method is motivated by generator objectives, a breakdown by generator and objective would be more informative than only aggregate dataset results. Section 5.7 reports gains for “unreported-objective” strata, but does not clearly identify their composition or sample counts. It is also unclear exactly which fixed objective library was used for the main results. The selected evaluation pools exclude appearance-only manipulations from the main scores, although these are evaluated separately. This is understandable given the mechanism, but makes it important to explain the selection protocol and scope of the generalisation claims a bit more precisely.

Questions for authors

Could you specify the controlled generator architectures, exact expert-loss definitions and expert checkpoints used in Sections 5.2–5.3? Which independent model reports synchronisation quality in Figure 2?

Was the same objective library fixed across all benchmark datasets, without choosing objectives based on the generator being evaluated? Which generators and how many source files are included in the “unreported-objective” strata?

The reproducibility statement says that the supplementary material includes source-linked manifests, objective/expert configurations and selected probe indices. I can only see a supplementary pdf uploaded, and I couldn't find these in there. Could you please clarify?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 9UxA

Thank you very much for your careful and constructive comments. We sincerely appreciate your questions regarding the controlled generator experiments, the independence of the evaluation reporter, the fixed objective library, benchmark-level objective disclosure, and reproducibility. We have expanded the implementation details, clarified the branch-level disclosure analysis, and added the probe-selection artifacts to the revised supplementary archive.

1. Controlled objectives and independent reporter

Table 10 specifies the tracker and generator setup. The three controlled generator variants share the same Wav2Lip-style backbone, identity-disjoint LRS2 training split, and training protocol. Only the generator objective (SyncNet, AVSyncNet, or AV-HuBERT) changes, while all other loss terms and their weights remain fixed (Section 5.3 and Appendix Table 10).

Table 10 also lists the SyncNet, AVSyncNet, and AV-HuBERT losses with their checkpoint sources. The AV-HuBERT loss uses the audio-to-visual KL divergence.

Figure 2 reports synchronization quality using the LRS2-pretrained VocaLiST model, which is excluded from generator optimization and the detector's objective library.

Table 10: Implementation and Figure 2 protocol

ComponentSetting
TrackerMediaPipe Face Landmarker (mediapipe==0.10.14, face_landmarker.task); fixed 40-point FACEMESH_LIPS indices; detection/presence/tracking thresholds of 0.5; valid frames receive equal weight in mesh centering, and each 1.0-second window requires at least 80% valid frames.
PreprocessingProcrustes head-pose removal on non-mouth landmarks and inter-ocular scale normalization; 25 fps trajectories linearly resampled to 50 Hz (gaps of at most 2 frames are cubic-spline interpolated, while longer gaps invalidate the window); 16 kHz mono audio normalized to −23-23 LUFS; 1.0-second windows with a 0.5-second hop.
Predictor and contextDefault fθf_\theta: 3 dilated temporal convolution blocks (512 channels, 1.0-second mesh input) predicting frozen WavLM Base+ layer-6 features (d=768d=768). Context uses frozen HuBERT Base layer-9 unit posteriors, pitch, and energy to form a 32-dimensional speech-content/prosody vector (p=32p=32); M=64M=64 candidate tangents, K=16K=16 scored probes, L=8L=8, and L0=32L_0=32.
Authentic splits and fitting7,000 / 3,500 / 3,500 / 6,000 authentic files for predictor/reference fitting, residual/probe design, 5%-FPR split-conformal calibration, and source testing, respectively. The predictor is trained with AdamW (learning rate 3×10−43\times10^{-4}, weight decay 0.010.01, cosine + smooth-L1L_1 loss).
Controlled generatorWav2Lip-style encoder–decoder on 96×9696\times96 lower-face crops, trained from random initialization on an identity-disjoint LRS2 split (Adam, β=(0.5,0.999)\beta=(0.5,0.999), learning rate 10−410^{-4}, batch size 32, 160k steps); fixed L1L_1 weight 1.01.0 and visual-quality discriminator weight 0.070.07. All three variants share the same backbone, initialization protocol, optimizer, training schedule, reconstruction loss, and visual-quality loss weights. Only the synchronization/generator objective (SyncNet, AVSyncNet, or AV-HuBERT) changes.
InterventionFigure 2 varies only λsync∈{0,0.03,0.10,0.30,1.0}\lambda_{\mathrm{sync}}\in\{0,0.03,0.10,0.30,1.0\}, starting from step 0. The λsync=0.10\lambda_{\mathrm{sync}}=0.10 generator corresponds to the SyncNet row in Table 2.
Sample unit1,800 authentic source clips, each paired with one generated variant per seed. Three seeds give 5,400 attempted fake variants per setting. The common valid-source intersection contains 1,781 sources, yielding 5,343 scored variants per setting; 57 attempted variants are excluded by the common-valid-source restriction. AUROC is computed separately for each seed against the same eligible authentic source pool, then averaged. Comparisons use the common valid-source intersection across arms and seeds. Bootstrap resamples identities while preserving paired sources, settings, and seeds.
ReporterOfficial VocaLiST LRS2-pretrained release; 96×9696\times96 aligned lower-face crops yielding 48×9648\times96 (H×WH\times W) lip inputs at 25 fps with 16 kHz, 80-bin mel spectrograms. The clip score is the mean zero-offset in-sync probability over 5-frame windows. The reporter is excluded from generator training and the removed library.
Detector basisBenchmarks: sample-specific cosine plane and frozen three-expert empirical library. Figure 2: analytic cosine plane and SyncNet empirical direction.
AuthorsOpenReview ↗

Response to Reviewer 9UxA

Declared expert scalar losses.

Each pixel-consuming expert is composed with the differentiable piecewise-affine mouth warp Vx\mathcal{V}_x.

ExpertLossCheckpoint
SyncNet−log⁡(clamp⁡(ssync,ε,1))-\log(\operatorname{clamp}(s_{\mathrm{sync}},\varepsilon,1)), 5-frame audio–visual cosine probability (ε=10−6\varepsilon=10^{-6}).Official Wav2Lip LRS2 expert release (lipsync_expert.pth).
AVSyncNet−log⁡p(sync∣v,a)-\log p(\mathrm{sync}\mid v,a), our binary cross-entropy objective using the AVSyncNet checkpoint (distinct from Yaman et al.'s three-way relative stabilized loss).Yaman et al. (2024) synchronization expert checkpoint (LRS2).
AV-HuBERTmean⁡tKL⁡(paudio∥pvisual)\operatorname{mean}_t\operatorname{KL}(p_{\mathrm{audio}}\Vert p_{\mathrm{visual}}), frame-level unit posteriors from the frozen kk-means cluster projection head.Official AV-HuBERT Large LRS3+VoxCeleb2 iter-5 checkpoint (large_vox_iter5.pt).

2. Fixed library and coverage

The three-expert empirical library is fitted on authentic data and frozen before calibration for all four benchmarks. The analytic cosine plane is computed separately for each sample (Section 3.2).

Table 11(a) reports scored clips by evaluated media file and distinct native-source counts separately. Table 11(b) assesses video and audio branches separately: HunyuanAvatar, MegActor-Sigma, and AniPortrait are each checked for their video branch, while joint clips require records for both visual and audio pipelines and versions. If an audio pipeline cannot be identified, the joint clip is classified as Unresolved.

We refined the disclosure classification from generator-level to branch-level because joint audio–visual edits may involve different video and audio pipelines. Since the available inventory does not cross-tabulate joint clips by generator, the previous stratum-specific AUROCs and gains cannot be reliably reassigned to the revised categories. We therefore removed these results rather than carrying forward potentially mismatched statistics.

Table 11(a): Scored-clip allocation and distinct native sources

DatasetGenerator / edit categoryScored direct clipsScored appearance-only clipsDistinct native source videos
MMDFHunyuanAvatar1,4870503
MMDFMegActor-Sigma1,5960547
MMDFAniPortrait1,5170521
MMDFAudio-only edits4000183
MMDFAppearance-only controls0500219
MMDF total5,000500---
HiFi-AVDFVeo 3.16680241
HiFi-AVDFSora 26410228
HiFi-AVDFKling 2.55970216
HiFi-AVDFWan 2.55730203
HiFi-AVDFSeedance 1.05180187
HiFi-AVDFPixVerse V54830171
HiFi-AVDFAudio-only edits5200232
HiFi-AVDFAppearance-only controls0400178
HiFi-AVDF total4,000400---

Clip counts refer to one eligible clip per evaluated media file. Distinct native-source counts are reported separately in Table 11(a); multiple evaluated media files may share the same native source video.

AuthorsOpenReview ↗

Response to Reviewer 9UxA

Table 11(b): Branch-level objective-disclosure evidence

Pipeline / sample groupEvidence availableWhat remains unverifiedDisclosure status
MMDF / HunyuanAvatar (1,487 total direct clips)Video pipeline/version identified; relevant video objective is not disclosed.Visual-only/joint split within the generator row; audio branch for joint clips.Unreported for video-only; joint clips unresolved.
MMDF / MegActor-Sigma (1,596 total direct clips)Video pipeline/version and relevant video objectives are documented.Visual-only/joint split within the generator row; audio branch for joint clips.Documented for video-only; joint clips unresolved.
MMDF / AniPortrait (1,517 total direct clips)Video pipeline/version and relevant video objectives are documented.Visual-only/joint split within the generator row; audio branch for joint clips.Documented for video-only; joint clips unresolved.
MMDF / joint audio–visual clips (1,700 aggregate)Aggregate edit-category count is reported in Table 1.Per-media-file audio pipeline/version and objective.Unresolved.
MMDF / audio-only edits (400)Audio-only category and scored-clip count are reported.Audio pipeline/version and objective.Unresolved.
HiFi-AVDF / six video generators (3,480 total direct clips)Generator families and versions are identified; relevant video objectives are not disclosed.Visual-only/joint split within generator rows; audio branch for joint clips.Unreported for video-only; joint clips unresolved.
HiFi-AVDF / joint audio–visual clips (1,520 aggregate)Aggregate edit-category count is reported in Table 1.Per-media-file audio pipeline/version and objective.Unresolved.
HiFi-AVDF / audio-only edits (520)Audio-only category and scored-clip count are reported.Audio pipeline/version and objective.Unresolved.

The joint rows are branch checks of subsets already counted in the video-generator rows of Table 11(a), not additional clips. The available aggregate allocation does not cross-tabulate joint clips by generator. For a joint clip, both modified branches must be identified and assessed; if either branch cannot be identified, the clip is Unresolved. We therefore do not carry forward the previous stratum-specific AUROC/gain claims under this revised classification.

3. Supplementary materials

We apologize that the earlier supplementary archive did not include the probe-selection artifacts needed to inspect the reported selection.We have now added the numerical candidate bank, the selected probe indices and displacement fields, the stored E-optimal selection trace, and a solver-verification utility to the revised archive.

The utility verifies the selection using the stored design matrices; it is intended for probe-selection verification and is not the full detector implementation.The archive also includes manifest schemas (field definitions and header-only templates), but does not include populated row-level source-linked manifests or benchmark media.

Reviewer_DCFpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the author proposed a new Outside the Objective (OtO) method to detect deepfake videos. The motivation is that, as zero-order synchronization has already been used in many methods’ training objectives, fake videos can perform quite well on this metric. Thus, OtO utilizes first-order responses as a metric, measuring a vision-to-acoustic predictor’s sensitivity to local mouth-mesh tangents. A further optimization is performed to use only the residual space orthogonal to the objective span. The experiment shows the model’s ability to detect fake videos.

Critical strengths
  • The model can be trained using only real videos.
  • The idea of using first-order responses is interesting, and it shows better performance than simple zero-order methods.
  • The ablation study is quite solid, and the overall performance surpasses that of all baselines on the selected baselines.
Critical weaknesses

-The method is quite limited to mouth-only audio–visual desynchronization scenarios. As also shown by the author, when they maintain the original character’s mouth movements and change only the appearance, the model’s performance clearly drops. Also, this method can be easily attacked by adding a first-order loss during training and letting the generator jointly optimize both first-order and zero-order objectives. I wonder whether these two objectives would mainly follow the same optimization direction without disrupting the overall training process much or introducing much additional computation.

-Another limitation I could find is that the authors first need prior knowledge of the generative method so that they can calculate the orthogonal residual space. For the MMDF and HiFi-AVDF examples, the improvement is because these datasets also contain some data generated using objectives similar to those used for the training data. I am really curious about the subset with objectives that are completely or substantially different from those used by the method to calculate the orthogonal residual. This will help me better understand the method’s generality and robustness.

-Also, I think that knowing the prior information about the generative method is similar to having real and synthetic pairs. This may make the comparison unfair, as you already know what the model will use as its objective and have a target around which to design your method. Also, the main benchmarks focus on areas where OtO performs well. Many baselines are designed for more general purposes.

Questions for authors

Check the weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer DCFp

Thank you very much for your thoughtful and constructive comments. We appreciate your careful examination of the scope of OtO, the role of the objective prior, and the evidence supporting objective coverage and robustness. In response, we have added full-pool comparisons, clarified the fixed objective library, expanded the objective-ablation experiments, and provided additional details about adaptive-generator attacks.

1. Scope and comparison

OtO targets edits that change the articulatory–acoustic response. On MMDF and HiFi-AVDF, the direct-pool AUROCs are 92.0 and 84.1, respectively, whereas the AUROCs on appearance-only edits are 58.3 and 56.8 (Tables 4–5). Including appearance-only clips gives reconstructed OtO full-pool AUROCs of 88.9 and 81.6 (Eq. 12 and Table 12). This reconstruction uses the shared authentic pools and the category counts in Table 1. For these two datasets, the supervised comparisons in Table 8 concern the direct pools.

Table 12: Evaluation pool counts

DatasetDirect fakeAppearance-only fakeAll fakeAuthentic
MMDF5,0005005,5002,000
HiFi-AVDF4,0004004,4001,800

Table 12: OtO AUROC (%)

DatasetDirectAppearance-onlyFull (reconstructed)
MMDF92.058.388.9
HiFi-AVDF84.156.881.6

Same-pool baseline comparison (AUROC, %)

DatasetMethodDirectAppearance-onlyFull
MMDFX-AVDT90.479.689.4
MMDFHAVIC88.581.787.9
MMDFOtO92.058.388.9
HiFi-AVDFX-AVDT79.873.479.2
HiFi-AVDFHAVIC76.477.976.5
HiFi-AVDFOtO84.156.881.6

The baseline direct-pool values are retained from the manuscript's Table 8 under its common training protocol, rather than taken from the baseline papers' native protocols. Full-pool values use the same authentic negatives and fixed scoring function across categories:

Afull=10Adirect+Aappearance11A_{\mathrm{full}}=\frac{10A_{\mathrm{direct}} + A_{\mathrm{appearance}}}{11}

All methods use identical eligible clips. X-AVDT is slightly above OtO on the MMDF full pool, whereas OtO retains an advantage on HiFi-AVDF. We therefore do not claim uniform full-pool superiority.

2. Objective prior

The public expert library is an explicit prior. Its empirical directions are fitted on authentic windows, without real/fake pairs or test-generator-specific expert selection. The matched columns in Table 2 are generator-informed diagnostics; the benchmark detector uses the same frozen library throughout (Section 3.2).

3. Objective coverage and library ablations

Table 2 holds the generator fixed and varies the removed directions. Matched removal gains 3.1–3.2 AUROC points over no projection, while the fixed union averages 90.9 compared with 88.5 without projection.

We have also revised Table 11(b) to assess modified branches rather than assign one objective-disclosure label to an entire generator row. HunyuanAvatar, MegActor-Sigma, and AniPortrait are assessed separately for their video branches. For joint clips, both the visual and audio branches must be identified and assessed. When the audio pipeline cannot be resolved, the clip is classified as Unresolved.

The aggregate inventory does not cross-tabulate joint clips by generator. Consequently, the previous disclosure-stratum AUROC/gain claims cannot be reliably reassigned to the revised categories. We have therefore removed these claims rather than carrying them over to the revised classification. Undisclosed objectives may overlap with the library, so we do not claim transfer to demonstrably out-of-library objectives.

Table 13: Objective-library ablations (AUROC, %)

Removed subspaceMMDFHiFi-AVDF
None88.479.3
Cosine plane only89.582.0
Cosine plane + fixed union92.084.1

On HiFi-AVDF, cosine-plane-only reaches 82.02 AUROC, a gain of 2.72 percentage points over None (79.30). The fixed union reaches 84.10, a further gain of 2.08 points over cosine-plane-only.

We additionally report the effect of omitting the generator's matching empirical expert from the fixed library.

Generator objectiveNoneMatching expert omittedUnionMatched
SyncNet87.488.290.090.6
AVSyncNet88.889.491.292.0
AV-HuBERT89.289.791.692.3
Mean88.589.190.991.6
AuthorsOpenReview ↗

Response to Reviewer DCFp

The first panel uses the benchmark direct pools. The second uses the controlled generators in Table 2 and removes the generator's matching empirical expert from the fixed library, retaining the other two experts and the sample-specific cosine plane.

The None setting removes neither component, whereas Matched retains the cosine plane and the matching empirical expert only. Every variant uses the same authentic splits and probe budget, with dependent bases, reference statistics, probe selection, and calibration refitted using authentic data only. Omitting the matching expert reduces AUROC by 1.8, 1.8, and 1.9 points relative to the fixed union for SyncNet, AVSyncNet, and AV-HuBERT, respectively. This diagnoses sensitivity to library coverage without identifying arbitrary undisclosed objectives.

4. Joint optimization and adaptive-generator attacks

Figure 2 varies the synchronization loss, while Table 6 tests adaptation to the detector. Full white-box adaptation lowers AUROC from 90.6 to 76.1. Adding the disclosed objective brings it back to 87.9 without fake-label retraining. This audit establishes sensitivity to score-aware adaptation (Sec 5.7 and 6).

Table 6: Adaptive-generator audit (AUROC, %)

The audit uses the matched SyncNet detector, comprising the analytic cosine plane and the SyncNet empirical basis. Black-box, gray-box, and white-box access respectively provide scalar score queries, independent surrogate gradients, and detector white-box gradients through a tracking surrogate.

AttackBeforeUnder attackAfter updateAttack changeUpdate recovery
Black-box90.688.289.8-2.4+1.6
Gray-box90.684.689.1-6.0+4.5
White-box90.676.187.9-14.5+11.8

Table 6 uses the matched SyncNet detector at the starting generator checkpoint (90.6 AUROC). Table 15 instead uses the frozen three-expert union detector; its continued-training control runs for 20k additional steps and reaches 89.8 AUROC. Joint response matching yields AUROCs of 86.9 with an independent predictor and 80.4 with the detector's predictor.

Table 15: Explicit response-matching comparison

Training objectiveAUROCStep timeΔ SyncΔ LPIPSGrad. cosine
Continued-training control89.81.00×00--
+ Independent predictor86.91.21×-0.0060.0040.18
+ Detector's predictor80.41.34×-0.0170.0110.31

Table 6-C: Independent attack configuration

Black-box access exposes scalar scores; gray-box access uses an independent predictor's surrogate gradients; white-box access exposes the frozen detector's scoring pipeline, with a differentiable tracking surrogate.

RegimeOptimization and budget
Black-boxOptimize a 32-dimensional additive audio-conditioning code per clip. Antithetic NES uses 2 Gaussian direction pairs per update, 50 updates, smoothing standard deviation 0.02, step size 0.01, and an ℓ₂ radius of 0.2 in standardized code coordinates. Total: 200 score queries per clip, excluding one final evaluation.
Gray-boxFine-tune generator weights for 10k steps with Adam, learning rate 2×10−52\times 10^{-5}, and batch size 32. Minimize the original generator objective plus 0.05 times an independently normalized Conformer/WavLM surrogate score, with its analytic cosine plane. No target predictor or empirical basis is exposed.
White-boxUse the same fine-tuning budget and original objective as gray-box, plus 0.05 times the target score normalized by authentic design-set scale. Predictor, whitening, projection, and score reference are frozen; backpropagation uses the frozen landmark surrogate specified for Table 15. Final scoring uses the original tracker.

The shared starting point is the SyncNet$\lambda_{\mathrm{sync}}=0.10$ generator with the matched SyncNet detector (90.6 AUROC). Training-based attacks use adaptation sources disjoint from evaluation; black-box optimization is performed per evaluated fake clip. These regimes describe different access and budget settings, not a budget-matched ranking. All use the same 1,800-source suite and three seeds.

Post-update fitting adds at most two additional disclosed-loss signature directions on the 3,500-file authentic design split and refits dependent residual statistics and calibration on their original authentic splits. For black-box adaptation, post-update fitting uses the disclosed attack wrapper and NES gradient estimates on authentic design files.

We would be very grateful if you could take a look at the corresponding revisions in the updated PDF, particularly Sections 3.2, 5.7, and 6; Tables 2, 6, 8, 11–13, and 15; and the additional attack configuration in Table 6-C. These changes clarify the scope of the detector, the role and coverage of the objective library, and the behavior under adaptive optimization.

Thank you for your careful reading and valuable suggestions.

Thanks for the rebuttal and the new experiment results. I think they have addressed most of my concerns. Also I have read the other reviewers' comments and I think this paper can provide insights to the community. I will raise my score to weak accept.

AuthorsOpenReview ↗

Response to Reviewer DCFp

Thank you very much for revisiting our response and for your thoughtful feedback. We truly appreciate your time and helpful comments.

Reviewer_myVROpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

The work presents a novel approach for audio-visual deepfake detection, focusing on lip-sync manipulations and articulatory audio-visual inconsistencies. The central issue that this work tackles is the rapid improvement of lip-sync generators that are specifically trained to bypass the simple, zero-order alignment tests that classic audio-visual consistency methods rely on. The authors propose Outside the Objective (OtO), a method which shifts the focus to the first-order acoustic responses that are extracted by applying small symmetric perturbations to the mouth mesh. In order to account for the before-mentioned issue, the authors construct a subspace defined by the components associated with the canonical cosine and known generator objectives (SyncNet, AVSyncNet and AV-HuBERT) that is later removed from the whitened acoustic-response space. The remaining components are used to predict a fakeness score. OtO achieves state-of-the-art results on four well-established benchmarks. Furthermore, the method's performance is supported by a suite of thorough evaluations (section 5).

Critical strengths

Novelty: OtO is a newly proposed approach that tackles a real technical issue in the field of audio-visual deepfake detection: the rapid improvement of lip-sync generators. While SotA methods focus detecting semantic or feature inconsistencies that are continuously improved by new generators, OtO focuses on detecting different types of forensic cues, specifically eliminating components associated with known generative objectives.

Methodology: the method has a strong mathematical foundation, making the methodology clearer and better motivated.

Experiments and results: the authors conduct a wide range of experiments which demonstrate the performance of OtO. Furthermore, the experiments are not simple ablations, but rather complex analyses which test the technical underlying implications of their method.

Critical weaknesses

Missing information: several implementation details and experimental choices are either not clear explained or not explained at all. This might lead to reproducibility concerns:

  1. Mouth-mesh model is missing.
  2. In section 5.1 the authors present a “controlled suite”. It is not clear if this suite is composed of samples generated by the authors using generators with objectives specified (SyncNet, AVSyncNet, AV-HuBERT) or the evaluation set is just a subset of the 4 datasets presented earlier.
  3. In section 5.2 the authors did not clearly state what is the generator used for which the λsync\lambda_{sync} is varied. Furthermore, the number of samples generated for each value of λsync\lambda_{sync} is not specified either. For a better understanding of the experimental setup, these details should be provided.
  4. In section 5.7 it is unclear what the generators are, what are they optimizing, what is their architecture, etc. This section requires substantial additional information

Limited applicability: in section 5.6 the authors observe that OtO performs well only on “visual articulation” and “joint” subsets, while performing at random chance for appearance-only manipulations. In today's scenarios where data is fully synthesized by large models like Sora with diverse semantic content this might represent a limitation. Nevertheless, the presented method can serve as a useful tool specifically designed for these kinds of attacks.

Minor:

  1. Related work is largely a rephrasing of the first paragraph from introduction. For a better understanding of the field the authors should consider enlarging this section.
  2. A mean over the "Generation objective" in Table 2 would have made the finding clearer (precisely that union is better on average).
  3. A whitening ablation would be useful to isolate the contribution of this design choice.
Questions for authors
  • Why are the authors using precomputed speech context-level approximations for generation objectives (SyncNet, AVSyncNet and AV-HuBERT) and not sample-level directions similar to the cosine similarity case? Is this a trade-off? If so, how much performance is lost and how much computation is saved?
3 comments
AuthorsOpenReview ↗

Response to Reviewer myVR

Thank you very much for your detailed and constructive feedback. We sincerely appreciate your comments on the implementation details, experimental design, computational efficiency, and presentation of the results. We have expanded the implementation description, added comparisons for empirical-basis estimation and response whitening, clarified the averaging procedure, and further specified the scope of the fully generated content experiments.

1. Implementation details and source units

The controlled suite is author-generated and separate from the four benchmark evaluation pools. We retain one deterministic eligible clip per evaluated media file, rather than one per original native source video; multiple evaluated media files may share the same native source. Table 11(a) reports scored-clip counts separately from distinct native-source counts.

For Figure 2, we use a Wav2Lip-style audio-conditioned encoder–decoder and vary the SyncNet loss coefficient. Each setting starts from 1,800 authentic sources and uses three generation seeds, resulting in 5,400 attempted variants per setting. Each seed is evaluated against the same eligible authentic pool, and AUROCs are averaged across seeds. We use an identity-clustered paired bootstrap that preserves paired sources and settings.

The three controlled generator variants share the same Wav2Lip-style backbone, identity-disjoint LRS2 training split, and training protocol. Only the generator objective (SyncNet, AVSyncNet, or AV-HuBERT) changes, while all other loss terms and their weights remain fixed.

Appendix B and Table 10 specify the mouth tracker, controlled generator setup, exact expert-loss definitions and checkpoint sources, and the evaluation-only VocaLiST reporter. Section 5.7 distinguishes the controlled interventions, source-linked benchmark composition, objective-disclosure strata, and adaptive audit (Tables 6 and 11). Table 6-C specifies the three attack regimes and their budgets.

2. Context pooling and response whitening

Per-sample empirical estimation requires expert differentiation and a regularized Riesz solve. Context pooling amortizes this computation while retaining the sample-specific analytic plane. The held-out median empirical-basis capture is 0.83 (Appendix B).

We report empirical-basis estimation and response-whitening comparisons separately; Appendix C provides a distinct audit of residual score covariance. The same-pool accuracy and runtime comparison shows that the sample-level row reaches 92.3/84.2 AUROC at 1.71 seconds per clip, compared with 92.0/84.1 at 0.68 seconds per clip for context-level pooling. Thus, context pooling provides similar AUROC with approximately 2.5× faster inference. All metric variants refit their dependent components on authentic data and retain the same evaluation pool and probe budget.

Table 14: Empirical-basis estimation comparison

Empirical basisMMDF AUROCHiFi-AVDF AUROCSeconds/clip
Context-level92.084.10.68
Sample-level92.384.21.71

Response-whitening comparison

Whitening metricMMDF no projection / projectedHiFi-AVDF no projection / projected
Identity87.9 / 90.678.7 / 81.8
Diagonal shrinkage88.2 / 91.479.1 / 83.0
Full shrinkage88.4 / 92.079.3 / 84.1

3. Macro-averages and presentation

Table 2 now reports column-wise macro-averages. The fixed union averages 90.9, compared with 88.5 without projection and 91.6 for generator-informed matching.

We have also revised the Introduction to explain the measured response, its authentic reference, and the removed directions before the formal methodological development. Section 2 now separates the supervision signal from the forensic cue.

Table 2: Added macro-average row (AUROC, %)

ObjectiveNoneSyncNetAVSyncNetAV-HuBERTRandomEnergyUnion
Mean88.589.089.189.088.789.090.9

4. Fully generated content

Joint audio–visual edits reach 92.8/84.7 AUROC on MMDF/HiFi-AVDF (Table 5). These are edit-category results on eligible clips; fully generated videos are not separately isolated in the current evaluation.

A usable track and speech permit scoring, but do not guarantee that the response is informative. We clarify this distinction in Section 6 and avoid extending the reported results beyond the evaluated edit categories.

We would be very grateful if you could take a look at the corresponding revisions in the updated PDF, particularly Sections 2, 5.1, 5.3, 5.7, and 6; Tables 2, 5, 10, 11, and 14; and Appendix B. These sections provide the additional implementation details, computational comparisons, and clarifications of the experimental scope.

Thank you again for your careful review and helpful suggestions.

Dear authors,

Thank you for your response and the updated manuscript. Most of my concerns have been addressed, especially the missing information and ablations (both on context-level vs sample-level and response whitening), yet the limited applicability still stands. I will keep my rating as Weak Acceptance.

AuthorsOpenReview ↗

Response to Reviewer myVR

Thank you very much for your careful review and constructive feedback. We sincerely appreciate your time and consideration.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response to the Area Chair and Reviewers

Thank you very much for your careful assessment and constructive feedback. We sincerely appreciate the time and effort you have devoted to reviewing our work. In the revised manuscript, we have clarified several aspects of the method, expanded the experimental comparisons, and improved the presentation and reproducibility of our results, while keeping the method and calibration protocol unchanged.

The main revisions are summarized below:

  • Additional comparisons. Tables 14 and 15 provide additional comparisons for empirical-basis estimation, response whitening, and explicit first-order response matching, with clearly specified experimental controls.
  • Scope of the detector. We distinguish direct-mechanism edits from appearance-only edits and report the reconstructed full-pool AUROC in Table 12. We also clarify that Table 6 evaluates adaptation to the final detection score, whereas Table 15 evaluates explicit first-order response matching using the frozen three-expert union detector.
  • Method and implementation. We distinguish the sample-specific analytic cosine plane from the fixed empirical objective library, provide additional implementation details for the controlled interventions in Table 10, and add macro-averages to Table 2. Table 13 further separates analytic-plane removal from empirical-library ablations and includes the omission of the matching expert.
  • Benchmark coverage and reproducibility. Table 11 reports aggregate evaluation-clip counts, distinct native-source counts, and branch-level objective-disclosure evidence, separately from expert-library coverage. We have also clarified the contents and limitations of the supplementary archive.

We would be very grateful if you could take a look at the corresponding revisions in the updated PDF, particularly Sections 3.2, 5.1, and 5.7; Tables 2, 6, and 10–15; and Appendices B and C. These revisions provide the additional methodological clarification, experimental evidence, and reproducibility details discussed above.

Thank you again for your thoughtful feedback and consideration.

Dear Reviewers,

The authors have responded to your questions. If you haven’t already done so, please review their responses and let them know whether your concerns have been addressed. Please use the discussion to raise any remaining concerns or follow-up questions.

Thank you for your time!

Best,

AC

Thank you very much for coordinating the discussion and for your careful consideration of our work. We sincerely appreciate your time and support.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.