- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes an audiovisual deepfake detector trained only on authentic data. It perturbs tracked mouth trajectories and measures how the predictions of a frozen visual-to-acoustic model change. These responses are compared with what is expected for authentic speech, conditioned on the observed audio. The method then removes response directions associated with specified generator objectives before computing an anomaly score. The motivation is that generators may learn to satisfy the same audiovisual consistency objectives that detectors rely on. The paper evaluates this through controlled changes to generator objectives, several ablations, and experiments on four datasets.
Critical strengths
I find the main idea interesting. If a generator is explicitly trained to satisfy a particular consistency measure, relying on that measure for detection may become less effective. Looking at local predictor responses, and removing directions related to these objectives, is a useful alternative to investigate.
There are several relevant experiments supporting the approach. In particular, the comparison between matched, mismatched, random and energy-matched removal is useful for understanding whether the choice of removed directions is important. The ablations suggest that removing these directions helps even when the model already uses audio conditioning. The reported projection gains of 3.6 and 4.8 AUROC points on MMDF and HiFi-AVDF are good, especially given that the method does not use fake examples for training.
Critical weaknesses
I found the paper difficult to follow, and some important implementation details remain unclear after reading the supplement. The distinction between the predictor’s response, the generator-loss directions and the conditional reference could be explained more directly before introducing the maths. More importantly, I could not find the architectures of the controlled generators or the exact scalar losses used for the SyncNet, AVSyncNet and AV-HuBERT experiments. Appendix B also says that the tracker version and confidence thresholds were fixed, but does not specify them. These details are important because the controlled experiments are a central part of the insights for the method.
There are enough datasets to make the evaluation meaningful, but I am less clear about the range of generators covered. Since the method is motivated by generator objectives, a breakdown by generator and objective would be more informative than only aggregate dataset results. Section 5.7 reports gains for “unreported-objective” strata, but does not clearly identify their composition or sample counts. It is also unclear exactly which fixed objective library was used for the main results. The selected evaluation pools exclude appearance-only manipulations from the main scores, although these are evaluated separately. This is understandable given the mechanism, but makes it important to explain the selection protocol and scope of the generalisation claims a bit more precisely.
Questions for authors
Could you specify the controlled generator architectures, exact expert-loss definitions and expert checkpoints used in Sections 5.2–5.3? Which independent model reports synchronisation quality in Figure 2?
Was the same objective library fixed across all benchmark datasets, without choosing objectives based on the generator being evaluated? Which generators and how many source files are included in the “unreported-objective” strata?
The reproducibility statement says that the supplementary material includes source-linked manifests, objective/expert configurations and selected probe indices. I can only see a supplementary pdf uploaded, and I couldn't find these in there. Could you please clarify?
6 comments
Response to Reviewer 9UxA
Declared expert scalar losses.
Each pixel-consuming expert is composed with the differentiable piecewise-affine mouth warp .
| Expert | Loss | Checkpoint |
|---|---|---|
| SyncNet | , 5-frame audio–visual cosine probability (). | Official Wav2Lip LRS2 expert release (lipsync_expert.pth). |
| AVSyncNet | , our binary cross-entropy objective using the AVSyncNet checkpoint (distinct from Yaman et al.'s three-way relative stabilized loss). | Yaman et al. (2024) synchronization expert checkpoint (LRS2). |
| AV-HuBERT | , frame-level unit posteriors from the frozen -means cluster projection head. | Official AV-HuBERT Large LRS3+VoxCeleb2 iter-5 checkpoint (large_vox_iter5.pt). |
2. Fixed library and coverage
The three-expert empirical library is fitted on authentic data and frozen before calibration for all four benchmarks. The analytic cosine plane is computed separately for each sample (Section 3.2).
Table 11(a) reports scored clips by evaluated media file and distinct native-source counts separately. Table 11(b) assesses video and audio branches separately: HunyuanAvatar, MegActor-Sigma, and AniPortrait are each checked for their video branch, while joint clips require records for both visual and audio pipelines and versions. If an audio pipeline cannot be identified, the joint clip is classified as Unresolved.
We refined the disclosure classification from generator-level to branch-level because joint audio–visual edits may involve different video and audio pipelines. Since the available inventory does not cross-tabulate joint clips by generator, the previous stratum-specific AUROCs and gains cannot be reliably reassigned to the revised categories. We therefore removed these results rather than carrying forward potentially mismatched statistics.
Table 11(a): Scored-clip allocation and distinct native sources
| Dataset | Generator / edit category | Scored direct clips | Scored appearance-only clips | Distinct native source videos |
|---|---|---|---|---|
| MMDF | HunyuanAvatar | 1,487 | 0 | 503 |
| MMDF | MegActor-Sigma | 1,596 | 0 | 547 |
| MMDF | AniPortrait | 1,517 | 0 | 521 |
| MMDF | Audio-only edits | 400 | 0 | 183 |
| MMDF | Appearance-only controls | 0 | 500 | 219 |
| MMDF total | 5,000 | 500 | --- | |
| HiFi-AVDF | Veo 3.1 | 668 | 0 | 241 |
| HiFi-AVDF | Sora 2 | 641 | 0 | 228 |
| HiFi-AVDF | Kling 2.5 | 597 | 0 | 216 |
| HiFi-AVDF | Wan 2.5 | 573 | 0 | 203 |
| HiFi-AVDF | Seedance 1.0 | 518 | 0 | 187 |
| HiFi-AVDF | PixVerse V5 | 483 | 0 | 171 |
| HiFi-AVDF | Audio-only edits | 520 | 0 | 232 |
| HiFi-AVDF | Appearance-only controls | 0 | 400 | 178 |
| HiFi-AVDF total | 4,000 | 400 | --- |
Clip counts refer to one eligible clip per evaluated media file. Distinct native-source counts are reported separately in Table 11(a); multiple evaluated media files may share the same native source video.
Response to Reviewer 9UxA
Table 11(b): Branch-level objective-disclosure evidence
| Pipeline / sample group | Evidence available | What remains unverified | Disclosure status |
|---|---|---|---|
| MMDF / HunyuanAvatar (1,487 total direct clips) | Video pipeline/version identified; relevant video objective is not disclosed. | Visual-only/joint split within the generator row; audio branch for joint clips. | Unreported for video-only; joint clips unresolved. |
| MMDF / MegActor-Sigma (1,596 total direct clips) | Video pipeline/version and relevant video objectives are documented. | Visual-only/joint split within the generator row; audio branch for joint clips. | Documented for video-only; joint clips unresolved. |
| MMDF / AniPortrait (1,517 total direct clips) | Video pipeline/version and relevant video objectives are documented. | Visual-only/joint split within the generator row; audio branch for joint clips. | Documented for video-only; joint clips unresolved. |
| MMDF / joint audio–visual clips (1,700 aggregate) | Aggregate edit-category count is reported in Table 1. | Per-media-file audio pipeline/version and objective. | Unresolved. |
| MMDF / audio-only edits (400) | Audio-only category and scored-clip count are reported. | Audio pipeline/version and objective. | Unresolved. |
| HiFi-AVDF / six video generators (3,480 total direct clips) | Generator families and versions are identified; relevant video objectives are not disclosed. | Visual-only/joint split within generator rows; audio branch for joint clips. | Unreported for video-only; joint clips unresolved. |
| HiFi-AVDF / joint audio–visual clips (1,520 aggregate) | Aggregate edit-category count is reported in Table 1. | Per-media-file audio pipeline/version and objective. | Unresolved. |
| HiFi-AVDF / audio-only edits (520) | Audio-only category and scored-clip count are reported. | Audio pipeline/version and objective. | Unresolved. |
The joint rows are branch checks of subsets already counted in the video-generator rows of Table 11(a), not additional clips. The available aggregate allocation does not cross-tabulate joint clips by generator. For a joint clip, both modified branches must be identified and assessed; if either branch cannot be identified, the clip is Unresolved. We therefore do not carry forward the previous stratum-specific AUROC/gain claims under this revised classification.
3. Supplementary materials
We apologize that the earlier supplementary archive did not include the probe-selection artifacts needed to inspect the reported selection.We have now added the numerical candidate bank, the selected probe indices and displacement fields, the stored E-optimal selection trace, and a solver-verification utility to the revised archive.
The utility verifies the selection using the stored design matrices; it is intended for probe-selection verification and is not the full detector implementation.The archive also includes manifest schemas (field definitions and header-only templates), but does not include populated row-level source-linked manifests or benchmark media.
Response to Reviewer 9UxA
Thank you very much for your careful and constructive comments. We sincerely appreciate your questions regarding the controlled generator experiments, the independence of the evaluation reporter, the fixed objective library, benchmark-level objective disclosure, and reproducibility. We have expanded the implementation details, clarified the branch-level disclosure analysis, and added the probe-selection artifacts to the revised supplementary archive.
1. Controlled objectives and independent reporter
Table 10 specifies the tracker and generator setup. The three controlled generator variants share the same Wav2Lip-style backbone, identity-disjoint LRS2 training split, and training protocol. Only the generator objective (SyncNet, AVSyncNet, or AV-HuBERT) changes, while all other loss terms and their weights remain fixed (Section 5.3 and Appendix Table 10).
Table 10 also lists the SyncNet, AVSyncNet, and AV-HuBERT losses with their checkpoint sources. The AV-HuBERT loss uses the audio-to-visual KL divergence.
Figure 2 reports synchronization quality using the LRS2-pretrained VocaLiST model, which is excluded from generator optimization and the detector's objective library.
Table 10: Implementation and Figure 2 protocol
mediapipe==0.10.14,face_landmarker.task); fixed 40-pointFACEMESH_LIPSindices; detection/presence/tracking thresholds of 0.5; valid frames receive equal weight in mesh centering, and each 1.0-second window requires at least 80% valid frames.