- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This work introduces a diagnostic benchmark used to conduct a thorough analysis of existing audio-visual deepfake detectors, focusing on the evidences used to provide a prediction and studying what happens when a specific evidence the detectors rely on does not survive a shift (specifically during OOD evaluations). The analysis focuses on four evidence routes: dataset artifacts (dataset specific shortcuts that should not relate to the task of deepfake detection, for example leading silence), single modality (the evidence is present in only one of the two modalities), generator forensic cue (a specific generative fingerprint that is tied to a specific deepfake detector) and audio-visual correspondence (dissimilarities between the two media streams). The authors argue that a single average AUC is not sufficiently insightful to analyse why a model fails in the OOD setup, so they introduce a metric for each evidence route and specific interventions to test their hypotheses. These are: leading silence trim, speaker identity (seen vs unseen evaluation), modality skew (difference between RV-FA AUC and FV-RA AUC) and generator overfit (in-domain vs cross-generator AUC). To further augment these results, the authors provide interventions such as: silence trim, identity splits, audio-visual correspondence breaking (rolling, shuffling, swapping audio).
One of the authors' findings is that the evidence route chosen by the model is determined by the training data rather than the model's architecture, which further raises concerns about data quality used in the field (e.g. AVDeepfake-1M leading silence shortcut). Furthermore, these evidences can survive the domain shift (e.g. training and testing on two different datasets but with a similar shortcut), which reduces the usefulness of simple OOD evaluations.
Critical strengths
The authors propose a thorough evaluation of the evidence routes chosen by audio-visual deepfake detectors. This evaluation is impactful for the field, as deepfake detection is severely affected by a lack of generalisation to new conditions: new generators, new semantic information, perturbations such as compressions, delays in real videos etc. To effectively counter these issues, researchers should be able to identify why audio-visual deepfake detection models fail in OOD setups before deploying solutions. This framework can be impactful and provide insightful information (specifically the route signatures, their measurements and the intervention protocols).
The authors make a clear distinction between shortcut reliance and OOD failure. The former implies learning a specific fingerprint from the training dataset which can still be present in the OOD setup, thus reporting an average metric over OOD is not sufficiently insightful. Furthermore, different target datasets (OOD testing setup) can preserve or break different shortcuts. Without a informative and meaningful analysis, it is unclear what causes the performance degradation.
Critical weaknesses
Novelty: the second stated contribution is consistent with other works cited by the authors (e.g. Smeu et al. 2025). In the cited work, the AVH-Align/sup AV1M-trained is shown to degrade when the “evidence” (the audio shortcut) is removed and the FAVC-trained model is not affected (AUC remains stable). I would rather reframe the claim as a thorough evaluation of the forensic cues reliance for deepfake detection decision using a complex suite of interventions and tests. (This is also supported by the third and fourth sentences from the “Related work” paragraph).
Readability; Soundness; Presentation: The main paper text is sometimes hard to follow and several metrics and procedures are not clearly defined before applying them.
- The evidence routes: authors present 4 evidence routes: dataset artifact, single modality (visual/audio), generator-specific cue, audio–visual correspondence. These are not mutually exclusive (e.g. a visual reliance can be caused by a generator-specific cue; leading silence can be both a dataset artifact and lead to modality skew towards the audio stream), thus pairing a model with a specific evidence route is not consistent. In this regard, a relevant example is AVH-Align/sup FAVC-trained model; for this model the authors concluded that the “visual“ route is the correct one on which the model relies on (Tab. 2). But the lack of audio-visual correspondence or reliance on leading silence does not result in a clear evidence route (as it is defined by the authors). In this particular case, the reliance may be on a visual generator cue or an identity cue or a visual shortcut which is unknown (similar to leading silence for audio).
- One of the main axis, generator overfit, is not measured throughout the entire paper. This raises a serious concern, since the authors clearly states that the four evidence measurements are important to identify the correct route.
- unsup_sync model and corr_sync method are mentioned but not properly introduced. The only explanations are provided in the appendix. This is problematic because the methods are important for the actual methodology.
- is not defined but the limits are set (Section 2). The authors should introduce the formula if they present the thresholds for decisions, otherwise the thresholds are not contextualized. Also, it is not clear how these thresholds were chosen. Furthermore, this metric is then utilized only in the appendix (App. J, Tab. 10). While this table is referenced in the main text, I think there is a discrepancy between the apparent importance of this metric (Section 2) and the actual usage.
- The distinction between manipulation artifacts which the model may rely on for prediction and shortcuts (dataset-induced, or methodological) is not clearly stated.
Questions for authors
- The authors mentioned paired bootstrapping for leading silence, speaker identity and modality-skew. While in the leading silence case it makes sense to pair bootstrap, I am not sure what is the reasoning behind pair bootstrapping in the other two cases (since the sampling subsets are distinct). Is there a particular reason for this protocol choice? What happens if instead of pair bootstrapping, an independent bootstrapping is used? (similar to the "generator overfit" axis).
8 comments
Response - Part 2
W5. Definitions of unsup_sync and corr_sync
The revision makes the scorer definitions self-contained in the main text, so each result can be interpreted without consulting the appendix. The revision will introduce each scorer at first use in the main text or setup table and state the score, readout, and training data in every results table.
- unsup_sync is an offset-prediction model trained on AV1M real clips without fake labels and evaluated over a ±15-frame window. We will distinguish its absolute-logit readout from the exploratory relative-peak readout.
- corr_sync currently refers to two different scorers. In Tables 10–11, it is the zero-parameter mean per-frame cosine between AV-HuBERT audio and visual features, maximised over shifts −3 to 3, with lower agreement indicating fake. This scorer will retain the name corr_sync. The FAVC-trained logistic head over eight such statistics, currently used in App. J and for the DeepSpeak 0.598 result, will receive a distinct name.
W6. Pairing metric and thresholds
The revision places the pairing quantities, their direction, and the reporting convention in one main-text definition block before any route verdict is reported. The revision will define them together in the main text:
- H: honest AUC of intact fakes against intact reals.
- H_T: honest AUC of the same fakes against perturbed reals.
- ΔH_T = H − H_T: positive values indicate a performance drop.
- R_T: AUC separating perturbed from intact reals, with perturbed reals as the positive class under the fake-oriented score; R_T = 0.5 indicates no aggregate separation.
Because H and H_T use the same fakes, ΔH_T isolates the effect of perturbing the real class. Table 10 currently uses the opposite sign and will be converted for consistency.
The thresholds were not calibrated and will be described only as a CI-based reporting convention: a dependency is supported when the lower CI limit is at least 0.70 for R_T and at least 0.05 for ΔH_T on the same declared break. The main paper will include the compact summary driving these labels; per-break detail will remain in the appendix.
W7. Manipulation artifacts and shortcuts
The revision gives the two terms complementary roles rather than treating them as mutually exclusive. A manipulation artifact is a cue introduced by synthesis or editing. A shortcut is a label-correlated cue on which a detector relies but which is unstable in the settings where it is expected to generalise. Thus, a manipulation artifact can function as a shortcut when it is predictive in the training setting but not stable across the intended deployment settings.
Leading silence and identity--label association are dataset-associated cues. The revision will report cue origin and measured dependence separately, use this terminology consistently in Related Work and the taxonomy figure, and retain an unresolved attribution whenever the intervention cannot identify the underlying cue.
Response - Part 3
Q1. Bootstrap procedures
We corrected the resampling description and re-ran the relevant analyses. Not all intervals were computed with paired bootstraps. The revision will state the actual procedures:
- Leading trim: paired, class-stratified resampling of the same clips.
- Identity: independent class-stratified resampling of two disjoint cohorts.
- Modality skew: independent resampling in the submitted analysis, despite the two cells sharing 151 real clips.
For leading trim, recomputing all 19 detectors from the Table 1 arms reproduces every estimate and significance decision. Resampling the naive and trimmed arms independently makes the intervals approximately 6× wider (median; range 3.7–35×), and 8 of the 12 significant effects no longer meet the original criterion. Since the two arms contain the same clips, paired resampling is the appropriate analysis.
For modality skew, we added a joint resampling analysis (new) that resamples shared reals once per draw and the two fake cells separately (18 detectors). The resulting intervals are the same or narrower (median width ratio 0.89; range 0.45–1.01), and no significance decision changes. The revision will use this joint procedure.
A methodology table will specify, for every axis, the sampling unit, pairing status, bootstrap type, and treatment of shared reals. The independent-bootstrap comparison and resampling code will be included in the appendix.
Response - Part 1
Dear Reviewer v3Hv,
Your comments motivated a substantial clarification of the framework. The revision separates input dependency from cue attribution, reduces the measured signature to the three contrasts actually available across the suite, and corrects the resampling description. Sec. 2 is now organised around these definitions and their associated interventions. Results marked (new) were obtained after submission.
W1. Novelty of the second contribution
The novelty of the revised contribution lies in a systematic, intervention-based study of forensic evidence use across detector families, together with fixed-architecture controls that isolate training-dependent sensitivity changes. The revised contribution is: fixed-architecture controls show training-dependent shifts in measured audio/visual and pairing sensitivity, within the scope of the evaluated data and representations. We will explicitly distinguish this from Smeu et al. (2025). Their AV1M- versus FAVC-trained AVH-Align comparison identifies an audio-artifact difference. Our study adds a common intervention suite across detector families, a fixed-architecture mixture sweep at fixed training size (10 seeds: FAVC FV-RA 0.505 to 0.980; DFTIMIT 0.488 to 0.846), and matched and same-source controls that isolate part of the effect to an audio-side label cue. The cross-backbone results constrain this claim. VFD shows a smaller rise in measured visual use (concat 0.528 to 0.649), while BA-TFD+ changes little and non-monotonically (0.679 to 0.710); both are three-seed studies. We will remove wording that presents them as equivalent replications. A new fixed-architecture correspondence experiment further shows sensitivity modulation, not improved transfer: replacing FAVC training fakes with pairing-break fakes raises the roll-15 honest-AUC drop from 0.07 to 0.90, while DFTIMIT FV-RA falls from 0.89 to 0.81.
W2. Readability, soundness, and definitions
The revision resolves the presentation issue by defining every measurement before it is used and by placing the full protocol at the start of Sec. 2. Sec. 2 will now begin with a notation-and-protocol table and follow the sequence: metrics → interventions → reporting labels. One sign convention will be used throughout. The table will define the three measured contrasts and their resampling units: paired leading-trim AUC change on the same clips; identity seen/unseen difference on disjoint, composition-matched cohorts; and RV-FA minus FV-RA AUC. It will then define the pairing quantities H, H_T, ΔH_T, and R_T, followed by the descriptive CI-based reporting convention. The untrained cosine scorer and trained logistic readout will be separately named and defined before their first results (W5). A compact interval summary will move into the main paper; detailed per-break results will remain in the appendix.
W3. Overlapping evidence categories
The revised taxonomy removes the false choice between overlapping concepts by reporting input dependency and cue attribution separately. The revision will report two separate quantities:
Accordingly, the FAVC-trained AVH-Align head will no longer be labelled a “visual route.” It will be described as visual dependence supported; visual cue unresolved. Pairing dependence is not supported by the tested breaks, but the tests do not exclude generator, identity, or another visual cue. The generator leave-out control was run on separate linear heads and will not be used to resolve this AVH-Align case.
W4. Generator overfit
We have corrected the signature to contain only the three contrasts measured across the suite. Generator overfit is therefore removed as a fourth coordinate. We will remove it from the abstract, Fig. 1, Sec. 2, contributions, and coverage table.
The cross-corpus values will instead be reported as cross-corpus transfer gaps, which are outcomes the study seeks to explain rather than route measurements. For the four detectors with known training corpora, these gaps are positive; for example, AVH-Align trained on FAVC has a 0.353 gap on AV1M.
The new within-FAVC generator leave-out study is a separate control, not a replacement fourth axis. It uses frozen-feature linear heads, equal training-fake counts, five seeds, and identical reals and test cells across arms. The largest mean gaps are 0.012 for visual (95% CI [0.010, 0.014]), 0.008 for audio+visual, and 0.001 for correspondence.