acceptodds
Under review as a conference paper at ICLR 2027

Training Data Redirects Evidence Routes in Audio-Visual Deepfake Detection

Abstract

Audio-visual deepfake detectors reach near-perfect in-domain AUC yet fail unpredictably under distribution shift, and accuracy alone does not say why. We argue that what matters is the detector's evidence route, the channel through which predictive evidence reaches its decision, and that the route is a property of the trained instance rather than of the architecture alone. AVRouteBench makes the route measurable in two steps: an intervention-based signature (leading silence, speaker identity, modality skew, generator overfit) that narrows the route to a few candidates, and a direct confirmation intervention, such as breaking the audio-visual pairing while leaving each stream intact, that tests each candidate. Two findings follow. First, training data can redirect the route within what a fixed architecture allows. With one frozen AV-HuBERT backbone and one logistic head, changing only the training mixture moves fake-video/real-audio AUC from 0.495 to 0.980. The effect survives a matched sweep that fixes class and cell balance and the within-identity real-fake pairing, and a same-source intervention traces part of it to the availability of an audio-side label cue. A published detector with identical architecture and code learns a visual route from one training source and an audio-dominated route from another, and on two other frozen backbones the size of the effect tracks the visual evidence a linear head can read from the representation. Second, the route sets the failure mode: which edits a detector still catches under shift depends on whether the evidence it reads survives the target's manipulation type. Audio-visual correspondence is the case we characterize in detail. Across 19 detector instances from ten families, AVRouteBench shifts detector evaluation from how accurate is it? to which evidence does it use, and will that evidence hold under shift?

Primary area:
interpretability and explainable AI
Keywords:
audio-visual deepfake detection, evidence routing, causal intervention, shortcut learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_v3HvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This work introduces a diagnostic benchmark used to conduct a thorough analysis of existing audio-visual deepfake detectors, focusing on the evidences used to provide a prediction and studying what happens when a specific evidence the detectors rely on does not survive a shift (specifically during OOD evaluations). The analysis focuses on four evidence routes: dataset artifacts (dataset specific shortcuts that should not relate to the task of deepfake detection, for example leading silence), single modality (the evidence is present in only one of the two modalities), generator forensic cue (a specific generative fingerprint that is tied to a specific deepfake detector) and audio-visual correspondence (dissimilarities between the two media streams). The authors argue that a single average AUC is not sufficiently insightful to analyse why a model fails in the OOD setup, so they introduce a metric for each evidence route and specific interventions to test their hypotheses. These are: leading silence trim, speaker identity (seen vs unseen evaluation), modality skew (difference between RV-FA AUC and FV-RA AUC) and generator overfit (in-domain vs cross-generator AUC). To further augment these results, the authors provide interventions such as: silence trim, identity splits, audio-visual correspondence breaking (rolling, shuffling, swapping audio).

One of the authors' findings is that the evidence route chosen by the model is determined by the training data rather than the model's architecture, which further raises concerns about data quality used in the field (e.g. AVDeepfake-1M leading silence shortcut). Furthermore, these evidences can survive the domain shift (e.g. training and testing on two different datasets but with a similar shortcut), which reduces the usefulness of simple OOD evaluations.

Critical strengths

The authors propose a thorough evaluation of the evidence routes chosen by audio-visual deepfake detectors. This evaluation is impactful for the field, as deepfake detection is severely affected by a lack of generalisation to new conditions: new generators, new semantic information, perturbations such as compressions, delays in real videos etc. To effectively counter these issues, researchers should be able to identify why audio-visual deepfake detection models fail in OOD setups before deploying solutions. This framework can be impactful and provide insightful information (specifically the route signatures, their measurements and the intervention protocols).

The authors make a clear distinction between shortcut reliance and OOD failure. The former implies learning a specific fingerprint from the training dataset which can still be present in the OOD setup, thus reporting an average metric over OOD is not sufficiently insightful. Furthermore, different target datasets (OOD testing setup) can preserve or break different shortcuts. Without a informative and meaningful analysis, it is unclear what causes the performance degradation.

Critical weaknesses

Novelty: the second stated contribution is consistent with other works cited by the authors (e.g. Smeu et al. 2025). In the cited work, the AVH-Align/sup AV1M-trained is shown to degrade when the “evidence” (the audio shortcut) is removed and the FAVC-trained model is not affected (AUC remains stable). I would rather reframe the claim as a thorough evaluation of the forensic cues reliance for deepfake detection decision using a complex suite of interventions and tests. (This is also supported by the third and fourth sentences from the “Related work” paragraph).

Readability; Soundness; Presentation: The main paper text is sometimes hard to follow and several metrics and procedures are not clearly defined before applying them.

  1. The evidence routes: authors present 4 evidence routes: dataset artifact, single modality (visual/audio), generator-specific cue, audio–visual correspondence. These are not mutually exclusive (e.g. a visual reliance can be caused by a generator-specific cue; leading silence can be both a dataset artifact and lead to modality skew towards the audio stream), thus pairing a model with a specific evidence route is not consistent. In this regard, a relevant example is AVH-Align/sup FAVC-trained model; for this model the authors concluded that the “visual“ route is the correct one on which the model relies on (Tab. 2). But the lack of audio-visual correspondence or reliance on leading silence does not result in a clear evidence route (as it is defined by the authors). In this particular case, the reliance may be on a visual generator cue or an identity cue or a visual shortcut which is unknown (similar to leading silence for audio).
  2. One of the main axis, generator overfit, is not measured throughout the entire paper. This raises a serious concern, since the authors clearly states that the four evidence measurements are important to identify the correct route.
  3. unsup_sync model and corr_sync method are mentioned but not properly introduced. The only explanations are provided in the appendix. This is problematic because the methods are important for the actual methodology.
  4. RT(f)R_T(f) is not defined but the limits are set (Section 2). The authors should introduce the formula if they present the thresholds for decisions, otherwise the thresholds are not contextualized. Also, it is not clear how these thresholds were chosen. Furthermore, this metric is then utilized only in the appendix (App. J, Tab. 10). While this table is referenced in the main text, I think there is a discrepancy between the apparent importance of this metric (Section 2) and the actual usage.
  5. The distinction between manipulation artifacts which the model may rely on for prediction and shortcuts (dataset-induced, or methodological) is not clearly stated.
Questions for authors
  • The authors mentioned paired bootstrapping for leading silence, speaker identity and modality-skew. While in the leading silence case it makes sense to pair bootstrap, I am not sure what is the reasoning behind pair bootstrapping in the other two cases (since the sampling subsets are distinct). Is there a particular reason for this protocol choice? What happens if instead of pair bootstrapping, an independent bootstrapping is used? (similar to the "generator overfit" axis).
8 comments
AuthorsOpenReview ↗

Response - Part 1

Dear Reviewer v3Hv,

Your comments motivated a substantial clarification of the framework. The revision separates input dependency from cue attribution, reduces the measured signature to the three contrasts actually available across the suite, and corrects the resampling description. Sec. 2 is now organised around these definitions and their associated interventions. Results marked (new) were obtained after submission.

W1. Novelty of the second contribution

The novelty of the revised contribution lies in a systematic, intervention-based study of forensic evidence use across detector families, together with fixed-architecture controls that isolate training-dependent sensitivity changes. The revised contribution is: fixed-architecture controls show training-dependent shifts in measured audio/visual and pairing sensitivity, within the scope of the evaluated data and representations. We will explicitly distinguish this from Smeu et al. (2025). Their AV1M- versus FAVC-trained AVH-Align comparison identifies an audio-artifact difference. Our study adds a common intervention suite across detector families, a fixed-architecture mixture sweep at fixed training size (10 seeds: FAVC FV-RA 0.505 to 0.980; DFTIMIT 0.488 to 0.846), and matched and same-source controls that isolate part of the effect to an audio-side label cue. The cross-backbone results constrain this claim. VFD shows a smaller rise in measured visual use (concat 0.528 to 0.649), while BA-TFD+ changes little and non-monotonically (0.679 to 0.710); both are three-seed studies. We will remove wording that presents them as equivalent replications. A new fixed-architecture correspondence experiment further shows sensitivity modulation, not improved transfer: replacing FAVC training fakes with pairing-break fakes raises the roll-15 honest-AUC drop from 0.07 to 0.90, while DFTIMIT FV-RA falls from 0.89 to 0.81.

W2. Readability, soundness, and definitions

The revision resolves the presentation issue by defining every measurement before it is used and by placing the full protocol at the start of Sec. 2. Sec. 2 will now begin with a notation-and-protocol table and follow the sequence: metrics → interventions → reporting labels. One sign convention will be used throughout. The table will define the three measured contrasts and their resampling units: paired leading-trim AUC change on the same clips; identity seen/unseen difference on disjoint, composition-matched cohorts; and RV-FA minus FV-RA AUC. It will then define the pairing quantities H, H_T, ΔH_T, and R_T, followed by the descriptive CI-based reporting convention. The untrained cosine scorer and trained logistic readout will be separately named and defined before their first results (W5). A compact interval summary will move into the main paper; detailed per-break results will remain in the appendix.

W3. Overlapping evidence categories

The revised taxonomy removes the false choice between overlapping concepts by reporting input dependency and cue attribution separately. The revision will report two separate quantities:

  • Supported input dependency: audio, visual, and/or a cross-modal relation; more than one may be supported.
  • Cue attribution: dataset artifact, generator, identity, or unresolved.

Accordingly, the FAVC-trained AVH-Align head will no longer be labelled a “visual route.” It will be described as visual dependence supported; visual cue unresolved. Pairing dependence is not supported by the tested breaks, but the tests do not exclude generator, identity, or another visual cue. The generator leave-out control was run on separate linear heads and will not be used to resolve this AVH-Align case.

W4. Generator overfit

We have corrected the signature to contain only the three contrasts measured across the suite. Generator overfit is therefore removed as a fourth coordinate. We will remove it from the abstract, Fig. 1, Sec. 2, contributions, and coverage table.

The cross-corpus values will instead be reported as cross-corpus transfer gaps, which are outcomes the study seeks to explain rather than route measurements. For the four detectors with known training corpora, these gaps are positive; for example, AVH-Align trained on FAVC has a 0.353 gap on AV1M.

The new within-FAVC generator leave-out study is a separate control, not a replacement fourth axis. It uses frozen-feature linear heads, equal training-fake counts, five seeds, and identical reals and test cells across arms. The largest mean gaps are 0.012 for visual (95% CI [0.010, 0.014]), 0.008 for audio+visual, and 0.001 for correspondence.

AuthorsOpenReview ↗

Response - Part 2

W5. Definitions of unsup_sync and corr_sync

The revision makes the scorer definitions self-contained in the main text, so each result can be interpreted without consulting the appendix. The revision will introduce each scorer at first use in the main text or setup table and state the score, readout, and training data in every results table.

  • unsup_sync is an offset-prediction model trained on AV1M real clips without fake labels and evaluated over a ±15-frame window. We will distinguish its absolute-logit readout from the exploratory relative-peak readout.
  • corr_sync currently refers to two different scorers. In Tables 10–11, it is the zero-parameter mean per-frame cosine between AV-HuBERT audio and visual features, maximised over shifts −3 to 3, with lower agreement indicating fake. This scorer will retain the name corr_sync. The FAVC-trained logistic head over eight such statistics, currently used in App. J and for the DeepSpeak 0.598 result, will receive a distinct name.

W6. Pairing metric and thresholds

The revision places the pairing quantities, their direction, and the reporting convention in one main-text definition block before any route verdict is reported. The revision will define them together in the main text:

  • H: honest AUC of intact fakes against intact reals.
  • H_T: honest AUC of the same fakes against perturbed reals.
  • ΔH_T = H − H_T: positive values indicate a performance drop.
  • R_T: AUC separating perturbed from intact reals, with perturbed reals as the positive class under the fake-oriented score; R_T = 0.5 indicates no aggregate separation.

Because H and H_T use the same fakes, ΔH_T isolates the effect of perturbing the real class. Table 10 currently uses the opposite sign and will be converted for consistency.

The thresholds were not calibrated and will be described only as a CI-based reporting convention: a dependency is supported when the lower CI limit is at least 0.70 for R_T and at least 0.05 for ΔH_T on the same declared break. The main paper will include the compact summary driving these labels; per-break detail will remain in the appendix.

W7. Manipulation artifacts and shortcuts

The revision gives the two terms complementary roles rather than treating them as mutually exclusive. A manipulation artifact is a cue introduced by synthesis or editing. A shortcut is a label-correlated cue on which a detector relies but which is unstable in the settings where it is expected to generalise. Thus, a manipulation artifact can function as a shortcut when it is predictive in the training setting but not stable across the intended deployment settings.

Leading silence and identity--label association are dataset-associated cues. The revision will report cue origin and measured dependence separately, use this terminology consistently in Related Work and the taxonomy figure, and retain an unresolved attribution whenever the intervention cannot identify the underlying cue.

AuthorsOpenReview ↗

Response - Part 3

Q1. Bootstrap procedures

We corrected the resampling description and re-ran the relevant analyses. Not all intervals were computed with paired bootstraps. The revision will state the actual procedures:

  • Leading trim: paired, class-stratified resampling of the same clips.
  • Identity: independent class-stratified resampling of two disjoint cohorts.
  • Modality skew: independent resampling in the submitted analysis, despite the two cells sharing 151 real clips.

For leading trim, recomputing all 19 detectors from the Table 1 arms reproduces every estimate and significance decision. Resampling the naive and trimmed arms independently makes the intervals approximately 6× wider (median; range 3.7–35×), and 8 of the 12 significant effects no longer meet the original criterion. Since the two arms contain the same clips, paired resampling is the appropriate analysis.

For modality skew, we added a joint resampling analysis (new) that resamples shared reals once per draw and the two fake cells separately (18 detectors). The resulting intervals are the same or narrower (median width ratio 0.89; range 0.45–1.01), and no significance decision changes. The revision will use this joint procedure.

A methodology table will specify, for every axis, the sampling unit, pairing status, bootstrap type, and treatment of shared reals. The independent-bootstrap comparison and resampling code will be included in the appendix.

Reviewer_XT97OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how training data influences the evidence used by audio-visual deepfake detectors and how this affects out-of-distribution generalization. The authors propose AVRouteBench, a diagnostic framework that combines a four-axis evidence signature with targeted confirmation interventions to identify the evidence routes used by detectors. Experiments across 19 detector instances from ten families show that conventional accuracy and shortcut scores cannot fully explain cross-domain failures. By changing the training data while keeping the architecture fixed, the authors demonstrate that a detector's reliance on audio or visual evidence can change substantially. Further experiments investigate the role of audio-visual correspondence in cross-domain detection. The work provides useful insights into evidence reliability and generalization in audio-visual deepfake detection.

Critical strengths
  1. Meaningful evidence-route analysis. The paper goes beyond conventional accuracy-based evaluation and introduces a framework for investigating the evidence used by deepfake detectors. The combination of evidence signatures and targeted interventions, particularly pairing breaks, provides a useful approach to understanding model decisions.
  2. Comprehensive and well-designed experiments. The paper includes controlled training-data interventions, matched comparisons, and cross-dataset evaluations. These experiments provide convincing evidence that training data can influence the evidence routes learned by detectors.
Critical weaknesses
  1. Limited validation across independent models and diagnostic datasets. Although the paper evaluates 19 detector instances, several share the same backbone, and the evidence-route signatures are primarily measured on FakeAVCeleb. Broader validation across independent architectures and diagnostic datasets would strengthen the generalizability of the conclusions.
  2. Limited coverage of the evidence-route signature. The proposed framework uses four diagnostic axes (leading silence, speaker identity, modality skew, and generator overfit), but detectors may rely on other evidence sources not covered by these axes. Although targeted interventions help confirm candidate routes, the paper provides limited evidence on whether unmodeled cues could lead to incorrect route assignments.
Questions for authors
  1. Could the authors further validate AVRouteBench on additional independent detector architectures or diagnostic datasets to assess the stability of the evidence-route signatures?
  2. How were the four diagnostic axes (leading silence, speaker identity, modality skew, and generator overfit) selected, and how would AVRouteBench handle detectors that rely on evidence sources not covered by these axes?
4 comments
AuthorsOpenReview ↗

Response - Part 1

Dear Reviewer XT97,

Thank you very much for the valuable review. The revision makes the scope of AVRouteBench explicit: it provides an intervention-based diagnosis for the tested models and contrasts, rather than an unqualified benchmark-wide route claim. We now distinguish partial replication of an individual measurement from validation of the full signature, report unmeasured evidence explicitly, and align the claims with this scope. Results marked (new) were obtained after submission.

W1. Validation across independent models and datasets

The current evaluation provides a controlled 19-instance suite and a partial out-of-corpus replication of modality skew; it does not claim full independent-architecture or full-signature replication. Eleven of the 19 instances are AV-HuBERT-derived, and we did not add a new independent architecture or a full second-corpus signature during the rebuttal period. Outside FAVC, we can report only a partial replication of the modality-skew contrast. Its ordering agrees with FAVC at Spearman 0.618 on LAV-DF (18 instances; this corpus is purity-flagged) and 0.67 on a purity-clean TIMIT pair (which has no identity coverage). These checks do not replicate the complete signature or validate the resulting route evidence labels. We will move these results into the main text together with those limitations, and restrict generalisation claims to the evaluated models, corpora, and contrasts.

W2. Coverage of the signature

The framework is intentionally structured as screening plus targeted confirmation, rather than as a complete basis for every possible cue. An unmodelled cue can lead to ambiguous attribution whenever an intervention changes multiple properties: trimming removes content as well as leading silence, and an audio swap changes speaker and content as well as pairing. The revision separates screening from confirmation. The three measured screening contrasts identify candidate dependencies; targeted interventions then test only the stated dependency. A table will state what every intervention changes and preserves, and the cue attribution will remain unresolved when the contrast is not sufficiently selective. For the FAVC-trained AVH-Align head, for example, visual dependence is supported, but the visual cue itself remains unresolved. We also added a within-FAVC generator leave-out control (new) on frozen-feature linear heads only. Leaving one of three generators out of training changes AUC by at most 0.012 for the visual head (fsgan; 5-seed 95% CI [0.010, 0.014]), 0.008 for audio+visual, and 0.001 for correspondence. This ceiling-adjacent control neither rules out generator cues elsewhere nor tests AVH-Align; we will state both limits explicitly.

AuthorsOpenReview ↗

Response - Part 2

Q1. Further validation on independent architectures or datasets

The new evidence beyond FAVC is a partial modality-skew replication, not a full second-corpus signature. The revision will report only the partial modality-skew checks described under W1 and will not present them as validation of the full route signature. We will correspondingly narrow the benchmark-wide generalisation claims.

Q2. Choice of axes and handling of unmeasured cues

AVRouteBench uses three screening contrasts because they provide explicit, testable contrasts for documented corpus-specific shortcuts and modality dependence: leading silence and identity are documented in these corpora (Smeu et al., 2025; Müller et al., 2021; Dong et al., 2023), while modality skew follows from factored manipulation cells. They are a diagnostic set, not a complete basis for all evidence sources. Throughout the revision, “four axes” will become “three measured screening contrasts plus targeted confirmation.” The within-FAVC generator leave-out is a separate control, and cross-corpus gaps are transfer outcomes rather than signature coordinates. For a new cue, AVRouteBench requires a defined contrast, an explicit account of what the intervention changes and preserves, and an uncertainty procedure. Cues without such a test are reported as unmeasured rather than folded into an existing route label.

Dear Reviewer XT97,

We would like to gently follow up on our response. In particular, we made the scope of AVRouteBench explicit, reported the partial out-of-corpus replication of the modality-skew contrast together with its limitations, and restructured the framework as three measured screening contrasts plus targeted confirmation. We also added a within-FAVC generator leave-out control and now report untested cues explicitly as unmeasured.

We hope these additions and clarifications address the concerns you raised, and we would be very glad to clarify anything further. Thank you again for your thoughtful review and your positive assessment of our work.

Thank you for the detailed response. The partial replication of modality skew on additional datasets provides useful supporting evidence, while your clarification of its limitations helps establish the scope of the findings. Separating the three screening contrasts from targeted confirmation, and explicitly acknowledging unmeasured cues and unresolved attributions, also addresses my concerns about the coverage of the evidence-route signature. Although broader validation across independent architectures and datasets remains desirable, I appreciate the additional experiments and the planned revisions. Overall, your response clarifies my main questions, and I maintain my positive assessment of the paper.

Reviewer_VGMJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AVRouteBench, an intervention-based framework for examining the evidence used by audio-visual deepfake detectors. It characterizes detectors through a signature measuring sensitivity to leading silence, speaker identity, modality skew, and generator shift, followed by targeted interventions that test candidate routes such as single-modality evidence and audio-visual correspondence. The study covers 19 detector instances grouped into ten families. Its central experiment holds an AV-HuBERT representation and linear head class fixed while varying the mixture of FakeAVCeleb and AV-Deepfake1M training data. The honest-subset AUC rises from 0.495 to 0.980 as the mixture changes, alongside a shift from audio-dominated toward visually informative behavior. Matched-unit sweeps, same-source feature recombination, single-stream controls, and evaluation on DFTIMIT further examine this result. The paper also studies the transfer behavior and limitations of correspondence-based detectors. A model that uses route signatures to rank out-of-distribution failure risk is presented as an exploratory analysis.

Critical strengths

The fixed-architecture mixture sweep is compelling. With the AV-HuBERT representation and linear head class held fixed, changing the training mixture raises honest-subset AUC from 0.495 to 0.980. The matched-unit sweep and DFTIMIT evaluation address important alternative explanations.

The paper examines evidence use through complementary controls rather than relying on aggregate accuracy. Single-stream probes, pairing breaks, and cell-purity audits help distinguish modality dependence from nominal manipulation labels and expose contamination in supposedly clean cells.

The reporting is unusually transparent. The authors disclose failed preregistered predictions, sensitivity to detector-family definitions, contaminated evaluation cells, and correspondence failures. These negative results make the supported conclusions easier to assess.

Critical weaknesses

AVRouteBench’s interventions do not uniquely identify a detector’s evidence route. The signature covers a limited set of cues, and a weak response does not exclude unmeasured or combined routes. Confirmation is available only for selected cases; its thresholds are descriptive rather than statistically calibrated. AVAD, for example, meets the 0.70 response-AUC threshold at 0.702, while its confidence interval crosses that threshold. Route labels should therefore be presented as conclusions supported by specific interventions, rather than exhaustive identifications.

The proposed data-to-route-to-transfer chain is established only in a limited setting. The natural mixture sweep changes several corpus properties together; the matched sweep retains differences in identity sets and generators. The same-source recombination offers stronger control but operates on pooled features and yields a mixed route at its strongest setting. No training intervention moves a detector into a correspondence route, and correspondence-feature restoration does not transfer reliably to DeepSpeak. The evidence supports training-induced changes in audio and visual cue use, alongside separate correspondence case studies, rather than one general mechanism spanning all route types.

The exploratory risk-ranking result is sensitive to analysis choices. The registered signature is individually significant on only one of five targets. Its joint pp-value changes from 0.013 to approximately 0.077 when AV-HuBERT-derived instances are merged into one family. The joint statistic was chosen after the per-target results were known, and the joint result becomes nonsignificant after accounting for the FAVC baseline AUC contained in the collapse outcome. These findings do not support a confirmatory claim that the signature predicts failure across detector families.

Generalization remains limited. Eleven of the 19 instances use AV-HuBERT-derived representations, complete signatures are measured on FakeAVCeleb, and the supervised and face-swap evaluations cover few distinct systems. Shared identities and donors may also reduce the effective sample size of some clip-level uncertainty estimates. The main mixture-sweep result remains valuable, but benchmark-wide claims need narrower wording.

Questions for authors

What prospective rule assigns route labels from the signature and confirmation interventions, and how does it handle mixed or unmeasured routes and uncertainty near a threshold?

Can the complete data-to-route-to-transfer chain be demonstrated within one architecture, including a variant whose training induces correspondence dependence?

Does the joint risk-ranking result survive a permutation analysis that accounts for the choice of joint statistic and alternative AV-HuBERT family partitions?

How stable are the mixture-sweep results with more training seeds, and do identity- or source-level confidence intervals change the conclusions of the pairing and identity interventions?

4 comments
AuthorsOpenReview ↗

Response - Part 1

Dear Reviewer VGMJ,

Our revision sharpens AVRouteBench into an intervention-based diagnostic framework: it reports the evidence dependencies supported by explicit tests, and it separates this central contribution from an exploratory risk-ranking analysis. We add controlled results, make the reporting rule and uncertainty explicit, and narrow claims exactly where the evidence requires. Results marked (new) were obtained after submission.

W1. What AVRouteBench identifies

AVRouteBench is designed to report the evidence dependencies supported by its declared interventions, rather than to claim exhaustive identification of every cue used by a detector. This is a meaningful diagnostic distinction: it makes each conclusion traceable to a concrete perturbation, its confidence interval, and the properties preserved by that perturbation. The revision makes this scope explicit in Sec. 2, Sec. 5.1, App. J, and the relevant captions. A detector may have multiple supported dependencies and/or an unresolved cue attribution.

We will replace confirmed / rejected with supported / no material response under the tested interventions / undetermined. These are transparent descriptive reporting labels, not calibrated statistical tests. For the pairing studies, supported requires the lower limit of the 95% CI to be at least 0.70 for response AUC and at least 0.05 for the honest-AUC drop on the same declared break (Q1).

AVAD illustrates the value of this interval-aware reporting. Its roll response AUC is 0.702 [0.644, 0.759], so it is appropriately marked borderline rather than given a threshold-based route label. Its honest-AUC drop is 0.136 [0.095, 0.176], which remains evidence of measured pairing sensitivity. We will provide one table containing every detector's tested dependency, intervention, CI, reporting label, and coverage.

W2. What the controlled training experiments establish

The revised paper establishes a focused mechanism: under fixed architecture, training data can substantially redirect measured audio versus visual evidence use. Across 10 seeds, FAVC FV-RA rises from 0.505 to 0.980 and the independent DFTIMIT result rises from 0.488 to 0.846. The revision will state this contribution directly while replacing broad causal language such as “route sets failure.” A revised table will state, for the natural sweep, matched sweep, same-source recombination, and the experiment below, what is held fixed and which confounds remain.

For correspondence, we added a fixed-architecture training experiment (new). We hold the head, audio/visual inputs, eight correspondence features, and training size fixed, and replace a fraction ρ of FAVC fakes with pairing-break fakes. The roll-15 honest-AUC drop increases from 0.07 to 0.39 to 0.90 for ρ = 0, 0.1, 1, while the response AUC on held-out shuffle and swap breaks increases from 0.90 to 0.99 to 1.00. Since the response is already present at ρ = 0, this experiment shows strengthening of an existing pairing dependence rather than the emergence of a new route.

This experiment also provides an informative negative result: transfer decreases rather than improves (DFTIMIT FV-RA: 0.89 to 0.81; DeepSpeak: 0.76 to 0.70). We therefore present it as evidence that training can modulate measured pairing sensitivity, while making no claim that this induced dependence universally improves transfer.

AuthorsOpenReview ↗

Response - Part 2

W3. Separating the central diagnostic result from exploratory risk ranking

The paper's central results do not depend on the risk-ranking analysis: they are the intervention-based diagnoses and fixed-architecture training controls. We will therefore relocate risk ranking to an explicitly exploratory appendix sensitivity analysis and remove it from the title, abstract, contributions, and conclusion.

For completeness, we will report its full sensitivity: only 1 of 5 targets is individually significant; the joint p = 0.013 becomes 0.077 when AV-HuBERT-derived instances are merged into one family, and 0.10 after residualising the FAVC baseline; the joint statistic was selected after inspecting per-target outcomes. This makes the auxiliary result appropriately interpretable without overstating it.

W4. Empirical scope and uncertainty

The study provides a controlled diagnosis across 19 detector instances, while the revision will make its empirical scope explicit: 11 instances are AV-HuBERT-derived, full signatures are measured on FAVC, and supervised and face-swap systems remain few. We will limit all benchmark-level claims to these evaluated models and corpora.

We also strengthened the uncertainty analysis. On the identity axis, an identity-clustered bootstrap (new) over 139 seen and 104 unseen identities (2,000 draws) preserves every effect direction and changes only 1 of 18 significance decisions (pooled_audio_linear becomes significant). These intervals remain conditional on the evaluated identities and fake set.

Pairing intervals are likewise conditional on the evaluated identities and fixed fake sample; shared swap donors remain an unmodelled dependence. Finally, the 10-seed mixture sweep quantifies optimisation variability, not model or corpus breadth.

AuthorsOpenReview ↗

Response - Part 3

Q1. Explicit, interval-aware reporting rule

We now make the reporting rule explicit and auditable. It is a retrospective descriptive convention for the current case studies, not a calibrated or prospective decision rule.

For pairing case studies, the declared interventions are a 15-frame roll, temporal shuffle, and same-stratum audio swap. For each scorer, the revised table will state which breaks were run and what else each break changes. For example, shuffle can introduce splice artifacts, and swap changes speaker and content as well as pairing.

Applied to 95% CIs, the convention is:

  • Supported: at least one declared break has a lower CI limit of at least 0.70 for response AUC and at least 0.05 for the honest-AUC drop on that same break.
  • No material response under the tested breaks: all declared breaks were run and their full CIs lie within [0.40, 0.60] for response AUC and [−0.02, 0.02] for the honest-AUC drop.
  • Undetermined: any other measured outcome.

The table will also flag borderline, mixed (more than one supported dependency), and unmeasured cases separately. AVAD is borderline under this convention. For trim, identity, and modality contrasts, we will report effect sizes and CIs without imposing a route-label threshold.

Q2. Fixed-architecture evidence for training-dependent pairing sensitivity

Within one fixed architecture, we now directly show that changing only the training fakes changes pairing sensitivity. Adding pairing-break fakes increases the roll-15 honest-AUC drop from 0.07 to 0.90 and the held-out response AUC from 0.90 to 1.00. Here, the honest-AUC drop is H − H_T: honest AUC with intact reals minus honest AUC when the same fakes are evaluated against rolled reals. The response AUC compares perturbed and intact reals, where 0.5 indicates no aggregate separation.

The same intervention reduces transfer (DFTIMIT: 0.89 to 0.81; DeepSpeak: 0.76 to 0.70). Thus, the experiment establishes training-dependent modulation of measured pairing sensitivity while also showing that stronger sensitivity alone is not a sufficient condition for better transfer. We will state this boundary clearly.

Q3. Robust interpretation of the joint analysis

Rather than rely on a joint statistic whose significance depends on analytic choices, we will treat this result only as exploratory. The revision will report the partition sensitivity (0.013 versus 0.077) and residualised result (p = 0.10), and move the entire analysis to an appendix sensitivity section. The primary intervention and training-control findings are unaffected.

Q4. Stable mixture-sweep result across 10 seeds

The main mixture-sweep finding now holds across 10 seeds (new); seeds 0–2 reproduce the submitted values.

λ00.250.50.751
FAVC FV-RA0.5050.8640.8990.9290.980
DFTIMIT0.4880.6340.6660.7010.846

The largest seed SD is 0.042 on FAVC and 0.040 on DFTIMIT. The seed mean rises monotonically with λ; individual seeds need not. The appendix will report mean ± SD, seed count, and checkpoint-selection rule for each λ. The other controlled sweeps remain three-seed studies.

The identity-clustered analysis changes 1 of 18 significance decisions (W4). We have not computed source- or donor-clustered pairing intervals; the current paired analysis conditions on the evaluated fake set, so shared swap-donor dependence remains a stated limitation.

Most of my concerns have been resolved, and I maintain my original score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision for the Area Chair

Dear Area Chair,

Thank you for handling our submission. A revised manuscript has been uploaded, with changes marked and the main text within the 10-page limit. This note summarises what changed.

1. New experiments and analyses

  • Ten-seed mixture sweep (VGMJ, v3Hv). The fixed-architecture result holds across ten seeds (FAVC FV-RA 0.505 → 0.980; independent DFTIMIT 0.488 → 0.846).
  • Pairing experiment at fixed architecture (VGMJ). Changing only the training fakes raises the roll-15 honest-AUC drop from 0.07 to 0.90, so pairing sensitivity depends on training. Transfer does not improve.
  • Uncertainty (VGMJ, v3Hv). We corrected the description of the resampling. Identity-clustered resampling changes 1 of 18 significance decisions and joint resampling changes 0 of 18. Complete per-detector intervals are posted and included in the appendix.
  • Generator leave-out (XT97, v3Hv). A within-FAVC control on linear heads; the largest gap is 0.012.

2. What changed in the manuscript

  • The signature is now three measured screening contrasts plus targeted confirmation. Generator overfit was not measured across the suite and is removed as an axis.
  • Route labels are replaced by a supported input dependency and a cue attribution, under an interval-based reporting convention. AVAD is reported as borderline.
  • Risk ranking moves to an exploratory appendix analysis, and benchmark-level claims are limited to the evaluated models and corpora.
  • Section 2 opens with a notation and protocol table that defines every quantity before it is used.

After revision, the paper's contribution is an intervention-based framework for evaluating which evidence a detector relies on, with fixed-architecture controls showing that training data changes that reliance. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.