- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the problem in cross-domain audio-visual deepfake detection where models rely on an “audio shortcut” and ignore manipulation evidence on the visual side. Rather than enforcing invariance through a loss, the paper proposes to solve this through the structural design of “routing,” i.e., which information is passed to the prediction head at inference time. The proposed method, TRIROUTE, decomposes frozen AV-HuBERT features into three branches and structurally excludes the residual branch from the prediction head. The model is trained on AV-Deepfake1M and evaluated by transfer to FakeAVCeleb (FAVC), LAV-DF, and AVLips.
Critical strengths
- The question posed (whether existing methods improve robustness by learning better representations, or by changing how the model uses its representations) is interesting.
- The authors explicitly exclude the individual use of each building block from their novelty claim and limit the contribution to the combination and the inference-time rule, which is honest.
- The experiments follow a transfer design in which training is performed on AV-Deepfake1M and evaluation is conducted on disjoint target datasets (FakeAVCeleb, LAV-DF, AVLips) that are never accessed during training, validation, or model selection.
Critical weaknesses
- The fragility of the nuisance partition remains unresolved. The main result of 0.838 depends on hash(speaker_id) mod 2, which is effectively a random binary speaker partition, yet a “random speaker regrouping” of the same kind drops to 0.694, and 4-way and 8-way hashes drop to 0.701 and 0.667 (Table 14). The paper does not explain why partitions of the same kind differ by 0.14, so the possibility that the result is a coincidence tied to one particular hash cannot be ruled out.
- There is no simple baseline that directly addresses the audio shortcut. Since the FV-RA failure is a matter of “relying on audio,” comparisons with a visual-only detector and with modality dropout (audio masking) during training are needed.
- The effect of push-pull is not separated. The difference between “three-way, no push-pull” (0.600) and TRIROUTE (0.838) is the effect of adding L_dom and L_adv at the same time, and there are no rows for L_dom only or L_adv only.
- Under leading-silence trimming in Table 17, TRIROUTE’s RV-FA drops from 0.999 to 0.810 (−0.189) and its overall AUC from 0.922 to 0.866, which is far larger than the RV-FA drop of the two-way baseline (−0.038). While the flatness of FV-RA is emphasized, the fact that performance on the audio-fake categories depends heavily on leading silence, a dataset-specific artifact, is hardly discussed.
- Although AV1M contains more than one million clips, only 8,000 clips are used for training, with batch size 1, 100 epochs, and 3 seeds. Batch size 1 introduces training instability. It is unclear whether the advantage of the proposed method remains when the amount of training data is increased.
Questions for authors
Clear responses to the weaknesses above would be appreciated.
6 comments
Response to Reviewer yx8m (Part 2)
W3. Isolating the two push–pull objectives
We appreciate this request because it allows us to identify which training signal contributes to the gain. We have now isolated L_dom and L_adv in a controlled ablation. We now evaluate each term separately with matched training and checkpoint selection (three seeds; new).
| Training objective | FAVC FV-RA | AVLips |
|---|---|---|
| Neither (FCD-style) | 0.671 ± 0.033 | 0.597 ± 0.030 |
L_dom only | 0.742 ± 0.027 | 0.694 ± 0.024 |
L_adv only | 0.703 ± 0.036 | 0.641 ± 0.028 |
| Both (TRIROUTE) | 0.838 ± 0.024 | 0.789 ± 0.021 |
Each objective is associated with an improvement over the no-push-pull setting under the original partition (+0.071 for L_dom, +0.032 for L_adv on FAVC); their combination improves by +0.167. We have not established a statistically supported interaction or synergy, and the estimates are sensitive to the choice of nuisance partition.
We also identified an untraceable original value: Table 1 reports 0.600 for 'three-way, no push-pull', whereas the reproducible re-run yields 0.671 ± 0.033 (seed values 0.687, 0.633, 0.693). We will correct Table 1 and revise the previously stated +0.238 gain to +0.167. The existing 0.3×/3× weight sweep assesses weight sensitivity, not whether either term is necessary.
W4. Leading-silence sensitivity and audio-fake performance
The reviewer correctly identifies an asymmetry between FV-RA and audio-fake robustness under test-only trimming. To investigate whether this is a train–test mismatch, we added a matched train-and-test trimming experiment. With the submitted checkpoints, test-time trimming changes FV-RA only from 0.838 to 0.833, but RV-FA falls from 0.999 to 0.810, FV-FA from 0.994 to 0.898, and overall AUC from 0.922 to 0.866. The RV-FA drop (−0.189) exceeds that of the two-way baseline (−0.038) and no-push-pull model (−0.050); after trimming, the two-way baseline scores 0.962 versus TRIROUTE's 0.810.
To test whether this reflects a train–test cue mismatch, we removed the first 13 frames (0.52 s) from both feature streams during source training and target evaluation, retraining seeds 40–42 (new).
| Change relative to original setting | Seed 40 | Seed 41 | Seed 42 | Mean |
|---|---|---|---|---|
| FV-RA | −0.091 | −0.011 | −0.047 | −0.050 (0.838 → 0.788) |
| RV-FA | −0.006 | −0.005 | −0.022 | −0.011 |
| FV-FA | −0.032 | −0.041 | −0.069 | −0.047 |
Removing the leading segment during both training and testing largely prevents the RV-FA collapse seen under test-only trimming. However, FV-RA now also decreases by 0.050 on average, indicating that the original FV-RA result is not wholly insensitive to the leading segment. We cannot isolate the precise cue within that fixed 0.52-second interval.
We will report the RV-FA, FV-FA, and overall changes beside FV-RA in the main text, describe the implemented trimming procedure precisely, and withdraw the claim that stable FV-RA under test-time trimming rules out audio-shortcut reliance. The two-way baseline has not been retrained with this cut, so the retrained-setting comparison remains incomplete.
Response to Reviewer yx8m (Part 3)
W5. Training scale, batch size, and reproducibility
Thank you for raising reproducibility and training stability. We have extended the seed analysis beyond the submitted runs and will report the broader distribution, alongside the fixed training protocol, so that the improvement and its variability can be assessed together. We use 8,000 AV1M source-training clips and 2,000 source-validation clips, batch size 1 for variable-length features and within-clip alignment, and up to 100 epochs with source-validation early stopping. All internal controls share these choices, but that does not establish robustness to batch size or dataset scale.
Additional runs reveal substantially greater variability than the submitted three-seed summary:
- Re-running the three submitted seed IDs yields 0.824 ± 0.022, compared with the submitted 0.838 ± 0.024.
- Six further seeds (46–51) yield 0.741, 0.706, 0.786, 0.821, 0.690, 0.613 (mean 0.726, SD 0.074).
- Across all nine runs, the reported mean is 0.763, SD 0.082. The expanded runs reveal a broader training-seed distribution than the initial three-seed estimate.
Five of the six added seeds are below the lowest originally submitted seed (0.813). Seed 51 is especially weak on FV-RA (0.613), with checkpoint selection at epoch 16 rather than epochs 88–97 for the other reported runs. This is consistent with an unstable training/selection process, but the results alone do not establish the cause.
We will use the nine-seed result as the main summary, list all seeds with source-validation AUC and selected epoch, and explicitly discuss the uncertainty. Baseline comparisons currently use only seeds 40–42; their observed differences (+0.035 over visual-only and +0.167 over no push–pull on those seeds) cannot be extrapolated to all nine runs without additional matched-seed evaluations.
We have not tested padded batches, gradient accumulation, or larger AV1M training subsets. The existing unsupervised scaling experiment concerns a different model and does not answer this question. We will state these gaps rather than imply training-scale robustness.
Response to Reviewer yx8m (Part 1)
Dear Reviewer yx8m,
Thank you for the detailed review and actionable suggestions. We have conducted new experiments addressing the main concerns: six additional nuisance partitions, matched visual-only and train-time audio-masking controls, separate push–pull ablations, matched leading-segment trimming, and an expanded seed analysis. The results identify where the proposed objectives add value, where simpler baselines are competitive, and which conditions affect stability. We provide the numerical evidence and corresponding manuscript revisions below.
W1. Fragility of the source nuisance partition
We have directly tested the reviewer's concern with six additional balanced speaker partitions, each evaluated over three seeds. This experiment clarifies both the promise and the sensitivity of nuisance-guided training. We retrained with six additional balanced binary speaker partitions, three seeds each (new).
md5(speaker_id) mod 2The original hash is the best of the seven. The earlier random-regrouping result (0.694) and the 4-/8-way results (0.701/0.667) are consistent with substantial partition sensitivity. Both the hash and the new binary partitions split speakers without known acquisition-condition labels. We therefore cannot explain the hash's advantage by a verified nuisance factor and acknowledge that it may reflect a favorable random realization.
An exploratory analysis finds a correlation between label decodability and gain across the seven partitions (r = 0.97); a speaker-mean-channel-feature k-means partition reaches 0.852. Neither establishes a reliable source-only selection criterion: seven partitions are too few for a robust conclusion, and the k-means procedure still uses speaker IDs. Greater label decodability could simply strengthen supervision without corresponding to a causal nuisance variable.
We will present the full distribution prominently and qualify the contribution as dependent on a suitable source-side grouping. We do not claim that arbitrary binary speaker partitions reliably deliver the originally reported gain.
W2. Visual-only and train-time audio-masking controls
We agree that the strongest test of TRIROUTE is against simpler ways to suppress audio shortcuts. We therefore added both a visual-only detector and a train-time audio-masking baseline under matched features and training settings. We trained both baselines using the same frozen features, seeds 40–42, and schedule (new).
The comparison establishes an important trade-off: visual-only is competitive on FV-RA, while TRIROUTE retains an audio-visual inference path needed for audio-manipulation cases. With only three seeds, the FAVC, AVLips, and DF-TIMIT differences do not justify a general superiority claim. The clearest paired improvement is on controlled LAV-DF, a list constructed post hoc. Across seven nuisance partitions, TRIROUTE's mean FAVC score (0.749) is below visual-only (0.803).
We also applied train-time audio masking to the two-way model with probability 0.6 (distinct from the original test-time masking):
Masking improves all three seeds, but remains below visual-only and the original-partition TRIROUTE result. This does not establish that TRIROUTE is superior to all modality-dropout strategies: symmetric dropout has not been evaluated.
A visual-only detector also cannot detect audio-only manipulations in the tested VidTIMIT RV-FA setting (0.472 versus TRIROUTE 0.731). However, FCD scores 0.763 there, and the leading-silence analysis in W4 reveals an audio-side weakness of TRIROUTE. We will report these trade-offs without claiming universal advantage.