- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies timing shortcuts in audio deepfake detection, focusing on distinguishing whether timing cues are merely available in a dataset from whether trained detectors actually rely on them. The authors analyze duration and silence statistics across several datasets, evaluate released detectors under matched-length conditions, and introduce counterfactual interventions that modify boundary silence while keeping the speech unchanged. They further investigate different training views and propose counterfactual timing invariance (CTI) to reduce reliance on timing cues. The main finding is that standard benchmark performance can hide shortcut reliance, while randomized cropping and consistency-based training can improve robustness to such shortcuts.
Critical strengths
- The analysis of the detectors' shortcut reliance on silence is interesting.
- The experiments are fairly comprehensive.
Critical weaknesses
- The paper is quite difficult to follow. The presentation is dense with many experimental conditions, abbreviations, metrics, controls and numerical comparisons introduced in a messy way. A substantial revision/restructuring is necessary.
- The overall novelty and significance of the findings are somewhat limited. Timing and silence artifacts in audio deepfake benchmarks have already been studied, as the paper itself acknowledged. The counterfactual framing makes the evidence cleaner, but I am less convinced that the empirical findings themselves substantially change our understanding of the problem.
- CTI improves some cross-dataset results but introduces clear trade-offs, including worse performance on some benchmark/transfer settings. This makes it unclear whether CTI should be viewed as a generally useful mitigation strategy or primarily as evidence supporting the authors’ counterfactual methodology.
Questions for authors
- Since timing and silence shortcuts in ASVspoof have already been established, what do the authors consider the main new scientific conclusion beyond providing a cleaner causal test of detector reliance? In particular, which finding would not reasonably be expected from the existing shortcut-learning literature?
- Random cropping appears to account for a substantial portion of the robustness improvement, even when silence is retained. Could the authors clarify what CTI provides beyond sufficiently diverse temporal cropping/augmentation, especially considering that CTI does not consistently improve performance across datasets?
- The paper shows cases where improving the standard benchmark score increases shortcut reliance. How sensitive is this conclusion to the particular fine-tuning setup and training budget? Would the same relationship hold when models are trained from scratch or selected using a more diverse validation distribution?
- Could the authors include additional experiments with more recent models to further validate the generality of the findings?