- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents ChainBench-ADD, a delivery-aware benchmark for audio deepfake detection. It constructs matched bona fide and spoof parents, applies five structured delivery families with 33 templates, and supports in-chain, intervention, and lineage-based evaluation. Experiments with several detectors show systematic differences across delivery families and reveal robustness changes under local delivery edits.
Critical strengths
- The matched-parent design and metadata-driven lineage structure provide a clear framework for controlled delivery analysis.
- The benchmark covers several delivery families and evaluates both ordinary detection and robustness under local changes, making the study useful for future benchmark development.
Critical weaknesses
- The delivery families are controlled simulations, so transfer to real platform, telephony, or replay audio remains uncertain.
- Many waveforms are derived from the same parents, but parent-level uncertainty is not reported; the effective sample size may be smaller than the waveform count suggests.
- The main splits share much of the operator inventory, while the complete data and external model assets are not in the supplement. Stronger unseen-route evaluation and a more reproducible release would help.
- The prose could be more direct, and the figures should be redrawn as clearer technical graphics with larger labels and higher resolution.
Questions for authors
- How sensitive are the results to the simulator parameters, and is there any validation on real delivery or replay recordings?
- Could the authors report parent-level confidence intervals and clarify which templates or operators are genuinely unseen in the held-out evaluation?
5 comments
Response to Reviewer Zf4N (2/2)
Weakness 2 & Question 2 (held-out evaluation): What is unseen, generalization to unseen routes, and reproducibility
Weakness 2: The main splits share much of the operator inventory, while the complete data and external model assets are not in the supplement. Stronger unseen-route evaluation and a more reproducible release would help. The prose could be more direct, and the figures should be redrawn as clearer technical graphics with larger labels and higher resolution.
Question 2 (excerpt): clarify which templates or operators are genuinely unseen in the held-out evaluation?
Thank you for these suggestions.
Which templates or operators are genuinely unseen? Template IDs are unseen, while individual operators are shared with training. Each Table 5 evaluation excludes one template (an ordered chain with codec/parameter settings) from training and development within its family. Sequence-sharing counts are:
| Complete sequence in remaining templates of the same family | Held-out templates (n) |
|---|---|
| Absent: the full sequence is also unseen | 20 |
| Present: the full sequence remains seen | 12 |
For example, holding out aac_single leaves opus_single, so the one-operation codec sequence remains seen. Codec identities and parameter ranges are not separately held out. Table 5 therefore tests held-out delivery-template configurations, including new compositions of familiar operators in the sequence-unseen cases.
Stronger unseen-route evaluation. The aac_reencode and opus_reencode templates share the complete codec→reencode sequence (compression followed by re-encoding). Holding out one evaluates a held-out codec-specific template on a familiar sequence. We complement this configuration-level test by excluding all templates with that sequence from training and development across all families. This tests an unseen complete composition while individual operators remain shared. We retrained both detectors and selected checkpoints and thresholds on the filtered development split. This exclusion removes 7.88% of training rows. The retrained models and their pooled-training counterparts are evaluated on the same 11,230 waveforms from 7,834 parents and 67 speakers:
| Model | Pooled-training EER % | Sequence-holdout EER % | EER increase, pp [speaker 95% CI] |
|---|---|---|---|
| Nes2Net | 1.73 | 2.20 | +0.47 [0.26, 0.82] |
| XLS-R+SLS | 2.22 | 2.53 | +0.31 [0.05, 0.51] |
Both models retain low EER on the fully excluded route, while the paired intervals show an increase relative to pooled training. This provides further support for the delivery generalization examined in Table 5: detectors can transfer to an unseen complete delivery composition built from familiar operators.
Reproducibility and presentation. The submitted supplement already contains construction/evaluation code, configurations, documentation and 20 audio examples. As explained in its README, the full 941,201-waveform corpus and third-party pretrained weights are excluded because of their size. The supplied configurations and adapters specify the required model identifiers and paths. The third-party pretrained weights can be obtained following the respective repositories' READMEs.
In the revision, we will clarify pretrained-weight downloads and runtime setup, and add development/test metadata, scores from the supplementary runs, checkpoint/score hashes and exact commands for independent score-based verification. We will also streamline the prose, standardize terminology and improve Figure 1’s resolution and label legibility.
Follow-up to Reviewer Zf4N
Thank you again for your thoughtful review. We hope our responses and revisions have clarified the points you raised. If any concerns remain, we would appreciate the opportunity to address them further during the discussion.
Response to Reviewer Zf4N (1/2)
Thank you for the constructive feedback and for recognizing the matched-parent design and linked delivery structure. We group overlapping weaknesses and questions by topic below. For the supplementary experiments, we retrained the two strongest baselines in Tables 6–7, Nes2Net and XLS-R+SLS, on training data combined across all five delivery families (pooled training), retaining the original speaker-disjoint splits. The sequence-holdout experiment additionally excludes the specified route.
Weakness 1 (real delivery) & Question 1: Parameter sensitivity and validation on real recordings
Thank you for raising these points. We tested parameter sensitivity with matched-source sweeps and collected paired Tencent Meeting/Zoom recordings for real-platform validation.
Sensitivity to simulator parameters. We varied target reverberation time (RT60) and Opus bitrate one at a time over the ranges used in our benchmark, in the chain resampling (16→8→16 kHz) → Opus → room impulse response (RIR). Each sweep reuses the same source clips, with the bitrate sweep using a smaller balanced subset. All other settings, operator-specific random draws, checkpoints and development-selected thresholds stay fixed. The table summarizes the observed accuracy range over all listed values:
Accuracy remains comparatively stable over 16–32 kbps, while increasing RT60 from 0.2 to 0.8 s lowers accuracy for both models. Parent- and speaker-clustered intervals support the RT60 endpoint difference. The bitrate-difference intervals include zero.
Validation on real platform recordings. Yes. We sent the same 128 held-out source clips from 39 speakers through Tencent Meeting and Zoom using shared computer audio, recording the received sound through Windows loopback with both microphones muted. We compared these recordings with versions of the same speech processed only by Opus at 24 kbps. The table reports accuracy using the same fixed checkpoints from these supplementary runs and development-selected thresholds:
The 95% confidence intervals use 1,000 speaker-level resamples, keeping all versions and utterances of a sampled speaker together. This real-delivery check covers platform shared-audio paths, complementing our simulated replay and telephony tests. The accuracy changes after platform delivery and with RT60 both occur with source speech held fixed. They support our finding that detection depends on delivery conditions even without a change in content or speaker.
Weakness 1 (source dependence) & Question 2 (uncertainty): Confidence intervals for shared source clips and speakers
We agree that shared parents and speakers can make the effective sample size smaller than the waveform count suggests. We therefore estimate EER (equal error rate) uncertainty with 1,000 paired bootstrap draws of whole parents or language-stratified whole speakers, keeping their related versions together and the checkpoints fixed. The combined test split contains 141,811 waveforms from 8,407 parents and 67 speakers:
SLS−Nes denotes XLS-R+SLS minus Nes2Net; pp denotes percentage points.
The intervals quantify EER uncertainty across source groups, keeping each group's versions together. Both paired-difference intervals are above zero, so Nes2Net's lower EER remains supported in these supplementary runs after accounting for shared parents and speakers.