acceptodds
Under review as a conference paper at ICLR 2027

ChainBench-ADD: A Delivery-Aware Dataset and Benchmark for Audio Deepfake Detection

Abstract

Existing audio deepfake datasets have greatly expanded evaluation across generators, languages, and domains, but most remain generation-centric and provide limited support for studying post-generation delivery. In practical misuse scenarios, forged audio may undergo platform re-encoding, telephony transmission, or replay-like recapture before detection. Existing delivery-related studies usually model such factors as isolated distortions and often lack control over transcript and speaker conditions. To address this gap, we introduce ChainBench-ADD, a delivery-aware dataset and benchmark for audio deepfake detection. It provides a controlled and reproducible approximation of common delivery factors, including transcoding/re-encoding, communication-channel degradation, playback/re-recording effects, and multi-hop delivery. These factors are organized into five delivery families using reusable operators, ordered templates, and realized chains, while each delivered sample is linked to a clean bona fide or spoof parent under fixed transcript and speaker conditions. The current release contains 941,201 waveforms derived from 55,813 parents and supports five protocol-aware tasks. Experiments show that ChainBench-ADD reveals family-level difficulty differences, sensitivity to controlled local edits, and robustness decay along accumulated delivery paths that conventional pooled evaluation cannot capture. We release the dataset and code to support reproducible delivery-aware evaluation.

Primary area:
datasets and benchmarks
Keywords:
Audio Deepfake Detection, Speech Spoofing, Robustness Evaluation, Delivery Chain Modeling, Telephony and Platform Distortions, Replay Simulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Zf4NOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ChainBench-ADD, a delivery-aware benchmark for audio deepfake detection. It constructs matched bona fide and spoof parents, applies five structured delivery families with 33 templates, and supports in-chain, intervention, and lineage-based evaluation. Experiments with several detectors show systematic differences across delivery families and reveal robustness changes under local delivery edits.

Critical strengths
  1. The matched-parent design and metadata-driven lineage structure provide a clear framework for controlled delivery analysis.
  2. The benchmark covers several delivery families and evaluates both ordinary detection and robustness under local changes, making the study useful for future benchmark development.
Critical weaknesses
  1. The delivery families are controlled simulations, so transfer to real platform, telephony, or replay audio remains uncertain.
  2. Many waveforms are derived from the same parents, but parent-level uncertainty is not reported; the effective sample size may be smaller than the waveform count suggests.
  3. The main splits share much of the operator inventory, while the complete data and external model assets are not in the supplement. Stronger unseen-route evaluation and a more reproducible release would help.
  4. The prose could be more direct, and the figures should be redrawn as clearer technical graphics with larger labels and higher resolution.
Questions for authors
  1. How sensitive are the results to the simulator parameters, and is there any validation on real delivery or replay recordings?
  2. Could the authors report parent-level confidence intervals and clarify which templates or operators are genuinely unseen in the held-out evaluation?
5 comments
AuthorsOpenReview ↗

Response to Reviewer Zf4N (1/2)

Thank you for the constructive feedback and for recognizing the matched-parent design and linked delivery structure. We group overlapping weaknesses and questions by topic below. For the supplementary experiments, we retrained the two strongest baselines in Tables 6–7, Nes2Net and XLS-R+SLS, on training data combined across all five delivery families (pooled training), retaining the original speaker-disjoint splits. The sequence-holdout experiment additionally excludes the specified route.

Weakness 1 (real delivery) & Question 1: Parameter sensitivity and validation on real recordings

Weakness 1 (excerpt): The delivery families are controlled simulations, so transfer to real platform, telephony, or replay audio remains uncertain.

Question 1: How sensitive are the results to the simulator parameters, and is there any validation on real delivery or replay recordings?

Thank you for raising these points. We tested parameter sensitivity with matched-source sweeps and collected paired Tencent Meeting/Zoom recordings for real-platform validation.

Sensitivity to simulator parameters. We varied target reverberation time (RT60) and Opus bitrate one at a time over the ranges used in our benchmark, in the chain resampling (16→8→16 kHz) → Opus → room impulse response (RIR). Each sweep reuses the same source clips, with the bitrate sweep using a smaller balanced subset. All other settings, operator-specific random draws, checkpoints and development-selected thresholds stay fixed. The table summarizes the observed accuracy range over all listed values:

Varied parameterTested valuesFixed settingParents (n)Nes2Net accuracy range (%)XLS-R+SLS accuracy range (%)
Target RT600.2, 0.4, 0.6, 0.8 sOpus 24 kbps51285.94–94.1484.38–93.75
Opus bitrate16, 24, 32 kbpsRT60 0.4 s25690.23–91.0290.23–91.80

Accuracy remains comparatively stable over 16–32 kbps, while increasing RT60 from 0.2 to 0.8 s lowers accuracy for both models. Parent- and speaker-clustered intervals support the RT60 endpoint difference. The bitrate-difference intervals include zero.

Validation on real platform recordings. Yes. We sent the same 128 held-out source clips from 39 speakers through Tencent Meeting and Zoom using shared computer audio, recording the received sound through Windows loopback with both microphones muted. We compared these recordings with versions of the same speech processed only by Opus at 24 kbps. The table reports accuracy using the same fixed checkpoints from these supplementary runs and development-selected thresholds:

ModelOpus-only accuracy (%)Tencent Meeting accuracy (%) [95% CI]Zoom accuracy (%) [95% CI]
Nes2Net100.0096.88 [93.75, 99.29]96.09 [92.86, 98.98]
XLS-R+SLS99.2295.31 [91.38, 98.61]95.31 [91.41, 98.50]

The 95% confidence intervals use 1,000 speaker-level resamples, keeping all versions and utterances of a sampled speaker together. This real-delivery check covers platform shared-audio paths, complementing our simulated replay and telephony tests. The accuracy changes after platform delivery and with RT60 both occur with source speech held fixed. They support our finding that detection depends on delivery conditions even without a change in content or speaker.

Weakness 1 (source dependence) & Question 2 (uncertainty): Confidence intervals for shared source clips and speakers

Weakness 1 (excerpt): Many waveforms are derived from the same parents, but parent-level uncertainty is not reported; the effective sample size may be smaller than the waveform count suggests.

Question 2 (excerpt): Could the authors report parent-level confidence intervals

We agree that shared parents and speakers can make the effective sample size smaller than the waveform count suggests. We therefore estimate EER (equal error rate) uncertainty with 1,000 paired bootstrap draws of whole parents or language-stratified whole speakers, keeping their related versions together and the checkpoints fixed. The combined test split contains 141,811 waveforms from 8,407 parents and 67 speakers:

QuantityEstimateParent-level 95% CISpeaker-level 95% CI
Nes2Net EER (%)3.85[3.60, 4.10][3.30, 4.32]
XLS-R+SLS EER (%)4.30[4.03, 4.58][3.66, 4.95]
EER difference (SLS−Nes, pp)+0.45[+0.29, +0.65][+0.15, +0.82]

SLS−Nes denotes XLS-R+SLS minus Nes2Net; pp denotes percentage points.

The intervals quantify EER uncertainty across source groups, keeping each group's versions together. Both paired-difference intervals are above zero, so Nes2Net's lower EER remains supported in these supplementary runs after accounting for shared parents and speakers.

AuthorsOpenReview ↗

Response to Reviewer Zf4N (2/2)

Weakness 2 & Question 2 (held-out evaluation): What is unseen, generalization to unseen routes, and reproducibility

Weakness 2: The main splits share much of the operator inventory, while the complete data and external model assets are not in the supplement. Stronger unseen-route evaluation and a more reproducible release would help. The prose could be more direct, and the figures should be redrawn as clearer technical graphics with larger labels and higher resolution.

Question 2 (excerpt): clarify which templates or operators are genuinely unseen in the held-out evaluation?

Thank you for these suggestions.

Which templates or operators are genuinely unseen? Template IDs are unseen, while individual operators are shared with training. Each Table 5 evaluation excludes one template (an ordered chain with codec/parameter settings) from training and development within its family. Sequence-sharing counts are:

Complete sequence in remaining templates of the same familyHeld-out templates (n)
Absent: the full sequence is also unseen20
Present: the full sequence remains seen12

For example, holding out aac_single leaves opus_single, so the one-operation codec sequence remains seen. Codec identities and parameter ranges are not separately held out. Table 5 therefore tests held-out delivery-template configurations, including new compositions of familiar operators in the sequence-unseen cases.

Stronger unseen-route evaluation. The aac_reencode and opus_reencode templates share the complete codec→reencode sequence (compression followed by re-encoding). Holding out one evaluates a held-out codec-specific template on a familiar sequence. We complement this configuration-level test by excluding all templates with that sequence from training and development across all families. This tests an unseen complete composition while individual operators remain shared. We retrained both detectors and selected checkpoints and thresholds on the filtered development split. This exclusion removes 7.88% of training rows. The retrained models and their pooled-training counterparts are evaluated on the same 11,230 waveforms from 7,834 parents and 67 speakers:

ModelPooled-training EER %Sequence-holdout EER %EER increase, pp [speaker 95% CI]
Nes2Net1.732.20+0.47 [0.26, 0.82]
XLS-R+SLS2.222.53+0.31 [0.05, 0.51]

Both models retain low EER on the fully excluded route, while the paired intervals show an increase relative to pooled training. This provides further support for the delivery generalization examined in Table 5: detectors can transfer to an unseen complete delivery composition built from familiar operators.

Reproducibility and presentation. The submitted supplement already contains construction/evaluation code, configurations, documentation and 20 audio examples. As explained in its README, the full 941,201-waveform corpus and third-party pretrained weights are excluded because of their size. The supplied configurations and adapters specify the required model identifiers and paths. The third-party pretrained weights can be obtained following the respective repositories' READMEs.

In the revision, we will clarify pretrained-weight downloads and runtime setup, and add development/test metadata, scores from the supplementary runs, checkpoint/score hashes and exact commands for independent score-based verification. We will also streamline the prose, standardize terminology and improve Figure 1’s resolution and label legibility.

AuthorsOpenReview ↗

Follow-up to Reviewer Zf4N

Thank you again for your thoughtful review. We hope our responses and revisions have clarified the points you raised. If any concerns remain, we would appreciate the opportunity to address them further during the discussion.

Reviewer_EDusOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ChainBench-ADD, an audio deepfake benchmark containing lots of waveforms across a few delivery families. Each sample is linked to a clean parent, so as to have controlled comparisons of delivery operators, parameters, ect. Experiments with five detectors show large performance outcomes.

Critical strengths

The matched-parent design allows controlled comparisons of delivery effects. Applying different chains to the same clean parent reduces confounding from speaker identity, content, and generator differences. The local intervention tasks allow specific changes to be examined individually. Speaker-disjoint splits further improve the evaluation design.

The benchmark covers several delivery conditions in a documented, reproducible framework, including codecs, packet loss, room responses, noise, and combinations of these effects.

Critical weaknesses

The experiments show behavior within the renderer, but seem not to establish transfer to real delivery conditions. The paper describes the operators as controlled approximations. Quality scores, ASR drift, and speaker similarity help assess what the transformations preserve, but do not result in delivery realism. No comparison with actual platform processing, communication recordings, or physical replay is reported to show whether the simulated conditions reproduce detector rankings or failure patterns. All six generators appear in both training and testing, and the unseen-template experiment uses the same operator implementation. These experiments support generalization within the benchmark, but indicated less about transfer beyond it. A direct comparison with existing delivery datasets would help establish the benchmark’s added value.

Questions for authors
  1. Do the definitions in Appendix D.3 imply that C-FFD and AURC-chain are exact rescalings? Please clarify their interpretation and explain how failure-free lineages with different observed depths should be compared.
  2. What are the distributions of lineage size, connected-component size, and maximum reachable distance? How many lineages have newly observed nodes at each distance? These statistics would help explain whether the flat (R_k) curves reflect robustness or sparse graph coverage.
  3. Is there evidence that the simulated conditions reproduce detector behavior on actual replay, communication, or platform-processed recordings? If this evaluation is unavailable, please limit the conclusions to robustness under the specified renderer. 4.The hard-decision metrics use a threshold estimated from pooled evaluation labels. Do the conclusions and detector rankings hold when the threshold is selected on the development split? It would be good to know uncertainty estimates that account for shared speakers and parents, especially for the close comparisons between Nes2Net and XLS-R+SLS.
5 comments
AuthorsOpenReview ↗

Response to Reviewer EDus (1/3)

Weakness 1 & Question 3: Shared generators/operators and transfer to real delivery

Weakness 1: The experiments show behavior within the renderer, but seem not to establish transfer to real delivery conditions. The paper describes the operators as controlled approximations. Quality scores, ASR drift, and speaker similarity help assess what the transformations preserve, but do not result in delivery realism. No comparison with actual platform processing, communication recordings, or physical replay is reported to show whether the simulated conditions reproduce detector rankings or failure patterns. All six generators appear in both training and testing, and the unseen-template experiment uses the same operator implementation. These experiments support generalization within the benchmark, but indicated less about transfer beyond it. A direct comparison with existing delivery datasets would help establish the benchmark’s added value.

Question 3: Is there evidence that the simulated conditions reproduce detector behavior on actual replay, communication, or platform-processed recordings? If this evaluation is unavailable, please limit the conclusions to robustness under the specified renderer.

Thank you for highlighting real-delivery behavior.

Shared generators and operator implementation. Yes. All six generators are shared between training and testing, and the unseen-template experiment uses the same operator implementation. Table 5 therefore tests new delivery-template configurations with familiar generators and operator implementations.

Comparison with actual platform processing. We compared recordings from the Tencent Meeting/Zoom clients with an Opus-only 24 kbps reference from our renderer. Both platforms received the same 128 held-out source clips (parents) from 39 speakers, balanced across English/Mandarin and bona fide/spoof. Mac computer-audio sharing and Windows receiver loopback were used with both microphones muted. All formal recordings were retained and pilot clips excluded.

We scored every version with newly trained Nes2Net and XLS-R+SLS models pooled across delivery families, using fixed checkpoints and development-selected decision thresholds. This holds the source speech and detector fixed. The table reports misclassification rates at those thresholds:

ModelPlatformOpus-only error (%)Received error (%) [95% CI]Shared errors (n)New errors (n)
Nes2NetTencent Meeting0.003.13 [0.71, 6.25]04
Nes2NetZoom0.003.91 [1.02, 7.14]05
XLS-R+SLSTencent Meeting0.784.69 [1.39, 8.62]15
XLS-R+SLSZoom0.784.69 [1.50, 8.59]15

“Shared” counts errors under both the reference and real path, and “new” counts errors on the real path alone. Speaker-clustered 95% confidence intervals (CIs) use 1,000 resamples, keeping all recordings from each sampled speaker together.

Nes2Net has lower observed fixed-threshold error under the reference and both platform paths. Both models also make new platform errors absent from the reference, supporting our finding that delivery can change detection outcomes with source speech and detector held fixed. These observations do not yet establish that the simulated conditions reproduce real-platform detector rankings or failure patterns.

Comparison with existing delivery datasets. We have not yet run a same-detector performance comparison on ADD-C or ReplayDF. Table 1 compares benchmark design: ChainBench-ADD combines multiple delivery families, explicit ordered templates and matched-parent local edits, enabling comparisons of operator, parameter and order changes on the same source speech.

In the revision, we will add the recording setup, fixed-threshold results and shared/new-error analysis, and distinguish real-platform observations from claims about simulation transfer.

AuthorsOpenReview ↗

Response to Reviewer EDus (2/3)

Questions 1 & 2: Path metrics, graph coverage and comparisons at the same observed depth

Question 1: Do the definitions in Appendix D.3 imply that C-FFD and AURC-chain are exact rescalings? Please clarify their interpretation and explain how failure-free lineages with different observed depths should be compared.

Question 2: What are the distributions of lineage size, connected-component size, and maximum reachable distance? How many lineages have newly observed nodes at each distance? These statistics would help explain whether the flat (R_k) curves reflect robustness or sparse graph coverage.

Metric identity and interpretation. Yes. C-FFD (censored first-failure distance) and AURC-chain (average correctness survival across graph distances) are exact rescalings. With Dmax⁡D_{\max} denoting the largest observed graph distance:

C−FFD=(Dmax⁡+1)AURC−chain.\mathrm{C - FFD}=(D_{\max}+1)\mathrm{AURC - chain}.

RkR_k is the fraction of lineages whose reachable versions through distance kk are all correctly classified. Let TgT_g be the first failure distance, capped at Dmax⁡+1D_{\max}+1 for a failure-free lineage. A lineage contributes to RkR_k when Tg>kT_g>k, so summing its survival indicators gives TgT_g. Averaging gives the identity. In Table 7, Dmax⁡=3D_{\max}=3: divide the reported AURC-chain percentage by 100, then multiply by 4 to obtain C-FFD. C-FFD expresses capped distance to first failure, while AURC-chain expresses average survival over the evaluated distances. They carry the same information on different scales.

Comparing failure-free lineages with different observed depths. We compare their capped scores together with their observed depths. Under Table 7's definition, failure-free lineages observed through distances 1 and 3 both receive the cap 4. The supported conclusions are “correct through 1” and “correct through 3,” respectively. The deeper lineage has verified correctness farther from the reference, while the shallower one remains untested beyond distance 1. Greater depth therefore describes a farther verified extent.

Lineage-size and connected-component-size distributions. For the original Table 7 test graph, a lineage groups processed versions of one source clip with the same delivery family and label. A node represents a distinct processing configuration, including operator order and codec/parameter categories. Edges connect configurations differing by an allowed single edit. A connected component contains nodes linked through such edges. The reference component contains the lineage's shortest observed processing chain:

Size nnLineages (nn waveforms)Lineages (nn nodes)All components (nn nodes)Reference components (nn nodes)
10078,36520,630
23829221,86610,571
31,0293,4772,3942,134
432,56029,858292292

Maximum reachable distance and newly observed nodes. Graph distance is the minimum number of edits from that reference chain:

Distance kkLineages with maximum reachable distance kkLineages with newly observed nodes at distance kk
020,63033,627
111,27212,997
21,6781,725
34747

Of the test graph's 133,403 waveforms, 81,105 are disconnected from the reference and excluded from these distance statistics. A lineage with no newly observed nodes at a larger distance remains in the RkR_k denominator with its previous correctness status.

Do the flat RkR_k curves reflect robustness or sparse graph coverage? We recomputed RkR_k within fixed cohorts observed at every distance through H=2H=2 or H=3H=3 in the existing graph. Results use the supplementary pooled detectors and development-selected thresholds. The table shows the last two distances of each set, with the distance-3 set a subset of the distance-2 set:

Common depth HHLineages (n)Graph distance kkNes2Net RkR_k (%)XLS-R+SLS RkR_k (%)
21,725196.7596.58
21,725295.4295.30
347297.8795.74
347397.8795.74

The cohort observed through distance 2 continues to lose correctness, while the 47-lineage cohort reaching distance 3 has no further drop at its final step. Together with the small number of lineages having newly observed nodes at larger distances, these results indicate that sparse graph coverage contributes to the flat pooled tail.

AuthorsOpenReview ↗

Response to Reviewer EDus (3/3)

Question 4: Calibration on development data and uncertainty in detector comparisons

Question 4: The hard-decision metrics use a threshold estimated from pooled evaluation labels. Do the conclusions and detector rankings hold when the threshold is selected on the development split? It would be good to know uncertainty estimates that account for shared speakers and parents, especially for the close comparisons between Nes2Net and XLS-R+SLS.

We focus these checks on Nes2Net and XLS-R+SLS, the two strongest baselines in Tables 6–7 and the close comparison highlighted in your question. Results use new pooled-training runs with development-selected checkpoints. The original five-model rankings in Tables 6–7 have not been rechecked with development-selected thresholds. BER is balanced error rate, PCR is pair consistency, and PJA is the percentage of pairs with both predictions correct.

Conclusions and rankings with development-selected thresholds. In these supplementary runs, the main conclusions and most rankings hold, with every tabulated metric changing by less than one percentage point. Parameter perturbation still yields the highest PCR/PJA and substitution the lowest for both models in this pooled comparison. Nes2Net retains lower BER and higher AURC-chain. Arrows show full-test-selected threshold → development-selected threshold on identical test scores:

MetricNes2NetXLS-R+SLS
BER %3.85 → 3.894.30 → 4.39
Substitution PCR %93.85 → 93.4594.12 → 93.63
Substitution PJA %91.69 → 91.1791.25 → 90.34
Parameter perturbation PCR %96.42 → 96.3096.73 → 96.30
Parameter perturbation PJA %95.43 → 95.1795.55 → 94.94
Order swap PCR %95.22 → 95.0295.30 → 94.97
Order swap PJA %92.46 → 92.1391.65 → 90.88
AURC-chain %95.06 → 94.6794.85 → 94.17

Nes2Net's substitution/order PJA advantage also persists. C-FFD shares the AURC-chain ranking because they are exact rescalings.

Uncertainty accounting for shared parents and speakers. Parent- and speaker-clustered intervals support Nes2Net's lower EER/BER and higher substitution/order PJA, while the parameter-PJA difference remains unresolved. Intervals use 1,000 paired whole-parent or language-stratified whole-speaker bootstrap draws with fixed checkpoints. For BER/PJA, development/test clusters are resampled independently and thresholds reselected on each development draw. EER (equal error rate) uses test draws only. Differences are XLS-R+SLS minus Nes2Net, in percentage points (pp):

MetricSLS−Nes difference, ppParent 95% CI, ppSpeaker 95% CI, pp
EER+0.45[+0.29, +0.65][+0.15, +0.82]
BER+0.50[+0.34, +0.70][+0.18, +0.86]
Substitution PJA−0.83[−1.29, −0.49][−1.58, −0.14]
Parameter perturbation PJA−0.24[−0.84, +0.21][−0.91, +0.27]
Order swap PJA−1.25[−1.78, −0.85][−2.15, −0.51]

The observed substitution consistency ranking still favors XLS-R+SLS despite its lower substitution PJA. Our reporting both metrics helps distinguish consistent predictions from predictions that remain correct across a delivery edit.

Reviewer_ZMo9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ChainBench-ADD, a dataset and benchmark for studying audio deepfake detection under post-generation delivery effects. The dataset contains 941,201 waveforms derived from 55,813 clean parents, covering English and Mandarin Chinese, six speech-generation systems, and five delivery families: direct, platform-like, telephony, simulated replay, and hybrid. Delivery processes are represented through reusable operators, ordered templates, and realized chains, with metadata linking each processed sample to its clean parent. The benchmark supports in-chain detection, operator substitution, parameter perturbation, order swap, and lineage-based robustness evaluation, together with leave-one-template-out testing. Experiments with five detectors show substantial performance variation across delivery families and demonstrate the usefulness of distinguishing prediction consistency from joint correctness.

Critical strengths

1.The benchmark provides a useful framework for controlled delivery-aware analysis. Combining multiple delivery families, explicit operator ordering, realized parameters, and parent-linked metadata enables more detailed analysis than reporting a single pooled detection score. The speaker-disjoint construction and parameter-matched order comparisons strengthen the framework by reducing important sources of confounding. This is a meaningful resource contribution for studying robustness to post-generation processing. 2.The experiments distinguish stable predictions from reliably correct predictions. Reporting both pair consistency and both-correct rate is particularly useful. Table 6 shows that high consistency can coexist with substantially lower joint correctness, preventing repeated errors from being interpreted as successful invariance. The evaluation also covers several detector architectures and demonstrates a clear challenge under replay-like and hybrid delivery: for example, Nes2Net’s EER increases from 13.03% under direct delivery to 19.21% under simulated replay and 21.44% under hybrid delivery.

Critical weaknesses
  1. The coverage of the path-based evaluation is unclear. Each source clip has at most four processed versions per non-direct family. A graph step can mean changing a parameter or swapping operations, so graph distance does not necessarily represent another transmission stage. Also, a short group can still count as correct at larger distances if all its observed versions are correctly classified, even when no versions were tested at those distances. Without reporting how many groups have samples at each distance, it is difficult to tell whether the flattening in Table 7 reflects detector robustness or limited test coverage.
  2. The comparison of the three intervention tasks in Table 6 needs better control over sample difficulty. The paper carefully matches the two samples within each pair, but does not show whether the three tasks use similarly difficult samples. For example, a task with more samples from difficult hybrid conditions could obtain a lower both-correct rate partly for that reason. The reported differences may therefore reflect both the type of change and the difficulty of the selected samples. A comparison under similar delivery conditions would help establish whether replacing an operator is more disruptive than changing a parameter or swapping the order.
Questions for authors
  1. For Table 7, how many groups of processed versions have samples at each graph distance? How are groups with no samples at larger distances counted? Does the flattening of the curves remain when the comparison includes only groups observed through the same distance?
  2. Could you report the number of test pairs in each task in Table 6, broken down by delivery family and label? Does operator substitution remain the most challenging intervention when the tasks are compared under similar delivery conditions and starting difficulty?
  3. In Table 5, can a held-out template share the same sequence of operations with templates used in training? For example, aac_single and opus_single both apply one codec operation. Please clarify whether the results demonstrate generalization to new codec or parameter settings, new operation sequences, or both.
6 comments
AuthorsOpenReview ↗

Response to Reviewer ZMo9 (1/3)

Thank you for the detailed review and constructive feedback. We appreciate your recognition of the matched-parent design and the distinction between prediction consistency and joint correctness. We address the overlapping weaknesses and questions together below.

Weakness 1 & Question 1: Coverage of processed versions and interpretation of the flat curves

Weakness 1: The coverage of the path-based evaluation is unclear. Each source clip has at most four processed versions per non-direct family. A graph step can mean changing a parameter or swapping operations, so graph distance does not necessarily represent another transmission stage. Also, a short group can still count as correct at larger distances if all its observed versions are correctly classified, even when no versions were tested at those distances. Without reporting how many groups have samples at each distance, it is difficult to tell whether the flattening in Table 7 reflects detector robustness or limited test coverage.

Question 1: For Table 7, how many groups of processed versions have samples at each graph distance? How are groups with no samples at larger distances counted? Does the flattening of the curves remain when the comparison includes only groups observed through the same distance?

Thank you for raising this point. Table 7 measures robustness as maintaining correct decisions across delivery versions of the same source clip. RkR_k is the fraction of all groups for which every observed, reachable version through distance kk is correctly classified. Groups that failed earlier remain in the denominator, so RkR_k measures cumulative correctness.

Groups tested at each graph distance. Counts are from the original Table 7 test graph. Each group contains versions with the same source, delivery family and label. Distance counts the minimum number of graph edits from its shortest observed processing chain, so a step need not add a transmission stage. Percentages are relative to all 33,627 groups:

Graph distance kkGroups observed at kkShare of all groups (%)
033,627100.00
112,99738.65
21,7255.13
3470.14

How are groups with no samples at larger distances counted? They remain in the denominator. A group whose observed versions are all correct also remains in the numerator at larger distances, while a group with an observed error remains failed. Its contribution therefore stays unchanged without further testing. Only the 47 groups with samples at distance 3 can fail anew between R2R_2 and R3R_3, so the pooled decrease is at most about 0.14 percentage points.

Does the flattening remain when only groups observed through the same distance are compared? Partly. The fixed depth-2 cohort still declines, while the 47-group depth-3 cohort remains flat from distance 2 to 3. Each cohort keeps the same groups, with samples at every compared distance. The depth-3 cohort is a subset of the depth-2 cohort. All curves use newly trained Nes2Net and XLS-R+SLS models pooled across delivery families, with development-selected checkpoints and thresholds:

CohortModelR0R_0 (%)R1R_1 (%)R2R_2 (%)R3R_3 (%)R4R_4 (%)
Existing, through distance 2Nes2Net97.5196.7595.42——
Existing, through distance 2XLS-R+SLS97.3996.5895.30——
Existing, through distance 3Nes2Net100.00100.0097.8797.87—
Existing, through distance 3XLS-R+SLS100.00100.0095.7495.74—
Complete-prefixNes2Net99.6196.8896.6884.1874.61
Complete-prefixXLS-R+SLS98.8397.6697.0786.1379.69

Dashes mark distances outside the evaluated horizon. The complete-prefix check uses 512 source clips and every prefix of resample→codec→noise→RIR and codec→resample→RIR→noise, with Opus at 24 kbps, noise at 20 dB signal-to-noise ratio and a room impulse response (RIR) with reverberation time RT60=0.4 s. RkR_k requires correctness through kk on both branches.

The final flat step reflects observed stability within those 47 groups, while limited coverage contributes to the pooled flattening. The losses on the fully observed paths illustrate our point: correctness at one version need not persist through further delivery changes.

We will report per-distance coverage alongside Table 7 and revise our interpretation of the curves that “flatten substantially” to account for the limited coverage at larger distances.

AuthorsOpenReview ↗

Response to Reviewer ZMo9 (2/3)

Weakness 2 & Question 2: Comparing intervention difficulty under matched conditions

Weakness 2: The comparison of the three intervention tasks in Table 6 needs better control over sample difficulty. The paper carefully matches the two samples within each pair, but does not show whether the three tasks use similarly difficult samples. For example, a task with more samples from difficult hybrid conditions could obtain a lower both-correct rate partly for that reason. The reported differences may therefore reflect both the type of change and the difficulty of the selected samples. A comparison under similar delivery conditions would help establish whether replacing an operator is more disruptive than changing a parameter or swapping the order.

Question 2: Could you report the number of test pairs in each task in Table 6, broken down by delivery family and label? Does operator substitution remain the most challenging intervention when the tasks are compared under similar delivery conditions and starting difficulty?

We agree that comparisons across the three tasks should control sample difficulty.

Test-pair counts by delivery family and label. Entries are eligible Table 6 test pairs, reported as bona fide (BF, genuine speech) / spoof:

Delivery familySubstitution BF / spoofParameter perturbation BF / spoofOrder swap BF / spoof
Platform-like1,507 / 3,1111,414 / 2,8350 / 0
Telephony0 / 01,338 / 2,4950 / 0
Simulated replay3,298 / 6,5530 / 02,733 / 5,409
Hybrid4,338 / 8,6610 / 02,740 / 5,432
Total pairs27,4688,08216,314

The substitution groups allow multi-position operator-name changes, including some reorderings. The common-base experiment below isolates a single edit with non-target settings fixed.

Within-family comparisons from the same starting waveform. We re-paired the test versions so both edits share the same starting sample_id. Parents are weighted equally, as are starting versions within each parent. Results use the supplementary pooled detectors and fixed development thresholds. Pair joint accuracy (PJA) is the proportion of pairs with both predictions correct. ΔPJA is substitution minus comparator PJA, in percentage points (pp). Negative values mean substitution is more challenging:

Family (comparator)Matched parents (n)ModelΔPJA, pp [speaker 95% CI]
Platform-like (parameter)428Nes2Net+0.93 [+0.11, +1.73]
Platform-like (parameter)428XLS-R+SLS+0.82 [-0.12, +1.81]
Simulated replay (order)5,750Nes2Net-4.89 [-5.82, -4.02]
Simulated replay (order)5,750XLS-R+SLS-3.97 [-4.83, -3.17]
Hybrid (order)3,878Nes2Net-1.99 [-2.64, -1.37]
Hybrid (order)3,878XLS-R+SLS-2.05 [-2.66, -1.49]

Substitution is more challenging than order swaps in simulated replay and hybrid for both models, with speaker-clustered intervals below zero. Platform-like shows slightly lower PJA for parameter perturbation. These same-start comparisons can still differ in non-target settings, which the following experiment fixes.

Comparison with non-target settings fixed. For each of 512 source clips (parents), all three edits start from the same hybrid waveform: resample (16→8→16 kHz) → Opus (24 kbps) → RIR (RT60=0.4 s). We replace resampling with narrowband band-limiting, change only the bitrate to 32 kbps, or swap the adjacent resample/codec operations. Content, speaker, label and starting delivery conditions are identical across tasks. Non-target settings and operator-specific random draws are shared. Results for the same two pooled detectors are:

ModelEditPre-edit accuracy (%)PJA (%)Retention on base-correct sources (%)Score drift / dev IQR
Nes2NetOperator substitution89.6585.5595.420.1009
Nes2NetParameter perturbation89.6588.2898.470.0271
Nes2NetOrder swap89.6587.3097.390.0438
XLS-R+SLSOperator substitution91.4188.4896.790.0676
XLS-R+SLSParameter perturbation91.4190.4398.930.0154
XLS-R+SLSOrder swap91.4190.6399.150.0211

Retention measures post-edit accuracy on exactly the same initially correct clips across tasks within each model. Score drift is normalized by that model’s development-score interquartile range (IQR).

For both detectors, substitution has the lowest PJA and retention and the largest score drift. Parent- and speaker-clustered intervals support its lower PJA than parameter perturbation. Thus, Table 6's substitution ranking persists when starting difficulty and non-target settings are controlled in this hybrid setting.

In the revision, we will include the common-base comparison in Section 4.3 and the pair counts and within-family comparisons in Appendix E.2.

AuthorsOpenReview ↗

Response to Reviewer ZMo9 (3/3)

Question 3: What is unseen when a delivery template is held out?

Question 3: In Table 5, can a held-out template share the same sequence of operations with templates used in training? For example, aac_single and opus_single both apply one codec operation. Please clarify whether the results demonstrate generalization to new codec or parameter settings, new operation sequences, or both.

Yes. Some held-out templates share the same sequence of operations with templates used in training. For example, holding out aac_single leaves opus_single and the same one-operation codec sequence in training. AAC also remains represented in other chains.

A template specifies an ordered processing chain with codec and parameter settings. Each of the 32 Table 5 evaluations removes one template ID from its family's training and development pools:

AspectOriginal Table 5 protocol
Template IDExcluded from train/dev in every evaluation
Codec identities and parameter valuesOther templates may reuse them. Codec identities and parameter ranges are not separately held out
Complete operator sequenceUnseen within the family for 20 held-out templates, shared for 12

Table 5 thus evaluates held-out delivery-template configurations built from familiar operators. Its results cover both template-specific codec/parameter configurations on familiar sequences and, in the sequence-unseen cases, transfer to new compositions of those operators.

Comments on the paper

1 comment
Area_Chair_MN9fOpenReview ↗

Reminder: Discussion Period Closes in 24 Hours

Dear Reviewers,

This is a gentle reminder that the discussion period will close in 24 hours.

If you have not yet done so, please review the authors’ responses to your comments and participate in the discussion. Please let the authors know whether their responses have addressed your concerns or whether you have any remaining questions.

Thank you again for your time and contribution to the review process.

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.