acceptodds
Under review as a conference paper at ICLR 2027

From Scores to Evidence: Auditable Decisions Can Improve Speech Deepfake Detection

Abstract

One score per utterance does not show which component measurements supported a speech deepfake decision. We ask whether adding two absolute score differences to cross-fitted logistic fusion lowers equal error rate (EER): one difference compares the passive and watermark scores, and the other compares their average with retrieval. The decision record retains four primary scores and an auxiliary neighbor-distance field for post hoc diagnosis. The record stores a passive detector score on original audio, a watermark score on a marked copy, a retrieval vote from ten nearest support examples, and a speaker-profile margin based on nearest bona fide and spoof distances. It also stores distance to the nearest support example. For spoof examples, retrieval excludes the target synthesis family. On 4,080 ASVspoof 5 Track 1 development examples with all four primary scores, the absolute-difference model reaches 8.43% EER. Cross-fitted scalar fusion reaches 11.91%, and the best of 22 passive WavLM runs reaches 6.71%. Across eight held-out families, the mean fold EER reduction is concentrated in A11 and A12. The controls do not identify what causes the reduction. The fixed rule averages passive and watermark scores, then averages that result with retrieval. Its minimum detection cost is lower than the learned model's. In an in-sample, post hoc analysis of the same examples, sorting by distance to the final-score threshold places 88.66% of observed errors among the 33.75% of examples closest to the threshold. This analysis does not evaluate a deployment policy or analyst utility.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
speech deepfake detection, score-level fusion, decision records, watermark detection, retrieval-based detection, cross-fitted logistic fusion, ASVspoof 5

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_GVNXOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a “decision record” for speech deepfake detection. Instead of only keeping the final score, it keeps several component scores: a passive detector score, a watermark-probe score, a retrieval score, a profile score, and some nearest-neighbour information. The final score is produced with logistic regression, and the main change compared with standard score fusion is the addition of two absolute-difference features between some of these scores. The main experiments are on 4,080 ASVspoof 5 development examples where all the required scores are available. The proposed model gets 8.43% EER compared with 11.91% for the same cross-fitted logistic fusion without the difference features. The paper also includes ablations and some post hoc analysis of which examples might be worth reviewing.

Critical strengths

One strength is that the paper does quite a lot of experiments around the score-fusion result. In particular, it compares against the same logistic model without the absolute-difference features, and also tests no-probe, shuffled-probe, squared-difference, single-gap, and product-term variants. The authors are also quite open about the fact that these ablations do not isolate what is causing the improvement.

The main 8.43% versus 11.91% EER result also seems reasonably supported statistically under their chosen protocol.

Critical weaknesses
  1. I am not convinced the main methodological contribution is strong enough. The main gain seems to come from adding nonlinear transformations of scores that are already available. But it is not clear that the particular proposed “decision record” construction is important.

For example, the no-probe nonlinear experiment gets 8.67% EER compared with 8.43% for the proposed model, and the confidence interval includes zero. The shuffled-probe experiment gets 8.78%, again very close. The squared-difference model actually has a slightly better result at 8.33%.

So to me the clearest result is that nonlinear score fusion helps compared with linear score fusion on this subset. I do not think the experiments establish that the particular absolute-difference construction is important, or that the watermark/probe score is providing a meaningful extra source of information. The paper itself also says that the probe does not have an independently established predictive benefit. This makes the actual contribution feel fairly limited for ICLR.

  1. I also do not think the “auditable decision” claim is really validated. Keeping the individual component scores around is potentially useful, but any score-fusion system could in principle keep its input scores rather than discarding them. I do not think the paper shows that this particular record makes decisions more auditable in practice. There is no user study, analyst study, or deployment experiment. The paper is explicit about this, which I appreciate.

Also, in the post hoc review experiment, simply ranking examples by distance to the final score threshold works better than the union of the proposed diagnostic cues. At the same review budget, the scalar-margin queue captures 88.66% of the errors at 22.15% precision, compared with 82.85% at 20.70% precision for the four-cue union. The extra fields may still tell you which component disagreed, but I do not think the paper shows that this leads to better auditing or error analysis in any practical sense.

  1. The evaluation is quite limited. The main experiments are only on the 4,080 development examples for which all score streams are available. This is a small selected subset of the available spoof examples: only around 3.7-4.9% per synthesis family are retained after matching. The paper itself notes that this subset is not representative of the full retrieval predictions. The gain is also quite concentrated. A11 and A12 account for 1.46 percentage points out of the 2.02-point mean improvement. There is also no result on the official evaluation set or on another dataset. Finally, the best passive WavLM system already gets 6.71% EER, which is better than the proposed 8.43%. When their decision-record approach is used in the WavLM setting, it gets 9.14%, again worse than the original passive model. So I am not convinced this gives a practically stronger detector either.
Questions for authors

What do you see as the main methodological contribution beyond nonlinear score fusion? The no-probe nonlinear and shuffled-probe experiments, and squared-difference model all perform very similarly to the proposed method. What evidence shows that the specific absolute-gap methodology is important?

What is the actual demonstrated benefit of the “auditable record” (aside from keeping the input scores available)? In the fixed-budget post hoc experiment, ranking by the final score margin actually finds more errors than the cue union. Is there any evidence that the record helps a human analyst make better decisions?

Does the result hold on the official ASVspoof 5 evaluation set, or on another dataset? I think it would be important to have results on more experiments in order to judge whether the result generalizes.

I also found the WavLM calibration result confusing. The raw WavLM score has 6.71% pooled EER, while the cross-fitted calibrated version has 11.79% pooled EER even though both have 5.00% fold EER. Can the authors explain why pooling the cross-fitted calibrated scores causes such a large degradation, and how this affects the interpretation of the 9.14% decision-record result?

2 comments

Cross-fitted nonlinear score fusion reaches 8.43% EER on the 4,080-example matched development subset, compared with 11.91% for scalar fusion. Nearby controls do not establish a distinct benefit from the proposed absolute gaps: the no-probe nonlinear control reaches 8.67%, with a paired interval including zero against the proposed model, and the squared-difference model reaches 8.33%. The direct passive-retrieval gap was not tested, so these results do not identify the probe as an independent source of predictive value.

The decision record retains input scores and supports post hoc inspection. At the same review load, the final-score-margin queue covers 88.66% of observed errors while the four-cue union covers 82.85%. No analyst study measures whether that information improves review decisions.

The evaluation is limited to 4,080 matched ASVspoof 5 development examples, and the best passive WavLM run has lower EER at 6.71%. No official evaluation set or independent corpus is reported. For the WavLM comparison, pooled EER uses one global threshold across cross-fitted scores, whereas fold EER averages fold-level thresholds. Passive calibration is 11.79% pooled and 5.00% fold EER; the record is 9.14% pooled and 6.56% fold EER. The record-versus-scalar-fusion pooled difference is −0.58% (9.14% − 9.72%), without paired uncertainty. These results do not establish generalization or a performance advantage over the best passive run.

AuthorsOpenReview ↗

WavLM pooled and fold-level interpretation

Thank you for pointing out the pooled/fold reversal. We recomputed the same 4,080 matched ASVspoof 5 development rows from the saved out-of-fold predictions. The passive one-feature cross-fit uses separate held-out-family logistic maps. Their raw-score slopes are all positive (6.313–6.651; intercepts −4.930 to −4.593), so each map preserves within-family ordering. Accordingly, mean fold EER remains 5.00%, matching raw WavLM. Pooled EER uses one threshold across folds and rises from 6.71% to 11.79%. At the pooled out-of-fold EER threshold (0.82286), spoof false-accept rates are 17.36% for A11, 58.00% for A13, and 18.97% for A15, while the other families are near zero. Same-fold positive/negative-pair AUC remains 0.9786; cross-fold-pair AUC falls from 0.9780 to 0.8942, and pooled AUC falls from 0.9781 to 0.9048. This supports a cross-fold score-scale/offset mismatch on these rows, rather than worse within-fold discrimination.

For the requested record/scalar comparison, pooled EER is 9.14% vs 9.72%. A paired speaker-cluster bootstrap of the pooled EER difference (record − scalar) gives −0.58% EER (95% percentile CI [−1.34%, −0.21%]), excluding zero (705 speakers, 2,000 resamples, seed 20260830). Mean fold EER is 6.56% for the record and 6.85% for scalar fusion. Against passive cross-fit, the record's pooled EER is lower (9.14% vs 11.79%), but its mean fold EER is higher (6.56% vs 5.00%). Thus the 9.14% result is a pooled-threshold comparison on this matched development subset; it does not establish better within-fold discrimination than passive WavLM or generalization beyond these rows.

Reviewer_xRtkOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a decision record for speech deepfake detection that retains passive, watermark-probe, retrieval, and speaker-profile scores alongside a final score. Its learned model adds two absolute score differences to logistic fusion. On a matched subset of the ASVspoof 5 development set, it reduces EER from 11.91% for linear score fusion to 8.43%. The paper also reports ablations, leave-one-family-out calibration, and paired uncertainty analyses

Critical strengths
  1. The paper makes a concrete comparison against linear score fusion and reports a meaningful EER improvement on its matched development subset.
  2. The ablations and uncertainty analyses are useful. The authors also distinguish several measured findings from deployment claims, which helps readers judge the limits of the evidence.
Critical weaknesses
  1. The proposed decision record is motivated by an auditability problem that the paper neither establishes clearly nor evaluates directly. The introduction does not identify representative systems that discard component measurements or a concrete audit failure caused by their absence. The retrospective error-ranking experiment does not test whether analysts make better decisions with the record; at the same review load, final-score threshold margin captures more errors than the proposed four-cue union (88.66% versus 82.85%). Figure 1 illustrates a missing probe, but the main comparison includes only examples with all four measurements available.
  2. The detection evidence does not isolate the proposed score differences or establish generalization. A nonlinear control without the probe reaches 8.67% EER, with a confidence interval for its difference from 8.43% that includes zero; a squared-difference model reaches 8.33%, and shuffling probe scores still yields 8.78%. The main result uses 4,080 matched development examples, comprising only about 3.7%–4.9% of spoof examples from each family, with no official evaluation-set or independent-corpus result. The mean fold improvement is concentrated in A11 and A12. On the matched examples, the proposed model also trails the best passive WavLM run in EER (8.43% versus 6.71%), and its minDCF is worse than the fixed retrieval rule under the stated costs.
Questions for authors
  1. Can the authors evaluate a locked model on the official evaluation set or an independent corpus, including examples where the probe is unavailable?
  2. What evidence distinguishes the proposed score differences from a general nonlinear transformation of the passive and retrieval scores?
  3. What measurable audit task does the decision record improve over showing an analyst the final score and its threshold margin?
1 comment

The measurable detection result is confined to the matched ASVspoof 5 development subset. The proposed model reaches 8.43% EER, below cross-fitted scalar fusion at 11.91%, but above the best of 22 passive WavLM runs at 6.71%. The no-probe nonlinear control reaches 8.67%, with a paired interval for the difference that includes zero; the squared-difference model reaches 8.33%. The direct passive-retrieval gap control is not reported, so the specific value of the proposed absolute differences remains unresolved.

The decision record retains component scores for inspection and ranks errors after scoring. At a shared load of 1,377 examples, a final-score-margin queue covers 88.66% of observed errors, compared with 82.85% for the four-cue union. The union retains which cues flagged each example, but the paper includes no analyst evaluation and does not show that these fields improve audit decisions.

The analysis uses 4,080 matched development examples and reports no official evaluation-set or independent-corpus results. It does not evaluate examples where the probe is unavailable. The findings are limited to this development-set comparison and post hoc description of retained scores; generalization and analyst utility are unestablished.

Reviewer_WAM2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies nonlinear score fusion for speech deepfake detection while retaining component measurements for inspection. Its decision record combines a passive detector score, a keyed probe measured on a marked copy, retrieval and profile scores, and nearest-neighbor distance. A cross-fitted logistic model adds two absolute score differences to the scalar features (Section 3, Equation (1)).

On 4,080 matched ASVspoof 5 development utterances, the proposed model reduces pooled EER from 11.91% for scalar fusion to 8.43% and ECE from 0.0840 to 0.0709 (Table 2). Most mean-fold EER improvement comes from A11 and A12. Nearby no-probe nonlinear and squared-difference controls reach 8.67% and 8.33% EER, respectively; the selected best passive WavLM run reaches 6.71%. Worked records and post hoc error rankings illustrate diagnostic use, but do not establish analyst utility or a causal detection benefit from displaying the fields.

Critical strengths
  1. A direct ablation supports a meaningful, narrowly defined improvement. The comparison uses the same cross-fitted fusion protocol with and without the two absolute-difference features. Pooled EER decreases by 3.48 pp, with nominal paired 95% CI [-4.60, -2.34] pp and held-out-fold CI [-5.85, -0.20] pp (Table 27, p. 21). The speaker-resampled interval also excludes zero (Table 29). ECE decreases by 0.0130, with paired interval [-0.0159, -0.0090] (Table 28). Alternative bona fide fold assignments preserve the pooled improvement (Table 31). These checks substantiate the result for the specified scores and evaluated population, without establishing general superiority.

  2. The diagnostic analysis exposes important failures and counterevidence. The paper reports the passive CNN's A12 reversal, weak standalone probe behavior, competitive nonlinear controls, and a stronger passive reference that outperforms the proposed system (Tables 6, 10-11, 27, and 34). Figure 4, p. 14, contrasts a corrected A12 spoof with an A13 failure involving conflicting passive and retrieval scores. Table 25 also shows that the cue union underperforms scalar-margin ranking at the same queue size. This transparency helps interpret aggregate performance and provides useful information about the system's limitations.

Critical weaknesses
  1. The added predictive and audit value remains unclear. The no-probe nonlinear control reaches 8.67% EER versus 8.43% for the proposed model, with a difference of -0.24 pp and 95% CI [-0.81, +0.46] pp; the squared-difference variant reaches 8.33% (Tables 2 and 27). Thus, the improvement over linear fusion does not establish an advantage over nearby nonlinear alternatives, although the proposed model has a small ECE advantage over the no-probe control (Table 28). The diagnostic cue union also captures fewer errors than scalar-margin ranking at equal budget (82.85% versus 88.66%; Table 25), and analyst utility remains untested. Compare minimal no-probe gap models under matched selection budgets and evaluate diagnostic value while holding the classifier fixed.

  2. Subset selection and cross-fold behavior limit generalization claims. Only 4,080 of 40,000 retrieval predictions enter the matched analysis, including 865 of 20,000 spoof examples (Tables 5 and 17); no independent evaluation-set or cross-corpus result is reported. WavLM pooled EER favors the record over cross-fitted passive calibration (9.14% versus 11.79%), whereas mean fold EER favors passive calibration (5.00% versus 6.56%); raw passive EER is 6.71%, selected from 22 runs (Table 11). This reversal warrants examining cross-fold score scales. The CNN improvement also appears within folds, but A11 and A12 account for 1.46 pp of its 2.02 pp mean-fold reduction (Table 30). Furthermore, whether all query classes and calibration-training rows use appropriately excluded support pools needs clarification (Sections 3-4; Tables 4 and 19); leakage is not established. Document these boundaries and missing-score selection, then evaluate an independently selected, frozen pipeline on disjoint data.

  3. Reproduction details are incomplete in the manuscript. Section 3 and Table 2 need exact distance/profile formulas, complete control-feature lists, and calibration fitting details. Redundant averages under L2 regularization also warrant a canonical five-score comparison. The reproducibility statement (p. 10) excludes full scores and checkpoints; the code archive was not independently inspected. Provide executable instructions and, where permitted, de-identified scores, fold assignments, and configurations.

  4. Uncertainty wording occasionally exceeds the evidence. Section 5, p. 8, describes favorable ordering against single-gap controls, but both paired intervals include zero: [-1.30, +0.14] and [-0.93, +0.22] pp (Table 27). Distinguish lower point estimates from statistically resolved improvements; an unresolved difference does not establish equivalence.

Questions for authors
  1. Clarification: support and fitting boundaries. For outer fold A12, what support pools are used for spoof queries, held-out bona fide queries, and calibration-training rows? Are retrieval/profile features recomputed under a common fold-specific exclusion? Please identify passive-stream training/checkpoint-selection data and confirm that standardization and feature/hyperparameter selection exclude outer test labels.

  2. Clarification and assessment-changing analysis: WavLM calibration. What explains pooled EER increasing from 6.71% to 11.79% after passive cross-fitted calibration while mean fold EER remains 5.00%? Please provide fold-specific score ranges or calibration parameters and paired uncertainty for the 9.14% versus 9.72% record/scalar-fusion comparison. This would clarify score-alignment versus discrimination effects.

  3. Assessment-changing experiment: gap/probe contribution. How does the full model compare with a minimal passive/retrieval model using a direct gap such as |s_p - s_r|, matched univariate nonlinear terms, and comparable regularization/selection? Does any advantage persist with WavLM inputs? Table 2 does not report this direct no-probe cross-score-gap comparison.

  4. Clarification and assessment-changing validation: selection. What causes the reduction to 3,215 bona fide and 865 spoof examples? Which data selected the retrieval configuration, passive checkpoints, gaps, and penalty? Was any labeled population reserved from these choices? Results on such an untouched population with the closest nonlinear controls would directly affect the recommendation.

1 comment

The main matched analysis contains 4,080 of 40,000 predictions, including 865 of 20,000 spoof predictions, after matching examples with all four primary scores. The paper reports the subset composition and held-out synthesis-family protocol, but does not establish that this development subset represents the full prediction set or an independent evaluation population.

The protocol states that spoof support excludes the held-out synthesis family, bona fide examples are assigned to folds by stable hash, and each calibrator is evaluated on examples it did not train on. It does not report the requested A12 row-level support memberships or establish a common exclusion rule for held-out bona fide queries and calibration rows.

For WavLM, pooled EER uses one global threshold over cross-fitted scores, whereas fold EER averages values computed separately within folds. The record's 9.14% pooled EER is 0.58% lower than scalar fusion at 9.72% (9.14% − 9.72%), but paired uncertainty for that contrast is not reported. Fold score ranges and calibration parameters are also absent.

The absolute-gap model reaches 8.43% EER, compared with 8.67% for the no-probe nonlinear control and 8.33% for the squared-difference model. The proposed-minus-no-probe interval includes zero, and the direct passive-retrieval gap was not evaluated. The paper's score fields, fixed averages, calibrator, and L2 penalty are described, but a complete score and fold package and an untouched population record are not supplied.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.