- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a “decision record” for speech deepfake detection. Instead of only keeping the final score, it keeps several component scores: a passive detector score, a watermark-probe score, a retrieval score, a profile score, and some nearest-neighbour information. The final score is produced with logistic regression, and the main change compared with standard score fusion is the addition of two absolute-difference features between some of these scores. The main experiments are on 4,080 ASVspoof 5 development examples where all the required scores are available. The proposed model gets 8.43% EER compared with 11.91% for the same cross-fitted logistic fusion without the difference features. The paper also includes ablations and some post hoc analysis of which examples might be worth reviewing.
Critical strengths
One strength is that the paper does quite a lot of experiments around the score-fusion result. In particular, it compares against the same logistic model without the absolute-difference features, and also tests no-probe, shuffled-probe, squared-difference, single-gap, and product-term variants. The authors are also quite open about the fact that these ablations do not isolate what is causing the improvement.
The main 8.43% versus 11.91% EER result also seems reasonably supported statistically under their chosen protocol.
Critical weaknesses
- I am not convinced the main methodological contribution is strong enough. The main gain seems to come from adding nonlinear transformations of scores that are already available. But it is not clear that the particular proposed “decision record” construction is important.
For example, the no-probe nonlinear experiment gets 8.67% EER compared with 8.43% for the proposed model, and the confidence interval includes zero. The shuffled-probe experiment gets 8.78%, again very close. The squared-difference model actually has a slightly better result at 8.33%.
So to me the clearest result is that nonlinear score fusion helps compared with linear score fusion on this subset. I do not think the experiments establish that the particular absolute-difference construction is important, or that the watermark/probe score is providing a meaningful extra source of information. The paper itself also says that the probe does not have an independently established predictive benefit. This makes the actual contribution feel fairly limited for ICLR.
- I also do not think the “auditable decision” claim is really validated. Keeping the individual component scores around is potentially useful, but any score-fusion system could in principle keep its input scores rather than discarding them. I do not think the paper shows that this particular record makes decisions more auditable in practice. There is no user study, analyst study, or deployment experiment. The paper is explicit about this, which I appreciate.
Also, in the post hoc review experiment, simply ranking examples by distance to the final score threshold works better than the union of the proposed diagnostic cues. At the same review budget, the scalar-margin queue captures 88.66% of the errors at 22.15% precision, compared with 82.85% at 20.70% precision for the four-cue union. The extra fields may still tell you which component disagreed, but I do not think the paper shows that this leads to better auditing or error analysis in any practical sense.
- The evaluation is quite limited. The main experiments are only on the 4,080 development examples for which all score streams are available. This is a small selected subset of the available spoof examples: only around 3.7-4.9% per synthesis family are retained after matching. The paper itself notes that this subset is not representative of the full retrieval predictions. The gain is also quite concentrated. A11 and A12 account for 1.46 percentage points out of the 2.02-point mean improvement. There is also no result on the official evaluation set or on another dataset. Finally, the best passive WavLM system already gets 6.71% EER, which is better than the proposed 8.43%. When their decision-record approach is used in the WavLM setting, it gets 9.14%, again worse than the original passive model. So I am not convinced this gives a practically stronger detector either.
Questions for authors
What do you see as the main methodological contribution beyond nonlinear score fusion? The no-probe nonlinear and shuffled-probe experiments, and squared-difference model all perform very similarly to the proposed method. What evidence shows that the specific absolute-gap methodology is important?
What is the actual demonstrated benefit of the “auditable record” (aside from keeping the input scores available)? In the fixed-budget post hoc experiment, ranking by the final score margin actually finds more errors than the cue union. Is there any evidence that the record helps a human analyst make better decisions?
Does the result hold on the official ASVspoof 5 evaluation set, or on another dataset? I think it would be important to have results on more experiments in order to judge whether the result generalizes.
I also found the WavLM calibration result confusing. The raw WavLM score has 6.71% pooled EER, while the cross-fitted calibrated version has 11.79% pooled EER even though both have 5.00% fold EER. Can the authors explain why pooling the cross-fitted calibrated scores causes such a large degradation, and how this affects the interpretation of the 9.14% decision-record result?
2 comments
WavLM pooled and fold-level interpretation
Thank you for pointing out the pooled/fold reversal. We recomputed the same 4,080 matched ASVspoof 5 development rows from the saved out-of-fold predictions. The passive one-feature cross-fit uses separate held-out-family logistic maps. Their raw-score slopes are all positive (6.313–6.651; intercepts −4.930 to −4.593), so each map preserves within-family ordering. Accordingly, mean fold EER remains 5.00%, matching raw WavLM. Pooled EER uses one threshold across folds and rises from 6.71% to 11.79%. At the pooled out-of-fold EER threshold (0.82286), spoof false-accept rates are 17.36% for A11, 58.00% for A13, and 18.97% for A15, while the other families are near zero. Same-fold positive/negative-pair AUC remains 0.9786; cross-fold-pair AUC falls from 0.9780 to 0.8942, and pooled AUC falls from 0.9781 to 0.9048. This supports a cross-fold score-scale/offset mismatch on these rows, rather than worse within-fold discrimination.
For the requested record/scalar comparison, pooled EER is 9.14% vs 9.72%. A paired speaker-cluster bootstrap of the pooled EER difference (record − scalar) gives −0.58% EER (95% percentile CI [−1.34%, −0.21%]), excluding zero (705 speakers, 2,000 resamples, seed 20260830). Mean fold EER is 6.56% for the record and 6.85% for scalar fusion. Against passive cross-fit, the record's pooled EER is lower (9.14% vs 11.79%), but its mean fold EER is higher (6.56% vs 5.00%). Thus the 9.14% result is a pooled-threshold comparison on this matched development subset; it does not establish better within-fold discrimination than passive WavLM or generalization beyond these rows.
Cross-fitted nonlinear score fusion reaches 8.43% EER on the 4,080-example matched development subset, compared with 11.91% for scalar fusion. Nearby controls do not establish a distinct benefit from the proposed absolute gaps: the no-probe nonlinear control reaches 8.67%, with a paired interval including zero against the proposed model, and the squared-difference model reaches 8.33%. The direct passive-retrieval gap was not tested, so these results do not identify the probe as an independent source of predictive value.
The decision record retains input scores and supports post hoc inspection. At the same review load, the final-score-margin queue covers 88.66% of observed errors while the four-cue union covers 82.85%. No analyst study measures whether that information improves review decisions.
The evaluation is limited to 4,080 matched ASVspoof 5 development examples, and the best passive WavLM run has lower EER at 6.71%. No official evaluation set or independent corpus is reported. For the WavLM comparison, pooled EER uses one global threshold across cross-fitted scores, whereas fold EER averages fold-level thresholds. Passive calibration is 11.79% pooled and 5.00% fold EER; the record is 9.14% pooled and 6.56% fold EER. The record-versus-scalar-fusion pooled difference is −0.58% (9.14% − 9.72%), without paired uncertainty. These results do not establish generalization or a performance advantage over the best passive run.