- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to assess the robustness of voice anti-spoofing models beyond accuracy. The authors propose PV-VASM, a model-agnostic framework that estimates how likely a detector is to fail under audio transformations, text-to-speech systems, and voice-cloning models. The framework combines several existing probabilistic bounds and introduces an adaptive method AdaVAU to select among them. The results show that verification is generally easier for simple audio transformations than for generated speech, while fine-tuning on a target generator can substantially improve robustness.
Critical strengths
-
The framework covers a much broader range of attacks than many existing verification methods. PV-VASM can also be applied to text-to-speech and voice-cloning systems, including generators that were not used during training.
-
Since PV-VASM treats the anti-spoofing model as a black box, it can in principle be used with different detector architectures. This makes the method easier to reuse and potentially more useful as an evaluation tool.
Critical weaknesses
-
One of my major concerns is that the evaluation is too narrow to support the broad claims of a general verification framework. Although PV-VASM is presented as model-agnostic, the experiments use only one anti-spoofing model. The paper indeed shows that the framework can work on one detector, but it remains unclear to other detectors with very different architectures or failure patterns.
-
PV-VASM estimates failure probability under a predefined distribution of transformations or generated samples. If that distribution does not match the attacks encountered in practice, the bound may not match/reflect real-world robustness meaningfully. This is important for voice attacks, where the space of possible generators, speakers, and settings is generally very large.
Questions for authors
-
How sensitive are the results to the choice of attack distribution? For example, if the distribution used for verification differs from the deployment, how should the guarantee be interpreted?
-
For the TTS and voice-cloning experiments, how much overlap is there between the generators used for verification and the data used to train the detector?
2 comments
Response to reviewer PysH
We thank the reviewer for their detailed feedback on our work. Below, we provide our answers to the weaknesses and questions raised by the reviewer. We also updated the text of the manuscript according to the provided answers (to include, where possible, additional experimental results and clarifications). We realize that the discussion phase itself is coming to an end, and posting our answers shortly before the deadline may feel inconvenient. However, we motivate it by the necessity of additional experiments and text rework. We would appreciate the reconsideration of the assessment of our work if the concerns of the reviewer are addressed sufficiently.
W1: We added verification of 2 more open-source models [Appendix A.2 Table 6]. We understand that evaluating the method across different models would strengthen the model-agnosticism claim, so we included it in the appendix.
W2: You are correct on that. Changing the parameter distribution directly impacts the upper bound of the expectation of r.v. Z, as it impacts the expectation itself.
We do not claim that the method can yield bounds that hold across multiple parameter distributions. Instead, we provide the framework that allows its user to choose an arbitrary parameter distribution that they find particularly applicable in their respective setting.
Q1: Please see the answer to W2.
Q2: TTS and VC models were present in the training set of the main model
No response from the authors
The authors chose not to include any response, so I will keep my original assessments.