- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper points out three main problems from existing literature in acoustic attacks and proposes a novel method of evaluating those attacks in a more realistic scenario. The paper analyzes that first, the attack success rate of the digital attack does not guarantee and reflect success rates of real-world OTA attack since there are more noises such as reverberation, attenuation, and spatial geometry. Second, the authors claim that the success rate of OTA depends on the attacker's knowledge of the room. To do so, the authors introduce knowledge gradient, ranging from native to oracle. The results have shown that knowing the exact RIR does not guarantee higher attack success, where Whisper for example Naive FGSM was more effective than RIR-aware FGSM. Third, the authors claim that stealth and effectiveness of the attack should be measured in different parts, meaning SNR for source and victim should be measured separately and calculate the projection cost.
Critical strengths
- The paper clearly points out evaluation flaws of the existing literature of OTA acoustic attacks.
- Measuring the effectiveness of existing OTA attacks in different RIRs and simulated room environments demonstrate the robustness of the attacks
- Knowledge gradient further demonstrates the knowledge of the attackers and measure the effectiveness of the crafted adversarial audio.
Critical weaknesses
-
While the authors critically points out the main concerns of the evaluation methods of existing attack literature, the paper itself does not have a simulation-to-real-world validation. Previous attack papers either processed the attack in digital-only or small sample of physical environment, the paper itself also evaluate in a simulated acoustic environment, which contradicts the claim. Did the authors consider measuring how close their simulated environment is to the real-world acoustic settings?
-
Though the introduction of knowledge gradient is interesting, the reviewer believes it did not clearly demonstrate the effectiveness of knowledge gradient. While the attack success rate does not have to be distinctively separated between possessing the knowledge gradient or not, at least the results should have demonstrated some variance through the introduction of knowledge gradient. One interesting result was the Naive FGSM was more effective than RIR-aware FGSM, but did the authors try to interpret why this was the case? Or was RIR-aware truly "RIR-aware"?
-
The reviewer is concerned about the novelty of the paper. The papers main contribution comes from differential RIR simulation, knowledge gradient, and Dual-Form SNR, while Qin et al. (2019) [1] already evaluated their methods through image source method, different room configuration both white-box and black-box, which is quite similar to what the authors were proposing as their contribution and novelty. If I may have missed, what do the authors believe their main contribution that separates from Qin et al. (2019) [1]?
References
[1] Qin, Yao, et al. "Imperceptible, robust, and targeted adversarial examples for automatic speech recognition." International conference on machine learning. PMLR, 2019.
Questions for authors
Please follow questions from Critical Weaknesses.