- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies audio jailbreak attacks on large audio language models. It examine whether the previous audio attack methods depend on specific frequency bands or model layers. The paper tests this by removing different parts of the attack noise and checking whether the attack still works. The results show that the apparent importance of certain frequency bands mainly comes from how the bands are divided and how much noise they contain, rather than the frequency itself. The paper also finds no clear evidence that the attack is concentrated in a particular model layer.
Critical strengths
-
The question and motivation is clear and straightforward. Understanding whether audio jailbreaks depend on particular frequency bands could help improve future defenses.
-
The presentation of this paper (including the figures and tables) are clear and well structured.
-
The experiments are thorough, which include careful controls, especially random bands with matched bandwidth, energy, and uniform noise scaling. These comparisons help distinguish the effect of frequency location from the amount of noise removed.
Critical weaknesses
-
The Introduction part does not seem to accurately describe the findings and insights in the previous papers. GRM optimizes attacks on selected bands (not showing that jailbreak signals are concentrated in particular bands, and the important bands may differ across attack methods or models) and Gupta et al. studies frequency filtering. These works do not directly show that jailbreak signals are concentrated in particular frequency bands.
-
The main findings are only based on AdvWave-P and mainly on model Qwen2-Audio. The additional datasets still use the same attack and model, while the LLaMA-Omni results do not establish the same findings. Whether this method still works for other similar frequency-based attacks (if any) need more experiments, to show the generalization.
-
The comparison with previous methods is not fully convincing. The paper evaluates the band selections of ALMGuard and GRM using AdvWave-P, rather than evaluating these methods under their original settings.
-
Another point worth noting is why the authors do not directly use audio from audio-specific jailbreak benchmarks such as AJailBench and JALMBench, but instead generate their own audio. These benchmarks also consider the possible effects of gender, accent, language, age, background sounds, etc. (for example, gender and language may correspond to different frequencies and thus affect the results).
If authors can address these problems, I am willing to raise the score.
Questions for authors
-
Please clarify which previous works directly show that audio jailbreak signals are concentrated in specific frequency bands.
-
Consider testing other attacks and audio language models to show generalization.
-
Why are the band selections of ALMGuard and GRM evaluated using AdvWave-P rather than their original settings?
-
I am curious about the frequency aspect here. How might gender, accent, language, and background noise affect the findings?