- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents AUDIO SAFETY BENCH, a policy-grounded benchmark for evaluating audio safety across diverse threat models, and AUDIO GUARD, a modular audio safety framework. The benchmark covers multilingual speech, non-speech harmful events, speaker attributes, voice impersonation, and voice–content compositional risks. AUDIO GUARD combines SoundGuard, which detects waveform-level acoustic cues using an ECAPA-TDNN encoder, with ContentGuard, which performs ASR-based semantic safety classification. A rule-based integration module combines their predictions into scenario-specific safety decisions. Experiments on the proposed benchmark and four external benchmarks demonstrate competitive safety detection performance compared with several audio large language models.
Critical strengths
- The paper addresses an important and timely problem in audio safety, extending beyond traditional transcript-based content moderation to include sound events, speaker attributes, and voice–content compositional risks.
- AUDIO SAFETY BENCH provides a relatively comprehensive evaluation setting, covering 17 languages, diverse speaker conditions, and multiple audio-related threat models. The inclusion of hard-benign examples is particularly useful for evaluating false positives.
- The modular design of AUDIO GUARD is practical and interpretable. Compared with monolithic audio-LLM judges, the proposed framework demonstrates strong detection performance and potentially more efficient deployment. The evaluation also includes external benchmarks, component-wise analyses, and statistical uncertainty estimates.
Critical weaknesses
- The threshold selection and calibration protocol are insufficiently specified. The compositional integration relies on category-specific thresholds, but their numerical values and selection procedures are not clearly described. It is unclear whether these thresholds are calibrated on independent validation data or adjusted across benchmarks. Since threshold selection directly affects safety accuracy, unsafe recall, and false-positive rates, the authors should clarify the calibration protocol and report the operating thresholds.
- The benchmark annotations and evaluation protocols require greater transparency. AUDIO SAFETY BENCH is a major contribution, but the complete fine-grained label inventory, per-category sample distributions, and training/validation/test partitioning criteria are insufficiently documented. The mapping from sound and content annotations to final safety labels also requires clarification. In particular, the authors should explicitly define whether Content-only Accuracy in Table 5 measures exact risk-category matching or binary Safe/Unsafe accuracy, and explain how annotations from the four external benchmarks are converted into the unified evaluation space.
- The distinction between speaker recognition and voice impersonation detection is unclear. SoundGuard uses a pretrained ECAPA-TDNN encoder to recognize speaker-related attributes, including public-figure voices. However, recognizing a public figure's voice does not necessarily distinguish genuine recordings from AI-generated voice clones. The authors should clarify whether SoundGuard explicitly detects synthesized or impersonated speech, whether genuine and cloned recordings of the same speaker receive different labels, and how voice-cloning misuse is evaluated in the ElevenLabs subset. Otherwise, the framework appears closer to speaker-aware content moderation than explicit impersonation detection.
- The choice of the SoundGuard backbone lacks sufficient justification. ECAPA-TDNN is primarily designed for speaker representation learning, whereas SoundGuard must also recognize diverse non-speech events. Comparisons with general-purpose audio encoders such as EAT under matched training conditions would help justify this design. Moreover, reporting the total parameters and computational cost of the complete AUDIO GUARD pipeline, including ASR and TextGuard, would better substantiate its claimed efficiency relative to large audio-language models.
- The current ablation study does not directly quantify the benefit of sound–content integration. Tables 5 and 6 report content-only and sound-only accuracy, respectively, while Table 2 evaluates joint sound–content accuracy. Since these metrics correspond to different prediction targets, their values cannot directly establish the benefit of fusion. A more informative ablation would evaluate SoundGuard-only, ContentGuard-only, and full AUDIO GUARD on identical test samples using the same binary Safe/Unsafe labels and metrics, including Accuracy, F1, unsafe recall, and FPR. This is particularly important for voice–content compositional scenarios.
Questions for authors
How are the integration thresholds calibrated, and are they fixed across benchmarks within each threat model?
What constitutes a correct prediction in Tables 2 and 5, and how are external-benchmark annotations mapped to these evaluation targets?
Does the ElevenLabs evaluation measure speaker-identity screening, voice-clone detection, or both? How are genuine and cloned recordings of the same speaker labeled?
Could you compare SoundGuard-only, ContentGuard-only, and full AudioGuard on identical samples using the same binary Safe/Unsafe labels, reporting unsafe recall and FPR, particularly for compositional risks and hard-benign cases?
4 comments
Q4. Does sound–content integration improve final safety decisions when evaluated under identical labels and metrics?
We appreciate this suggestion and agree that the benefit of sound–content integration should be distinguished from the individual recognition accuracy of the two components.
The key motivation for AudioGuard is that audio safety cannot always be decomposed into independent sound and content moderation decisions. Many real-world policies are inherently compositional: the same semantic content or acoustic cue can lead to different safety decisions depending on their combination and deployment context.
For example, detecting sexual content without recognizing that the speaker has a child-like voice may miss the relevant compositional violation. Conversely, blocking all child-like voices or all sensitive acoustic events would produce unnecessary false positives. Similarly, public-figure voice impersonation combined with misinformation can create a policy-relevant risk that independent moderation systems may not adequately represent.
Our integration module explicitly models these relationships through policy-grounded rules over SoundGuard and ContentGuard predictions. This provides two key advantages:
- Effectiveness: The system can identify risks requiring joint acoustic and semantic evidence, which transcript-only moderation cannot capture.
- Flexibility: The same detectors can support different audio safety policies and deployment scenarios through configurable integration rules without retraining the underlying models.
Our existing component-wise evaluations establish that SoundGuard and ContentGuard recognize complementary signals, but they use different prediction targets and therefore do not by themselves quantify the gain from fusion under a common binary label. The current evidence supports the design motivation and overall end-to-end effectiveness.
Dear Reviewer,
Thank you again for your constructive feedback. We have addressed your questions in our rebuttal and provided additional clarifications.
We would greatly appreciate it if you could take another look at our responses when you have a chance. Please let us know if any concerns remain or if further clarification would be helpful.
Thank you for your time and consideration!
Official Comment by Reviewer jQ5Q
Thank you for the clarifications. However, two central concerns remain unresolved. First, the current evaluation does not establish whether SoundGuard can distinguish genuine voices from cloned voices of the same speaker, which is essential to the impersonation-detection claim. Second, as the authors also acknowledge, the existing component-wise results do not demonstrate the actual benefit of sound–content integration. The requested same-sample, same-label comparison of SoundGuard-only, ContentGuard-only, and the full AudioGuard is still missing. These are important empirical gaps in the central claims of the paper, so I maintain my original rating.
We sincerely thank the reviewer for the detailed and technically insightful feedback. We appreciate the recognition of the practical and interpretable design of AudioGuard. Below, we clarify the calibration procedure, evaluation metrics, architectural choices, and benefits of compositional integration.
Q1. How are integration thresholds selected, and are they fixed across benchmarks?
Thank you for raising this important methodological question.
AudioGuard uses a policy-grounded integration mechanism in which each rule is expressed as a conjunction of threshold conditions over sound/voice cues and semantic content risks. The thresholds are configured according to the relevant threat model and remain fixed across benchmarks evaluated under the same threat-model configuration. The separate 10% calibration partition is used for threshold selection; evaluation examples are not used to adjust thresholds on a per-benchmark basis.
Q2. What exactly do the reported accuracy metrics measure, and how are benchmark labels mapped into the evaluation space?
We appreciate this request and agree that the distinctions between the different evaluation metrics should be more explicit.
Our evaluation considers three related but distinct prediction targets:
These metrics serve different purposes. In particular, the component-wise metrics quantify the recognition capabilities of individual modules, whereas the end-to-end metric evaluates their combined behavior. We agree that the paper should not rely on comparisons between different prediction targets as direct evidence of fusion improvements.
For external benchmarks, the available unsafe categories are collapsed into a single unsafe label, and results are reported at the binary safe/unsafe level rather than as category-wise classification accuracy. Consequently, the external scores should be interpreted as binary safety detection results, not fine-grained taxonomy-matching accuracy.
Q3. Why is ECAPA-TDNN appropriate for general sound-event detection, and how efficient is the complete pipeline?
Thank you for the suggestion. Our selection of ECAPA-TDNN is primarily motivated by its efficient architecture and strong pretrained acoustic representations.
SoundGuard is designed as a lightweight detector of safety-relevant acoustic cues rather than a general-purpose audio understanding model. We initialize it with the pretrained SpeechBrain ECAPA-TDNN encoder and attach a multi-label classification head trained on our audio-native risk categories. This design enables reuse of pretrained acoustic features while keeping the task-specific training component relatively small.
Although ECAPA-TDNN was originally developed for speaker representation learning, the relevant question for our application is whether its learned representations support accurate recognition of the audio-native cues needed by the safety taxonomy. Our experiments demonstrate sound-only accuracy of 0.887 on Speech, 0.876 on Non-Speech, and 0.918 on ElevenLabs, supporting its effectiveness within the evaluated risk categories.
Regarding efficiency, Table 2 reports the observed end-to-end deployment latency: 1.423 seconds for AudioGuard, compared with 3.245 seconds for Gemini 3 and 2.542 seconds for GPT-Audio. These measurements include the AudioGuard pipeline rather than SoundGuard alone.
We emphasize that these measurements are not compute-matched: AudioGuard is evaluated locally, whereas the proprietary models are accessed through APIs with additional network and service overhead. We therefore interpret these results as observed deployment latency under the available interfaces, not as proof of universally lower intrinsic computational cost.
The modular architecture also enables SoundGuard and ContentGuard to operate in parallel and allows deployments to enable only the components required for the relevant threat model.