- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces AU-BENCH, a benchmark comprising 18 audio CAPTCHA tasks spanning transcription, spatial inference, emotion recognition, counting, and audio matching, alongside two metrics: Audio-Temporal Reasoning Depth (ATRD) and Acoustic Perceptual Load (APL) to characterize task complexity and acoustic degradation. Experiments comparing native multimodal models and ASR-based systems with human participants reveal substantial weaknesses under severe noise and on certain nonverbal audio tasks, while showing that chain-of-thought prompting yields mixed results.
Critical strengths
- This paper’s investigation of audio CAPTCHAs is valuable, offering a perspective that has been overlooked in existing audio LLM evaluation benchmarks.
- The dual-gradient complexity framework proposed in this paper is novel and offers useful insights for the design of future benchmarks.
Critical weaknesses
- A major weakness of this benchmark paper is its insufficient comparison with and discussion of existing audio LLM evaluation benchmarks, such as MMAU, MMAU-Pro, MMAR, MMSU, VoiceBench, and OpenAudioBench. Although these benchmarks may not specifically target audio CAPTCHAs, the capabilities they assess overlap substantially with those examined in this work. The paper therefore needs to clarify what is novel about the proposed benchmark in terms of the capabilities it evaluates.
- Another major limitation is the narrow range of models evaluated in this benchmark, which omits several advanced model families, such as SALMONN, Qwen-Omni, Audio-Flamingo, Step-Audio, Kimi-Audio and Baichuan-Audio.
Questions for authors
- Which model and evaluation protocol produced Figure 6? Does the reported Pass@1 include autonomous browser interaction, or only answers to directly supplied audio?