acceptodds
Under review as a conference paper at ICLR 2027

The Acoustic Stalemate: Benchmarking Large Language Models via Audio-based CAPTCHAs

Abstract

Audio CAPTCHAs represent one of the most challenging real-world tests for multimodal AI systems, requiring robust auditory perception and logical inference under diverse acoustic degradation. However, existing benchmarks largely focus on uni-modal language or vision tasks, leaving a critical gap in systematic evaluation of audio-language models. Thus, we introduce Au-Bench, the first interactive, open-source benchmark specifically designed to systematically evaluate the auditory perception and reasoning capabilities of large language models through 18 diverse audio CAPTCHA tasks spanning Content-based and Rule-based paradigms. We propose a dual-gradient complexity framework comprising Audio-Temporal Reasoning Depth (ATRD) to quantify the cognitive, multi-step reasoning burden, and Acoustic Perceptual Load (APL) to formally measure physical signal degradation through SNR, reverberation index, and spectral masking density. Our fully interactive web-based testing platform systematically evaluates state-of-the-art native large audio-language models and cascaded ASR-LLM systems under both zero-shot and Chain-of-Thought settings. Comprehensive experiments reveal that current foundation models perform substantially below human baselines, demonstrating acute vulnerability under both physical acoustic degradation and multi-step temporal reasoning while displaying highly distinct failure topologies. Au-Bench aims to shift the research frontier from conversationally fluent assistants toward robust, perceptually grounded agents capable of navigating real-world web interactions.

Primary area:
datasets and benchmarks
Keywords:
Audio CAPTCHAs, Large Language Models, Multimodal Agents, Benchmark, Audio Perception, Temporal Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_C49DOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces AU-BENCH, a benchmark comprising 18 audio CAPTCHA tasks spanning transcription, spatial inference, emotion recognition, counting, and audio matching, alongside two metrics: Audio-Temporal Reasoning Depth (ATRD) and Acoustic Perceptual Load (APL) to characterize task complexity and acoustic degradation. Experiments comparing native multimodal models and ASR-based systems with human participants reveal substantial weaknesses under severe noise and on certain nonverbal audio tasks, while showing that chain-of-thought prompting yields mixed results.

Critical strengths
  • This paper’s investigation of audio CAPTCHAs is valuable, offering a perspective that has been overlooked in existing audio LLM evaluation benchmarks.
  • The dual-gradient complexity framework proposed in this paper is novel and offers useful insights for the design of future benchmarks.
Critical weaknesses
  • A major weakness of this benchmark paper is its insufficient comparison with and discussion of existing audio LLM evaluation benchmarks, such as MMAU, MMAU-Pro, MMAR, MMSU, VoiceBench, and OpenAudioBench. Although these benchmarks may not specifically target audio CAPTCHAs, the capabilities they assess overlap substantially with those examined in this work. The paper therefore needs to clarify what is novel about the proposed benchmark in terms of the capabilities it evaluates.
  • Another major limitation is the narrow range of models evaluated in this benchmark, which omits several advanced model families, such as SALMONN, Qwen-Omni, Audio-Flamingo, Step-Audio, Kimi-Audio and Baichuan-Audio.
Questions for authors
  • Which model and evaluation protocol produced Figure 6? Does the reported Pass@1 include autonomous browser interaction, or only answers to directly supplied audio?
Reviewer_YejxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces AU-Bench a benchmark with 18 AudioCAPTCHA tasks. The tasks span two dimensions: (1) content-based: characters, numeric and words with diverse noise conditions, (2) rule-based: emotion, music, math, matching etc.

The dataset itself is compiled using ESD emotional speech dataset, Google Cloud TTS and DSP simulation.

The authors propose assessments on 2 axis based on difficulty of the CAPtCHA tasks: (1) Audio-Temporal Reasoning Depth (ATRD): labeled by humans on the atomic units needed to solve a task, (2) Acoustic Perceptual Load (APL): objective measure as a weighted mix of SNR, RT60 and spectral masking density.

Several models are evaluated on the benchmarks including Gemini, Mimo, in 2 configurations zero-shot and chain-of-thought reasoning, against human raters and cascaded ASR + LLM.

The results indicate that the audio-LLMs outperform humans on clean transcriptions however fall short against humans in noisy, degraded environments and emotion / spatial tasks. CoT effects are mixed.

Critical strengths
  • Audio CAPTCHA is an under-explored area and are of sufficient interest.
  • The tasks definitions and choices are broad and well designed / covered across lexical, affect, cognitive load, acoustic conditions.
  • Disentangling the tasks on logical reasoning and perceptual quality is intuitive and good framing.
  • Several popular models included in the benchmark evaluations including cascaded system.
  • Human evaluation brings crucial validity to the benchmark.
Critical weaknesses
  • No per task APL is reported. No correlation between APL/ATRD and accuracy
  • Evaluation is missing open sourced audio LLMs like Qwen3 Omni, Kimi-Audio) - which is crucial for reproduction in the scientific community and helps adoption.
  • It is weird to use ASR+LLM setup for emotion or spatial tasks by construction - it shouldn't be framed as a finding.
  • Some of the claims are too strong and the data doesn't check out - Example: CoT consistently improves Math and Category count - results don't back it.
  • The security conclusions are not supported with any real-world CAPTCHA tests (reCAPTCHA or hCaptcha audio).
Questions for authors
  • There are certain references to Section 3.4 - however the section doesn't exist - please fix them.
  • Why does the character task give different numbers in Table 1 vs. Table 2 - SNR 20dB - Reverb 0? and table 3?
  • What is the chance accuracy for each task?
  • What noise types were used for SNR ablations?
Reviewer_P2sFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces AU-BENCH, a benchmark of 18 audio CAPTCHA task types for evaluating auditory perception, rule following, and reasoning. It describes task difficulty using Audio-Temporal Reasoning Depth (ATRD) and Acoustic Perceptual Load (APL), compares native audio multimodal models, ASR/ASR–LLM settings, and humans in a web-based task interface, and studies performance under noise, reverberation, and masking. Results show substantial variation across tasks and acoustic conditions, with task- and model-dependent effects of CoT prompting.

Critical strengths
  1. The task suite covers several practically relevant audio capabilities. The 18 task types include digit, character, and word transcription as well as emotion, spatial judgments, matching, and counting. This coverage can expose differences between transcription ability and other forms of audio understanding. The human baseline provides a useful reference, making the evaluation more informative than a transcription-only test.
  2. The controlled acoustic experiments provide concrete observations about robustness. On the Character task, the authors vary SNR, reverberation, and distractor type and compare models with humans. For example, in Table 3, reducing SNR from 20 to 0 dB lowers Gemini-3.1-Pro accuracy from 83% to 65% without reverberation, but from 82% to 30% at RT60=1.5 s. These results illustrate performance changes under combined acoustic degradation and offer a useful test design for further robustness research.
Critical weaknesses
  1. The validity of the two-dimensional framework as a diagnostic tool is not sufficiently established. Section 4.2.4 interprets failures under low APL and high ATRD as reasoning bottlenecks. However, clear audio does not guarantee that a model has correctly identified the words, emotion, or sound source. The evaluation lacks a direct control that supplies correct perceptual information while retaining the subsequent rule-processing requirement, leaving the locus of failure uncertain. Figure 4 also presents a proposed task decomposition rather than a step-by-step validation against the CoT explanations actually returned by the models. Comparing final accuracy with and without CoT does not establish which intermediate judgments were correct, where errors occurred, or whether the estimated ATRD corresponds to the steps reflected in the models’ observed responses. In addition, the APL weights are calibrated using human performance on Character, and the appendix reports a joint-degradation check on the same task, but the validity of these fixed weights on other tasks remains untested. This limits their use as a cross-task difficulty measure. The evidence currently supports a useful way to organize tasks more strongly than the central diagnostic claim of distinguishing perceptual from reasoning failures and quantifying difficulty across tasks (Sections 3.3 and 4.2.4; Limitations).
  2. The benchmark’s additional value for evaluating audio reasoning in strong models is not sufficiently demonstrated. Counting the entries in Table 1, Gemini-3.1-Pro with CoT already reaches at least 90% accuracy on 7 of the 12 rule-based tasks. High scores do not by themselves invalidate a benchmark, but the paper should more directly establish which unresolved capabilities the remaining challenges measure, whether they reliably distinguish strong models on tasks humans can complete, and whether the remaining gaps reflect auditory perception or rule-based reasoning. Meanwhile, human accuracy on Lepture, Word, and Word* is only 18%, 56%, and 49%, making some low scores more naturally interpretable as extreme stress tests rather than direct evidence of practical CAPTCHA usability or higher-order reasoning difficulty. The native-audio evaluation also covers only Mimo and two Gemini versions, without validation across additional model families. The results do not yet sufficiently establish a set of tasks with a clear capability interpretation that can continue to support comparisons among strong models, leaving the diagnostic value and application scope in need of more targeted evidence.
Questions for authors
  1. Does the returned CoT follow the proposed task decomposition, and do errors arise in perception or subsequent processing? Please provide the CoT prompts and representative successful and failed examples, comparing the visible reasoning output, where available, with the steps in Figure 4. Check the correctness of key perceptual judgments and subsequent categorization or counting operations, and identify where errors arise. In addition, an existing rule-based task could be evaluated with correct perceptual intermediate information: for example, supply an accurate transcript for Category Count while retaining its categorization and counting requirements, and compare with raw-audio input. These controls would more directly test the proposed decomposition and error attribution; agreement in visible CoT alone would not establish the model’s internal mechanism.
  2. Does APL remain effective on tasks that were not used for calibration? Please keep the current weights fixed and evaluate the relationship between APL and both human and model accuracy on at least one other task type, comparing it with simpler measures based only on SNR or RT60. This would help assess whether the combined metric adds explanatory or predictive value across tasks.
  3. Which tasks remain meaningfully challenging for multiple strong models while being reliably solvable by humans? Please specify the size of this subset, the capabilities and degradation conditions it covers, and performance differences between strong models with uncertainty. Use error analysis to distinguish gaps in transcription, acoustic-attribute recognition, and rule processing. For the low human success rates on Lepture and the word tasks, clarify the roles of acoustic quality, language background, rules, and playback conditions, and discuss these tasks separately. This would more directly establish the benchmark’s value for comparing strong models and its practical applications.
  4. Could the evaluation include audio-capable representatives from additional model families, such as Qwen and Seed? The native-audio group currently contains Mimo and two Gemini versions. Consider representatives from Qwen’s Audio/Omni line and from the Seed family that were available at the relevant evaluation date and support native audio understanding. Please use the same tasks, audio, and scoring rules, with equivalent instructions and comparable inference budgets, and report model versions, evaluation dates, and audio-input constraints. This would help assess whether rankings, CoT effects, and task-specific bottlenecks are consistent across model families.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.