acceptodds
Under review as a conference paper at ICLR 2027

Making Audio Matter: MMALR for Benchmarking Audio Listening, Understanding, and Reasoning with Counter-Prior Construction

Abstract

Large audio-language models (LALMs) have made substantial progress across a wide range of audio tasks. To evaluate them consistently, Audio Question Answering (AQA) has become a mainstream paradigm with a unified cross-task interface. However, existing evaluations usually rest on an implicit assumption: a model's response is taken as its analysis of the audio. But in practice, the textual content alone may induce a preference for a particular answer, which arises from the model's textual prior. The effect of the prior becomes evident when audio is replaced with random noise: a considerable portion of questions can still be correctly answered, suggesting that high accuracy on existing benchmarks may not stem from genuine audio processing capabilities. Meanwhile, whether textual priors still affect model predictions with normal audio remains unexamined. To fill this gap, we introduce MMALR, a multi-domain, multi-task benchmark for audio listening, understanding, and reasoning that uses counter-prior construction to make audio evidence play a greater role in AQA. Specifically, MMALR anchors each correct answer in audio evidence and turns plausible textual predictions into distractors, making audio indispensable for reliable answering. We systematically evaluate open- and closed-source LALMs as well as human participants. Results show that under uninformative-audio conditions, MMALR yields the lowest model accuracy among all compared benchmarks. Furthermore, paired predictions under uninformative- and normal-audio conditions on MMALR show that models differ in their textual tendencies, while most models still exhibit consistent incorrect tendencies under the normal-audio condition. MMALR thus provides a more reliable benchmark for evaluating LALMs' audio listening, understanding, and reasoning, and a basis for analyzing their text-audio modality dependency.

Primary area:
datasets and benchmarks
Keywords:
Audio question answering, Large audio-language models, Textual Priors

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_QqFaOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Review of the Paper "MMALR — Making Audio Matter: Benchmarking Audio Listening, Understanding, and Reasoning with Counter-Prior Construction"

Summary

The paper proposes a novel audio understanding benchmark, MMALR, that different from the majority of previous works actually emphasizes the listening aspect of audio understanding and tries to avoid including questions that can be inferred via text alone. As the paper shows, the majority of previous benchmarks, notably MMAU, still achieve high accuracy scores when using noisy/silence data. This questions the reasonability of these benchmarks. The paper overall is well written, easy to follow with a clear story and a clear goal.

I think that this paper should be accepted, due to the large impact it might have on future benchmarks and the potential to replace a majority of previous text-heavy audio benchmarks.

  • It seems somewhat funny that after so many different benchmarks, little impact has been given to models actually using the audio content for responses. This fact alone makes this benchmark very impactful and important for proper benchmarking of audio understanding performance.
  • On a side note, due to the results of this paper, one would argue that recent models that benchmarked MMAU/MMAR might only have achieved improvements due to the textual prior, not due to better training or better audio understanding capabilities.
Critical strengths
  • Text-solvability is a genuine issue in LALM evaluation.
  • The proposed Textual Prior Errors metric is a reasonable metric for diagnosis. The paper shows that noise-correct sets intersect in only 6/1000 samples.
  • Reasonable QA construction by using text-only blind test then audio QA test and a third-stage review.
Critical weaknesses
  • Not much to say as weaknesses here, with the exception of some extra statistics like inter-annotator agreement and somewhat limited human evaluation ( 3 annotators ).
Questions for authors

Personally the best paper of my batch, easy to follow and understand and focuses on an important problem: LALM's think in terms of text not in terms of audio, leading to a mismatch between benchmark performance and real world use.

Reviewer_xugxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MMALR is 1K audio QA benchmark including sound, music, speech, and mixes. The authors evaluated 14 audio language models and human annotators in normal/noise/silent conditions.

Critical strengths
  • The authors demonstrate an interesting problem that textural priors alone can let models answer many audio questions
  • The method, construction process, broad model, and human evaluations are useful contributions of the paper.
Critical weaknesses
  • The benchmark is based on the MMAR task, while the ADQA bench already filters text solvable questions. The appendix mentions MMAU Pro as including open ended and multi choice questions. A matched closed ended comparison, if comparable, could strengthen the context.
  • The text prior dominance result has no baseline provided. Reporting the proposed least plausible answer baseline could be a useful baseline.
  • The paper reports human accuracy to be about 66.6%, but the paper should explain errors and report the annotator agreement statistic or a detailed analysis of human errors.
  • Since the data is collected from online platforms (YT and Bilibili), the authors should inform whether the licenses allow redistribution.
Questions for authors
  • Can you share your insights on how choosing the least text plausible option performs?
  • Can you please include a matched closed-ended MMAU Pro subset evaluation?
  • Can you please include an error analysis of the reported human accuracy?
  • Which baseline should be used for prior dominance analysis?
Reviewer_QWA7OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces MMALR, a 1,000-item multiple-choice audio QA benchmark covering sound, speech, music and their four mixtures, organized into four task layers (signal, perception, semantic, cultural) with 17 subcategories adapted from MMAR. The central idea is "counter-prior construction". Annotators write audio-grounded questions from clips of public videos; textual preferences are then elicited from human annotators and from DeepSeek-V4-Flash given only the question; items the text alone answers are discarded; and the text-only predictions that differ from the answer become distractors, alongside acoustically confusable options. Items receive captions and chain-of-thought annotations, are refined by two LLMs, and pass a three-stage human inspection. Fourteen LALMs are evaluated under normal, Gaussian-noise and silent audio, with three human annotators as reference. Under noise, MMALR yields the lowest accuracy among MMAU, MMAR and MMALR for every model (16–25%); under normal audio Gemini 3.1 Pro reaches 65.0%, the best open model 38.5%, and humans 66.6%. A paired analysis on each model's "textual-prior error" set finds that wrong answers under normal audio usually repeat the prior's wrong answer, which the authors call Textual Prior Dominance.

Critical strengths
  1. New human-annotated data across seven domains, includingy mixtures. Every item carries a caption, a CoT rationale, a domain and a subcategory, and human accuracy is measured by three annotators who did not take part in construction, under both normal and silent conditions.
  2. Sample-level paired analysis across conditions. Fixing questions and options across noise, silence and normal audio allows the ER/AR/OWT decomposition (Eq. 2), a useful vocabulary for describing how models move from a text-only answer once audio is present.
  3. A clear demonstration that existing benchmarks are partly text-solvable. They show that many recent LALMs exceed chance on MMAU and MMAR with noise in place of audio, which is a useful warning for the field.
Critical weaknesses
  1. Low noise-condition accuracy is guaranteed by construction and does not by itself show audio dependence. Distractors are built from what text-only humans and an LLM predict, so by design the correct answer is the option text-only reasoning finds less plausible. The resulting accuracies are below chance: below-chance accuracy indicates that the label is anti-correlated with the textual prior, which is itself an exploitable bias: a strategy of choosing the least textually plausible option would score above chance. A benchmark free of text cues should produce chance accuracy under noise, not below it.
  2. The Textual Prior Dominance analysis is circular. The TPE set consists of items a model answers identically and wrongly under noise and silence, and those wrong answers are largely the counter-prior distractors. A model that cannot extract the evidence from audio will fall back to the same answer under normal audio. An intervention is needed: for example, a parallel version of the same items with neutral distractors (acoustic confusions only) to measure how much counter-prior distractors lower normal-audio accuracy at fixed difficulty, or a probe of whether the model's own caption of the clip contains the decisive evidence while its MCQ answer follows the prior.
  3. The most relevant prior work is missing, and the novelty is narrower than claimed. RUListening (Zang et al., 2025) already constructs music-QA distractors from text-only model plausibility to increase perceptual dependence, and it is not cited. MMAU-Pro (Kumar et al. 2026) is another uncited reference. The taxonomy (four layers, 16 subcategories extended to 17), the data source (clips from internet videos) and the CoT annotation closely follow MMAR, and ADQA-Bench already filters for audio dependence. The paper's claim to be "the first to explicitly establish this controlled divergence" (Appendix I) should be revised and the differences from these works stated precisely.
  4. The human ceiling suggests substantial label ambiguity. Humans reach only 66.6% with audio, essentially level with Gemini 3.1 Pro at 65.0%. Several appendix examples show why. The time-signature item's CoT concedes "the meter is not distinctly perceptible" while asserting 5/4 (Appendix A.2). The aesthetic item "How are the solos of top performers?" has the answer "Occasional mishaps". The bitrate item asks the listener to distinguish 16, 32, 64 and 128 kbps encodings in audio that has been re-encoded. Several speech items (e.g., the father's inhalation) rest on interpretation. The paper reports no inter-annotator agreement and no item-level analysis; items that all three humans miss should be audited.
  5. Statistics and construction details are incomplete. With 1,000 items across 17 subcategories and 7 domains, some cells hold roughly 45–55 items, yet claims such as "the signal layer may be the key bottleneck" are made without intervals. Textual priors are from a single LLM family, and Figure 6 shows that only 6 items are answered correctly under noise by all models.
Questions for authors
  1. What accuracy does an anti-prior strategy (choose the option a text-only LLM rates least likely) achieve on MMALR, MMAU and MMAR?
  2. Can you construct a neutral-distractor version of a subset of items and compare normal-audio accuracy and ER/AR between the two versions?
  3. What is the inter-annotator agreement, and how many items are missed by all three human annotators? What happens to model accuracy when those items are removed?
  4. In what format and bitrate will audio be released, and how are signal-layer labels such as encoding bitrate validated after platform re-encoding?
  5. How many items did the LLM refinement stage change in their options or answer position? Is the answer-letter distribution balanced?
  6. How sensitive are the results to the choice of the prior-eliciting LLM? Would distractors built from a second LLM family change the noise-condition accuracies?
Reviewer_d5sPOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

They propose MMALR, a benchmark of 1,000 multiple-choice questions for evaluating audio-language models across different audio domains and task types. The benchmark is designed to test whether models rely on textual clues in the question and answer options or use evidence from the audio. The evaluation is thorough, covering a range of open- and closed-source models under normal-audio and uninformative-audio conditions. The annotation and quality-checking process also seems carefully designed, with human review at several stages.

Critical strengths

MMALR tackles a meaningful problem which audio-language models can answer some audio questions from the wording alone. Its counter-prior construction and paired normal-audio, noise, and silence evaluations offer a useful way to study this behavior. The benchmark also covers a broad range of audio tasks, and the authors describe multiple rounds of human quality checks.

Critical weaknesses

The main concern is that the experiments do not isolate the effect of counter-prior construction from differences in audio, questions, and task composition across benchmarks. As a result, lower noise-condition accuracy on MMALR supports the claim that its questions are less text-solvable for the evaluated models, but it does not by itself establish that counter-prior distractors caused the difference. This limits how strongly the paper can attribute its results to the proposed construction method.

Questions for authors
  1. How much of the improvement can be attributed specifically to counter-prior construction? Table 7 shows that MMALR has lower noise-condition accuracy than MMAU and MMAR. However, these benchmarks also differ in their audio clips, questions, and task composition. Table 8 further shows that the mean normal-minus-noise accuracy gap is slightly smaller on MMALR than on the comparison benchmarks. Do you have a controlled comparison using the same clips and questions with conventional versus counter-prior distractors? If such a comparison is unavailable, could you clarify which conclusions about the effectiveness of counter-prior construction can be established from the current cross-benchmark results? This affects how confidently the observed differences can be attributed to the proposed construction method.

  2. What evidence distinguishes textual prior dominance from failure to recognize the relevant acoustic cue? Section 4.5 acknowledges that retaining the same wrong answer can arise either because the model fails to extract decisive audio evidence or because it remains influenced by its textual preference despite the conflicting evidence. Both explanations produce the same outcome under Eq. 2, but the conclusion describes models as neglecting relevant audio evidence. Is there any analysis showing that models recognized the decisive acoustic information on the retained-error samples? If these explanations cannot be distinguished with the current evaluation, could you clarify the intended scope of “Textual Prior Dominance”? The answer matters for interpreting the paper’s central behavioral finding.

  3. How should the conditional error-retention rates be interpreted and compared across models? Equation 2 computes ER and AR only on each model’s own TPE set, whose coverage varies from 43.0% to 76.1% in Table 9. Consequently, the models are evaluated on different subsets when their retention rates are compared. In addition, the statement that approximately 29% and 39% “of their errors” remain consistent with textual-prior errors appears to use a different denominator from Eq. 2. Could you clarify the denominator and report the retained-error counts as fractions of both the full benchmark and all normal-audio errors? Does the cross-model interpretation remain similar when evaluated on a shared subset? This would clarify the prevalence of the phenomenon beyond the selected TPE sets.

  4. Does the statistical analysis distinguish reduced text solvability from systematic selection of attractive wrong answers? Section 4.4.1 uses right-tailed tests against chance, while most models score below the 25% guessing level on MMALR. Given that the construction explicitly favors plausible wrong distractors, failure to exceed chance could reflect systematic attraction to those distractors. Could you clarify how this affects the interpretation that textual cues have been suppressed, and provide uncertainty estimates or tests that distinguish chance-level from below-chance performance? Also, how were the nonuniform chance levels for MMAU and MMAR derived? If individual items have different option counts, was the null distribution computed using their individual guessing probabilities? These details affect the statistical support for the comparison in Fig. 5.

1 comment

Thank the authors for the detailed response, and their response addressed most of my concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.