- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Review of the Paper "MMALR — Making Audio Matter: Benchmarking Audio Listening, Understanding, and Reasoning with Counter-Prior Construction"
Summary
The paper proposes a novel audio understanding benchmark, MMALR, that different from the majority of previous works actually emphasizes the listening aspect of audio understanding and tries to avoid including questions that can be inferred via text alone. As the paper shows, the majority of previous benchmarks, notably MMAU, still achieve high accuracy scores when using noisy/silence data. This questions the reasonability of these benchmarks. The paper overall is well written, easy to follow with a clear story and a clear goal.
I think that this paper should be accepted, due to the large impact it might have on future benchmarks and the potential to replace a majority of previous text-heavy audio benchmarks.
- It seems somewhat funny that after so many different benchmarks, little impact has been given to models actually using the audio content for responses. This fact alone makes this benchmark very impactful and important for proper benchmarking of audio understanding performance.
- On a side note, due to the results of this paper, one would argue that recent models that benchmarked MMAU/MMAR might only have achieved improvements due to the textual prior, not due to better training or better audio understanding capabilities.
Critical strengths
- Text-solvability is a genuine issue in LALM evaluation.
- The proposed Textual Prior Errors metric is a reasonable metric for diagnosis. The paper shows that noise-correct sets intersect in only 6/1000 samples.
- Reasonable QA construction by using text-only blind test then audio QA test and a third-stage review.
Critical weaknesses
- Not much to say as weaknesses here, with the exception of some extra statistics like inter-annotator agreement and somewhat limited human evaluation ( 3 annotators ).
Questions for authors
Personally the best paper of my batch, easy to follow and understand and focuses on an important problem: LALM's think in terms of text not in terms of audio, leading to a mismatch between benchmark performance and real world use.
Thank the authors for the detailed response, and their response addressed most of my concerns.