- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents LAMAR-Bench, a unified benchmark for long-form and multi-audio understanding. It evaluates three related capabilities: fine-grained audio understanding, cross-audio relation reasoning, and decision making under explicit goals and constraints. The benchmark is substantial, containing 1597 questions across 10 tasks, 935 source materials, and 606.7 hours of audio, with individual questions containing up to 19 clips and 59.9 minutes of summed input.
The benchmark covers speaker and paralinguistic analysis, acoustic localization, exhaustive retrieval, long-range semantic links, temporal reconstruction, audio-version comparison, splice-impact judgment, candidate selection, and constraint verification. This is a meaningful and relatively broad evaluation agenda. However, the breadth also creates validity concerns: the data combines timestamps or transformation-based labels with subjective judgments about semantic disruption and acoustic naturalness, while many tasks reuse shared automatically generated annotations. The paper does not provide sufficiently independent evidence on annotation accuracy, subjective-label agreement, or error propagation to validate the benchmark.
Results from 14 baselines (together with text-only LLMs) show substantial difficulty in direct localization, complete evidence recovery, detailed audio-difference analysis, and reliable adaptation to order or goal changes. Nevertheless, the paper does not provide a cross-benchmark comparison that the 14 baselines exhibit behavior uniquely revealed by LAMAR-Bench. It mainly reports their scores on the proposed benchmark. Thus, the claim that LAMAR-Bench exposes distinctions not captured by existing benchmarks remains insufficiently substantiated. Overall, these findings are informative, but the conclusions are also based on specific data, and the main results warrant further analysis of uncertainty and sensitivity.
Critical strengths
-
LAMAR-Bench targets a meaningful gap in current audio-language evaluation by combining long-form recordings, multiple related audio inputs, fine-grained evidence recovery, cross-audio comparison, and goal-conditioned decisions. Its three-family organization provides a coherent progression from perception to relational reasoning and decision making, while the 10 tasks cover capabilities that are rarely evaluated together.
-
The benchmark goes beyond simple recognition by testing sub-second acoustic localization, exhaustive retrieval, long-range semantic links, audio-version differences that may preserve the words, temporal reconstruction, and multi-constraint verification. The paired analyses of answer format, presentation order, and changing goals are particularly valuable because they reveal gaps between multiple-choice recognition, direct evidence recovery, repeatability, and reliable goal adaptation.
-
The paper describes a shared-fact construction and quality-control pipeline and evaluates 14 hosted and open-source systems, alongside question-only and transcript-only references. The breadth of the model evaluation make the reported results useful to scholars, even though the current version still needs stronger validation, statistical controls, and cross-benchmark comparisons.
Critical weaknesses
-
LAMAR-Bench offers no advantage in size compared with MMAU-Pro, currently the most widely used benchmark. With only 1597 QA pairs, it may not be able to comprehensively evaluate LALMs.
-
LAMAR-Bench combines timestamps or transformation-based labels with subjective judgments about semantic disruption and acoustic naturalness, while many tasks reuse shared automatically generated annotations. The paper does not provide sufficiently independent evidence on annotation accuracy, subjective-label agreement, or error propagation to validate the benchmark.
-
This paper does not provide a cross-benchmark comparison that the 14 baselines exhibit behavior uniquely revealed by LAMAR-Bench. It mainly reports their scores on the proposed benchmark. Thus, the claim that LAMAR-Bench exposes distinctions not captured by existing benchmarks remains insufficiently substantiated.
Questions for authors
Can the authors clarify whether the final benchmark was independently rechecked after annotation and adjudication, and report the revision/removal rate, task-specific agreement, and final spot-check size? These details are needed to determine whether the reported quality statistics reflect the reliability of the benchmark.