acceptodds
Under review as a conference paper at ICLR 2027

LAMAR-Bench: Diagnosing Long-Form and Multi-Audio Understanding and Reasoning

Abstract

Audio-language models are increasingly able to process long recordings and multiple related audios. Existing benchmarks capture important parts of these settings, but the evaluation landscape remains fragmented, with fine-grained paralinguistic and speech-artifact analysis, long-range semantic dependencies, detailed audio-version comparison, and goal-conditioned decisions still underexplored. We introduce LAMAR-Bench, a capability-based benchmark that organizes long-form and multi-audio evaluation into three complementary families: fine-grained understanding, cross-audio relation reasoning, and decisions under goals and constraints. It contains 1,597 questions across 10 tasks, constructed from shared audio facts with traceable supervision, with up to 19 audio clips and 59.9 minutes of input per question. Evaluating 14 audio-language and omni-modal systems reveals substantial room for improvement: the best paralinguistic-analysis and acoustic-localization accuracies reach only 51.09% and 54.38%, respectively, while detailed cross-audio difference analysis remains challenging. We further conduct paired analyses over answer format, presentation order, and requested goals, revealing limitations in direct localization, order robustness, and goal adaptation. These findings highlight persistent gaps in evidence-grounded understanding and reliable goal-conditioned reasoning over long and multiple audio inputs.

Primary area:
datasets and benchmarks
Keywords:
Long-form audio, Multi-audio reasoning, Audio-language models, Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_jbFXOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents LAMAR-Bench, a unified benchmark for long-form and multi-audio understanding. It evaluates three related capabilities: fine-grained audio understanding, cross-audio relation reasoning, and decision making under explicit goals and constraints. The benchmark is substantial, containing 1597 questions across 10 tasks, 935 source materials, and 606.7 hours of audio, with individual questions containing up to 19 clips and 59.9 minutes of summed input.

The benchmark covers speaker and paralinguistic analysis, acoustic localization, exhaustive retrieval, long-range semantic links, temporal reconstruction, audio-version comparison, splice-impact judgment, candidate selection, and constraint verification. This is a meaningful and relatively broad evaluation agenda. However, the breadth also creates validity concerns: the data combines timestamps or transformation-based labels with subjective judgments about semantic disruption and acoustic naturalness, while many tasks reuse shared automatically generated annotations. The paper does not provide sufficiently independent evidence on annotation accuracy, subjective-label agreement, or error propagation to validate the benchmark.

Results from 14 baselines (together with text-only LLMs) show substantial difficulty in direct localization, complete evidence recovery, detailed audio-difference analysis, and reliable adaptation to order or goal changes. Nevertheless, the paper does not provide a cross-benchmark comparison that the 14 baselines exhibit behavior uniquely revealed by LAMAR-Bench. It mainly reports their scores on the proposed benchmark. Thus, the claim that LAMAR-Bench exposes distinctions not captured by existing benchmarks remains insufficiently substantiated. Overall, these findings are informative, but the conclusions are also based on specific data, and the main results warrant further analysis of uncertainty and sensitivity.

Critical strengths
  1. LAMAR-Bench targets a meaningful gap in current audio-language evaluation by combining long-form recordings, multiple related audio inputs, fine-grained evidence recovery, cross-audio comparison, and goal-conditioned decisions. Its three-family organization provides a coherent progression from perception to relational reasoning and decision making, while the 10 tasks cover capabilities that are rarely evaluated together.

  2. The benchmark goes beyond simple recognition by testing sub-second acoustic localization, exhaustive retrieval, long-range semantic links, audio-version differences that may preserve the words, temporal reconstruction, and multi-constraint verification. The paired analyses of answer format, presentation order, and changing goals are particularly valuable because they reveal gaps between multiple-choice recognition, direct evidence recovery, repeatability, and reliable goal adaptation.

  3. The paper describes a shared-fact construction and quality-control pipeline and evaluates 14 hosted and open-source systems, alongside question-only and transcript-only references. The breadth of the model evaluation make the reported results useful to scholars, even though the current version still needs stronger validation, statistical controls, and cross-benchmark comparisons.

Critical weaknesses
  1. LAMAR-Bench offers no advantage in size compared with MMAU-Pro, currently the most widely used benchmark. With only 1597 QA pairs, it may not be able to comprehensively evaluate LALMs.

  2. LAMAR-Bench combines timestamps or transformation-based labels with subjective judgments about semantic disruption and acoustic naturalness, while many tasks reuse shared automatically generated annotations. The paper does not provide sufficiently independent evidence on annotation accuracy, subjective-label agreement, or error propagation to validate the benchmark.

  3. This paper does not provide a cross-benchmark comparison that the 14 baselines exhibit behavior uniquely revealed by LAMAR-Bench. It mainly reports their scores on the proposed benchmark. Thus, the claim that LAMAR-Bench exposes distinctions not captured by existing benchmarks remains insufficiently substantiated.

Questions for authors

Can the authors clarify whether the final benchmark was independently rechecked after annotation and adjudication, and report the revision/removal rate, task-specific agreement, and final spot-check size? These details are needed to determine whether the reported quality statistics reflect the reliability of the benchmark.

Reviewer_ZopgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LAMAR-Bench is a benchmark for evaluating audio-language and omni-modal models on long-form and multi-audio inputs. It organizes tasks into fine-grained understanding, cross-audio relation reasoning, and goal/constraint-based decisions, built from shared audio facts with traceable supervision. The authors evaluate many hosted and open-source systems plus textual baselines using task-specific metrics. They also run paired diagnostics on answer format, presentation order, and requested goals. Results show persistent weaknesses in fine-grained perception, precise localization and retrieval, audio difference analysis, and adaptation to order or goal changes, with validation through task-specific checks and human review.

Critical strengths

A critical strength is the benchmark’s capability-based design: it jointly evaluates fine-grained long-form audio understanding, cross-audio relation reasoning, and goal/constraint-conditioned decisions using shared, traceable audio facts and task-specific metrics. This is consequential because prior audio benchmarks largely isolate these abilities or cover them unevenly, so LAMAR-Bench enables a more diagnostic assessment of whether models can recover precise evidence, relate multiple recordings, and act under explicit goals. A second critical strength is the paired diagnostic methodology, which goes beyond aggregate accuracy to probe answer format, presentation order, and requested-goal changes, including repeated decision trials. This matters because it exposes failures that a standard leaderboard would hide—such as recognizing an answer in multiple-choice form but failing to localize it, changing selections under order reversal, or persisting with the same candidate when the goal changes.

Critical weaknesses

A key weakness is limited external validity and contamination control: because the benchmark uses public-source audio and evaluates opaque hosted models without systematic decontamination or private held-out checks, strong scores may not fully reflect genuine long-form and multi-audio reasoning. A second concern is comparability: the benchmark reports no overall score and equally averages heterogeneous metrics with different chance levels, weakening model ranking and leaderboard utility. Third, data construction relies partly on generative AI for transcription, annotation, question/reference generation, and speech synthesis; despite human review, initial answer-correctness and uniqueness pass rates indicate residual label risk, and the final spot-check sampling is insufficiently detailed. Finally, the speech-centered coverage and dependence on opaque hosted model versions and planned releases limit generality and independent reproducibility.

Questions for authors
  1. Across the multi-audio tasks, what are the proportions of same-source excerpts, edited versions of the same recording, and heterogeneous or synthesized candidates, and how do these proportions vary by task type?

  2. Which specific tasks (A1–A5, B1–B2, C1–C3) rely primarily on same-source/edited audio versus heterogeneous/independent audio, and does the paper provide a task-level breakdown rather than only an aggregate multi-audio subset description?

  3. Does the paper report a per-question distribution of the distance between evidence anchors and answer targets, or any minimum-span criterion, to show that questions require long-form cross-span integration rather than local reading?

  4. Are there truncated-input, local-window, or nearby-only baselines that test whether models can answer from a few adjacent key cues without integrating distant evidence?

  5. Given the reported multiple-choice versus direct-localization gap, how does the benchmark ensure that correct multiple-choice answers reflect genuine long-span evidence recovery rather than answer recognition or surface cues?

  6. Does the evaluation analyze performance by evidence distance or required search span, so that long-form reasoning is measured separately from short-range pattern matching?

Reviewer_MZQvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LAMAR-Bench presents 1,597 questions across 10 tasks that cover fine-grained audio understanding, cross-audio reasoning, and goal-conditioned decision-making. The questions draw on 935 source materials totaling 606.7 hours of audio, with up to 19 clips and 59.9 minutes of input per question. The authors evaluate 14 audio and omni-modal systems, along with text-based baselines. Their results show weaknesses in precise localization and audio-difference analysis. Additional experiments find gaps between multiple-choice recognition and direct evidence recovery, as well as sensitivity to presentation order and requested goals.

Critical strengths
  • The paper covers a broad, capability-based range. The 10-task suite spans a wide range of tasks with a fine-grained task breakdown.
  • Multi-step checks for evaluation.
  • Examples are clearly written to understand each task.
  • Vast range of model evaluation.
Critical weaknesses
  • The annotation details and the gold answers verification are not clearly stated. It would be helpful to state how the timestamp annotations are done, as this is very time and resource-consuming. How were the timestamps validated?
  • The information on tools or models used to generate transcripts, word alignments, speaker assignments, acoustic-event annotations, and semantic links is not clear.
  • The question distribution among the tasks and the human-performance baseline are missing.
Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.