acceptodds
Under review as a conference paper at ICLR 2027

LongAudioSpan: Spanning the Duration and Depth of Audio Comprehension

Abstract

General audio comprehension now covers speech, sound, and music over durations from seconds to hours, driven by large audio-language models (LALMs) that are increasingly omni-modal. Yet the benchmarks that test them still rely on clips of seconds, where scores saturate and models converge; recent long-form efforts extend duration but evaluate long audio much as short clips are. We introduce LongAudioSpan, a benchmark that spans both duration and depth: it pairs audio from 10 minutes to over 2 hours with 3,240 questions across three cognitive levels, namely perception, understanding, and reasoning. Two paths supply the questions, differing in how question content is sourced and how ground truth is obtained. Native QA extracts questions from the audio's content, posing each as a multiple-choice item and an open-ended one graded by detailed rubrics. Anchor QA instead injects ground truth, planting acoustic anchors into the audio and building a perception-to-reasoning chain scored only to the first error. A fully automated pipeline constructs every item through structured captioning, QA generation, and adversarial critic feedback. Evaluating 12 LALMs on LongAudioSpan, we find the hard part comes before reasoning: distilling a few relevant facts from a long, redundant signal. This difficulty grows with audio length and falls hardest on perception, especially temporal grounding. LongAudioSpan is available at https://huggingface.co/datasets/audiospan/LongAudioSpan.

Primary area:
datasets and benchmarks
Keywords:
audio-language models, benchmark, long-form audio, audio understanding

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_xxFoOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LongAudioSpan evaluates long-form speech understanding in English and Chinese across perception, understanding, and reasoning. It combines caption-derived Native QA with edit-based Anchor QA, introduces multiple evaluation formats, and evaluates 12 audio-language models.

Critical strengths
  • Relevant focus on localization and reasoning over long audio across languages and durations.

  • Complementary QA paths, paired answer formats, and prefix-based chain scoring.

  • Multi-stage quality control combining automated verification, adversarial filtering, refinement, and human review.

Critical weaknesses
  1. The construction of structured captions and the rationale for using them to generate questions require further explanation.
  • The paper notes that prior caption-then-QA pipelines depend on caption quality, yet adopts a similar approach. Please clarify how transcripts, semantic segments, acoustic descriptions, and cross-segment dependencies are converted into the final structured captions, which model or program performs each step, and whether final assembly is rule-based merging or LLM processing. If LLMs are used, provide their names, specific versions or API snapshots, and complete prompt templates, including input fields and output formats. Also explain why questions are generated from the assembled captions rather than directly from the preceding outputs.
  1. Anchor placement and surrounding-context requirements lack quantitative characterization.
  • Please explain how selected anchor locations are ensured to contain sufficient context for understanding and reasoning questions. Report the distribution of anchor positions and how much surrounding audio is needed to answer these questions, including the distribution of context lengths and representative examples.
  1. The actual evidence scope required by the questions is insufficiently analyzed.
  • Recording duration is not equivalent to the context needed to answer a question. For Native QA and Anchor QA, separately report the cumulative evidence duration, the span from the earliest to the latest evidence, and the proportion of questions requiring multiple segments at each cognitive level. These analyses should distinguish locating local evidence within long audio from integrating distant information, clarifying which long-audio abilities the benchmark measures.
  1. The information advantage of the text baselines needs clarification, and an independently executed chunk-based cascade is missing.
  • The Caption baseline achieves the highest average scores in Table 3, but receives the same captions used to construct the questions. The table or its notes should identify the models generating the transcripts and captions, any manual revisions, and their relationship to the question-authoring materials. Results using the authoring captions should be labeled as a diagnostic reference. More importantly, add a practical cascade: divide audio into approximately 30–60-second chunks, independently generate transcripts and captions, and use a text LLM to aggregate them and answer questions. A same-model pipeline that first generates text and then answers using only that text could also be evaluated. Report Native QA and Anchor QA results separately, and explain qualitatively or quantitatively which additional long-range abilities the benchmark requires beyond short-chunk processing.
  1. The overlap between the judge and the evaluated text model introduces a potential self-preference risk.
  • GPT-5.4 both answers the text-baseline questions and judges open-ended responses, potentially favoring its own outputs in Rubric scoring. Rescore representative responses using independent judges from other model families to check whether GPT-5.4’s relative advantage and the model rankings remain stable. This concern primarily applies to Rubric and does not directly explain Accuracy or Chain, which use exact option matching.
Questions for authors

See Weakness

Reviewer_D7wQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LongAudioSpan, a benchmark spanning both duration and depth, pairing audio from 10 minutes to over two hours with 3,240 questions across perception, understanding, and reasoning. Native QA poses questions in multiple-choice and open-ended formats, while Anchor QA plants acoustic anchors and builds a perception-to-reasoning chain scored only to the first error. Evaluating 12 LALMs, the authors find that the hard part comes before reasoning: distilling a few relevant facts from a long, redundant signal.

Critical strengths
  1. Broad coverage. The benchmark covers bilingual, real-world audio ranging from 10 minutes to over two hours across diverse genres and topics.

  2. Matched evaluation formats. Paired multiple-choice and open-ended questions enable direct comparison between answer selection and free-form response generation.

  3. Complementary question paths. Native QA assesses natural content, while controlled acoustic anchors provide verifiable reference points for targeted evaluation.

  4. Thoughtful construction pipeline. Audio-based evidence verification, adversarial filtering, critic feedback, and human review provide multiple layers of quality control.

Critical weaknesses
  1. Unclear how much audio each question requires. The paper uses long recordings, but it is unclear how much of each recording is actually needed to answer a question. Comparing results on relevant short clips and full recordings would help.
  2. The claim that models are guessing needs more evidence. The model may understand the relevant passage even if it reports the wrong timestamp. Moving the anchor to another passage and checking whether the answers change would help test this claim.
Questions for authors

see weakness

Reviewer_RQaAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors introduce LongAudioSpan, a benchmark of 360 in-the-wild long-form audio recordings spanning 10 minutes to over 2 hours paired with 3,240 questions organized including perception, understanding, reasoning. The benchmark is built via a fully automated two-phase pipeline that produces Native QA and Anchor QA. 12 recent LALMs plus GPT-5.4 text-only baselines are evaluated.

Critical strengths
  1. The Anchor QA mechanism supplies reliable ground truth in wild audio and suppresses shortcut answering, addressing a real gap in caption-then-QA benchmarks.
  2. The quality-assurance stack is more rigorous than most peers, using a tiered solver gate to keep only audio-dependent items plus a critic loop and human review.
Critical weaknesses
  1. Ground truth and judging are entirely LLM-generated with no reported inter-annotator agreement, no human-vs-judge correlation, and no measured error rate for the verification gate, so headline reliability is unquantified.
  2. The central claim that perception is the bottleneck is confounded, because questions are authored from the captions and open models are truncated well below full duration, so much of the gap reflects input-length limits.
Questions for authors
  1. How well does the LLM-judge rubric correlate with human scores, and what is the precision and recall of the solver gate on a human-labeled sample?
  2. How much of the closed-vs-open and audio-vs-text gap survives if every model is given only the audio window it can actually process? 3. What evidence shows the injected anchors are perceptually natural rather than detectable via synthesis artifacts?
Reviewer_XUQNOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper provides a benchmark on long audio reasoning which involves a three-level cognitive framework, adding depths to the evaluation of audio reasoning abilities. NativeQA comes in MCQ and OEQ, where MCQ is evaluated by accuracy, OEQ is evaluated by weighting the scores of semantic/temporal accuracies of essential and supporting evidences. AnchorQA injects audio events and ask about them, chaining the three-level questions in the score so that early mistakes invalidate the later correct answers. This paper also provides an automated construction pipeline which is potentially scalable on creating similar benchmarks. The benchmark is used for an evaluation on existing audio and omni-modal models to show the perception is the bottleneck.

Critical strengths

The benchmark adds weights on perception evaluation which involves localization and counting. As the existing models are not typically trained to do this well, (although they may be aware of the events, just not be able to express them in timestamps, as section App. D.5 shows), it hints on the next directions models need to be improved.

Critical weaknesses
  1. About anchor operations: The statistics of the anchor events added in the audios is not given. It would be helpful to see if the lengths, the noise levels, etc. affect the perception accuracy. It seems possible that the anchor events were too short or insignificant and are diluted in the process of downsampling of the audio models. 

  2. Unfair comparison between the text-only baseline and the audio reasoning performances: (1) The audio captioning was done on 5 minutes segments, which may have been easier for the captioning models to spot and describe events. (2) Section 5.3 mentions that the performance degradation due to length happens in text-only baseline as well, and audio inputs are generally much longer than text inputs. So stats may be needed for the average context lengths for the text-only baselines and the audio input, since the performance gap may come from the input lengths.

  3. Unfair comparison between the open/closed models: The closed-source models were used in the benchmark creation and filtering passes, (gemini-3.1-pro for acoustic captioning, qwen3-omni-flash for evidence verification) which may explain the reason why closed-source models perform significantly well on NatureQA while the gap is not as big in the AnchorQA. Because the benchmark is questioning on the problems that the closed models can sense and verify. 

Nitpick: 

The term "link" appears for the first time in section 5, which needs definition. 

In section 5.2, long-form audio..., grey cells in Table 3 are not for the open models with length limits.

Figure 4b is poorly explained. What is the score of the audio models? Is it the average of all the models tested?

Questions for authors
  1. In the caption pipeline, the ASR transcript is not a multi-talker result, and yet the semantic segmentation uses speaker changes as a cue. How are the speaker information extracted?

  2. Despite the claim of the paper being scalable, it still involves a human review step. Do you have stats on how significant the human review was on the creation of the benchmark?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.