acceptodds
Withdrawn from ICLR 2027

Benchmarking Audio-Language Models on Real-World Long-Form Audio Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large audio language models (LALMs) have made remarkable progress at the second- and minute-level scale, yet understanding hour-level audio remains a fundamental bottleneck. Existing benchmarks rely on short clips or artificially concatenated segments, and so fail to assess long-range comprehension in real-world scenarios such as podcasts or lengthy broadcasts. We introduce VoiceGiraffe, a bilingual, open-domain benchmark for extreme long-context audio understanding. It comprises 1,500 curated triplets under a dual-level taxonomy: single-hop questions for fine-grained grounding and acoustic perception, and multi-hop questions requiring cross-segment evidence aggregation. We evaluate a broad suite of open-source and proprietary LALMs against human performance, and complement the primary multiple-choice protocol with open-ended evaluation that removes candidate options. Three findings emerge. First, VoiceGiraffe is far from saturation, and only one end-to-end LALM surpasses the human reference. Second, inference paradigms are model-dependent. End-to-end inference favours models with native long-context capacity, and cascaded aggregation stabilizes models overwhelmed by hour-scale audio. Third, long-range memory persistence is the key bottleneck, where current models reason over prominent evidence once localized but fail to retain event states over hour-scale context. VoiceGiraffe thus offers a challenging, diagnostic testbed for long-form audio understanding.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Audio Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_igxFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents VOICE Giraffe, a bilingual benchmark for evaluating how well large audio-language models understand long, real-world recordings such as sports broadcasts, e-sports commentary, TV dramas, news, and podcasts. Its main contribution is a two-tier question taxonomy that moves from single-hop perception tasks like temporal localization, semantic understanding, acoustic event detection, and paralinguistic analysis to multi-hop reasoning tasks like causal alignment and event tracking. The benchmark is built through a careful pipeline of segmentation, captioning, question generation, and model-plus-human verification, while avoiding redistribution of copyrighted audio. The authors validate it by testing a broad set of open-source and proprietary audio models alongside human performance under several inference settings. They find that long-form audio understanding remains difficult, that the best inference approach depends on the model, and that weak long-range memory and fine-grained paralinguistic perception are major bottlenecks.

Critical strengths

The strongest strength is the paper’s controlled diagnostic evidence: comparing the same questions on full hour-long audio versus short evidence clips isolates long-context difficulty from question hardness, and the paradigm comparisons reveal model-dependent trade-offs between acoustic fidelity and textual aggregation. This is consequential because it turns the benchmark from a mere harder dataset into a tool that identifies long-range memory and inference-pipeline bottlenecks, which is the most persuasive basis for acceptance despite novelty concerns.

Critical weaknesses

The most apparent weakness is that the paper’s positioning against prior work relies heavily on duration buckets and coarse checklist comparisons, making the benchmark’s distinctiveness and innovation hard to assess; classifying audio understanding mainly by seconds versus minutes is shallow and does not show why hour-scale context demands fundamentally new capabilities rather than longer inputs or better aggregation. This is consequential because the central claimed contribution is benchmark novelty, and without a deeper task- and capability-level comparison, the acceptance case rests largely on scale rather than demonstrated conceptual advance. A second weakness is that the evaluation confounds benchmark difficulty with inference-pipeline artifacts: cascaded captioning and external reasoning models introduce text bottlenecks, information loss, and hallucination risks, so low scores may reflect the evaluation design as much as model ability. The human reference is also sampled and limited, and raw audio is not redistributed, which complicates reproducibility and fair comparison. These issues are consequential because they weaken the paper’s ability to substantiate its conclusions about model limitations and benchmark difficulty, and they would need stronger justification or additional controls to support acceptance.

Questions for authors
  1. Beyond duration buckets, can you provide a task- and capability-level comparison with prior audio benchmarks to clarify what fundamentally new challenge hour-scale audio poses, rather than merely longer inputs or better aggregation?

  2. How do you disentangle benchmark difficulty from inference-pipeline artifacts, e.g., by reporting controlled ablations across E2E, self-cascaded, and LRM-cascaded settings, including an oracle-caption or text-only upper bound to quantify information loss and hallucination?

Reviewer_pNaWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces VOICEGIRAFFE, a bilingual benchmark with 1,500 questions based on 123 continuous recordings (113.1 hours) across five domains. It evaluates 13 audio-language models under end-to-end and caption-based inference, complemented by short-clip and human reference analyses.

Critical strengths

The benchmark uses continuous, hour-scale recordings rather than artificially concatenated clips, providing a realistic test of long-form audio understanding. Its five domains, two languages, and six question types cover both acoustic perception and cross-segment reasoning. The comparison of end-to-end, cascaded, and reasoning-enhanced inference is informative. Short-clip experiments and human reference results help characterize the performance gap associated with full-length audio.

Critical weaknesses

The audio-dependence filtering needs more detail. Section 3.2 and Appendix D state that a text-only model sees the question and clip captions, but do not fully specify the model, decision criteria, or rejection rates. Caption-only accuracy on the final benchmark would help establish how much genuine audio evidence is required.

Some results need verification. MiniCPM-o-4.5 scores only 3.73% with self-cascading, below the 25% chance level for four-choice questions (Table 2). Appendix K also reports the Gemini-3.1-Pro clip gain as 36.42 instead of 26.42 points (66.95 - 40.53). Invalid-output rates and an answer-extraction check would improve confidence.

The memory bottleneck claim is not fully isolated. Although Appendix L compares Causal Alignment and Event Tracking, their question formats and possibly their difficulty also differ. Matched questions or controlled evidence spans would better separate long-term memory demands from other task differences.

The human reference may not be fully comparable to model results. Eight annotators answer 150 questions each, giving 1,200 responses for a 1,500-question benchmark. It is unclear whether human and model scores use the same subset of questions.

Questions for authors
  1. Which text-only model, filtering criteria, and rejection rates are used, and what is the final benchmark accuracy using captions alone?

  2. What explains the 3.73% MiniCPM-o-4.5 self-cascading accuracy, and how many outputs fail answer extraction?

  3. Can the authors verify the Gemini-3.1-Pro clip improvement in Appendix K (reported 36.42 vs. calculated 26.42 points)?

  4. How do Causal Alignment and Event Tracking compare when question format and evidence demands are controlled?

  5. Are the human and model results computed on the same subset, and how many distinct questions are covered by the human reference?

Reviewer_DAR2OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the performance of a set of open-source and proprietary large audio LMs against human performance.

Critical strengths

Extensive evaluation demonstrates authors’ unreserved appreciation of third parties developed audio LLMs, to which they themselves made no obvious technical contribution.

Critical weaknesses

Very limited technical novelty: Although audio LLMs are attracting substantial research interests themselves, exhaustive and repetitive benchmarking their performance across different datasets alone presents no novel methodology design or clearly set out actionable items.

Questions for authors

Please address the above issues for possible future submissions.

Reviewer_2gHZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a new benchmark called VoiceGiraffe for hour-long audio understanding by Large Audio Language Models (LALMs). Their contributions come in the form of data, tasks, a data curation pipeline, and various evaluation settings.

Data:

  • 1500 four-option MCQs over 126 real-world recordings.
  • Bilingual data in English and Chinese across five domains: sports, e-sports, TV drama, news, interview podcasts.
  • Audio data contains a mix of speech, sound effects, and music.

Tasks:

  • Tier1: Single-hop perception tasks: temporal localization, semantic content, acoustic events, paralinguistics.
  • Tier2: Multi-hop reasoning tasks: Causal alignment (ordering a chain of events) and event tracking (counting or which event never happened)

Data curation pipeline:

  • Segment audio into chunks of 30-40s.
  • Qwen3-Omni generates captions for each audio chunk.
  • Qwen3.5-Omni (Plus or Flash not specified) generates global captions for the entire audio clip.
  • Gemini-3.0 generates questions with templated adversarial distractors.
  • 3 filters: a text-only LLM filter, acoustic verification, and human verification by 8 human annotators.

Evaluaiton/Inference settings:

  • End-to-End (E2E) on the full audio: only four proprietary APIs belonging to Qwen and Gemini model families.
  • Self Cascaded (SC): the model captions 30s windows, then the captions are concatenated and fed along with the question and options as text input to the model, it answers in text form.
  • Large Reasoning Model - Cascaded (LC): the model captions 30s windows, then the captions are concatenated and fed along with the question and options as text input to the LRM model, then the LRM model answers in text form. GPT-5.2 and Gemini-3.1-Pro were evaluated as LRM models.

Headline claims:

  • The benchmark is far from saturated.
  • Only one E2E model - Qwen3.5-Omni-Plus beats human reference.
  • The best inference paradigm depends on the model.
  • Long-range memory is the key bottleneck: models do better on causal tasks than on event-tracking questions, while humans show the reverse.
Critical strengths
  • Important, underserved problem: hour-scale, real-world, mixed-content audio is common in practice but poorly benchmarked. Popular audio benchmarks like Big Bench Audio, Full-Duplex Bench, etc. measure short clips or live conversations.
  • Uniformly long, acoustically dense recordings: all recordings are> 30 min (mean ~55 min) from domains rich in non-speech sound like sports, e-sports, dramas in two languages.
  • Task types complementary benchmarks lack: paralinguistics, counting, absence of events.
  • Transparency: All captioning, global caption, and QA-generation prompts are reproduced verbatim (App. P-R). This makes the benchmark auditable, and many critiques are only possible because of it.
  • Several inference paradigms and ablations are done rather than a single number: E2E vs SC vs LC, 2 LRMs are tried, open-ended vs closed-ended QA is done (App. J).
  • Copyright-aware release intent: they release IDs rather than audio.
Critical weaknesses

Dataset builder vs. evaluated models:

This is my biggest concern because both families of models used to build the dataset - Qwen and Gemini series are the only ones evaluated for the E2E inference paradigm. This gives a weak signal and lacks sufficient evidence on how good the eval genuinely is.

  • Qwen3-Omni writes the clip captions (Sec. 3.2, App. P), audits them (App. D), and is evaluated as well.
  • Qwen3.5-Omni writes the global captions from which multi-hop chains are minted (Sec. 3.2 Step 2), and is the only E2E model above human-reference.
  • Gemini writes every question and distractor (Sec. 3.2 says Gemini3.0, App. R says Gemini-3.1-Pro) and is the best cascaded model and the best external reasoner.

This is a huge bias in dataset curation because it most likely is the case that the kind of captions that Qwen3-Omni and Qwen3.5-Omni generate have all the features that are best understood by Qwen3.5-Omni at inference time, so in a way you are upper bounded on the difficulty of tasks created or information extracted by your data extraction model's capabilities, which is Qwen3.5-Omni in this case. It would be helpful to see performance of other models in the family on the evals. Also, no reasoning is given for why a certain model was chosen at any stage of data curation.

External cross-verification:

MuLA-Bench (which came out concurrently around ICLR's submission deadline) has Gemini-3.1-Pro and Qwen-3.5-Omni-Plus statistically tied (61.31 vs 60.92), whereas this benchmark's E2E eval has Qwen3.5-Omni-Plus 35 points ahead (76.0 vs 40.5) - again suggesting a high bias in this eval because of the data generation pipeline using the Qwen family of models.

Interpretation of this eval for models incapable of E2E paradigm:

  • None of the open-source models are evaluated in the E2E inference setting due to limitations to inference at 1hr scale. What this means is that the open-source models' capabilities are evaluated on only two aspects:

    • their ASR capabilities in short context (30-40s).
    • their long-context text backbone's perception and reasoning capabilities.

    Thus, this defeats the purpose of an hour-long audio capability evaluation for open-source models as part of this paper. It definitely helps in giving a signal on the ASR capabilities of these models at small windows and the long-context capabilities of their text backbones, but does not give any signal on their hour-long audio capabilities, which is the main highlight of this paper.

Headline claims are contradicted or unsupported by the paper's own evidence:

  • Page 1 line 22: "Only one end-to-end LALM surpasses the human reference", Page 2 line 96: "only Qwen3.5-Omni-Plus surpasses the human reference", Page 7 line 362: "only Qwen3.5-Omni-Plus surpasses the human reference" - there is not sufficient evidence to make this claim, especially considering that the model that surpasses the human reference was in fact used for caption generation in the dataset on top of which the QA pairs were generated. Either correct the claim with E2E in context along with more models benchmarked in E2E settings, or remove the extreme exaggeration.
    • Cascaded Gemini-3.1-Pro (75.13) and Gemini-2.5-Pro (72.6) beat humans (70.51).
    • 9 of 13 Gemini reasoner configurations in Table 3 do too, up to 78.3.
  • Page 1 line 22: "far from saturation", Page 7 line 371 "far from saturation" - saturation is a subjective term; given the evaluation on a limited number of models in an E2E setting, such a claim should not be made.
    • A few of the tasks are saturated, like Temp. Loc. at 92, Sem. Con. at 97.5.
  • Page 2 line 102: "but can also bottleneck strong proprietary systems when the external reasoning model is less capable than the evaluated model itself" - holds only for GPT-5.2 in this limited context, while this is plausibly true in general, sufficient evidence hasn't been provided to claim this, as only two proprietary model families were evaluated against two reasoning models.
  • Page 3 line 141: "no systematic evaluation protocol exists for hour-scale naturalistic audio" - this might be true for E2E audio inference that's created as part of this, but this can't be claimed out of context, as prior evaluation systems exist for transcription, diarization, retrieval, and summarization.
  • Page 3 line 153: "we bridge this gap with VoiceGiraffe designed to probe these capabilities in the extreme long-context regime" - firstly, retract the word extreme, as it's mentioned in Appendix M that median audio token count in E2E inference is 42K (up to 62K) - while the token count definitely falls in the long-context regime, it's not extreme by today's standards where SOTA models often have 1M context windows. Secondly, since most of the models that are part of this study were not run in E2E inference mode because of their limitations, it's difficult to state that the gap has been bridged for audio models, as the evals have fed data to these models in a text form.
  • Page 4 line 166: "hour-scale naturalistic audio drawn from intact real-world recordings rather than synthetic concatenations" - this is not entirely true because Appendix A Table 4 mentions that in the domain of TV dramas, the content was taken from Friends, The Big Bang Theory, Modern Family; each of these has episodes in the 20- 24 min range, while the claim is that the min. duration is > 50 mins; thus, the audio source must have concatenated episodes before they were downloaded.

Some non-critical but still major flaws in the paper:

  • Nataive capacity is ignored: Qwen3-Omni supports ~40min of audio but was run at 30s, MOSS-Audio supports ~55min and Phi-4-Multimodal supports ~30min.
  • The authors don't highlight that, for models with thinking mode on, how did they remove the thinking traces? This is crucial, as this has the potential to introduce bias. The concern arises from the fact that in Qwen3-Omni, thinking mode leads to significantly degraded performance across most tasks with LRM disabled, whereas the performance drop is much smaller when LRM is enabled.
  • Page 7 line 361: "extremely challenging" - retract the word extremely; it's challenging for sure, as models don't perform well.

Factual inconsistencies observed:

  • The human reference is miscredited and too small:
    • Stated protocol: 8 annotators x 150 questions (75 per language, Sec. 4, App. F) = 1200 judgements.
    • Fig. 6 - when given to AI for translation, reads the following in the footer: 150 questions in total, 3 questionnaires. Please choose one of them to complete, indicating they chose 50 questions (possibly per language), which contradicts the previous point.
  • Page 2 line 83 mentions 123 recordings that total 113 hours; the same is mentioned in Table 1 as well, but this contradicts Tables 5 and 6 in Appendix A, which mention 126 recordings with 116 hours.
  • In Table 2, Gemini-2.5-Pro and gemini-3.1-Pro are marked as False for Thinking, whereas in fact there's no explicit way to switch off thinking entirely for them.
  • Fig. 4 plots the human point at ~(66, 75) in all panels vs (52.2, 61.1) in Table 2. Every model point matches; only the human point is off.
  • Typos and garbled text: "Semantic Conetent"
  • Broken sentence fragment: Page 1 line 41: "The computation overhead imposed by dense tokens, coupled with limited exposure to longer contexts during training".

Suggestions for authors:

  • Move Appendix I inside Section 3 as it provides critical details important for the paper.
  • Fig. 5 is too small, and the legend is unreadable. I suggest enlarging this image and not keeping two images side-by-side in this figure.
Questions for authors
  • Builder model bias: Could you rebuild the dataset using a different model family that's not used for evaluation?
  • Could you add more models in the E2E eval that are not used in the data curation process?
  • How did you decide which models to use at various stages of the data curation process? Could you add the reasoning in the appendix?
  • Why are open models run as only 30-second captioners? Could you rather split your dataset into smaller subsets so that you can evaluate open-source models in E2E paradigm?
  • How was the reasoning stripped from the thinking models?
  • Why did you choose not to evaluate OpenAI, Gemini Flash, or other open models with longer native audio capabilities like Nemotron 3 Nano Omni, etc.?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.