acceptodds
Under review as a conference paper at ICLR 2027

LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMs

Abstract

Recent advancements in omnimodal large language models (OmniLLMs) have significantly improved the comprehension of audio and video inputs. However, current evaluations primarily focus on short audio and video clips ranging from 10 seconds to 5 minutes, failing to reflect the demands of real-world applications, where videos typically run for tens of minutes. To address this critical gap, we introduce LVOmniBench, a new benchmark designed specifically for the cross-modal comprehension of long-form audio and video. This dataset comprises high-quality videos sourced from open platforms that feature rich audio-visual dynamics. Through rigorous manual selection and annotation, LVOmniBench comprises 275 videos, ranging in duration from 10 to 90 minutes, and 1,014 question-answer (QA) pairs. LVOmniBench aims to rigorously evaluate the capabilities of OmniLLMs across domains, including long-term memory, temporal localization, fine-grained understanding, and multimodal perception. Our extensive evaluation reveals that current OmniLLMs encounter significant challenges when processing extended audio-visual inputs. Open-source models generally achieve accuracies below 40%, whereas Gemini 3.8-Flash reaches a peak accuracy of approximately 69%. We anticipate that this dataset, along with our empirical findings, will stimulate further research and the development of advanced models capable of resolving complex omnimodal understanding problems within long inputs.

Primary area:
datasets and benchmarks
Keywords:
OmniLLMs, Audio-Video Understanding, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cutWOpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LVOmniBench, a benchmark for evaluating multimodal large language models on long-form audio-visual understanding. Unlike existing benchmarks that mainly use short clips or static audio-image pairs, LVOmniBench contains 275 videos lasting 10–90 minutes and 1014 manually annotated multiple-choice questions. The questions cover perception, understanding, inference, and logical reasoning, with difficulty levels ranging from low to high. They are designed to require complementary information from both audio and video. The authors evaluate several open-source and proprietary models. Results show that long-form audio-visual understanding remains difficult, especially for counting, music perception, temporal localization, spatial reasoning, and cross-modal alignment. This benchmark holds certain value for MLLMs. However, the models used in the experiments are not state-of-the-art, and some experimental conclusions may have become outdated. The main results (Tab. 2) conclusions and the models used in the experimental analysis do not correspond one-to-one; in particular, the analysis omits Gemini 3.8 Flash, the claimed best-performing model. There are notable weakness in both the experimental conclusions and experimental completeness.

Critical strengths
  1. This paper addresses a clear gap in current multimodal evaluation by focusing on long-form audio-visual understanding, which is more representative of real-world applications than short video benchmarks.

  2. LVOmniBench contains 275 videos with durations ranging from 10-90 minutes, covering diverse domains and providing a challenging testbed for long-context multimodal reasoning.

  3. The benchmark organizes questions into multiple capability categories and difficulty levels, enabling more fine-grained analysis of model strengths and weaknesses than a single aggregate accuracy score.

  4. The manual annotation and unimodal filtering process are designed to reduce questions solvable from only audio, only video, or prior knowledge, making the benchmark more targeted toward audio-visual integration.

Critical weaknesses
  1. The experimental setup does not appear to reflect the SOTA proprietary models as of the submission deadline. Models such as Gemini 3.1 Pro, GPT-series, and Claude-series are not included. Therefore, the conclusion about the SOTA proprietary models based primarily on Gemini 3.8 Flash is insufficiently comprehensive.

  2. The open-source models omit several potentially competitive models, such as Gemma 4 and Nemotron-3-Nano-Omni. In addition, the non-omnimodal baselines, including Qwen3-VL and Qwen2-Audio, are relatively outdated. Consequently, the open-source baselines may not provide an accurate picture of the latest progress in audio-visual understanding.

  3. In Section 4, only Table 2 reports results for Gemini 3.8 Flash, whereas most of the subsequent analyses focus on Gemini 3.0 Flash. In particular, Figure 4, Figure 5, and Table 3 do not provide corresponding analyses of Gemini 3.8 Flash. The same issue occurs in Supplementary Material A. The reported experimental analyses may be incomplete and may not fully support the paper's claims regarding its performance and limitations.

Questions for authors

As noted in the 3rd of the Critical Weaknesses, why are the main experimental conclusions drawn from Gemini 3.8 Flash, while the experimental analyses are based on Gemini 3.0 Flash?

Reviewer_kjfwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LVOmniBench evaluates how well omnimodal language models understand long videos together with their audio. It contains 275 videos, averaging about 34.5 minutes, and 1,014 manually written multiple-choice questions. The questions cover nine task types, three difficulty levels, and speech, environmental sounds, and music. The authors combine model-assisted checks with human review to revise or remove questions that can be answered using a single modality. They evaluate several models, report video-only, audio-only, and speech-transcript comparisons, and analyze model errors. Gemini 3.8 Flash achieves 69.5% accuracy, while the best tested open-source model reaches 39.5%.

Critical strengths
  1. The problem is practically relevant. Many useful questions about long videos require both audio and visual evidence: identifying someone from an earlier introduction, finding a scene based on something that was said, or counting events subject to a temporal condition. The Toby example in Figure 1 illustrates this well. Including environmental sounds and music also makes the benchmark more than a test of video frames and speech transcripts.

  2. The dataset reflects substantial annotation effort. The videos are selected from a larger candidate pool, questions and answer choices are written by humans, and final filtering decisions are also made manually. Checking whether questions can be answered from video, audio, or the question text alone is a sensible measure. The difficulty labels and their reported agreement provide further information about the dataset.

  3. The experimental analysis is clear. The paper reports results by task, difficulty, and audio type, alongside modality comparisons and manual error analysis. Some results clearly demonstrate the value of audio: Gemini 3.0 Flash improves from 46.2% with video alone to 59.0% with both audio and video. These analyses provide a starting point for further investigation of model weaknesses.

Critical weaknesses
  1. It is unclear how many questions actually test long-range understanding, rather than finding a local event in a long video. This directly concerns the paper’s main contribution. Some examples, such as counting Toby’s appearances, clearly involve multiple moments. Others, such as locating the race and then counting people on the podium, appear mainly to require finding a short segment and answering a local question. Both tasks are useful, but they test different abilities. The paper reports source-video duration and overall difficulty, but not how far apart the necessary evidence is or how many segments an answer requires. It also does not compare full videos with shorter inputs that retain all necessary evidence. Low scores could therefore reflect difficulties with local perception or event search, rather than an inability to retain and combine information across time. Reporting these cases separately would clarify the benchmark’s purpose and findings.

  2. The paper does not distinguish failures caused by missing input from failures to understand the input. On page 8, the authors state that open-source models typically truncate audio by default because of input-length limits. If the information needed for an answer is removed, an incorrect answer does not establish that the model cannot understand audio or reason across audio and video. The authors should report how much audio each model actually receives and whether the required evidence is retained. The speech-transcript experiments need the same clarification: if the transcript contains the full recording but the raw audio is truncated, better performance with the transcript may simply reflect access to more information. Moreover, Table 3(b) shows that adding raw audio improves Qwen3-Omni’s accuracy from 35.1% to 39.5%, indicating that it does use some audio information.

  3. The paper needs to explain more clearly what the new benchmark tests beyond existing benchmarks. Video-MME already includes 30–60-minute videos and raw-audio evaluation, and OmniVideoBench also includes long-duration audio-visual questions. A longer average video duration alone is therefore not enough to establish the added contribution. In particular, comparing this average with the overall averages of existing datasets containing many short clips is insufficient. A more convincing comparison would focus on their long-video subsets and clarify what is new about the evaluation in LVOmniBench. For example, how do these benchmarks compare in their coverage of questions that require linking distant moments and using both audio and visual evidence? Collecting long videos and manually writing questions that require complementary audio-visual information is still valuable, but the paper needs to show which important gaps in existing evaluations it fills.

  4. The reported dataset statistics are inconsistent. Multiplying 275 videos by the stated average duration of 2,069.7 seconds gives approximately 158 hours, rather than the 140 hours reported in the main text.

Questions for authors
  1. What were the specific criteria for filtering questions during manual annotation?
Reviewer_9wEDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LVOmniBench contains 1,014 manually curated multiple-choice questions over 275 videos lasting 10–90 minutes. It evaluates joint audio-video understanding across task types and difficulty levels. Gemini-3.8-Flash reaches 69.5% accuracy, while the strongest tested open model reaches 39.5%.

Critical strengths
  • S1. Useful long-form resource. Videos average 34:29, substantially extending the temporal setting of shorter audio-visual benchmarks (Table 1). Manual curation targets complementary audio and visual evidence.
  • S2. Informative modality controls. Video-only, audio-only, and ASR comparisons (Table 3) expose differences between access to speech information and effective use of raw audio.
Critical weaknesses
  • W1. Input coverage confounds the capability conclusions. Appendix F.2 uses 768, 64, or 16 frames across open models, versus 1 fps for Gemini; §4.3 also identifies audio truncation. Without per-model audio/video coverage, the gap cannot be attributed specifically to cross-modal reasoning.
  • W2. Long-context necessity is insufficiently established. Long source videos alone do not show how much temporal evidence each question needs. Sections 3.2–3.3 provide no quantitative evidence-span breakdown or evidence-window control, making retrieval difficulty hard to separate from reasoning difficulty.
Questions for authors
  • Clarification (W1): How much audio and video does each model actually receive, and how often is answer evidence truncated?
  • Clarification (W2): What evidence-span annotations or existing localized-window results establish the need for long-context processing?
1 comment

Thank you for the clarifications and the new evidence breakdown. Long-context reasoning. The breakdown clarifies that 38.4% of questions require evidence from multiple moments. For these questions, could you report the distribution of time spans between the earliest and latest pieces of evidence needed to answer? I am keeping my Weak Reject recommendation because the evidence breakdown does not yet establish how often the benchmark tests integration across widely separated moments. This distinction matters for interpreting the benchmark scores and assessing its central contribution to long-context audio-visual evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.