- Rating
- 3(+2)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces LVOmniBench, a benchmark for evaluating multimodal large language models on long-form audio-visual understanding. Unlike existing benchmarks that mainly use short clips or static audio-image pairs, LVOmniBench contains 275 videos lasting 10–90 minutes and 1014 manually annotated multiple-choice questions. The questions cover perception, understanding, inference, and logical reasoning, with difficulty levels ranging from low to high. They are designed to require complementary information from both audio and video. The authors evaluate several open-source and proprietary models. Results show that long-form audio-visual understanding remains difficult, especially for counting, music perception, temporal localization, spatial reasoning, and cross-modal alignment. This benchmark holds certain value for MLLMs. However, the models used in the experiments are not state-of-the-art, and some experimental conclusions may have become outdated. The main results (Tab. 2) conclusions and the models used in the experimental analysis do not correspond one-to-one; in particular, the analysis omits Gemini 3.8 Flash, the claimed best-performing model. There are notable weakness in both the experimental conclusions and experimental completeness.
Critical strengths
-
This paper addresses a clear gap in current multimodal evaluation by focusing on long-form audio-visual understanding, which is more representative of real-world applications than short video benchmarks.
-
LVOmniBench contains 275 videos with durations ranging from 10-90 minutes, covering diverse domains and providing a challenging testbed for long-context multimodal reasoning.
-
The benchmark organizes questions into multiple capability categories and difficulty levels, enabling more fine-grained analysis of model strengths and weaknesses than a single aggregate accuracy score.
-
The manual annotation and unimodal filtering process are designed to reduce questions solvable from only audio, only video, or prior knowledge, making the benchmark more targeted toward audio-visual integration.
Critical weaknesses
-
The experimental setup does not appear to reflect the SOTA proprietary models as of the submission deadline. Models such as Gemini 3.1 Pro, GPT-series, and Claude-series are not included. Therefore, the conclusion about the SOTA proprietary models based primarily on Gemini 3.8 Flash is insufficiently comprehensive.
-
The open-source models omit several potentially competitive models, such as Gemma 4 and Nemotron-3-Nano-Omni. In addition, the non-omnimodal baselines, including Qwen3-VL and Qwen2-Audio, are relatively outdated. Consequently, the open-source baselines may not provide an accurate picture of the latest progress in audio-visual understanding.
-
In Section 4, only Table 2 reports results for Gemini 3.8 Flash, whereas most of the subsequent analyses focus on Gemini 3.0 Flash. In particular, Figure 4, Figure 5, and Table 3 do not provide corresponding analyses of Gemini 3.8 Flash. The same issue occurs in Supplementary Material A. The reported experimental analyses may be incomplete and may not fully support the paper's claims regarding its performance and limitations.
Questions for authors
As noted in the 3rd of the Critical Weaknesses, why are the main experimental conclusions drawn from Gemini 3.8 Flash, while the experimental analyses are based on Gemini 3.0 Flash?
Thank you for the clarifications and the new evidence breakdown. Long-context reasoning. The breakdown clarifies that 38.4% of questions require evidence from multiple moments. For these questions, could you report the distribution of time spans between the earliest and latest pieces of evidence needed to answer? I am keeping my Weak Reject recommendation because the evidence breakdown does not yet establish how often the benchmark tests integration across widely separated moments. This distinction matters for interpreting the benchmark scores and assessing its central contribution to long-context audio-visual evaluation.