acceptodds
Under review as a conference paper at ICLR 2027

MS-OmniBench: When Listening Across Video Streams Is Necessary

Abstract

A camera may record the sound of an impact while its cause is visible only from another viewpoint. However, existing benchmarks for Omni-modal Large Language Models (Omni-LLMs) offer limited evaluation of reasoning that requires both sound and complementary information across streams. To address this gap, we introduce MS-OmniBench, a benchmark designed around questions that jointly require acoustic evidence and multiple streams. MS-OmniBench comprises 3,109 human-reviewed or human-authored questions across diverse real-world settings, organized into twelve subtasks spanning four families: synchronous cross-stream understanding, asynchronous cross-segment reasoning, audio-anchored retrieval, and cross-modal consistency verification. Answer relevant evidence is distributed across camera viewpoints, source-specific audio tracks, and non-adjacent temporal segments, with annotated evidence chains identifying the supporting streams and time ranges. We evaluate seventeen open-weight and proprietary Omni-LLMs and four existing agentic systems. The best proprietary and open-weight end-to-end models achieve 61.9% and 52.0% accuracy, respectively, while the strongest existing agentic system reaches 49.8%. We further introduce MESA, a simple Multi-stream Evidence Search Agent that organizes observations into per-stream event timelines and uses them to guide targeted media inspection and cross-stream answer verification. With Gemini 3.8 Flash, MESA reaches 65.6% accuracy, improving by 3.7 percentage points over the same backbone in the end-to-end setting. MS-OmniBench provides a testbed for studying how models use sound to connect complementary observations across streams. The benchmark will be released at **MS-OmniBench**.

Primary area:
datasets and benchmarks
Keywords:
Audio-Visual Reasoning, Multi-Stream Video Understanding, Omni-Modal Large Language Models, Multimodal Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_weh2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces MS-OmniBench, a benchmark for evaluating Omni-modal LLMs on questions that require both acoustic evidence and complementary information distributed across multiple streams. It covers diverse real-world scenarios and task families, with human-verified questions, explicit evidence chains, and a construction pipeline designed to screen out shortcut solutions and ensure audio and multi-stream dependence. The authors evaluate many end-to-end Omni-LLMs and existing agentic systems, and they propose MESA, a multi-stream evidence-search agent that uses per-stream event timelines to guide targeted inspection and cross-stream verification. Validation shows current models still struggle on these jointly audio-dependent, multi-stream reasoning tasks, while MESA improves over end-to-end baselines and reduced-input diagnostics support the benchmark’s intended reliance on audio and multiple streams.

Critical strengths

The critical strength is that the paper clearly articulates the complexity of cross-stream and cross-modal understanding and operationalizes it through a comprehensive ontology. It decomposes the problem into four task families and twelve subtasks—synchronous cross-stream understanding, asynchronous cross-segment reasoning, audio-anchored retrieval, and cross-modal consistency verification—and further specifies three query formats and explicit evidence chains distributed across camera viewpoints, source-specific audio tracks, and non-adjacent temporal segments. This is the paper’s strongest contribution: it turns “joint audio–multi-stream reasoning” from a vague slogan into concrete, annotatable task categories and evidence requirements.

Critical weaknesses

The most consequential weakness is that the paper does not empirically establish the benchmark’s necessity and uniqueness relative to prior work; it motivates the gap qualitatively and provides internal reduced-input ablations, but offers no head-to-head comparison, overlap analysis, or evidence that existing benchmarks fail to isolate the same joint audio–multi-stream capability. It also lacks a systematic analysis of existing Omni-LLM failures, reporting aggregate scores and input-removal diagnostics rather than an error taxonomy or subtask-level failure attribution. Consequently, readers cannot tell whether the observed model weaknesses are specific to this benchmark’s intended construct or whether MS-OmniBench diagnoses failure modes that prior benchmarks miss. This is critical because the paper’s central claim—that MS-OmniBench is needed to study joint acoustic and multi-stream reasoning—depends on exactly these comparative and diagnostic demonstrations.

Questions for authors
  1. What is the main argument you want to make through the MESA performance? You mentioned four stages, including planning, localization, discrimination, and decision. Do other agents have similar processes or not?
  2. You report "The aggregate results are strongly non-monotonic. Gemini improves from 60.2% on two-stream questions to 66.5% on five-stream questions, whereas Qwen decreases from 52.5% to 42.2%" in Section 5.3. Have you considered the possible reasons? When will the multi-stream video provides sufficient information and when will it confuse the omni models?
  3. What are the application scenarios of MS-OmniBench? Would a more fine-grained understanding lead to better practical performance? And is there some coupling relationship between the performance gaps across different models and their performance across different tasks?
  4. In Related Works, do you consider add some previous works about cross-stream omni understanding to emphasize the characteristics of MS-OmniBench?
  5. About the "Question Generation" part, do you use models to assist this process? If you do, how do you complete the human-AI interaction protocol?
3 comments

We sincerely thank Reviewer weh2 for the constructive feedback. We appreciate the recognition of our task taxonomy, question formats, and explicit cross-stream evidence chains. We provide a comparison of benchmark protocols, supplementary failure diagnostics, and clarifications below.

W1/Q4: Distinction from prior benchmarks and related work.

R1: Thank you for the suggestion. We distinguish related benchmarks by their input structures and evidence requirements. As outlined in Table A, JointAVBench studies strict audio-visual dependency, including reasoning across scenes within a video, and OmniVideoBench filters unimodal shortcuts and provides reasoning traces. EgoExoBench includes synchronous and asynchronous visual relations across viewpoints, while CrossView evaluates multi-camera visual reasoning. X-Stream checks multi-stream necessity, while its Section 3.4 reports that approximately 30% of questions incorporate audio or speech. Its broader taxonomy also includes Single-stream Understanding alongside cross-stream cooperation and reference alignment, as detailed in its Appendix B.2. MS-OmniBench focuses on questions that require acoustic evidence and complementary scene streams jointly.

MS-OmniBench makes acoustic necessity and multi-stream necessity joint inclusion criteria for every retained question. Each item requires evidence that remains insufficient after removing answer-relevant scene audio or restricting the input to any one supplied scene stream. Its evidence chains record which acoustic, visual, and cross-stream associations support the answer. For example, the bike-repair case in Figure 6 uses a click in the egocentric audio to identify the relevant instant, while the tool is visible only in the silent exocentric view. Answering requires connecting the acoustic anchor to complementary visual evidence.

Our empirical checks address this joint requirement. As reported in Table 6, removing scene audio reduces Gemini accuracy from 61.9% to 36.4% and Qwen accuracy from 50.6% to 24.2%; isolated-stream averages fall to 21.4% and 19.7%. To examine the concern that an average can conceal a sufficient individual stream, two independent annotators additionally assess 304 distinct question–stream inputs on which models answer correctly, with gold answers, model predictions, other scene streams, and evidence annotations hidden. Initial agreement is 96.1%, and adjudication identifies no single-stream-sufficient input in this sample. Together with item-level human verification, these results support the intended joint evidence requirement.

MS-OmniBench further provides explicit stream-and-time evidence chains and a diagnostic breakdown of perception, association, and temporal integration errors. These annotations and analyses support studying how models use acoustic cues to connect complementary observations across streams.

Table A: Comparison of published benchmark protocols. The comparison concerns evidence requirements and input structure.

BenchmarkScene inputAudio-related requirementJoint inclusion rule for acoustic evidence and multiple streams
[WorldSense](Hong et al., 2026)Single videoCoupled audio and visual understandingNot formulated over separate scene streams
[Daily-Omni](Zhou et al., 2025)Single videoAudio and visual reasoning and temporal alignmentNot formulated over separate scene streams
[JointAVBench](Chao et al., 2026)Single video, including multiple scenesJoint audio and visual dependencyNot formulated over separate scene streams
[OmniVideoBench](Li et al., 2026)Single videoFiltering of questions answerable from one modalityNot formulated over separate scene streams
[EgoExoBench] / [CrossView] (He et al., 2026, and Shah et al., 2026)Multiple viewsVisual cross-view reasoningNot an acoustic necessity requirement
[X-Stream] (Sun et al., 2026)2 to 5 concurrent streamsAudio or speech involved in approximately 30% of questionsNecessity of multiple streams is checked. Audio necessity is not required throughout the benchmark
MS-OmniBench2 to 6 scene streamsAcoustic necessity checked per retained itemJoint inclusion criterion with human verification

W2: Systematic analysis of model failures.

R2: Thank you for the suggestion. We rerun each backbone on questions it answers incorrectly in the original end-to-end evaluation, using a chain-of-thought prompt. From diagnostic reruns that remain incorrect and contain sufficient reported evidence for attribution, we sample 600 traces per model across the twelve subtasks. We include all eligible errors from smaller subtasks and sample approximately evenly from the remaining subtasks. The original predictions define the error pool, while the CoT reruns provide the traces for analysis. We distinguish five categories:

  1. Localization: The trace identifies an incorrect stream or time window.
  2. Acoustic perception: The trace identifies the relevant window but describes sound or speech incorrectly.
  3. Visual perception: The trace identifies the relevant window but describes an action, object, or attribute incorrectly.
  4. Cross-stream association: Local observations agree with the reference evidence, but the correspondence between entities, views, or streams is incorrect.
  5. Temporal integration or decision: Local observations agree with the reference evidence, but ordering, counting, deduplication, synchrony, or final option selection is incorrect.

We use Claude Opus 5.5 to compare the reported observations with the annotated evidence chains and assign the first applicable category in the listed order. For validation, we compare the automated labels with independent human annotations. As reported in Table B2, the labels agree on 91 of 100 Gemini traces and 89 of 100 Qwen traces, giving 90.0% exact agreement overall.

The breakdown in Table B1 shows that cross-stream association and temporal integration or decision account for 46.0% of sampled Gemini errors and 43.2% of sampled Qwen errors. Acoustic and visual perception account for 43.3% and 39.2%, respectively. Cross-stream association is the largest category in consistency verification for both models, while temporal integration or decision is the largest category in asynchronous reasoning. Models therefore struggle both to perceive individual observations and to connect them across streams and time. These proportions refer to the sampled diagnostic traces. We have included this analysis in Appendix D.7.

Table B1: Diagnostic error attribution by task family. Entries report counts and row percentages.

Model/familyLocalizationAcousticVisualCross-streamTemporal/decisionTotal
Gemini 3.8 Flash
Sync.25(12.2)27(13.2)67(32.7)65(31.7)21(10.2)205
Async.11(7.1)30(19.5)50(32.5)7(4.5)56(36.4)154
Retr.23(16.5)22(15.8)38(27.3)11(7.9)45(32.4)139
Verif.5(4.9)14(13.7)12(11.8)49(48.0)22(21.6)102
All64(10.7)93(15.5)167(27.8)132(22.0)144(24.0)600
Qwen3-Omni-30B
Sync.40(19.0)31(14.7)58(27.5)76(36.0)6(2.8)211
Async.35(22.3)22(14.0)39(24.8)12(7.6)49(31.2)157
Retr.23(16.9)20(14.7)42(30.9)13(9.6)38(27.9)136
Verif.8(8.3)14(14.6)9(9.4)41(42.7)24(25.0)96
All106(17.7)87(14.5)148(24.7)142(23.7)117(19.5)600

Table B2: Agreement between human and automated labels on identical traces.

BackboneHuman-checked tracesMatching labelsExact agreement (%)
Gemini 3.8 Flash1009191.0
Qwen3-Omni-30B1008989.0
Overall20018090.0

Q1: The purpose of MESA and its relation to existing agents.

A1: Thank you for the question. MESA examines whether organizing and verifying distributed evidence improves answering with the same backbone. Full MESA improves Gemini from 61.9% to 65.6% and Qwen from 50.6% to 52.1%. Removing timelines and derived summaries reduces accuracy to 63.5% and 51.1%. Removing the auxiliary prediction retains 64.8% and 51.6%. These results support the usefulness of evidence organization and targeted inspection within this implementation.

VideoAgent and Deep Video Discovery employ iterative retrieval and inspection, while omni-modal agents incorporate active perception and multimodal evidence search. MESA organizes these operations around model-generated per-stream timelines and an option-by-evidence matrix. We present it as a simple baseline for distributed acoustic and visual evidence search. In the Table 1 (main paper), its improvements on MHop and Manip coexist with decreases on MemA and TrigLoc, indicating that the benefits depend on the task.

Q2: When additional streams help or confuse models.

A2: Thank you for the question. We consider source-specific stream structures and the effect of supplying all available streams for the same benchmark questions.

Source-specific evidence. As reported in Table 12, Gemini achieves 65.4%, 72.1%, and 84.6% across EasyCom groups with two, three, and four streams; Qwen achieves 60.1%, 70.4%, and 74.4%. Across the corresponding STARSS groups, Qwen achieves 49.0%, 50.0%, and 37.4%. An important difference is the evidence supplied by these streams. As described in Table 3, EasyCom provides source-specific microphone observations, whereas STARSS provides directional views extracted from panoramic video. Complementary acoustic observations can help resolve ambiguous events, while more views do not necessarily improve answers. This interpretation is consistent with the observed source-wise patterns, although the groups contain different questions.

Additional streams for the same questions. The FULL/ALL-STREAMS comparison in Table 6 already examines additional inputs on the same questions. FULL supplies the specified streams; ALL-STREAMS also supplies the remaining streams from the source case. As reproduced in Table C below, Gemini increases from 61.9% to 62.5%, while Qwen decreases from 50.6% to 45.6%. Thus, the additional observations help Gemini slightly but reduce Qwen's accuracy by 5.0 percentage points. Their value depends on both the evidence supplied and the backbone's ability to integrate it.

Table C: Overall FULL and ALL-STREAMS accuracy reproduced from Table 6.

BackboneFULL (%)ALL-STREAMS (%)Change (pp)
Gemini 3.8 Flash61.962.5+0.6
Qwen3-Omni-30B50.645.6−5.0

Q3: Applications and task-dependent model differences.

A3: Thank you for the question. The benchmark evaluates skills relevant to meeting assistance, AR and activity assistance, and multitrack media analysis. Meeting assistance requires linking speech to participants, gestures, and referenced objects. Activity assistance requires identifying the visible cause of a sound that is off-screen in another view. Multitrack media analysis requires associating instrument-specific audio with visual performers. Figures 1 and 2 illustrate why resolving the specific person, object, or action matters beyond recognizing the overall scene.

The model gaps also depend on the task. For example, Table 1 shows that Gemini leads Qwen overall at 61.9% versus 50.6% and on SpkBind at 65.7% versus 56.1%, whereas Qwen leads on XCount at 43.6% versus 38.6%. Nemotron leads the open-weight models overall at 52.0%, yet Qwen achieves higher SpkBind accuracy at 56.1% versus 49.4%. These reversals show why overall rankings do not capture performance on identity binding or counting. These task-specific results characterize strengths relevant to the corresponding application scenarios.

Q5: Model-assisted generation and the human–AI protocol.

A5: Thank you for the question. We use Gemini 3 Flash for event extraction, cross-stream alignment, question planning and writing, and shortcut screening. Planning specifies the supporting streams, evidence hops, and distractors before the question is written. As described in Section 3.3 and Appendices B.2–B.4, screening checks structural validity, no-media answerability under two option orders, muted-input answerability, and isolated-stream answerability.

Human annotators inspect and revise model-generated candidates and also author questions directly. A separate reviewer checks the inclusion criteria, with a third reviewer handling disagreements. As reported in Table 5, 2,391 candidates enter review-stage quality control, 193 are automatically removed, and 50 are removed during human review. The retained pool contains 1,999 model-assisted questions and 149 human-authored controlled variants. Direct human authoring contributes 961 retained questions from 1,032 candidates. The 149 controlled variants and 961 directly authored questions form the 1,110-question human-authored group, giving 3,109 questions overall. Human inspection of the supporting media determines final inclusion.

The analysis in Appendix D.5 also examines a possible construction-model advantage. In Table 11, Gemini 3.8 Flash achieves 59.2% on model-assisted questions and 66.9% on human-authored questions, while Gemini 3 Flash achieves 59.7% on both. Neither model shows an accuracy advantage on the model-assisted subset within these evaluation groups.

Reviewer_tLM3OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MS-OmniBench to evaluate models' joint use of audio and information from multiple video streams. Questions span a range of real-world settings, are annotated with supporting streams and time intervals, and undergo automated screening and human review. The paper compares several model and agent families, uses reduced-input controls to examine information dependencies, and introduces MESA, which answers questions through timeline retrieval and targeted local inspection.

Critical strengths
  1. The construction process explicitly targets questions that require audio and information from multiple streams, and retains the evidence locations for each item. The initial human review of every item, combined with automated shortcut screening, provides a concrete way to check these requirements during sample selection.

  2. The evaluation covers a broad range of models and includes controls that remove audio, restrict input to a single stream, or provide transcripts. Breakdowns by data source, question provenance, and input structure help reveal differences that aggregate rankings may obscure.

  3. The comparison between MESA and direct answering uses the same backbone, and the appendix reports call costs and detailed execution traces. I also appreciate that the authors report performance decreases on some tasks, which helps readers understand the procedure's benefits and limitations.

Critical weaknesses
  1. Because SINGLE averages performance across streams, it may conceal questions that can already be answered from one particular stream. Although every item received an initial human review, it would be useful to report agreement from an independent reassessment and estimate the proportion of items that may still fail to meet the information-necessity requirement. Reporting results for each item and individual stream, followed by a blind reassessment of items that models consistently answer correctly from one stream, would help determine whether evidence from another stream is actually needed.

  2. The current comparison leaves some uncertainty about which components account for MESA's gains. MESA combines precomputed timelines, repeated local inspections, and an additional direct-answer prediction. Although the appendix reports costs at question time, it does not compare MESA with simple repeated answering or local-inspection strategies under similar budgets. Such baselines, ablations that remove the timelines or auxiliary answer, and reporting the amortized cost of timeline construction would clarify the sources of the gains.

  3. Models are evaluated with different video resolutions and numbers of sampled frames, which may substantially affect tasks requiring fine-grained identification of people or actions. The authors appropriately frame the leaderboard as a system-level comparison, although this still makes it difficult to separate model capability from input-processing choices. Evaluating a representative subset of models under matched resolution and sampling budgets would make differences in model capability easier to interpret.

Questions for authors

My questions are included in the weaknesses above.

3 comments

We sincerely thank Reviewer tLM3 for the constructive review. We appreciate the recognition of our evidence-grounded construction pipeline, reduced-input controls, and reporting of MESA's costs and subtask-level trade-offs. We address the concerns below.

W1: Single-stream shortcuts and independent reassessment.

R1: Thank you for the suggestion. We examine whether individual scene streams provide sufficient evidence through a blind reassessment of question–stream inputs on which models answer correctly.

Construction verification. An initial author or annotator prepares each candidate, a different reviewer checks the inclusion criteria jointly, and a third reviewer handles disagreements. Initial confirmation rates are 78.5% for model-assisted candidates and 92.3% for directly human-authored candidates. In Table A, we distinguish these rates from the final retention counts and exclude the 149 human-authored controlled variants from the model-assisted track.

Table A: Initial candidate confirmation by a separate reviewer and final outcomes.

Verification trackCandidatesInitially confirmed, n (%)RemovedRetained
Model-assisted candidates2,0491,608 (78.5)501,999
Direct human-authored candidates1,032953 (92.3)71961

Independent reassessment. We first include all 104 question–stream inputs on which both models answer correctly from the same isolated stream. Excluding these inputs, we randomly sample 100 further successes from each model and prevent overlap between the two samples. This gives 304 distinct question–stream inputs.

Two annotators independently assess each input using only the question, five options, applicable reference media, and the isolated scene stream. They are blind to the remaining scene streams, gold answer, annotated evidence chain, and model predictions. Both judge the single-stream evidence insufficient on 292 inputs, giving 96.1% initial agreement. On the other twelve, one annotator judges the evidence sufficient and the other insufficient. A third independent reviewer adjudicates these cases against the complete cross-stream evidence chains and confirms that the isolated evidence only rules out some distractors; the association needed to uniquely justify the answer remains unavailable. The final audit therefore identifies no violation of multi-stream necessity among the 304 inspected inputs.

W2: MESA component attribution, inference controls, and timeline cost.

R2: Thank you for the suggestion. We compare MESA with repeated direct answering and simple local inspection, and examine the effects of removing timelines and derived summaries or the auxiliary prediction.

Local-inspection control. This baseline uses no precomputed timelines. It divides each stream into fixed-duration coarse intervals and uses the question to guide inspection of finer 15–30-second windows. It uses only single-stream perception tools, view_clip and listen_clip, with ref_locate available for reference-media questions, under the same cap of twelve search-media calls. Returned descriptions are appended to a running scratchpad, from which the model selects one final answer. The baseline uses neither joint cross-stream comparison tools nor an option-by-evidence matrix or an auxiliary end-to-end prediction.

MESA without timelines or summaries. This ablation removes the cached timelines and summaries, so neither stream summaries nor timeline queries are available. Localization therefore uses the same coarse-to-fine search without a timeline index. The downstream MESA procedure remains intact: compare_streams and compare_windows test competing options in the option-by-evidence matrix, and the Decide stage combines decisive-window re-inspection with the auxiliary prediction. The local-inspection baseline tests how far local observations and text aggregation suffice; this ablation tests the effect of removing the index from MESA's verification procedure.

Components and inference controls. As shown in Table C, five direct passes with permuted answer options and vote aggregation achieve 62.6% for Gemini and 51.0% for Qwen. Simple local inspection under the same cap of twelve search calls achieves 63.1% and 50.9%, below full MESA at 65.6% and 52.1%. Removing timelines and derived summaries reduces accuracy to 63.5% and 51.1%. Without the auxiliary prediction, accuracy remains at 64.8% and 51.6%. The gains therefore persist without the direct answer, and the timelines and summaries contribute beyond local inspection alone.

Table C: MESA component ablations and inference controls.

ConfigurationGemini Acc. (%)Gemini media calls / tokens per itemQwen Acc. (%)Qwen media calls / tokens per item
Direct answering61.91.0 / 38k50.61.0 / 41k
Repeated direct answering + aggregation (5 passes)62.65.0 / 190k51.05.0 / 205k
Simple local inspection + aggregation63.19.6 / 149k50.910.4 / 168k
MESA without timelines or summaries63.59.8 / 152k51.110.6 / 171k
MESA without auxiliary prediction64.88.1 / 106k51.69.1 / 121k
Full MESA65.69.1 / 144k52.110.1 / 162k

Timeline construction and amortized cost. Constructing the timelines costs an average of 118k tokens per case for Gemini and 128k for Qwen. Sharing them across 3,109 questions from 581 cases adds approximately 22k and 24k tokens per question. As detailed in Table D, total amortized costs reach 166k and 186k tokens, compared with 190k and 205k for five direct passes. MESA achieves 3.0 and 1.1 percentage points higher accuracy at lower total token consumption over this workload. The first-query estimate includes timeline construction, whereas subsequent queries reuse the cached timelines. We have included the cost details in Appendix E.4 and the controls and ablations in Appendix E.5.

Table D: Timeline construction, question-time inference, and workload amortization.

Cost measureGemini 3.8 FlashQwen3-Omni-30B
Timeline construction per case4.1 calls / 118k tokens (773 s media)4.1 calls / 128k tokens (773 s media)
Amortized timeline construction per question≈0.76 calls / 22k tokens≈0.76 calls / 24k tokens
Question-time inference, including auxiliary prediction9.1 calls / 144k tokens10.1 calls / 162k tokens
Estimated first query total from mean component costs≈13.2 calls / 262k tokens≈14.2 calls / 290k tokens
Average query cost with cached timelines9.1 calls / 144k tokens10.1 calls / 162k tokens
Total per question amortized over the workload≈9.9 calls / 166k tokens≈10.9 calls / 186k tokens
Question-time search media duration170 s180 s
Auxiliary full input media duration542 s542 s

W3: Matched resolution and frame-sampling budgets.

R3: Thank you for the suggestion. We compare Gemini and Qwen using 180p video copies and 32 sampled frames per stream, and vary Gemini's frame count and Qwen's pixel cap, as shown in Table E.

At the common external resolution and frame count, Gemini achieves 57.90%, compared with Qwen's 50.60% at its original pixel cap and 48.18% at the larger cap. Gemini's original configuration achieves 61.95%; using 180p inputs with 16 frames per stream yields 56.06%. For Qwen, increasing the pixel cap from 14,400 to 28,800 to 57,600 yields 47.57%, 50.60%, and 48.18%, respectively.

Input configuration affects absolute accuracy, while Gemini remains ahead in the evaluated settings. The comparison with Gemini's original configuration changes both resolution and sampling, so it does not isolate resolution alone. These controls align video copy resolution and frame count while retaining model-specific resizing and visual token allocation. We have added these input controls to Appendix D.2.

Table E: Resolution and frame-sampling controls.

ModelVideo copySampling per streamQwen pixel cap (pixels/frame)Overall (%)
Gemini 3.8 FlashSource resolution (original)Default (~1 fps)N/A61.95
Gemini 3.8 Flash180p32 framesN/A57.90
Gemini 3.8 Flash180p16 framesN/A56.06
Qwen3-Omni-30B180p32 frames14,40047.57
Qwen3-Omni-30B180p (original)32 frames28,80050.60
Qwen3-Omni-30B180p32 frames57,60048.18
Reviewer_7gfYOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces MS-OmniBench, a benchmark of 3,109 five-option questions for omni-modal LLMs in which answering requires both acoustic evidence and information spread across two or more video streams. Questions cover 4 task families and 12 subtasks, are built from six public datasets through a Gemini-assisted pipeline with shortcut screening, and every item is human-verified to require both audio and multiple streams. The authors evaluate 17 end-to-end models and 4 existing agents; the best proprietary and open-weight models reach 61.9% and 52.0%. They also propose MESA, a budgeted evidence-search agent that improves Gemini 3.8 Flash to 65.6%, and provide analyses by stream count, evidence-chain length, question format, and authoring origin.

Critical strengths
  1. The paper targets a real gap: prior benchmarks test either audio dependence on a single stream or multi-stream reasoning without audio, and this is the first to require sound as the link between streams.
  2. The setting is realistic and useful. The source recordings are the kinds of inputs omni-modal models will actually face, and the questions are ones a real user would ask, so progress here should transfer directly to assistant, meeting, and AR applications.
  3. Every item is human-verified against explicit necessity criteria, and each question ships with an annotated evidence chain, which is more rigorous than the sampled audits most benchmarks rely on.
  4. The authors anticipate the concern that Gemini builds the questions and Gemini tops the leaderboard, and show with a model-assisted versus human-authored split that this does not explain the ranking.
  5. The evaluation is broad, covering 17 end-to-end models and 4 agents under a frozen prompt, and the fine-grained analyses by stream count, chain length, question format, and source go beyond a plain leaderboard.
Critical weaknesses
  1. I would have liked to see a human baseline. Right now there is no way to know whether 61.9% is close to the ceiling or far from it, or whether the weakest subtasks are actually hard or just hard for current models.
  2. The paper never looks at why models fail. Every question comes with an evidence chain, so it would have been easy to check whether models stumble on the audio hop, the visual hop, or the link between streams. That analysis is missing.
  3. The human verification has no agreement numbers, and no result has error bars. Small differences like MESA's 1.5-point gain on Qwen3-Omni could be noise.
  4. Several variables in the dataset move together. Most questions have two streams, stream count depends on which source the clip came from, and chain length depends on whether a model or a person wrote the question. Because of this, a few of the findings in Section 5.3 end up comparing sources rather than the thing they claim to compare.
Questions for authors
  1. How often does MESA's final answer simply match the end-to-end prediction it receives as an auxiliary signal? If it is most of the time, the agent is closer to a re-ranker than an evidence searcher.

Beyond these, my questions follow from the weaknesses above.

5 comments

We sincerely thank Reviewer 7gfY for the constructive feedback. We appreciate the recognition of our joint acoustic and multi-stream setting, item-level evidence verification, and broad evaluation. We address the concerns below.

W1: Human performance on MS-OmniBench.

R1: Thank you for the suggestion. We evaluate human performance on all 3,109 questions with five English-proficient evaluators who do not participate in question construction or necessity verification. Each evaluator answers a disjoint subset, with questions distributed approximately evenly across evaluators within each subtask. We collect one human response per question. Evaluators receive the scene streams, question, answer options, and applicable reference media, without gold answers or evidence annotations.

As shown in Table A, human accuracy reaches 92.0%, compared with 61.9% for Gemini 3.8 Flash and 52.0% for Nemotron-3-Nano-Omni. Humans also achieve 88.2% on XCount and 88.3% on Manip, where Gemini scores 38.6% and 49.4%. The substantial gaps on these two subtasks indicate that current models remain far from human performance even where their scores are lowest. We have included human performance in the Table 1 (main paper) and documented the evaluation protocol in Section 5.1 and Appendix C.4.

Table A: Human and end-to-end accuracy (%).

SystemOverallSync.Async.Retr.Verif.XCountManip
Human92.094.392.289.390.688.288.3
Gemini 3.8 Flash61.965.066.755.657.338.649.4
Nemotron-3-Nano-Omni52.052.855.647.451.237.346.9

W2: Systematic analysis of model failures.

R2: Thank you for the suggestion. We rerun each backbone on questions it answers incorrectly in the original end-to-end evaluation, using a chain-of-thought prompt. From diagnostic reruns that remain incorrect and contain sufficient reported evidence for attribution, we sample 600 traces per model across the twelve subtasks. We include all eligible errors from smaller subtasks and sample approximately evenly from the remaining subtasks. The original predictions define the error pool, while the CoT reruns provide the traces for analysis. We distinguish five categories:

  1. Localization: The trace identifies an incorrect stream or time window.
  2. Acoustic perception: The trace identifies the relevant window but describes sound or speech incorrectly.
  3. Visual perception: The trace identifies the relevant window but describes an action, object, or attribute incorrectly.
  4. Cross-stream association: Local observations agree with the reference evidence, but the correspondence between entities, views, or streams is incorrect.
  5. Temporal integration or decision: Local observations agree with the reference evidence, but ordering, counting, deduplication, synchrony, or final option selection is incorrect.

We use Claude Opus 5.5 to compare the reported observations with the annotated evidence chains and assign the first applicable category in the listed order. For validation, we compare the automated labels with independent human annotations. As reported in Table B2, the labels agree on 91 of 100 Gemini traces and 89 of 100 Qwen traces, giving 90.0% exact agreement overall.

The breakdown in Table B1 shows that cross-stream association and temporal integration or decision account for 46.0% of sampled Gemini errors and 43.2% of sampled Qwen errors. Acoustic and visual perception account for 43.3% and 39.2%, respectively. Cross-stream association is the largest category in consistency verification for both models, while temporal integration or decision is the largest category in asynchronous reasoning. Models therefore struggle both to perceive individual observations and to connect them across streams and time. These proportions refer to the sampled diagnostic traces. We have included this analysis in Appendix D.7.

Table B1: Error attribution by family, n(row %)

Model/familyLocalizationAcousticVisualCross-streamTemporal/decisionTotal
Gemini 3.8 Flash
Sync.25(12.2)27(13.2)67(32.7)65(31.7)21(10.2)205
Async.11(7.1)30(19.5)50(32.5)7(4.5)56(36.4)154
Retr.23(16.5)22(15.8)38(27.3)11(7.9)45(32.4)139
Verif.5(4.9)14(13.7)12(11.8)49(48.0)22(21.6)102
All64(10.7)93(15.5)167(27.8)132(22.0)144(24.0)600
Qwen3-Omni-30B
Sync.40(19.0)31(14.7)58(27.5)76(36.0)6(2.8)211
Async.35(22.3)22(14.0)39(24.8)12(7.6)49(31.2)157
Retr.23(16.9)20(14.7)42(30.9)13(9.6)38(27.9)136
Verif.8(8.3)14(14.6)9(9.4)41(42.7)24(25.0)96
All106(17.7)87(14.5)148(24.7)142(23.7)117(19.5)600

Table B2: Human–automated agreement on the same traces.

ModelnMatchesAgreement (%)
Gemini1009191.0
Qwen1008989.0
Overall20018090.0

W3: Human verification and statistical uncertainty.

R3: Thank you for the comment. We clarify the verification procedure and provide paired confidence intervals below.

Human verification. An initial author or annotator prepares each candidate, and a different reviewer checks full-input sufficiency, acoustic necessity, and multi-stream necessity. A third reviewer handles disagreements. As indicated in Table C, the reviewer initially confirms 1,608 of 2,049 model-assisted candidates and 953 of 1,032 directly human-authored candidates, corresponding to 78.5% and 92.3%. These are confirmation rates before adjudication; the model-assisted track excludes the 149 human-authored controlled variants.

We also conduct an independent reassessment of 304 distinct question–stream inputs on which models answer correctly. Two annotators see only the question, options, applicable reference media, and the isolated stream; the other streams, gold answer, annotated evidence chain, and model predictions are hidden. Both judge the isolated evidence insufficient on 292 inputs, giving 96.1% initial agreement. A third independent reviewer examines the twelve disagreements against the complete cross-stream evidence chains and confirms that all twelve also lack the association needed to justify a unique answer. No audited input violates multi-stream necessity. These agreement figures concern single-stream sufficiency in the selected sample.

Table C: Initial candidate confirmation by a separate reviewer and final outcomes.

Verification trackCandidatesInitially confirmed, n (%)RemovedRetained
Model-assisted candidates2,0491,608 (78.5)501,999
Direct human-authored candidates1,032953 (92.3)71961

Statistical uncertainty. MESA corrects 292 Gemini errors and introduces 178 regressions, giving 114 additional correct answers. For Qwen, 256 corrections and 209 regressions give 47 additional correct answers. The paired estimates in Table D correspond to gains of 3.67 and 1.51 percentage points, with 95% Wald intervals of [2.31, 5.03] and [0.15, 2.87]. Both 95% intervals exclude zero. We have added verification results to Appendix C.2 and confidence intervals to Appendix E.6.

Table D: Paired MESA accuracy gains on 3,109 questions.

BackboneDirect (%)MESA (%)CorrectionsRegressionsNet additional correctGain (pp)95% paired Wald CI (pp)
Gemini 3.8 Flash61.965.6292178114+3.67[+2.31, +5.03]
Qwen3-Omni-30B50.652.125620947+1.51[+0.15, +2.87]

W4: Confounding in the fine-grained analysis.

R4: Thank you for the comment. We examine the performance patterns within individual sources and authoring groups. Appendix D.6 already provides these breakdowns to help interpret the pooled results.

Stream count. As shown in Table 12 (Appendix), mean accuracy across the seventeen models increases from 55.8% to 62.7% to 67.7% across EasyCom groups with two, three, and four streams. Across the corresponding STARSS groups, it decreases from 49.5% to 45.8% to 43.5%. Stream count therefore has different associations with accuracy across sources.

Evidence-chain length. For model-assisted questions, mean accuracy decreases from 51.0% at three hops to 43.2% at four or more hops, as reported in Table 13 (Appendix). The corresponding human-authored groups both achieve 52.1%. Pooling these groups obscures the relationship with authoring origin.

These patterns show that neither stream count nor annotated chain length consistently orders question difficulty across sources and authoring groups. Because the comparisons involve different questions, we interpret them as associations rather than isolated effects of adding streams or hops. We have clarified the interpretation of these comparisons in Section 5.3.

Q1: Dependence on the auxiliary end-to-end prediction.

A1: Thank you for the question. MESA agrees with the direct prediction on 75.9% of Gemini questions and 71.2% of Qwen questions. Among questions answered incorrectly by the direct model, agreement falls to 51.7% and 55.4%. Table E gives the agreement statistics alongside the paired changes and auxiliary-answer ablation.

MESA corrects 292 Gemini errors and introduces 178 regressions, giving 114 additional correct answers. For Qwen, 256 corrections and 209 regressions give 47 additional correct answers. These paired results show both successful corrections and regressions during evidence search.

The auxiliary prediction enters only at the Decide stage; the preceding stages use timelines and inspected media. Without this prediction, MESA reaches 64.8% for Gemini and 51.6% for Qwen, above direct answering at 61.9% and 50.6%. The full configuration achieves 65.6% and 52.1%. Thus, the auxiliary answer helps the final decision, while the improvement over direct answering remains when that answer is removed. We have added the ablation and agreement results to Appendices E.5 and E.6.

Table E: Agreement, paired changes, and auxiliary prediction ablation.

StatisticGemini 3.8 FlashQwen3-Omni-30B
Exact answer agreement (%)75.971.2
Agreement on direct answer errors (%)51.755.4
Direct wrong → MESA correct (n)292256
Direct correct → MESA wrong (n)178209
Correction rate among direct answer errors (%)24.716.7
Regression rate among correct direct answers (%)9.213.3
Direct answering (%)61.950.6
MESA without auxiliary prediction (%)64.851.6
Full MESA (%)65.652.1

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.