- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces MS-OmniBench, a benchmark for evaluating Omni-modal LLMs on questions that require both acoustic evidence and complementary information distributed across multiple streams. It covers diverse real-world scenarios and task families, with human-verified questions, explicit evidence chains, and a construction pipeline designed to screen out shortcut solutions and ensure audio and multi-stream dependence. The authors evaluate many end-to-end Omni-LLMs and existing agentic systems, and they propose MESA, a multi-stream evidence-search agent that uses per-stream event timelines to guide targeted inspection and cross-stream verification. Validation shows current models still struggle on these jointly audio-dependent, multi-stream reasoning tasks, while MESA improves over end-to-end baselines and reduced-input diagnostics support the benchmark’s intended reliance on audio and multiple streams.
Critical strengths
The critical strength is that the paper clearly articulates the complexity of cross-stream and cross-modal understanding and operationalizes it through a comprehensive ontology. It decomposes the problem into four task families and twelve subtasks—synchronous cross-stream understanding, asynchronous cross-segment reasoning, audio-anchored retrieval, and cross-modal consistency verification—and further specifies three query formats and explicit evidence chains distributed across camera viewpoints, source-specific audio tracks, and non-adjacent temporal segments. This is the paper’s strongest contribution: it turns “joint audio–multi-stream reasoning” from a vague slogan into concrete, annotatable task categories and evidence requirements.
Critical weaknesses
The most consequential weakness is that the paper does not empirically establish the benchmark’s necessity and uniqueness relative to prior work; it motivates the gap qualitatively and provides internal reduced-input ablations, but offers no head-to-head comparison, overlap analysis, or evidence that existing benchmarks fail to isolate the same joint audio–multi-stream capability. It also lacks a systematic analysis of existing Omni-LLM failures, reporting aggregate scores and input-removal diagnostics rather than an error taxonomy or subtask-level failure attribution. Consequently, readers cannot tell whether the observed model weaknesses are specific to this benchmark’s intended construct or whether MS-OmniBench diagnoses failure modes that prior benchmarks miss. This is critical because the paper’s central claim—that MS-OmniBench is needed to study joint acoustic and multi-stream reasoning—depends on exactly these comparative and diagnostic demonstrations.
Questions for authors
- What is the main argument you want to make through the MESA performance? You mentioned four stages, including planning, localization, discrimination, and decision. Do other agents have similar processes or not?
- You report "The aggregate results are strongly non-monotonic. Gemini improves from 60.2% on two-stream questions to 66.5% on five-stream questions, whereas Qwen decreases from 52.5% to 42.2%" in Section 5.3. Have you considered the possible reasons? When will the multi-stream video provides sufficient information and when will it confuse the omni models?
- What are the application scenarios of MS-OmniBench? Would a more fine-grained understanding lead to better practical performance? And is there some coupling relationship between the performance gaps across different models and their performance across different tasks?
- In Related Works, do you consider add some previous works about cross-stream omni understanding to emphasize the characteristics of MS-OmniBench?
- About the "Question Generation" part, do you use models to assist this process? If you do, how do you complete the human-AI interaction protocol?
3 comments
W2: Systematic analysis of model failures.
R2: Thank you for the suggestion. We rerun each backbone on questions it answers incorrectly in the original end-to-end evaluation, using a chain-of-thought prompt. From diagnostic reruns that remain incorrect and contain sufficient reported evidence for attribution, we sample 600 traces per model across the twelve subtasks. We include all eligible errors from smaller subtasks and sample approximately evenly from the remaining subtasks. The original predictions define the error pool, while the CoT reruns provide the traces for analysis. We distinguish five categories:
- Localization: The trace identifies an incorrect stream or time window.
- Acoustic perception: The trace identifies the relevant window but describes sound or speech incorrectly.
- Visual perception: The trace identifies the relevant window but describes an action, object, or attribute incorrectly.
- Cross-stream association: Local observations agree with the reference evidence, but the correspondence between entities, views, or streams is incorrect.
- Temporal integration or decision: Local observations agree with the reference evidence, but ordering, counting, deduplication, synchrony, or final option selection is incorrect.
We use Claude Opus 5.5 to compare the reported observations with the annotated evidence chains and assign the first applicable category in the listed order. For validation, we compare the automated labels with independent human annotations. As reported in Table B2, the labels agree on 91 of 100 Gemini traces and 89 of 100 Qwen traces, giving 90.0% exact agreement overall.
The breakdown in Table B1 shows that cross-stream association and temporal integration or decision account for 46.0% of sampled Gemini errors and 43.2% of sampled Qwen errors. Acoustic and visual perception account for 43.3% and 39.2%, respectively. Cross-stream association is the largest category in consistency verification for both models, while temporal integration or decision is the largest category in asynchronous reasoning. Models therefore struggle both to perceive individual observations and to connect them across streams and time. These proportions refer to the sampled diagnostic traces. We have included this analysis in Appendix D.7.
Table B1: Diagnostic error attribution by task family. Entries report counts and row percentages.
| Model/family | Localization | Acoustic | Visual | Cross-stream | Temporal/decision | Total |
|---|---|---|---|---|---|---|
| Gemini 3.8 Flash | ||||||
| Sync. | 25(12.2) | 27(13.2) | 67(32.7) | 65(31.7) | 21(10.2) | 205 |
| Async. | 11(7.1) | 30(19.5) | 50(32.5) | 7(4.5) | 56(36.4) | 154 |
| Retr. | 23(16.5) | 22(15.8) | 38(27.3) | 11(7.9) | 45(32.4) | 139 |
| Verif. | 5(4.9) | 14(13.7) | 12(11.8) | 49(48.0) | 22(21.6) | 102 |
| All | 64(10.7) | 93(15.5) | 167(27.8) | 132(22.0) | 144(24.0) | 600 |
| Qwen3-Omni-30B | ||||||
| Sync. | 40(19.0) | 31(14.7) | 58(27.5) | 76(36.0) | 6(2.8) | 211 |
| Async. | 35(22.3) | 22(14.0) | 39(24.8) | 12(7.6) | 49(31.2) | 157 |
| Retr. | 23(16.9) | 20(14.7) | 42(30.9) | 13(9.6) | 38(27.9) | 136 |
| Verif. | 8(8.3) | 14(14.6) | 9(9.4) | 41(42.7) | 24(25.0) | 96 |
| All | 106(17.7) | 87(14.5) | 148(24.7) | 142(23.7) | 117(19.5) | 600 |
Table B2: Agreement between human and automated labels on identical traces.
| Backbone | Human-checked traces | Matching labels | Exact agreement (%) |
|---|---|---|---|
| Gemini 3.8 Flash | 100 | 91 | 91.0 |
| Qwen3-Omni-30B | 100 | 89 | 89.0 |
| Overall | 200 | 180 | 90.0 |
Q1: The purpose of MESA and its relation to existing agents.
A1: Thank you for the question. MESA examines whether organizing and verifying distributed evidence improves answering with the same backbone. Full MESA improves Gemini from 61.9% to 65.6% and Qwen from 50.6% to 52.1%. Removing timelines and derived summaries reduces accuracy to 63.5% and 51.1%. Removing the auxiliary prediction retains 64.8% and 51.6%. These results support the usefulness of evidence organization and targeted inspection within this implementation.
VideoAgent and Deep Video Discovery employ iterative retrieval and inspection, while omni-modal agents incorporate active perception and multimodal evidence search. MESA organizes these operations around model-generated per-stream timelines and an option-by-evidence matrix. We present it as a simple baseline for distributed acoustic and visual evidence search. In the Table 1 (main paper), its improvements on MHop and Manip coexist with decreases on MemA and TrigLoc, indicating that the benefits depend on the task.
Q2: When additional streams help or confuse models.
A2: Thank you for the question. We consider source-specific stream structures and the effect of supplying all available streams for the same benchmark questions.
Source-specific evidence. As reported in Table 12, Gemini achieves 65.4%, 72.1%, and 84.6% across EasyCom groups with two, three, and four streams; Qwen achieves 60.1%, 70.4%, and 74.4%. Across the corresponding STARSS groups, Qwen achieves 49.0%, 50.0%, and 37.4%. An important difference is the evidence supplied by these streams. As described in Table 3, EasyCom provides source-specific microphone observations, whereas STARSS provides directional views extracted from panoramic video. Complementary acoustic observations can help resolve ambiguous events, while more views do not necessarily improve answers. This interpretation is consistent with the observed source-wise patterns, although the groups contain different questions.
Additional streams for the same questions. The FULL/ALL-STREAMS comparison in Table 6 already examines additional inputs on the same questions. FULL supplies the specified streams; ALL-STREAMS also supplies the remaining streams from the source case. As reproduced in Table C below, Gemini increases from 61.9% to 62.5%, while Qwen decreases from 50.6% to 45.6%. Thus, the additional observations help Gemini slightly but reduce Qwen's accuracy by 5.0 percentage points. Their value depends on both the evidence supplied and the backbone's ability to integrate it.
Table C: Overall FULL and ALL-STREAMS accuracy reproduced from Table 6.
| Backbone | FULL (%) | ALL-STREAMS (%) | Change (pp) |
|---|---|---|---|
| Gemini 3.8 Flash | 61.9 | 62.5 | +0.6 |
| Qwen3-Omni-30B | 50.6 | 45.6 | −5.0 |
Q3: Applications and task-dependent model differences.
A3: Thank you for the question. The benchmark evaluates skills relevant to meeting assistance, AR and activity assistance, and multitrack media analysis. Meeting assistance requires linking speech to participants, gestures, and referenced objects. Activity assistance requires identifying the visible cause of a sound that is off-screen in another view. Multitrack media analysis requires associating instrument-specific audio with visual performers. Figures 1 and 2 illustrate why resolving the specific person, object, or action matters beyond recognizing the overall scene.
The model gaps also depend on the task. For example, Table 1 shows that Gemini leads Qwen overall at 61.9% versus 50.6% and on SpkBind at 65.7% versus 56.1%, whereas Qwen leads on XCount at 43.6% versus 38.6%. Nemotron leads the open-weight models overall at 52.0%, yet Qwen achieves higher SpkBind accuracy at 56.1% versus 49.4%. These reversals show why overall rankings do not capture performance on identity binding or counting. These task-specific results characterize strengths relevant to the corresponding application scenarios.
Q5: Model-assisted generation and the human–AI protocol.
A5: Thank you for the question. We use Gemini 3 Flash for event extraction, cross-stream alignment, question planning and writing, and shortcut screening. Planning specifies the supporting streams, evidence hops, and distractors before the question is written. As described in Section 3.3 and Appendices B.2–B.4, screening checks structural validity, no-media answerability under two option orders, muted-input answerability, and isolated-stream answerability.
Human annotators inspect and revise model-generated candidates and also author questions directly. A separate reviewer checks the inclusion criteria, with a third reviewer handling disagreements. As reported in Table 5, 2,391 candidates enter review-stage quality control, 193 are automatically removed, and 50 are removed during human review. The retained pool contains 1,999 model-assisted questions and 149 human-authored controlled variants. Direct human authoring contributes 961 retained questions from 1,032 candidates. The 149 controlled variants and 961 directly authored questions form the 1,110-question human-authored group, giving 3,109 questions overall. Human inspection of the supporting media determines final inclusion.
The analysis in Appendix D.5 also examines a possible construction-model advantage. In Table 11, Gemini 3.8 Flash achieves 59.2% on model-assisted questions and 66.9% on human-authored questions, while Gemini 3 Flash achieves 59.7% on both. Neither model shows an accuracy advantage on the model-assisted subset within these evaluation groups.
We sincerely thank Reviewer weh2 for the constructive feedback. We appreciate the recognition of our task taxonomy, question formats, and explicit cross-stream evidence chains. We provide a comparison of benchmark protocols, supplementary failure diagnostics, and clarifications below.
W1/Q4: Distinction from prior benchmarks and related work.
R1: Thank you for the suggestion. We distinguish related benchmarks by their input structures and evidence requirements. As outlined in Table A, JointAVBench studies strict audio-visual dependency, including reasoning across scenes within a video, and OmniVideoBench filters unimodal shortcuts and provides reasoning traces. EgoExoBench includes synchronous and asynchronous visual relations across viewpoints, while CrossView evaluates multi-camera visual reasoning. X-Stream checks multi-stream necessity, while its Section 3.4 reports that approximately 30% of questions incorporate audio or speech. Its broader taxonomy also includes Single-stream Understanding alongside cross-stream cooperation and reference alignment, as detailed in its Appendix B.2. MS-OmniBench focuses on questions that require acoustic evidence and complementary scene streams jointly.
MS-OmniBench makes acoustic necessity and multi-stream necessity joint inclusion criteria for every retained question. Each item requires evidence that remains insufficient after removing answer-relevant scene audio or restricting the input to any one supplied scene stream. Its evidence chains record which acoustic, visual, and cross-stream associations support the answer. For example, the bike-repair case in Figure 6 uses a click in the egocentric audio to identify the relevant instant, while the tool is visible only in the silent exocentric view. Answering requires connecting the acoustic anchor to complementary visual evidence.
Our empirical checks address this joint requirement. As reported in Table 6, removing scene audio reduces Gemini accuracy from 61.9% to 36.4% and Qwen accuracy from 50.6% to 24.2%; isolated-stream averages fall to 21.4% and 19.7%. To examine the concern that an average can conceal a sufficient individual stream, two independent annotators additionally assess 304 distinct question–stream inputs on which models answer correctly, with gold answers, model predictions, other scene streams, and evidence annotations hidden. Initial agreement is 96.1%, and adjudication identifies no single-stream-sufficient input in this sample. Together with item-level human verification, these results support the intended joint evidence requirement.
MS-OmniBench further provides explicit stream-and-time evidence chains and a diagnostic breakdown of perception, association, and temporal integration errors. These annotations and analyses support studying how models use acoustic cues to connect complementary observations across streams.
Table A: Comparison of published benchmark protocols. The comparison concerns evidence requirements and input structure.