acceptodds
Under review as a conference paper at ICLR 2027

EvaOmni: An Evidence-Grounded Benchmark for Audio-Visual Reasoning

11 views

Abstract

Audio-visual reasoning requires models to locate key audio and visual evidence and integrate information across modalities. However, existing benchmarks that rely primarily on multiple-choice questions typically score models solely on answer correctness, making it difficult to assess whether they can identify the evidence supporting their answers. Moreover, textual or unimodal shortcuts may allow models to answer correctly without analyzing the necessary audio-visual evidence, further weakening these benchmarks’ ability to assess audio-visual reasoning. To address these issues, we introduce EvaOmni, an evidence-grounded benchmark comprising 420 videos and 1,200 question-answer pairs across 15 capabilities and 3 levels: retrieval and grounding, basic understanding, and complex reasoning. Experts annotate audio and visual evidence and construct video-grounded distractors; text-only, audio-only, and visual-only checks filter questions that can be answered without both modalities. Models must return both answers and supporting evidence intervals. We propose the Coverage Grounded Score (C.G.) to measure how much of the annotated evidence a model identifies, and the Bottleneck Grounded Score (B.G.) to penalize poor temporal grounding in either modality; both incorporate answer correctness. Across 7 open-weight and 15 proprietary Omni LLMs, the strongest model achieves 80.0% answer accuracy but only 35.8% B.G., compared with 94.3% for human experts. At Level 3, its 79.1% accuracy contrasts with 12.4% B.G. These results show why audio-visual reasoning should be evaluated through both answers and the supporting evidence. We will publicly release EvaOmni to support the continued evolution of evaluation systems and further advance model development.

Primary area:
datasets and benchmarks
Keywords:
EvaOmni: An Evidence-Grounded Benchmark for Audio-Visual Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_uUevOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces EvaOmni, it's an audio-visual reasoning benchmark that evaluates both answer correctness and temporal grounding of the audio and visual evidence. It contains 1,200 questions from 420 videos across three capability levels. The benchmark shows a large gap between answer accuracy and evidence localization. Evaluations from 22 models, modality ablations, and a human performance baseline were done. Together they show that current models often answer correctly without grounding on both modalities.

Critical strengths
  1. Requiring models to localize supporting evidence addresses a real weakness of many video understanding benchmarks. The bottleneck score is interesting because this way strong grounding in one modality cannot hide failure in the other.

  2. The evaluation is relatively comprehensive. It's nice to see the results of 22 different models, ablations with different modality combinations, and human baseline comparisons. The large gap between answer accuracy and grounded performance is a useful empirical finding.

Critical weaknesses
  1. The benchmark has only 1-12 minute videos for audio-visual cues. This narrows the data distribution and limits the broader claim about evidence-grounded reasoning, especially for long real-world videos where evidences might be sparser. This also makes the contribution weaker as comapred to some prior works like Video-MME2, ANCHOR, and LVOmniBench. The task taxonomy is different compared to prior works, but the derived conclusions seem barely distinguishable from them.

  2. The final answer remains multiple choice, so it's possible that the model can select the correct option from partial evidence while separately generating plausible timestamps. The metric shows answer-grounding disagreement, but does not make it firm that the predicted evidence caused the answer.

Questions for authors
  1. Beyond the different task taxonomy, what new empirical conclusion does EvaOmni provide over the prior benchmarks? A direct comparison would help clarify the contribution.

  2. How do you verify that the predicted evidence actually supports the model’s answer, rather than being a plausible timestamp generated separately after selecting an option from partial evidence? An intervention such as masking the predicted evidence could be an idea.

6 comments
AuthorsOpenReview ↗

Response to Reviewer uUev (part 1/2)

We sincerely thank Reviewer uUev for recognizing the thoroughness of our experiments and the value of our findings to the community. Below, we address each of your comments and questions in detail. All corresponding revisions have been incorporated into the updated manuscript (highlighted in blue).


W1 & Q1: Video Duration Design, Comparison to Prior Work, and New Empirical Findings

(1) Rationale for Choosing a 1–12 Minute Duration (Appendix A.1 & B.1)

Selecting a 1–12 minute duration balances native context capacity and second-level annotation precision:

  • Zero Downsampling Distortion: At 1.0 FPS1.0\text{ FPS}, 1–12 minutes yields 60–720 frames, fitting entirely within the native context windows of mainstream Omni LLMs. This avoids the sparse sampling required by ultra-long benchmarks, ensuring fair and precise second-level temporal grounding.
  • High-Precision Bimodal Annotation: Bounding videos to 12 minutes makes rigorous expert annotation and cross-verification of fine-grained bimodal boundaries feasible.
  • Sufficient to Expose Core Bottlenecks: In Level 3 tasks (requiring reasoning across evidence gaps ≥4\ge 4 min and ≥5\ge 5 discrete segments), the top model (Gemini3.7-Flash) achieves only 12.4% B.G. (vs. 93.0% for humans). Thus, the primary bottleneck is fundamental bimodal retrieval and alignment rather than video length.

Nevertheless, we fully acknowledge the significance of evaluating longer videos, and we have incorporated this point into Appendix F.2, Future Work.

(2) Distinctions from Prior and Concurrent Benchmarks (Section2,Tabel1 & Appendix E)

Video-MME-v2 [1] focuses primarily on video-only understanding, while LVOmniBench [2] targets ultra-long audio-video understanding; however, both rely solely on answer accuracy without question-level bimodal timestamps (Table 1). Concurrent work (arXiv:2610.22964v1) ANCHOR [3] evaluates MLLM Chain-of-Thought (CoT) processes; however, its evaluation framework is relatively simplistic, lacks hierarchical task design, and remains limited in scope (few Omni LLMs).

In contrast, EvaOmni applies a rigorous shortcut filtering pipeline that reduces shortcut rates to 1.0%–3.6% (vs. 8.2%–29.4% previously). Crucially, it is the piror benchmark to jointly evaluate answer accuracy, evidence retrieval recall, and localization precision via composite metrics (Acc., C.G., B.G.) across 22 models.

References:

  • [1] Fu et al., Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding, arXiv.
  • [2] Tao et al., LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMs, arXiv.
  • [3] Julin et al., ANCHOR: Audio-Visually Grounded Chain-of-Thought Reasoning Benchmark, arXiv.

(3) New Empirical Findings Enabled by EvaOmni (Secs. 4.3–4.4, Appendix D)

We detail the main findings in Sections 4.3 and 4.4, with supplementary results in Appendix D. Below, we highlight partial key conclusions:

  1. Rising Accuracy vs. Collapsing Grounding in Complex Reasoning (Finding 2, Table 2): From Level 2 to Level 3, answer accuracy increases for 11 of 22 models (e.g., Gemini3.7-Flash Acc. rises from 74.0% to 79.1%), whereas grounding plummets (C.G.: 66.2% →\rightarrow 49.4%; B.G.: 29.9% →\rightarrow 12.4%). Unlike accuracy-only benchmarks that suggest strong reasoning, EvaOmni shows models often eliminate wrong options via partial cues without grounding the full evidence chain.
  2. Enhanced Discriminability & Unmasking Inflated Performance (Finding 1): Accuracy-only benchmarks show narrow or inverted gaps between open-weight and proprietary models (Gemini2.5-Flash vs. Qwen3-Omni-Instruct-30B-A3B: 73.1% vs. 71.9% on Daily-Omni; 52.3% vs. 54.0% on WorldSense; 52.4% vs. 38.4% on OmniVideoBench). On EvaOmni, their Acc. / C.G. / B.G. diverge sharply to 49.5% / 41.8% / 17.5% vs. 25.2% / 4.0% / 0.4%. Qwen3-Omni-Instruct-30B-A3B's C.G. and B.G. drop 21.2 and 24.8 points below its accuracy; this represents a 43.8×43.8\times relative B.G. gap behind Gemini2.5-Flash, exposing severe grounding deficits in small open-weight models.
  3. ASR Cannot Replace Native Audio for Grounding (Finding 4, Table 4): In MiMo-V2.5, Video+Audio outperforms Video+ASR in B.G. by 9.1 points (20.7% vs. 11.6%), proving text transcripts cannot capture non-speech acoustic cues or fine-grained temporal alignment.
  4. The strongest models still lag significantly behind the reference upper bound (Table 2). Leading models' Grounding Scores remain much lower than the reference bounds (71.7/35.8 vs. 96.0/94.3). This reveals substantial room for future model development.
  5. The shortcut rate of EvaOmni is significantly lower than that of previous audio-visual reasoning benchmarks (Figure 1, Table 8). This establishes a new foundation for reliably evaluating the current capabilities of Omni LLMs.
AuthorsOpenReview ↗

Response to Reviewer uUev (part 2/2)

W2 & Q2: Verifying Whether Evidence Supports the Answer and Ruling Out "Partial Evidence + Post-Hoc Timestamp Generation"

We fully agree with your insightful observation: in a multiple-choice question (MCQ) setting, a model could indeed select the correct option using partial evidence and separately generate plausible-looking timestamps; moreover, temporal overlap alone does not logically equate to causal proof (accordingly, we explicitly define C.G. and B.G. as interpretable proxy metrics for evidence-grounded reasoning in the Abstract, Section 3.5, Appendix D.3, and Appendix F.1–F.2). To address your concern, we validate this through both metric mechanism design and causal intervention experiments (Appendix A.6):

(1) The B.G. Metric and Diagnostic Analysis Directly Capture and Penalize "Partial-Evidence Answering + Post-Hoc Timestamp Guessing"

  • If a model selects the correct option using only a single modality or local cues and generates timestamps post-hoc via random guessing or priors, it is extremely difficult to simultaneously and accurately guess the exact start and end boundaries of all ground-truth evidence across both audio and visual modalities in 1–12 minute videos that have undergone rigorous four-stage shortcut filtering.
  • Because our B.G. metric is defined as s×min⁡(IoUaudio,IoUvisual)s \times \min(\mathrm{IoU}_{\mathrm{audio}}, \mathrm{IoU}_{\mathrm{visual}}) , any reliance on partial evidence from a single modality is directly penalized by a low IoU in the other modality. Our diagnostic analysis in Table 5 of the main paper directly quantifies the phenomenon you raised: for Gemini3.7-Flash, out of its 80.0% overall accuracy, 38.2% of all samples (accounting for 47.8% of its correct answers) fall into the "correct answer but ungrounded bimodal evidence (C,¬GC, \neg G)" category. In other words, one of EvaOmni's core contributions is precisely to explicitly isolate and penalize such inflated scores achieved by answering correctly via partial evidence without complete evidence localization.

(2) Sufficiency and Necessity Causal Intervention Experiments (Appendix A.6, Table 6 & Table 7)

To further empirically verify whether the ground-truth evidence intervals overlapped by the model's predicted timestamps causally drive its correct answers, we conducted two complementary causal intervention experiments on Qwen3.8-Omni-Flash and Gemini3.7-Flash in Appendix A.6:

  1. Sufficiency Test (Appendix Table 6): Explicitly providing ground-truth (Oracle) evidence timestamps in the prompt to guide the model's attention to those intervals.
  2. Necessity Test (Appendix Table 7): Replacing the ground-truth annotated evidence intervals in the video with random non-evidence segments of equal duration from the same video (preserving total video duration while removing core evidence content).

The experimental results are as follows:

ModelWithout HintWith Evidence HintΔ\Delta Accuracy
Qwen3.8-Omni-Flash58.3%86.1%+27.8%
Gemini3.7-Flash80.0%90.9%+10.9%

Table 1: Sufficiency test (Answer Accuracy, %).

ModelWithout RemovalWith Equal-Length Random ReplacementΔ\Delta Accuracy
Qwen3.8-Omni-Flash58.3%10.2%−48.1%
Gemini3.7-Flash80.0%11.8%−68.2%

Table 2: Necessity test (Answer Accuracy, %).

Together, these two intervention experiments establish a rigorous causal chain:

  • Ruling Out Answering via Non-Evidence Partial Cues or Commonsense Priors (Necessity): In the Necessity Test, once the annotated evidence intervals are masked/replaced while retaining the rest of the video context, the accuracy of the two models plummets by 48.1% and 68.2%, dropping to 10.2% and 11.8%, respectively. This demonstrates that models cannot select the correct answer merely by relying on dataset priors, text commonsense, or background cues outside the evidence intervals.
  • Establishing Tight Coupling Between Predicted Evidence and Answers (Sufficiency): The substantial gains of +27.8 pp+27.8\text{ pp} and +10.9 pp+10.9\text{ pp} in the Sufficiency Test confirm that the annotated intervals contain the decisive information required to answer the questions. Combined with the Necessity Test, this proves that the annotated evidence intervals are both necessary and sufficient for correct reasoning. Therefore, we can broadly conclude that the evidence annotated in the benchmark is relatively qualified. Notably, Table 2 shows that the Grounded Score shares a similar trend with Accuracy and exhibits strong discriminability (Gemini3.7-Flash 35.8 B.G. vs. Qwen2.5Omni-7B 0.3 B.G.). This indicates that our evaluation system effectively distinguishes model capabilities.
AuthorsOpenReview ↗

Gentle follow-up regarding our rebuttal

Dear Reviewer uUev,

Following up on our rebuttal submitted a couple of days ago, we wanted to kindly check if our detailed response has satisfactorily resolved the concerns you raised.

Thank you very much for your time and continued effort in reviewing our work. We look forward to any further feedback you might have.

Best regards, The Authors

Reviewer_GRvYOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces EvaOmni, an evidence-grounded benchmark for evaluating audio-visual reasoning in omni-modal language models. EvaOmni propose a multimodal benchmark with 1,200 questions across 420 videos forcing the models to answer using multiple modalities. The authors conduct very clear and complete experiments, and the conclusions could be very useful for the community.

Critical strengths
  1. Making sure that a model is not over relying on a modality could be very important to know it’s working properly
  2. Road evaluation across 22 models, with human baselines, modality ablations, and alternative grounding metrics.
Critical weaknesses
  1. The paper fails to properly compare to previous work. SFT 20K already investigate the modality forcing of models for example. NExT-GQA and AVHaystacks already evaluate this for temporal data.
  2. Numerical results from Tab 2 are inconsistent: Gemini2.5-Pro’s level-specific B.G. scores imply an overall score of approximately 35.6% under the stated dataset composition, whereas 29.4% is reported.
  3. Low scores may reflect timestamp prediction or output-format failures alongside reasoning limitations. Failure rates are unreported, and some shortcut evaluations reuse models involved in filtering.
Questions for authors
  1. Could the authors clarify the proximity to the other works tackling it?
  2. It would be better if the authors could provide more clarity over the numerical inconsistencies.
4 comments
AuthorsOpenReview ↗

Response to Reviewer GRvY (part 1/2)

We sincerely thank Reviewer GRvY for recognizing the thoroughness of our experiments and the value of our findings to the community. Below, we address each of your comments and questions in detail. All corresponding revisions have been incorporated into the updated manuscript (highlighted in blue).


W1 & Q1: Comparison to previous work

Thank you for pointing out these relevant works. In addition to our comparisons with omni-modal benchmarks (e.g., Daily-Omni, WorldSense, OmniVideoBench) in Section 2 and Table 1, we have added dedicated citations and a detailed discussion of NExT-GQA [1], SF20K [2], and AVHaystacksQA [3] in Appendix E of the revised manuscript. Specifically, EvaOmni differs from these works in three key aspects regarding modality constraints and temporal grounding:

  1. From Unimodal to Joint Audio-Visual Grounding (vs. NExT-GQA [1]): While NExT-GQA extends VideoQA with temporal grounding to mitigate shortcuts, it strictly focuses on vision-only grounding. In contrast, EvaOmni targets omni-modal reasoning, where answering questions requires integrating cues across both audio and visual streams, and requires models to localize supporting temporal intervals in both modalities simultaneously.
  2. From Answer-Only Scoring to Evidence-Grounded Evaluation (vs. SF20K [2]): Although SF20K investigates story-level understanding and modality necessity in short films, it relies on answer-only evaluation without question-level bimodal evidence interval annotations. Consequently, it cannot verify whether a model's correct answer is genuinely supported by the relevant audio-visual segments. EvaOmni provides expert-annotated audio and visual evidence intervals for all 1,200 questions and introduces the Coverage Grounded Score (C.G.) and Bottleneck Grounded Score (B.G.) to evaluate answer accuracy, evidence retrieval and grounding capability.
  3. Intra-Video Hierarchical Reasoning with Tri-Modal Shortcut Filtering (vs. AVHaystacksQA [3]): AVHaystacksQA focuses on cross-video retrieval and multi-video needle-in-a-haystack reasoning. Conversely, EvaOmni diagnoses fine-grained intra-video audio-visual reasoning, organized into a 3-level, 15-capability taxonomy, and reinforced by video-anchored distractor design and systematic text-only, audio-only, and vision-only shortcut filtering.

References:

  • [1] Xiao et al., Can I Trust Your Answer? Visually Grounded Video Question Answering, CVPR 2024.
  • [2] Ghermi et al., Long Story Short: Story-level Video Understanding from 20K Short Films, IJCV 2026.
  • [3] Chowdhury et al., MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks, NeurIPS 2025.

W2 & Q2: Clarification on numerical results in Table 2

Thank you for catching this! The apparent inconsistency stemmed from two factors during manuscript preparation: (1) a transcription error in the Gemini 2.5 Pro results, and (2) an outdated EvaOmni breakdown in the Appendix prior to the final round of quality-control filtering that had not been updated.

In the finalized EvaOmni benchmark (detailed in Appendix A.7 of the revised manuscript), the distribution of the 1,200 QA pairs across the three levels is: 551 in Level 1 (45.92%), 454 in Level 2 (37.83%), and 195 in Level 3 (16.25%). The verified scores for Gemini 2.5 Pro in Table 2 are:

ModelLevel 1 Acc.Level 1 C.G.Level 1 B.G.Level 2 Acc.Level 2 C.G.Level 2 B.G.Level 3 Acc.Level 3 C.G.Level 3 B.G.Overall Acc.Overall C.G.Overall B.G.
Gemini 2.5 Pro76.473.942.264.754.322.372.646.19.571.462.029.4

Table 1: Verified performance (%) of Gemini 2.5 Pro across capability levels and overall.

Under the finalized dataset composition (N1=551,N2=454,N3=195,Ntotal=1200N_1 = 551, N_2 = 454, N_3 = 195, N_{\text{total}} = 1200), the overall scores align exactly with the weighted averages across levels:

  • Overall Acc.=(551×76.4+454×64.7+195×72.6)/1200=71.36%≈71.4%\text{Overall Acc.} = (551 \times 76.4 + 454 \times 64.7 + 195 \times 72.6) / 1200 = 71.36\% \approx \mathbf{71.4\%}
  • Overall C.G.=(551×73.9+454×54.3+195×46.1)/1200=61.97%≈62.0%\text{Overall C.G.} = (551 \times 73.9 + 454 \times 54.3 + 195 \times 46.1) / 1200 = 61.97\% \approx \mathbf{62.0\%}
  • Overall B.G.=(551×42.2+454×22.3+195×9.5)/1200=29.36%≈29.4%\text{Overall B.G.} = (551 \times 42.2 + 454 \times 22.3 + 195 \times 9.5) / 1200 = 29.36\% \approx \mathbf{29.4\%}

We have updated both Table 2 and Appendix A.7 in the revised manuscript to ensure full consistency.

AuthorsOpenReview ↗

Response to Reviewer GRvY (part 2/2)

W3-a: Analysis of output format failure rates

Thank you for raising this important question. While some smaller open-weight models occasionally deviate from the strict JSON schema due to limited instruction-following capabilities or overly long reasoning traces, we do not automatically penalize minor syntax deviations. As long as a response contains a recoverable answer option and timestamp intervals, we extract them via a fallback parser or manual verification; a response is counted as a parsing failure (scored as 0) only when required fields are missing or truncated beyond recovery. We now report the complete parsing failure rates across all 22 models in Appendix D.5 (Table 10):

MetricQwen2.5-Omni-7BQwen3-Omni-Instruct-30B-A3BLarge Open-Weight (MiMo Series)Gemini3.7-FlashAll 15 Proprietary Models (Range)
Failures (Count )1929618–352418–36
Failure Rate (%)16.0%8.0%1.5%–2.9%2.0%1.5%–3.0%

Table 2: Summary of output parsing failure counts and rates.

While smaller open-weight models exhibit higher parsing failure rates (8.0%–16.0%), all large open-weight models (MiMo series) and proprietary models have very low parsing failure rates (1.5%–3.0%). Thus, formatting errors have a negligible impact on the performance of leading models.


W3-b: Independence of shortcut evaluation from filtering models

We appreciate this rigorous observation. First, we note that in Appendix D.1 (Table 8), each two-model intersection test already includes a model completely disjoint from the Appendix A.4 filtering pipeline (filtering only used Gemini3.1-Pro-Preview and Qwen3-Omni-Instruct-30B-A3B, whereas text-only Qwen3.8-Flash, audio-only Audio-Flamingo-3, and video-only Qwen3VL-Instruct-30B-A3B were never used for filtering).

To completely rule out any circularity concerns, we conducted an additional single-model shortcut evaluation using newly introduced models that are entirely independent of the filtering pipeline (GPT-6.1-Sol for Text-only; Qwen3.8-Omni-Flash for Audio-only and Vision-only):

Modality Ablation Setting (Independent Models)Daily-OmniWorldSenseOmniVideoBenchEvaOmni (Ours)
Text-only (GPT6.1-Sol)48.3%26.7%28.4%8.7%
Audio-only (Qwen3.8-Omni-Flash)63.2%48.4%33.1%19.8%
Vision-only (Qwen3.8-Omni-Flash)46.8%40.6%28.7%14.3%

Table 3: Single-model shortcut accuracy (%).

Across all three settings, EvaOmni exhibits substantially lower shortcut susceptibility than Daily-Omni, WorldSense, and OmniVideoBench (40.2%–84.3% relative reduction, with baseline shortcut rates 1.7x to 5.6x higher than EvaOmni). This confirms that EvaOmni's robustness against shortcuts stems from its intrinsic question design and video-anchored distractors, rather than overfitting to the filtering models.

AuthorsOpenReview ↗

Gentle follow-up regarding our rebuttal

Dear Reviewer GRvY,

Following up on our rebuttal submitted a couple of days ago, we wanted to kindly check if our detailed response has satisfactorily resolved the concerns you raised.

Thank you very much for your time and continued effort in reviewing our work. We look forward to any further feedback you might have.

Best regards, The Authors

AuthorsOpenReview ↗

Gentle follow-up regarding our rebuttal

Dear Reviewer GRvY,

We sincerely appreciate your insightful comments and efforts in reviewing our work. Since the rebuttal phase is drawing to a close within a day, we would like to gently check in to see if there are any further points you would like us to address. Should our rebuttal have resolved all your concerns, we would be grateful if you might consider re-evaluating and potentially raising your score. Thank you once again for your time and consideration.

Warm regards,

The Authors

Reviewer_bbfxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EvaOmni, an evidence-grounded benchmark for audio-visual reasoning in Omni LLMs. The authors argue that existing benchmarks rely mainly on multiple-choice accuracy, which cannot determine whether a correct answer is grounded in audio-visual evidence, and that question design may introduce textual, logical, or unimodal shortcuts. EvaOmni contains 420 videos and 1,200 questions, organized into 3 levels and 15 fine-grained capabilities. Experts annotate separate audio and visual evidence intervals for each question, followed by human cross-validation and model-assisted shortcut screening. At evaluation time, models must output both an answer and modality-specific evidence intervals. The paper proposes two metrics that incorporate answer correctness: C.G. measures coverage of annotated evidence, while B.G. takes the minimum of audio and visual IoU, requiring accurate localization in both modalities.

The authors evaluate 22 open-weight and proprietary models alongside a human baseline. The main findings include the following. Leading models still trail humans substantially in precise localization: the strongest model, Gemini3.7-Flash, achieves 80.0% accuracy, 71.7% C.G., and 35.8% B.G., compared with 96.8%, 96.0%, and 94.3% for humans, while open-weight models with fewer than 100B total parameters lag clearly behind proprietary models in evidence localization. Localization quality also declines at higher levels, whereas accuracy does not decrease monotonically; on Level 3, Gemini3.7-Flash reaches 79.1% accuracy but only 12.4% B.G. Moreover, correct answers and successful localization do not necessarily coincide, as many correct answers fail dual-modal localization and some well-localized predictions still select the wrong answer. The authors conclude that audio-visual reasoning should be evaluated through both answers and supporting evidence, and that precise temporal grounding is the most prominent weakness of current models.

Critical strengths

Strength 1: A valuable shift in evaluation paradigm

The paper systematically analyzes two limitations of existing audio-visual benchmarks: scoring based solely on answer correctness cannot determine whether a correct answer is grounded in audio-visual evidence, and question and option design may introduce textual, logical, or unimodal shortcuts. By intersecting correct predictions across multiple models and runs, the authors quantify how answerable existing benchmarks are under text-only and unimodal inputs, providing empirical support for these concerns. Building on this, the paper requires models to output both an answer and modality-specific evidence intervals, and introduces complementary metrics: C.G. measures evidence coverage, while B.G. takes the minimum across modalities to require accurate localization in both. This paradigm moves evaluation from whether a model answers correctly to whether it identifies the evidence supporting its answer, offering meaningful direction for Omni LLM evaluation.

Strength 2: Systematic and careful benchmark design

EvaOmni is organized into 3 levels and 15 fine-grained capabilities according to reasoning and evidence complexity, each with a clear operational definition. For example, long-range reasoning requires evidence whose start times are at least four minutes apart, and multi-evidence chains require at least five temporally separated pieces of evidence. In data construction, the authors select videos uploaded after January 2025 to mitigate contamination, exclude videos with hard-coded subtitles to prevent OCR from substituting for audio understanding, and require distractors to be drawn from audio-visual content actually present in the video. Together, these choices give the benchmark a clear hierarchical structure and strong diagnostic value.

Strength 3: Comprehensive experiments with diagnostic analyses

The paper evaluates 22 open-weight and proprietary models and provides a human baseline for reference. The analysis goes beyond leaderboard comparisons: the modality ablation shows that native audio outperforms ASR transcripts; the four-way decomposition of answering and grounding outcomes reveals a substantial mismatch between correct answers and successful localization; and the paper further includes fine-grained capability analysis, IoU threshold sensitivity analysis, and cost-efficiency analysis. The authors also state their conclusions with appropriate caution, repeatedly clarifying the scope of their analyses

Critical weaknesses

Limitation 1: Grounded scores do not validly measure evidence-grounded reasoning

B.G. and C.G. only measure temporal agreement between predicted and reference intervals, yet the paper treats them as measures of grounded reasoning. This interpretation is questionable in both directions.

First, low scores need not reflect poor evidence localization. Requiring absolute timestamps couples two distinct abilities: identifying relevant evidence in the content, and mapping it onto the timeline. The latter depends on a model’s temporal encoding and training data rather than on evidence reasoning. The paper neither specifies how models obtain frame timing nor distinguishes these failure types, yet attributes all low scores to poor localization. For instance, Qwen2.5-Omni-7B achieves 16.4% Acc. but only 0.8% on the lenient C.G., which evidence perception alone can hardly explain.

Second, high scores do not show that the evidence was used. Since answers and evidence are produced in a single output, the reported intervals may be post-hoc matches to an already chosen answer. That Level 3 accuracy exceeds Level 2 while B.G. drops sharply further suggests a disconnect between answer selection and evidence use. Appendix D.3 acknowledges that temporal overlap does not establish causal use of evidence, but the main text still presents grounded scores as measures of grounded reasoning, which is an overclaim.

I suggest standardizing temporal information across models, adding an oracle-answer localization experiment, and conducting evidence-removal tests with equal-length random removals as a control.

Limitation 2: Circularity in the shortcut analysis

EvaOmni is filtered with Gemini3.1-Pro-Preview and Qwen3-Omni-30B-A3B under unimodal and text-only conditions (Appendix A.4), and the shortcut comparison in Table 6 uses these same models. EvaOmni’s low shortcut rates (e.g., 1.8% text-only intersection) thus largely follow from adversarial filtering against them. Because the other benchmarks were not filtered this way, the comparison is unfair and does not independently support the claim of reduced shortcuts. The evaluation should be repeated with models not involved in filtering.

Limitation 3: Unverified annotation reliability and a questionable human baseline

B.G. relies on a single set of reference intervals, but annotation consistency is never quantified: the cross-validation in Appendix A.3 makes only qualitative pass/fail judgments and reports no inter-annotator agreement on intervals. Given that temporal boundaries are subjective and an answer may be supported by multiple valid evidence sets, IoU against one reference systematically penalizes other reasonable localizations.

Moreover, the human B.G. of 94.3% (vs. 96.8% Acc.) implies an average weaker-modality IoU of about 0.97 on correct answers, which is unusual for independent participants. The number of participants, questions covered, and their relationship to the annotators are not reported. If true inter-annotator agreement is substantially lower, the reported 58.5-point human–model gap would be considerably overstated. I suggest reporting agreement from double annotation on a subset and providing the full human-baseline protocol.

Limitation 4: No step-level evaluation of reasoning and evidence

Appendix A.2 annotates reasoning chains for Level 3 multi-hop questions (e.g., Steps 1–3 in Fig. 2), yet the evaluation protocol (Appendix C) requires only an answer and question-level evidence intervals, without reasoning steps, step-to-evidence mapping, or dependencies among evidence. C.G. and B.G. therefore cannot tell whether a model correctly relates the evidence: it may localize every interval accurately yet combine them in the wrong order or causal structure. Leaving the annotated chains unused limits Level 3’s diagnostic value for complex reasoning. I suggest structured step-level outputs on Level 3, compared step by step against the annotated chains. For models exposing their reasoning, its correctness on well-localized samples could also be analyzed, noting that such traces may not faithfully reflect the actual decision process.

Questions for authors
  1. Annotation consistency. Was the agreement between independent annotators on evidence intervals measured at any stage (e.g., IoU between independent annotations of the same question)? If so, what is the typical agreement? In addition, how are cases handled where an answer can be supported by multiple valid evidence segments, such as an event that occurs more than once?

  2. Human baseline. How many participants took part, how were the 1,200 questions allocated among them, and what was their background? In particular, did they receive the same guidelines or training as the annotators regarding how evidence boundaries should be marked? Given that the human B.G. implies an average weaker-modality IoU of about 0.97, these details are important for interpreting the reported human–model gap.

  3. Reasoning chain annotations. Are reasoning chains annotated for all Level 3 questions or only for the multi-hop reasoning category? Is there a specific reason they were not incorporated into the evaluation

4.Shortcut evaluation with independent models. Since Gemini3.1-Pro-Preview and Qwen3-Omni-30B-A3B were used both to filter EvaOmni (Appendix A.4) and to compute the shortcut rates in Table 6, have the authors evaluated shortcut susceptibility using models that were not involved in filtering? If so, do EvaOmni’s shortcut rates remain substantially lower than those of the other benchmarks?

6 comments
AuthorsOpenReview ↗

Response to Reviewer bbfx (part 1/2)

We sincerely thank Reviewer bbfx for recognizing our progression toward an evidence-grounded evaluation paradigm, our hierarchical benchmark design, and our diagnostic analyses. Below, we address all four weaknesses (W1–W4) and four questions (Q1–Q4) point by point. All corresponding revisions have been incorporated into the updated manuscript (highlighted in blue).


W1: Grounded scores do not validly measure evidence-grounded reasoning

(1) Standardizing Temporal Input and Disentangling the Causes of Low Grounded Scores

Regarding temporal encoding, all models uniformly take video input sampled at 1 fps and use their officially recommended inference interfaces, with no differential treatment on the evaluation side. Therefore, we believe that temporal encoding mechanisms are part of each model's internal design and should not be a concern of the evaluation benchmark.

We agree with the reviewer that "identifying relevant content" and "mapping it to absolute timestamps" represent two distinct capabilities. Regarding the extreme case of Qwen2.5-Omni-7B (16.4% Acc. vs. 0.8% C.G.), our inspection of its raw predictions revealed that the model failed to output validly formatted JSON timestamp intervals on approximately 16% of the questions; under our scoring protocol (Appendix D.3), such cases receive a Grounding Score of 0. In contrast, stronger models rarely suffer from instruction-following failures (e.g., ~8% for Qwen3-Omni-Instruct-30B-A3B and only ~2% for Gemini3.7-Flash). Therefore, aside from smaller open-weight models being bottlenecked by instruction-following capabilities, low Grounded Scores primarily reflect genuine limitations in cross-modal evidence retrieval and temporal localization. Therefore, we believe that the ability to effectively output standard timestamp formats can also reflect differences in model capabilities. In Appendix D.5, Table 10, we supplement detailed results regarding parsing failures across various models. For proprietary models and some larger open-source models, the impact of format parsing failures is relatively small (failure rate <= 3.0%). For such models, a low Grounded Score still primarily reflects evidence retrieval failure (C.G.) and insufficiently precise grounding (B.G.).

(2) Causal Verification: Sufficiency and Necessity Experiments

We agree that temporal interval overlap alone does not logically prove that a model causally utilized the evidence. We have ensured that both the main text (Abstract, Section 3.5) and the Appendix (D.3, F.1–F.2) explicitly frame C.G. and B.G. as interpretable proxy metrics for "answer correctness with evidence grounding". Representing evidence through a time window is a deliberate trade-off. This approach ensures that the evaluation remains efficient and objective , which is why we adopted this methodology. To a certain extent, it reflects both the quantity of correctly retrieved evidence (C.G.) and the precision of evidence grounding (B.G.).

To empirically test whether model answers genuinely depend on the annotated evidence intervals, we conducted two complementary experiments:

  1. Sufficiency Test (Table 1): We explicitly provide the ground-truth evidence timestamp intervals in the prompt to test whether guiding the model to the annotated intervals improves answer accuracy.
  2. Necessity Test (Table 2): We replace the ground-truth evidence intervals in the video with random non-evidence segments of equal duration from the same video.
ModelWithout HintWith Evidence HintΔ\Delta Accuracy
Qwen3.8-Omni-Flash58.3%86.1%+27.8%
Gemini3.7-Flash80.0%90.9%+10.9%

Table 1: Sufficiency test (Answer Accuracy, %).

ModelWithout RemovalWith Equal-Length Random ReplacementΔ\Delta Accuracy
Qwen3.8-Omni-Flash58.3%10.2%−48.1%
Gemini3.7-Flash80.0%11.8%−68.2%

Table 2: Necessity test (Answer Accuracy, %).

Together, these two sets of experiments establish two key conclusions:

  • Sufficiency: Explicitly providing ground-truth evidence intervals boosts accuracy by +27.8% and +10.9%, respectively, confirming that our annotated intervals indeed capture the decisive information required to answer the questions.
  • Necessity: Replacing the annotated evidence with equal-length random segments causes model accuracy to plummet to 10.2% and 11.8% (drops of −48.1% and −68.2%, respectively). If models relied primarily on dataset priors, general world knowledge, or post-hoc timestamp matching, removing the true evidence would not trigger such a drastic performance collapse. Tables 1 and 2 jointly verify that model performance on EvaOmni is tightly coupled with the annotated evidence.
AuthorsOpenReview ↗

Response to Reviewer bbfx (part 2/2)

W2 & Q4: Circularity in the shortcut analysis

We would like to clarify that in Appendix Table 8, each two-model intersection test already includes one model that is completely disjoint from the filtering pipeline in Appendix A.4 (specifically: Qwen3.8-Flash for text-only, Audio-Flamingo-3 for audio-only, and Qwen3VL-Instruct-30B-A3B for video-only; whereas the filtering pipeline used only Gemini3.1-Pro-Preview and Qwen3-Omni-Instruct-30B-A3B).

To thoroughly eliminate any concerns regarding circularity, we supplemented our evaluation with single-model shortcut experiments using models completely independent of the filtering pipeline (GPT-6.1-Sol for text-only; Qwen3.8-Omni-Flash for audio-only and vision-only).

Evaluation Setting (Independent Models)Daily-OmniWorldSenseOmniVideoBenchEvaOmni (Ours)
Text-only (GPT-6.1-Sol)48.3%26.7%28.4%8.7%
Audio-only (Qwen3.8-Omni-Flash)63.2%48.4%33.1%19.8%
Vision-only (Qwen3.8-Omni-Flash)46.8%40.6%28.7%14.3%

Table 3: Single-model shortcut answerability (%) evaluated using models completely independent of the filtering pipeline.

Across all three settings, EvaOmni's shortcut answerability is 2 to 5 times lower than that of Daily-Omni, WorldSense, and OmniVideoBench. This confirms that EvaOmni's resistance to shortcuts stems from its intrinsic question design and video-anchored distractor mechanisms, rather than adversarial overfitting to the filtering models.


W3, Q1 & Q2: Unverified annotation reliability and a questionable human baseline

(1) Annotation Reliability (Response to Q1)

  • Handling Non-Unique or Repeated Evidence: During our multi-round human cross-verification (Appendix A.3), temporal evidence quality was a core review criterion; any degree of temporal discrepancy triggered a "return for re-annotation" or "sample discard" protocol. Specifically, approximately 22% of the questions did not pass the first round of review, among which ~20% involved inaccurate temporal evidence annotations. After multiple rounds of iterative revision and cross-verification, all questions in the final version passed temporal evidence accuracy checks, and any items with non-unique evidence or disputed boundaries were either revised or discarded.
  • Quantitative Verification of Interval Consistency: After the benchmark was finalized, 5 independent experts who did not participate in the annotation process re-verified a random sample of 100 QA pairs (see Appendix A.5 for details). Among these, only 3 pairs exhibited minor boundary deviations of 1–2 seconds.

(2) Human Baseline (Response to Q2)

As detailed in Appendix B.2, the human baseline evaluation was conducted by 6 independent senior Ph.D. students (with backgrounds in computer science and linguistics) who had no involvement in dataset construction. The 1,200 questions were randomly partitioned into mutually exclusive subsets, with each participant independently completing approximately 200 questions. Prior to testing, participants received the exact same annotation guidelines given to the original annotators. During the test, participants were allowed to pause and replay the videos to pinpoint timestamps accurately. As noted in Appendix B.2, while allowing replay compensates for human working memory limitations, it introduces an asymmetry relative to the single-pass inference of models; consequently, we position the human score as an "upper-bound reference baseline" rather than a strictly parity-matched perceptual comparison with models.


W4 & Q3: No step-level evaluation of reasoning and evidence

Currently, reasoning chain annotations are provided solely for Level 3 multi-hop reasoning and are used exclusively for human quality control rather than formal evaluation, primarily due to two considerations: (1) Objectivity and Reproducibility: Because valid reasoning paths are not unique, introducing a Judge Model for step-level evaluation easily introduces subjective bias, thereby compromising evaluation reproducibility; and (2) Model Compatibility: Not all evaluated models support outputting structured reasoning steps. Nevertheless, we strongly agree with the reviewer that conducting fine-grained "process correctness" analyses for models capable of outputting reasoning traces is highly valuable; accordingly, we have incorporated step-level evaluation as a core direction for future work (Appendix F.2, Future Work).

AuthorsOpenReview ↗

Gentle follow-up regarding our rebuttal

Dear Reviewer bbfx,

Following up on our rebuttal submitted a couple of days ago, we wanted to kindly check if our detailed response has satisfactorily resolved the concerns you raised.

Thank you very much for your time and continued effort in reviewing our work. We look forward to any further feedback you might have.

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary to AC

Dear AC,

Thank you very much for your time and effort in handling our manuscript. As the rebuttal phase draws to a close, we summarize the key points raised by the three reviewers and our corresponding responses below to facilitate a quick overview of the review progress:

Strengths:

  • Reviewer bbfx appreciated our innovative shift in the evaluation paradigm, rigorous benchmark design, and the insightful findings revealed by our large-scale experiments.
  • Reviewer GRvY recognized our benchmark's strict reliance on audio-visual dual-modality information and the overall completeness of our experiments.
  • Reviewer uUev acknowledged that our evidence-driven design effectively addresses the limitations of existing video understanding benchmarks, commended the diagnostic value of our bottleneck metric, and recognized the comprehensiveness of our experiments.

Weaknesses & Questions:

  • [bbfx & uUev] Distinction between evidence-grounded evaluation and evidence-based reasoning: Our benchmark primarily emphasizes the complementarity between evidence and answers to enhance evaluation trustworthiness and interpretability, rather than testing causal reasoning from evidence to answer. Investigating whether models select an answer first and match evidence post-hoc would require per-question analysis of their chain-of-thought, which falls beyond the scope of this work; additionally, we included ablation experiments to validate the sufficiency and necessity of our evidence annotations (see bbfx-W1 and uUev-W2/Q2).
  • [bbfx & GRvY] Circularity in shortcut analysis: The appendix already contains evaluation results independent of the data filtering pipeline; furthermore, we added comparative shortcut analyses across different models to independently demonstrate EvaOmni's advantage in mitigating shortcut biases (see bbfx-W2/Q4 and GRvY-W3).
  • [bbfx] Annotation reliability and human baseline details: We provided a thorough explanation of our complete annotation protocol as well as specific experimental details regarding the human baseline (see bbfx-W3/Q1/Q2).
  • [bbfx] Evaluation of reasoning processes: Out of consideration for objectivity, reproducibility, and broad model compatibility, this work does not currently evaluate intermediate reasoning steps directly. We fully recognize its value and have designated it as a key direction for future extension (see bbfx-W4/Q3).
  • [GRvY & uUev] Core distinctions from prior work: Centered around evidence-grounded evaluation, we established a rigorous annotation pipeline and scoring mechanism to effectively improve the trustworthiness and interpretability of Omni-LLMs in audio-visual reasoning. This perspective has been underexplored in previous studies and represents one of our core contributions (see GRvY-W1/Q1 and uUev-W1/Q1).
  • [GRvY] Updates to numerical results: Transcription errors occurred in the reported results for a few individual models, which have now been thoroughly corrected across both the main text and the appendix of the revised manuscript (see GRvY-W2/Q2).
  • [GRvY] Causes of low model scores and formatting error analysis: We supplemented statistical results on output format parsing. The data shows that while smaller open-source models may exhibit a higher rate of formatting errors, the impact on larger open-source and proprietary models is minimal (see GRvY-W3).
  • [uUev] Selection of video duration range: Considering fine-grained annotation precision and model context window length, we constrained video durations to 1–12 minutes. We attach great importance to evaluating longer-form videos and have outlined this as a future direction (see uUev-W1/Q1).

Thank you again to you and all reviewers for the time and dedication invested in evaluating our work. All aforementioned revisions have been mapped to specific sections and appendix. Should any new feedback or questions arise before the rebuttal deadline, we will respond promptly to address any further concerns.

Best regards,

The EvaOmni Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.