- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces EvaOmni, it's an audio-visual reasoning benchmark that evaluates both answer correctness and temporal grounding of the audio and visual evidence. It contains 1,200 questions from 420 videos across three capability levels. The benchmark shows a large gap between answer accuracy and evidence localization. Evaluations from 22 models, modality ablations, and a human performance baseline were done. Together they show that current models often answer correctly without grounding on both modalities.
Critical strengths
-
Requiring models to localize supporting evidence addresses a real weakness of many video understanding benchmarks. The bottleneck score is interesting because this way strong grounding in one modality cannot hide failure in the other.
-
The evaluation is relatively comprehensive. It's nice to see the results of 22 different models, ablations with different modality combinations, and human baseline comparisons. The large gap between answer accuracy and grounded performance is a useful empirical finding.
Critical weaknesses
-
The benchmark has only 1-12 minute videos for audio-visual cues. This narrows the data distribution and limits the broader claim about evidence-grounded reasoning, especially for long real-world videos where evidences might be sparser. This also makes the contribution weaker as comapred to some prior works like Video-MME2, ANCHOR, and LVOmniBench. The task taxonomy is different compared to prior works, but the derived conclusions seem barely distinguishable from them.
-
The final answer remains multiple choice, so it's possible that the model can select the correct option from partial evidence while separately generating plausible timestamps. The metric shows answer-grounding disagreement, but does not make it firm that the predicted evidence caused the answer.
Questions for authors
-
Beyond the different task taxonomy, what new empirical conclusion does EvaOmni provide over the prior benchmarks? A direct comparison would help clarify the contribution.
-
How do you verify that the predicted evidence actually supports the model’s answer, rather than being a plausible timestamp generated separately after selecting an option from partial evidence? An intervention such as masking the predicted evidence could be an idea.
6 comments
Response to Reviewer uUev (part 2/2)
W2 & Q2: Verifying Whether Evidence Supports the Answer and Ruling Out "Partial Evidence + Post-Hoc Timestamp Generation"
We fully agree with your insightful observation: in a multiple-choice question (MCQ) setting, a model could indeed select the correct option using partial evidence and separately generate plausible-looking timestamps; moreover, temporal overlap alone does not logically equate to causal proof (accordingly, we explicitly define C.G. and B.G. as interpretable proxy metrics for evidence-grounded reasoning in the Abstract, Section 3.5, Appendix D.3, and Appendix F.1–F.2). To address your concern, we validate this through both metric mechanism design and causal intervention experiments (Appendix A.6):
(1) The B.G. Metric and Diagnostic Analysis Directly Capture and Penalize "Partial-Evidence Answering + Post-Hoc Timestamp Guessing"
- If a model selects the correct option using only a single modality or local cues and generates timestamps post-hoc via random guessing or priors, it is extremely difficult to simultaneously and accurately guess the exact start and end boundaries of all ground-truth evidence across both audio and visual modalities in 1–12 minute videos that have undergone rigorous four-stage shortcut filtering.
- Because our B.G. metric is defined as , any reliance on partial evidence from a single modality is directly penalized by a low IoU in the other modality. Our diagnostic analysis in Table 5 of the main paper directly quantifies the phenomenon you raised: for
Gemini3.7-Flash, out of its 80.0% overall accuracy, 38.2% of all samples (accounting for 47.8% of its correct answers) fall into the "correct answer but ungrounded bimodal evidence ()" category. In other words, one of EvaOmni's core contributions is precisely to explicitly isolate and penalize such inflated scores achieved by answering correctly via partial evidence without complete evidence localization.
(2) Sufficiency and Necessity Causal Intervention Experiments (Appendix A.6, Table 6 & Table 7)
To further empirically verify whether the ground-truth evidence intervals overlapped by the model's predicted timestamps causally drive its correct answers, we conducted two complementary causal intervention experiments on Qwen3.8-Omni-Flash and Gemini3.7-Flash in Appendix A.6:
- Sufficiency Test (Appendix Table 6): Explicitly providing ground-truth (Oracle) evidence timestamps in the prompt to guide the model's attention to those intervals.
- Necessity Test (Appendix Table 7): Replacing the ground-truth annotated evidence intervals in the video with random non-evidence segments of equal duration from the same video (preserving total video duration while removing core evidence content).
The experimental results are as follows:
| Model | Without Hint | With Evidence Hint | Accuracy |
|---|---|---|---|
Qwen3.8-Omni-Flash | 58.3% | 86.1% | +27.8% |
Gemini3.7-Flash | 80.0% | 90.9% | +10.9% |
Table 1: Sufficiency test (Answer Accuracy, %).
| Model | Without Removal | With Equal-Length Random Replacement | Accuracy |
|---|---|---|---|
Qwen3.8-Omni-Flash | 58.3% | 10.2% | −48.1% |
Gemini3.7-Flash | 80.0% | 11.8% | −68.2% |
Table 2: Necessity test (Answer Accuracy, %).
Together, these two intervention experiments establish a rigorous causal chain:
- Ruling Out Answering via Non-Evidence Partial Cues or Commonsense Priors (Necessity): In the Necessity Test, once the annotated evidence intervals are masked/replaced while retaining the rest of the video context, the accuracy of the two models plummets by 48.1% and 68.2%, dropping to 10.2% and 11.8%, respectively. This demonstrates that models cannot select the correct answer merely by relying on dataset priors, text commonsense, or background cues outside the evidence intervals.
- Establishing Tight Coupling Between Predicted Evidence and Answers (Sufficiency): The substantial gains of and in the Sufficiency Test confirm that the annotated intervals contain the decisive information required to answer the questions. Combined with the Necessity Test, this proves that the annotated evidence intervals are both necessary and sufficient for correct reasoning. Therefore, we can broadly conclude that the evidence annotated in the benchmark is relatively qualified. Notably, Table 2 shows that the Grounded Score shares a similar trend with Accuracy and exhibits strong discriminability (
Gemini3.7-Flash35.8 B.G. vs.Qwen2.5Omni-7B0.3 B.G.). This indicates that our evaluation system effectively distinguishes model capabilities.
Gentle follow-up regarding our rebuttal
Dear Reviewer uUev,
Following up on our rebuttal submitted a couple of days ago, we wanted to kindly check if our detailed response has satisfactorily resolved the concerns you raised.
Thank you very much for your time and continued effort in reviewing our work. We look forward to any further feedback you might have.
Best regards, The Authors
Response to Reviewer uUev (part 1/2)
We sincerely thank Reviewer uUev for recognizing the thoroughness of our experiments and the value of our findings to the community. Below, we address each of your comments and questions in detail. All corresponding revisions have been incorporated into the updated manuscript (highlighted in blue).
W1 & Q1: Video Duration Design, Comparison to Prior Work, and New Empirical Findings
(1) Rationale for Choosing a 1–12 Minute Duration (Appendix A.1 & B.1)
Selecting a 1–12 minute duration balances native context capacity and second-level annotation precision:
Gemini3.7-Flash) achieves only 12.4% B.G. (vs. 93.0% for humans). Thus, the primary bottleneck is fundamental bimodal retrieval and alignment rather than video length.Nevertheless, we fully acknowledge the significance of evaluating longer videos, and we have incorporated this point into Appendix F.2, Future Work.
(2) Distinctions from Prior and Concurrent Benchmarks (Section2,Tabel1 & Appendix E)
Video-MME-v2 [1] focuses primarily on video-only understanding, while LVOmniBench [2] targets ultra-long audio-video understanding; however, both rely solely on answer accuracy without question-level bimodal timestamps (Table 1). Concurrent work (arXiv:2610.22964v1) ANCHOR [3] evaluates MLLM Chain-of-Thought (CoT) processes; however, its evaluation framework is relatively simplistic, lacks hierarchical task design, and remains limited in scope (few Omni LLMs).
In contrast, EvaOmni applies a rigorous shortcut filtering pipeline that reduces shortcut rates to 1.0%–3.6% (vs. 8.2%–29.4% previously). Crucially, it is the piror benchmark to jointly evaluate answer accuracy, evidence retrieval recall, and localization precision via composite metrics (Acc., C.G., B.G.) across 22 models.
References:
(3) New Empirical Findings Enabled by EvaOmni (Secs. 4.3–4.4, Appendix D)
We detail the main findings in Sections 4.3 and 4.4, with supplementary results in Appendix D. Below, we highlight partial key conclusions:
Gemini3.7-FlashAcc. rises from 74.0% to 79.1%), whereas grounding plummets (C.G.: 66.2% → 49.4%; B.G.: 29.9% → 12.4%). Unlike accuracy-only benchmarks that suggest strong reasoning, EvaOmni shows models often eliminate wrong options via partial cues without grounding the full evidence chain.Gemini2.5-Flashvs.Qwen3-Omni-Instruct-30B-A3B: 73.1% vs. 71.9% on Daily-Omni; 52.3% vs. 54.0% on WorldSense; 52.4% vs. 38.4% on OmniVideoBench). On EvaOmni, their Acc. / C.G. / B.G. diverge sharply to 49.5% / 41.8% / 17.5% vs. 25.2% / 4.0% / 0.4%.Qwen3-Omni-Instruct-30B-A3B's C.G. and B.G. drop 21.2 and 24.8 points below its accuracy; this represents a 43.8× relative B.G. gap behindGemini2.5-Flash, exposing severe grounding deficits in small open-weight models.MiMo-V2.5, Video+Audio outperforms Video+ASR in B.G. by 9.1 points (20.7% vs. 11.6%), proving text transcripts cannot capture non-speech acoustic cues or fine-grained temporal alignment.