acceptodds
Under review as a conference paper at ICLR 2027

TraceAV-Bench: Benchmarking Multi-Hop Trajectory Reasoning over Long Audio-Visual Videos

Abstract

Real-world audio-visual understanding requires chaining evidence that is sparse, temporally dispersed, and split across the visual and auditory streams, whereas existing benchmarks largely fail to evaluate this capability. They restrict videos to short clips, isolate modalities, or reduce questions to one-hop perception. We introduce TraceAV-Bench, the first benchmark to jointly evaluate multi-hop reasoning over long audio-visual trajectories and multimodal hallucination robustness. TraceAV-Bench comprises 2,000 rigorously validated multiple-choice questions over 578 long videos, totaling 339.5 hours, spanning 4 evaluation dimensions and 15 sub-tasks. Each question is grounded in an explicit multi-hop evidence trajectory that averages 3.68 hops across a 15.1-minute temporal span. The dataset is built by a three-step pipeline followed by a strict quality assurance process. Evaluation of multiple representative OmniLLMs on TraceAV-Bench reveals that the benchmark poses a persistent challenge across all models, with the strongest closed-source model (Gemini 3.1 Pro) reaching only 68.29% on general tasks, and the best open-source model under direct inference (Ming-Flash-Omni-2.0) reaching 51.70%, leaving substantial headroom. Further analysis shows that current models struggle to gather and combine evidence across long audio-visual videos, with performance varying across modalities and reasoning tasks. We hope TraceAV-Bench will facilitate progress toward reliable long-form audio-visual reasoning in OmniLLMs. The code and data are available at https://anonymous.4open.science/r/TraceAV-Bench-Submission-5FD4.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Reasoning, MLLM Benchmark, Audio, Video

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_U31WOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces TraceAV-Bench, a benchmark dataset designed to evaluate long-term audio-visual understanding and false premise verification, which employs evidence trails that explicitly annotate temporal positions and source modalities. The dataset comprises 578 video clips and 2,200 multiple-choice questions, covering 4 dimensions and 15 subtasks. On average, each question requires 3.68 evidence jumps and spans 15.1 minutes. The data was generated through processes such as visual caption generation, audiovisual fusion, and trace-driven question generation, and was subsequently filtered automatically and reviewed manually.

Critical strengths

S1. A clearly defined evaluation objective. The evidence trajectoires simultaneously annotates the temporal location and source modality of each jump, which supports a more granular analysis than simply measuring the accuracy of the final answer. Distinguishing between AVR, VR, AR, and MH also avoids treating every question indiscriminately as joint audiovisual reasoning.

S2. The experiments are relatively comprehensive. The Evidence-hop masking reduced the average general-task accuracy of the four models from 43.84% to 28.87% (Mask-1) and 22.02% (Mask-All). The score of an independent text-only model was also close to or below the reported random baseline. Combined with monomodal control experiments and analysis of active methods, this evidence supports the argument that the benchmark dataset contains non-trivial problems that rely on source video.

Critical weaknesses

W1. The description of the actual input configuration for long-video reasoning is insufficient. To my knowledge, current models vary significantly in how they sample video inputs, and the context is insufficient to support the full information contained in such long videos. For long video, I would like to see more details regarding the actual input configurations. These details determine whether low scores result from information loss during the input stage or from failed reasoning after observing the evidence. They also determine whether others can reproduce the reported ranking results.

W2. The criteria for comparing related work are not clearly defined. In Table 1, all previous benchmarks in the “Multi-hop” column are marked with an “×,” but the paper does not define what constitutes “multi-hop.” To my knowledge, OmniVideoBench annotates step-by-step reasoning chains for each question, recording the modality and supporting evidence for each step. This is conceptually similar to the modality-annotated evidence trails proposed in this paper, and the authors themselves cite OmniVideoBench as one of their video data sources. LVOmniBench also includes a “Logic” category, which explicitly requires multi-step, cross-modal reasoning chains. I recall that some previous works have also referred to their tasks as “multi-hop”; I hope the authors will clarify the distinction between the definition of “multi-hop” in this paper and that used in previous works.

Questions for authors

Q1. Can the authors provide a table of actual long-video input configurations categorized by model?

Q2. Can the authors clearly define the “Multi-hop” column in Table 1 and explain why benchmarks such as OmniVideoBench and LVOmniBench do not meet this definition?

Q3. Can the authors report the accuracy of human responses or a small-scale human upper bound?

Q4. Could the authors add a matching control group to the masking analysis—specifically, non-evidence clips with the same total masking duration—and report what percentage of questions can still be answered when only a single step is retained?

5 comments

We thank the reviewer for the thoughtful feedback and constructive suggestions. We address each concern below.

Response to W1 & Q1:

Our evaluation adopts model-specific maximum practical input configurations. Table 1 summarizes the settings.

Table 1. Model-specific audio-visual input configurations.

ModelVideo SamplingAudio Input
Ming-Flash-Omni-2.0768 framesModel-native preprocessing
Qwen3-Omni-30B-A3B768 framesModel-native preprocessing
OmniVinci-9B128 framesModel-native preprocessing (≤60 min)
MiniCPM-o 4.564 framesModel-native preprocessing
Qwen2.5-Omni-7B384 framesModel-native preprocessing
Gemma 4-E4B60 framesAudio + video
Video-SALMONN 2+384 framesModel-native preprocessing
HumanOmni-7B32 framesModel-native preprocessing
VideoLLaMA2.1-AV-7B16 framesModel-native preprocessing
Baichuan-Omni-1.5512 framesModel-native preprocessing
Gemini Series1 FPSNative API processing

Over-length Handling. We uniformly sample (NmN_m) frames across the complete video timeline. If the input exceeds model limits, we progressively reduce image resolution (1280 → 960 → 640 → 480 px on the long edge), followed by frame count (Nm→⌊Nm/2⌋→⌊Nm/4⌋→⋯N_m \rightarrow \lfloor N_m/2 \rfloor \rightarrow \lfloor N_m/4 \rfloor \rightarrow \cdots), always preserving full-timeline coverage. Audio is initially submitted in full where supported. If audio must be reduced, we halve its budget and select 30-second windows distributed across the original timeline, synchronized with the corresponding video segments. The implementation details are available in the /eval directory of our anonymous git repository, and we have updated Appendix H with more detailed evaluation settings to facilitate reproducibility.

Response to W2 & Q2:

We thank the reviewer for this insightful comment. We recognize that OmniVideoBench and LVOmniBench are excellent contributions to audio-visual reasoning evaluation.

We emphasize that Multi-hop in Table 1 of the paper refers to multi-hop evidence dependency, rather than merely multi-step reasoning. Specifically, our criterion requires questions to integrate evidence from multiple temporally separated events, with explicit temporal and modality annotations for each hop. Crucially, the evidence hops should provide complementary information needed to determine the answer.

OmniVideoBench provides valuable modality-tagged atomic reasoning steps, including supporting evidence and inference. However, its construction protocol does not explicitly establish the joint necessity of temporally separated evidence hops as a benchmark-wide requirement. LVOmniBench includes challenging multi-step cross-modal reasoning, particularly in its Logical category, but does not specify explicit, timestamped multi-hop evidence trajectories as a benchmark-wide annotation requirement.

In contrast, TraceAV-Bench enforces this multi-hop dependency throughout its construction pipeline, with an average of 3.68 hops spanning 15.1 minutes. Furthermore, our Mask-1/Mask-All and four supplementary experiments (see response to Q4) empirically demonstrate substantial performance degradation when evidence hops are removed or isolated, supporting the necessity of integrating the distributed evidence.

We will explicitly state this operational definition in Table 1 of the paper to distinguish trajectory-grounded multi-hop reasoning from general multi-step reasoning.

Response to Q3:

We conduct an additional human evaluation in which two PhD students independently answer the same 200 questions sampled from TraceAV-Bench. The questions are selected using stratified sampling across task categories and difficulty levels to ensure broad coverage. Their category counts are summarized in Table 2.

Table 2. Human evaluation subset.

CategoryNumber of TasksQuestions
General Reasoning12140
Multimodal Hallucination (MH)360
Total15200

We develop a dedicated evaluation interface (see Appendix I), where participants could access only the original videos, questions, and answer options, without ground-truth answers or evidence annotations. The interface supported video seeking, pausing, and playback-speed adjustment. Human and model performance are compared in Table 3.

Table 3. Human performance compared with representative OmniLLMs (accuracy, %).

EvaluatorGeneralMHOverall
Human 186.43%90.00%87.50%
Human 290.71%93.33%91.50%
Human Average88.57%91.67%89.50%
Gemini 3.1 Pro66.16%84.64%71.41%
Gemini 3 Flash58.92%79.04%64.64%
Ming-Flash-Omni 2.050.16%67.52%55.09%
Qwen3-Omni47.11%67.52%52.91%
Gemma 439.62%67.68%47.59%

Response to Q3 (continued):

Human evaluators achieve an average accuracy of 89.50%, substantially exceeding the strongest evaluated OmniLLM (71.41%). Importantly, this high accuracy is achieved solely from the original videos and questions, without access to ground-truth answers or annotated evidence trajectories, demonstrating that the benchmark questions are largely solvable from the source videos alone. Nevertheless, questions requiring long-range evidence localization and multi-hop integration remain challenging even for humans.

The human advantage may stem from their ability to actively search for relevant evidence by seeking through the video timeline, adjusting playback speed, and integrating audio-visual information more effectively. Humans also demonstrate greater robustness on multimodal hallucination tasks, achieving 91.67%, suggesting a stronger ability to distinguish video-grounded evidence from misleading information.

Response to Q4:

Beyond the Mask-1 and Mask-All experiments reported in Appendix B, we conduct four complementary evaluations on five OmniLLMs. (a) Gold Evidence Clips concatenates all audio-visual evidence hops in temporal order, (b) Gold Evidence Text supplies the original captions corresponding to those hops, (c) Non-evidence Masking masks non-evidence segments with the duration-matched black screen and mute, and (d) Single-Hop Only evaluates each audio-visual hop separately, resulting in over 8,000 hop-level scenarios per model.

For the SL (Spatiotemporal Localization) task, we preserve the original source-video timestamps in the Gold Evidence Clips and Single-Hop Only prompts, and explicitly prefix timestamps to the Gold Evidence Text captions. Overall accuracy is summarized in Table 4, with task-level results reported in Tables 5–8.

Table 4. Full Video and four complementary evaluations (overall accuracy, %).

ModelFull VideoGold ClipsGold TextNon-evidence MaskSingle-Hop Only
Gemini 3 Flash64.64%86.00%92.45%67.91%30.82%
Qwen3-Omni52.91%80.00%90.27%52.45%28.85%
OmniVinci45.00%77.91%89.18%43.82%28.26%
Gemma 447.59%73.14%87.82%46.82%28.12%
Video-SALMONN 2+39.59%71.95%86.27%38.91%29.17%

Table 5. Gold Evidence Clips (accuracy, %).

ModelIRTSETFCRBCRCMMSLSRVCSCESBMGen. AvgV2AA2VTSFMH AvgOverall
Gemini 3 Flash85.71%83.51%77.42%89.04%84.27%88.24%95.59%81.82%89.38%88.46%82.95%80.15%86.29%89.57%82.97%82.53%85.28%86.00%
Qwen3-Omni80.71%78.35%70.16%84.93%77.53%80.00%93.83%74.55%82.30%83.08%72.73%74.05%80.38%82.61%75.55%78.92%79.04%80.00%
OmniVinci80.00%76.29%66.13%83.56%67.42%77.65%92.95%72.73%76.99%83.08%77.27%72.52%78.16%76.52%80.79%73.49%77.28%77.91%
Gemma 477.14%62.89%71.77%82.19%68.54%71.76%92.07%66.06%71.68%73.85%70.45%74.05%74.60%72.17%72.93%60.84%69.44%73.14%
Video-SALMONN 2+71.43%58.76%62.90%82.19%76.40%68.24%93.83%71.52%79.20%63.85%75.00%68.70%74.29%64.35%67.69%66.27%66.08%71.95%

Table 6. Gold Evidence Text (accuracy, %).

ModelIRTSETFCRBCRCMMSLSRVCSCESBMGen. AvgV2AA2VTSFMH AvgOverall
Gemini 3 Flash93.57%88.66%89.52%89.04%83.15%88.24%99.56%84.85%93.81%93.08%88.64%87.79%91.05%94.78%97.38%95.78%96.00%92.45%
Qwen3-Omni94.29%87.63%84.68%89.04%79.78%84.71%99.12%82.42%89.38%90.00%86.36%86.26%88.83%94.35%95.63%90.96%93.92%90.27%
OmniVinci90.71%84.54%85.48%87.67%78.65%82.35%98.68%78.79%89.38%93.85%81.82%86.26%87.75%90.00%94.76%93.98%92.80%89.18%
Gemma 490.71%86.60%79.84%86.30%73.03%81.18%99.12%73.94%89.82%82.31%82.95%83.97%85.52%90.87%97.38%92.17%93.60%87.82%
Video-SALMONN 2+88.57%82.47%79.03%78.08%74.16%80.00%99.12%70.91%90.27%73.85%77.27%85.50%83.49%92.17%96.51%90.36%93.28%86.27%

Response to Q4 (continued):

Table 7. Non-evidence Masking (accuracy, %).

ModelIRTSETFCRBCRCMMSLSRVCSCESBMGen. AvgV2AA2VTSFMH AvgOverall
Gemini 3 Flash80.71%52.58%63.71%82.19%56.18%50.59%70.93%72.12%34.96%86.92%61.36%62.60%63.75%76.96%73.36%87.35%78.40%67.91%
Qwen3-Omni45.00%53.61%37.10%43.84%48.31%41.18%29.96%56.36%39.38%75.38%57.95%61.83%47.68%63.04%65.07%65.66%64.48%52.45%
OmniVinci47.14%42.27%35.48%58.90%37.08%35.29%29.96%53.94%32.30%63.08%63.64%52.67%44.06%43.91%43.23%42.17%43.20%43.82%
Gemma 441.43%29.90%36.29%35.62%30.34%35.29%15.42%54.55%33.63%50.77%53.41%57.25%38.35%75.65%71.18%53.61%68.16%46.82%
Video-SALMONN 2+42.14%40.21%25.81%27.40%28.09%29.41%31.72%47.88%38.94%49.23%48.86%46.56%38.54%44.78%37.99%35.54%39.84%38.91%

Table 8. Single-Hop Only (accuracy, %).

ModelIRTSETFCRBCRCMMSLSRVCSCESBMGen. AvgV2AA2VTSFMH AvgOverall
Gemini 3 Flash32.59%31.01%25.05%24.23%23.25%22.91%25.68%48.83%23.65%45.69%34.33%43.57%31.52%30.76%25.43%32.34%29.21%30.82%
Qwen3-Omni28.86%30.49%20.42%21.15%24.84%19.50%23.99%44.88%19.06%44.26%34.33%43.98%29.10%29.31%24.89%31.48%28.27%28.85%
OmniVinci32.09%27.13%22.93%28.63%23.57%21.05%21.92%40.93%20.94%42.82%32.67%35.48%28.37%30.28%24.57%29.77%27.99%28.26%
Gemma 425.87%31.78%28.90%16.30%19.75%25.39%23.74%43.27%22.19%43.30%31.67%41.08%29.49%26.90%24.46%23.36%24.97%28.12%
Video-SALMONN 2+30.10%32.30%26.01%20.26%16.88%29.41%22.31%43.99%18.85%44.50%30.00%41.91%29.17%32.00%23.80%32.91%29.18%29.17%

The Overall columns in Tables 4–8 use question-weighted accuracy; the main Table 1 in the paper reports the macro-average across task categories. The matched Non-evidence Masking control largely preserves Full Video performance (38.91–67.91%), contrasting with the substantial degradation observed under evidence masking in Appendix B. This indicates that the performance drops under Mask-1/Mask-All are primarily associated with removing annotated evidence. Furthermore, Single-Hop Only achieves just 28.12–30.82% accuracy across individual hop scenarios, substantially below Gold Evidence Clips (71.95–86.00%). Together, these results provide complementary support for the importance of integrating evidence distributed across multiple hops.

Reviewer_KxrsOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TraceAV-Bench, it's a benchmark for multi-hop reasoning over long audio-visual videos. It contains 2200 multiple choice questions over 578 videos, with explicit evidence trajectories. They had an average of 3.68 hops and 15.1 minutes. The benchmark covers many aspects in multimodal reasoning like joint audio-visual reasoning, uni-modality reasoning, and hallucination. Results show a non-trivial gap between current systems and reliable evidence integration.

Critical strengths
  1. The benchmark combines long form AV understanding with explicit multi-hop evidence trajectories. The trajectory annotations make it possible to analyze how performance changes with factors like hop count, temporal span, and evidence position. These make the benchmark more diagnostic than reporting only the overall QA accuracy.

  2. The hallucination tasks test whether evidence from one modality causes models to invent unsupported content in the other. Together with the modality and evidence-masking analyses, this provides a useful guidance of where current models fail to integrate long AV context.

Critical weaknesses
  1. TraceAV-Bench somewhat overlaps with prior works. I found some overlaps with LVOmniBench in long-form AV understanding, VideoMME2 in multi-step reasoning, and ANCHOR in evidence-grounded AV reasoning. Its main niche area is still quite distinct where it combines explicit multi-hop trajectories with hallucination tasks, but I think the paper does not show very clearly that this combination measures a new capability or leads to conclusions that the other benchmarks didn't find. To make the benchmark useful in practice, I would like to see some insights or conclusion for model development that can only be measured using TraceAV-Bench.

  2. Most questions and chain of evidence are generated through a multi-stage LLM pipeline, while humans audit only a sample from each batch. It is therefore unclear whether models need the full annotated trajectory to answer each question, or can rely on partial evidence or option elimination.

Questions for authors
  1. What capability or empirical conclusion does TraceAV-Bench reveal that the prior benchmarks do not?

  2. What fraction of the final items was independently checked end to end by humans against the original video, including every evidence hop? Can you also test whether the questions remain answerable after individual hops are removed, to verify that the complete trajectory is necessary?

5 comments

We thank the reviewer for the thoughtful feedback and constructive suggestions. We address each concern below.

Response to W1 & Q1:

We thank the reviewer for this insightful comment. We acknowledge that LVOmniBench, OmniVideoBench, Video-MME-v2, and ANCHOR are valuable contributions to multimodal LLM reasoning evaluation.

However, TraceAV-Bench differs in its evaluation focus. Table 1 summarizes how its question-level multi-hop requirement compares with the other benchmarks. OmniVideoBench provides modality-tagged reasoning chains. LVOmniBench emphasizes comprehensive long-form audio-visual understanding, including multi-step logical reasoning. Video-MME-v2 evaluates progressive reasoning and consistency across related questions. ANCHOR focuses on grounding model-generated reasoning in perceptual evidence.

Table 1. Multi-hop evidence requirements across benchmarks.

BenchmarkFocusIs multi-hop reasoning required for every question?
LVOmniBenchLong-form AV understanding✗
OmniVideoBenchAV reasoning chains✗
Video-MME-v2Progressive video reasoning✗
ANCHOREvidence-grounded reasoning✗
TraceAV-BenchLong-form AV multi-hop reasoningYes: Each question requires at least two temporally separated evidence hops.

Here, ✗ means that multi-hop reasoning is not required for every question as a benchmark-wide criterion; it does not mean that the benchmark lacks multi-step reasoning tasks or annotations.

As OmniLLMs develop stronger multimodal capabilities, evaluating their ability to reason over long audio-visual videos becomes increasingly important. While many conventional long-video QA tasks can be answered using evidence concentrated at specific moments, real-world understanding often requires connecting events and clues scattered across both time and modalities. Multi-hop reasoning therefore represents a natural yet challenging capability. TraceAV-Bench explicitly models temporally dispersed, modality-tagged evidence dependencies within individual questions, enabling systematic analysis of multi-hop evidence integration and hallucination robustness.

Importantly, this design yields several distinctive empirical findings:

1. Long-form audio-visual reasoning remains challenging. Even the strongest evaluated model, Gemini 3.1 Pro, achieves only 71.41% overall accuracy, while open-source models perform considerably worse, highlighting substantial room for improvement in long-form audio-visual reasoning.

2. Evidence distribution matters. Accuracy decreases with increasing evidence hops, while questions with early evidence are answered 6.91% more accurately on average (see Section 4.2).

3. Agentic reasoning introduces reliability trade-offs. OmniAgent-RL improves General accuracy by 12.91 percentage points but reduces MH accuracy by 10.38 points (see Appendix L).

4. Reasoning and hallucination robustness are not equivalent. General reasoning and MH performance exhibit almost no per-video correlation.

Overall, TraceAV-Bench offers a more fine-grained evaluation of long-form audio-visual reasoning, revealing key limitations of current OmniLLMs and providing insights into future model development.

Response to W2 & Q2:

As detailed in Appendix G, 330 questions (15.00%) underwent independent, source-grounded human auditing by three annotators, covering every annotated evidence hop against the original audio-visual content. The audit verified answer correctness, temporal alignment, modality consistency, and hop necessity, achieving strong inter-annotator agreement (Fleiss' κ = 0.82).

Our original Mask-1 and Mask-All experiments (see Appendix B) evaluate four OmniLLMs: Qwen3-Omni, OmniVinci, Gemma 4, and Video-SALMONN 2+. Under Mask-1, their General accuracy decreases from 48.43% to 29.95%, 47.38% to 33.12%, 40.93% to 28.12%, and 38.63% to 24.27%, respectively. On average, General accuracy drops from 43.84% (Full) to 28.87% (Mask-1) and 22.02% (Mask-All), while MH accuracy decreases from 54.46% to 28.03% and 20.96%. These results demonstrate the importance of individual evidence hops for multi-hop reasoning.

To provide stronger empirical validation, we conduct four complementary experiments on five OmniLLMs: (a) Gold Evidence Clips concatenates all audio-visual evidence hops in temporal order, (b) Gold Evidence Text supplies the original captions corresponding to those hops, (c) Non-evidence Masking masks non-evidence segments with the duration-matched black screen and mute, and (d) Single-Hop Only evaluates each audio-visual hop separately, resulting in over 8,000 hop-level scenarios per model.

For the SL (Spatiotemporal Localization) task, we preserve the original source-video timestamps in the Gold Evidence Clips and Single-Hop Only prompts, and explicitly prefix timestamps to the Gold Evidence Text captions. Overall accuracy is summarized in Table 2, with task-level results reported in Tables 3–6.

Response to W2 & Q2 (continued):

Table 2. Full Video and four complementary evaluations (overall accuracy, %).

ModelFull VideoGold ClipsGold TextNon-evidence MaskSingle-Hop Only
Gemini 3 Flash64.64%86.00%92.45%67.91%30.82%
Qwen3-Omni52.91%80.00%90.27%52.45%28.85%
OmniVinci45.00%77.91%89.18%43.82%28.26%
Gemma 447.59%73.14%87.82%46.82%28.12%
Video-SALMONN 2+39.59%71.95%86.27%38.91%29.17%

Table 3. Gold Evidence Clips (accuracy, %).

ModelIRTSETFCRBCRCMMSLSRVCSCESBMGen. AvgV2AA2VTSFMH AvgOverall
Gemini 3 Flash85.71%83.51%77.42%89.04%84.27%88.24%95.59%81.82%89.38%88.46%82.95%80.15%86.29%89.57%82.97%82.53%85.28%86.00%
Qwen3-Omni80.71%78.35%70.16%84.93%77.53%80.00%93.83%74.55%82.30%83.08%72.73%74.05%80.38%82.61%75.55%78.92%79.04%80.00%
OmniVinci80.00%76.29%66.13%83.56%67.42%77.65%92.95%72.73%76.99%83.08%77.27%72.52%78.16%76.52%80.79%73.49%77.28%77.91%
Gemma 477.14%62.89%71.77%82.19%68.54%71.76%92.07%66.06%71.68%73.85%70.45%74.05%74.60%72.17%72.93%60.84%69.44%73.14%
Video-SALMONN 2+71.43%58.76%62.90%82.19%76.40%68.24%93.83%71.52%79.20%63.85%75.00%68.70%74.29%64.35%67.69%66.27%66.08%71.95%

Table 4. Gold Evidence Text (accuracy, %).

ModelIRTSETFCRBCRCMMSLSRVCSCESBMGen. AvgV2AA2VTSFMH AvgOverall
Gemini 3 Flash93.57%88.66%89.52%89.04%83.15%88.24%99.56%84.85%93.81%93.08%88.64%87.79%91.05%94.78%97.38%95.78%96.00%92.45%
Qwen3-Omni94.29%87.63%84.68%89.04%79.78%84.71%99.12%82.42%89.38%90.00%86.36%86.26%88.83%94.35%95.63%90.96%93.92%90.27%
OmniVinci90.71%84.54%85.48%87.67%78.65%82.35%98.68%78.79%89.38%93.85%81.82%86.26%87.75%90.00%94.76%93.98%92.80%89.18%
Gemma 490.71%86.60%79.84%86.30%73.03%81.18%99.12%73.94%89.82%82.31%82.95%83.97%85.52%90.87%97.38%92.17%93.60%87.82%
Video-SALMONN 2+88.57%82.47%79.03%78.08%74.16%80.00%99.12%70.91%90.27%73.85%77.27%85.50%83.49%92.17%96.51%90.36%93.28%86.27%

Table 5. Non-evidence Masking (accuracy, %).

ModelIRTSETFCRBCRCMMSLSRVCSCESBMGen. AvgV2AA2VTSFMH AvgOverall
Gemini 3 Flash80.71%52.58%63.71%82.19%56.18%50.59%70.93%72.12%34.96%86.92%61.36%62.60%63.75%76.96%73.36%87.35%78.40%67.91%
Qwen3-Omni45.00%53.61%37.10%43.84%48.31%41.18%29.96%56.36%39.38%75.38%57.95%61.83%47.68%63.04%65.07%65.66%64.48%52.45%
OmniVinci47.14%42.27%35.48%58.90%37.08%35.29%29.96%53.94%32.30%63.08%63.64%52.67%44.06%43.91%43.23%42.17%43.20%43.82%
Gemma 441.43%29.90%36.29%35.62%30.34%35.29%15.42%54.55%33.63%50.77%53.41%57.25%38.35%75.65%71.18%53.61%68.16%46.82%
Video-SALMONN 2+42.14%40.21%25.81%27.40%28.09%29.41%31.72%47.88%38.94%49.23%48.86%46.56%38.54%44.78%37.99%35.54%39.84%38.91%

Table 6. Single-Hop Only (accuracy, %).

ModelIRTSETFCRBCRCMMSLSRVCSCESBMGen. AvgV2AA2VTSFMH AvgOverall
Gemini 3 Flash32.59%31.01%25.05%24.23%23.25%22.91%25.68%48.83%23.65%45.69%34.33%43.57%31.52%30.76%25.43%32.34%29.21%30.82%
Qwen3-Omni28.86%30.49%20.42%21.15%24.84%19.50%23.99%44.88%19.06%44.26%34.33%43.98%29.10%29.31%24.89%31.48%28.27%28.85%
OmniVinci32.09%27.13%22.93%28.63%23.57%21.05%21.92%40.93%20.94%42.82%32.67%35.48%28.37%30.28%24.57%29.77%27.99%28.26%
Gemma 425.87%31.78%28.90%16.30%19.75%25.39%23.74%43.27%22.19%43.30%31.67%41.08%29.49%26.90%24.46%23.36%24.97%28.12%
Video-SALMONN 2+30.10%32.30%26.01%20.26%16.88%29.41%22.31%43.99%18.85%44.50%30.00%41.91%29.17%32.00%23.80%32.91%29.18%29.17%

The Overall columns in Tables 2–6 use question-weighted accuracy; the main Table 1 in the paper reports the macro-average across task categories. The results provide complementary evidence supporting our multi-hop design. First, Gold Evidence Clips improve accuracy by 21.36–32.91 percentage points, while Gold Evidence Text further raises performance to 86.27–92.45%, demonstrating the importance of accessing and interpreting the relevant evidence. Second, Non-evidence Masking largely preserves performance, suggesting that the improvements cannot be explained simply by removing arbitrary video content. Third, Single-Hop Only achieves just 28.12–30.82%, substantially below Gold Evidence Clips (71.95–86.00%).

Response to W2 & Q2 (continued):

Together with Mask-1/Mask-All experiments, these results consistently show that isolated evidence hops are generally insufficient and that successful reasoning depends on integrating complementary information distributed across the annotated trajectories.

We further address the possibility of answering questions through textual cues or option elimination. As reported in Appendix J, we evaluated DeepSeek-V4.1-Flash using only questions and answer options, without any audio-visual input. Its accuracy reaches only 14.17% on General tasks and 20.13% on MH tasks, both below the random-guess baselines (20.29% and 20.39%). These results indicate that textual shortcuts and option elimination do not provide a reliable solution strategy across the benchmark.

Reviewer_GeCGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TraceAV-Bench, a benchmark for long-form audio-visual reasoning and multimodal hallucination robustness. It contains 2,200 multiple-choice questions over 578 videos, covering 15 tasks across four dimensions: audio-visual joint reasoning, visual-centric reasoning, audio-centric reasoning, and multimodal hallucination. Each question is accompanied by a timestamped, modality-tagged evidence trajectory, with a reported average of 3.68 hops spanning 15.1 minutes. The benchmark is constructed through visual captioning, audio-visual fusion, and trajectory-based question generation, followed by automated filtering and human auditing. The authors evaluate open-source models and closed-source Gemini models, and provide additional analyses involving evidence masking, text-only shortcuts, representative errors, and agentic inference methods. The strongest evaluated model achieves 68.29% macro-averaged accuracy on the 12 general tasks.

Critical strengths
  1. Combining audio and visual evidence spread across long videos is an important challenge. TraceAV-Bench addresses this setting and provides timestamped evidence trajectories with modality labels, which could be useful for developing and evaluating long-video reasoning systems.
  2. The authors include several complementary analyses, such as evidence-masking experiments, an independent text-only shortcut evaluation, human auditing, and evaluations of agentic methods. These help assess data quality and whether models rely on the annotated evidence.
Critical weaknesses
  1. I randomly checked about 30 of the released questions and found several serious issues. Some questions reveal the answer in their own wording (video244 Q90; video527 Q99), while a question described as multi-hop has only one annotated hop and zero temporal span (video574 Q199). The authors should explain how these items passed filtering and assess how common such problems are. These issues raise concerns about the dataset’s quality.
  2. The error analysis discusses both missing evidence and failed evidence composition, but does not quantify how often each occurs. This makes it difficult to judge whether models mainly struggle to find the evidence or to combine it.
Questions for authors
  1. How common are the data issues identified above, and would correcting them change the reported results?
  2. The paper discusses both missing evidence and failed evidence composition, but does not quantify their relative contributions. How does performance change when models are given the correct evidence clips or verified evidence text instead of the full video?
  3. Could you provide human accuracy on the benchmark, at least on a representative subset? This would help assess how difficult the questions are for people and how large the gap between models and humans is.
5 comments

We thank the reviewer for the thoughtful feedback and constructive suggestions. We address each concern below.

Response to W1 & Q1:

To address these concerns, we conducted a comprehensive re-examination of all 2,200 questions, focusing on question difficulty and evidence trajectory annotations.

1. Question Wording and Difficulty.

1.1 The Highlighted Questions Remain Challenging and Discriminative.

Regarding the two highlighted examples, video244 Q90 is a visual counting question involving a 52.5-minute video and four evidence hops, while video527 Q99 is a speech-context understanding question involving a 10.1-minute video and four evidence hops. The former evaluates the identification and counting of distinct processing stages, while the latter concerns distinguishing the narrator's descriptions of visited cruise destinations from other mentioned locations.

Importantly, both questions are explicitly categorized as Easy in our benchmark. They are consistent with this difficulty designation, while the annotated evidence trajectories capture the temporal information involved.

We further examined the original predictions of seven models on these two questions, as summarized in Table 1:

Table 1. Model predictions on the two questions.

QuestionGemini 3.1 ProGemini 3 FlashMing-Flash-Omni 2.0Qwen3-OmniGemma 4HumanOmniBaichuan-Omni-1.5
video244 Q90✓✓✓✓✗✗✗
video527 Q99✓✓✓✗✗✓✗

Despite their Easy labels, only 4/7 models correctly answer each question. Stronger models (e.g., Gemini 3.1 Pro) generally succeed, whereas several weaker models fail, demonstrating that these examples retain non-trivial model discrimination.

1.2 Comprehensive Dataset-Wide Verification.

To investigate whether similar patterns are prevalent, we additionally screened all 2,200 questions using DeepSeek-V4.1-Flash, followed by manual inspection. We identified 19 potentially flagged questions (0.86%), which are all labeled as Easy. We then compared model accuracy on these questions against the complete Easy subset (410 questions) and the full benchmark (Table 2):

Table 2. Model accuracy on flagged questions, the Easy subset, and the full benchmark.

ModelFlagged (19)Easy (410)Full Benchmark
Gemini 3.1 Pro100.00%87.32%71.41%
Gemini 3 Flash89.47%81.71%64.64%
Ming-Flash-Omni 2.073.68%71.22%55.09%
Qwen3-Omni73.68%72.20%52.91%
Gemma 452.63%61.71%47.59%
HumanOmni42.11%50.49%37.64%
Baichuan-Omni-1.542.11%45.85%29.23%

The Full Benchmark column in Table 2 uses question-weighted accuracy; the main Table 1 in the paper reports the macro-average across task categories. After accounting for question difficulty, the flagged subset does not exhibit a consistent accuracy advantage across models, while the results preserve differences across models.

1.3 Text-Only Evaluation Reveals No Systematic Shortcut Advantage.

Furthermore, in Appendix J, we provide an independent Question-only evaluation using DeepSeek-V4.1-Flash, without access to video or audio; results are reported in Table 3:

Table 3. Question-only accuracy of DeepSeek-V4.1-Flash versus random guessing (%).

MetricDeepSeek-V4.1-FlashRandom Guess
General Avg.14.17%20.29%
MH Avg.20.13%20.39%

The model performs below random guessing on General tasks and approximately at chance level on MH tasks. These results suggest that question-only information does not provide a consistent advantage over random guessing, even for a strong text-only model.

We emphasize that including questions of varying difficulty is an intentional benchmark design choice. Easy questions assess relatively straightforward reasoning operations, but they can still challenge current models when the relevant information is embedded in long audio-visual inputs. These results also demonstrate the benchmark's ability to discriminate among models with different capabilities.

Response to W1 & Q1 (continued):

2. Multi-hop Verification.

We also inspected the evidence trajectories across the entire benchmark and identified six affected cases: video127 Q42, video368 Q122, video568 Q201, video161 Q59, video574 Q199, and video508 Q188.

By tracing the intermediate artifacts in intermediate/step3_agentic_question_generation/event_blocks/ and intermediate/step4_quality_assurance/benchmark_full/, we confirmed that all six questions were originally constructed with multiple evidence hops. The missing hops result from a processing-script issue during evidence-chain aggregation.

Thus, this is a formatting issue rather than a failure of the multi-hop construction. Importantly, the evaluation process of the benchmark supplies models with audio-visual input and multiple-choice questions. Therefore, restoring the omitted hops does not change the model inputs, predictions, or accuracy. We have recovered the complete trajectories from the intermediate artifacts and updated the corresponding annotations.

Response to W2 & Q2:

1. Complementary Experiments.

Beyond the Mask-1 and Mask-All experiments reported in Appendix B, we conduct four complementary evaluations on five OmniLLMs. (a) Gold Evidence Clips concatenates all audio-visual evidence hops in temporal order, (b) Gold Evidence Text supplies the original captions corresponding to those hops, (c) Non-evidence Masking masks non-evidence segments with the duration-matched black screen and mute, and (d) Single-Hop Only evaluates each audio-visual hop separately, resulting in over 8,000 hop-level scenarios per model.

For the SL (Spatiotemporal Localization) task, we preserve the original source-video timestamps in the Gold Evidence Clips and Single-Hop Only prompts, and explicitly prefix timestamps to the Gold Evidence Text captions. Overall accuracy is summarized in Table 4, with task-level results reported in Tables 5–8.

Table 4. Full Video and four complementary evaluations (overall accuracy).

ModelFull VideoGold ClipsGold TextNon-evidence MaskSingle-Hop Only
Gemini 3 Flash64.64%86.00%92.45%67.91%30.82%
Qwen3-Omni52.91%80.00%90.27%52.45%28.85%
OmniVinci45.00%77.91%89.18%43.82%28.26%
Gemma 447.59%73.14%87.82%46.82%28.12%
Video-SALMONN 2+39.59%71.95%86.27%38.91%29.17%

Table 5. Gold Evidence Clips (accuracy, %).

ModelIRTSETFCRBCRCMMSLSRVCSCESBMGen. AvgV2AA2VTSFMH AvgOverall
Gemini 3 Flash85.71%83.51%77.42%89.04%84.27%88.24%95.59%81.82%89.38%88.46%82.95%80.15%86.29%89.57%82.97%82.53%85.28%86.00%
Qwen3-Omni80.71%78.35%70.16%84.93%77.53%80.00%93.83%74.55%82.30%83.08%72.73%74.05%80.38%82.61%75.55%78.92%79.04%80.00%
OmniVinci80.00%76.29%66.13%83.56%67.42%77.65%92.95%72.73%76.99%83.08%77.27%72.52%78.16%76.52%80.79%73.49%77.28%77.91%
Gemma 477.14%62.89%71.77%82.19%68.54%71.76%92.07%66.06%71.68%73.85%70.45%74.05%74.60%72.17%72.93%60.84%69.44%73.14%
Video-SALMONN 2+71.43%58.76%62.90%82.19%76.40%68.24%93.83%71.52%79.20%63.85%75.00%68.70%74.29%64.35%67.69%66.27%66.08%71.95%

Table 6. Gold Evidence Text (accuracy, %).

ModelIRTSETFCRBCRCMMSLSRVCSCESBMGen. AvgV2AA2VTSFMH AvgOverall
Gemini 3 Flash93.57%88.66%89.52%89.04%83.15%88.24%99.56%84.85%93.81%93.08%88.64%87.79%91.05%94.78%97.38%95.78%96.00%92.45%
Qwen3-Omni94.29%87.63%84.68%89.04%79.78%84.71%99.12%82.42%89.38%90.00%86.36%86.26%88.83%94.35%95.63%90.96%93.92%90.27%
OmniVinci90.71%84.54%85.48%87.67%78.65%82.35%98.68%78.79%89.38%93.85%81.82%86.26%87.75%90.00%94.76%93.98%92.80%89.18%
Gemma 490.71%86.60%79.84%86.30%73.03%81.18%99.12%73.94%89.82%82.31%82.95%83.97%85.52%90.87%97.38%92.17%93.60%87.82%
Video-SALMONN 2+88.57%82.47%79.03%78.08%74.16%80.00%99.12%70.91%90.27%73.85%77.27%85.50%83.49%92.17%96.51%90.36%93.28%86.27%

Response to W2 & Q2 (continued):

Table 7. Non-evidence Masking (accuracy, %).

ModelIRTSETFCRBCRCMMSLSRVCSCESBMGen. AvgV2AA2VTSFMH AvgOverall
Gemini 3 Flash80.71%52.58%63.71%82.19%56.18%50.59%70.93%72.12%34.96%86.92%61.36%62.60%63.75%76.96%73.36%87.35%78.40%67.91%
Qwen3-Omni45.00%53.61%37.10%43.84%48.31%41.18%29.96%56.36%39.38%75.38%57.95%61.83%47.68%63.04%65.07%65.66%64.48%52.45%
OmniVinci47.14%42.27%35.48%58.90%37.08%35.29%29.96%53.94%32.30%63.08%63.64%52.67%44.06%43.91%43.23%42.17%43.20%43.82%
Gemma 441.43%29.90%36.29%35.62%30.34%35.29%15.42%54.55%33.63%50.77%53.41%57.25%38.35%75.65%71.18%53.61%68.16%46.82%
Video-SALMONN 2+42.14%40.21%25.81%27.40%28.09%29.41%31.72%47.88%38.94%49.23%48.86%46.56%38.54%44.78%37.99%35.54%39.84%38.91%

Table 8. Single-Hop Only (accuracy, %).

ModelIRTSETFCRBCRCMMSLSRVCSCESBMGen. AvgV2AA2VTSFMH AvgOverall
Gemini 3 Flash32.59%31.01%25.05%24.23%23.25%22.91%25.68%48.83%23.65%45.69%34.33%43.57%31.52%30.76%25.43%32.34%29.21%30.82%
Qwen3-Omni28.86%30.49%20.42%21.15%24.84%19.50%23.99%44.88%19.06%44.26%34.33%43.98%29.10%29.31%24.89%31.48%28.27%28.85%
OmniVinci32.09%27.13%22.93%28.63%23.57%21.05%21.92%40.93%20.94%42.82%32.67%35.48%28.37%30.28%24.57%29.77%27.99%28.26%
Gemma 425.87%31.78%28.90%16.30%19.75%25.39%23.74%43.27%22.19%43.30%31.67%41.08%29.49%26.90%24.46%23.36%24.97%28.12%
Video-SALMONN 2+30.10%32.30%26.01%20.26%16.88%29.41%22.31%43.99%18.85%44.50%30.00%41.91%29.17%32.00%23.80%32.91%29.18%29.17%

The Overall columns in Tables 4–8 use question-weighted accuracy; the main Table 1 in the paper reports the macro-average across task categories. The four experiments reveal several findings. First, Gold Evidence Clips improve accuracy by 21.36–32.91 percentage points, while Gold Evidence Text further raises performance to 86.27–92.45%, indicating substantial challenges in evidence access and audiovisual information extraction. Second, Non-evidence Masking largely preserves the models' performance, suggesting that the gains from Gold Evidence Clips are not simply due to removing arbitrary content. Third, Single-Hop Only achieves just 28.12–30.82%, far below Gold Evidence Clips (71.95–86.00%), highlighting the importance of integrating evidence across multiple hops.

These findings further reinforce the validity of our multi-hop design: successful reasoning requires not only locating relevant evidence across long videos but also interpreting and integrating information distributed across multiple hops.

2. Error Analysis.

Building on these experiments, we further quantify the two error patterns discussed in our original analysis. Table 9 summarizes their contributions. (1) Missing Evidence, where an incorrect prediction becomes correct when Gold Evidence Clips are provided, and (2) Failed Evidence Composition, where the prediction remains incorrect despite access to all annotated evidence clips.

Table 9. Error analysis.

ModelFull ErrorsMissing Evidence ProxyFailed Evidence Composition
Gemini 3 Flash77876.61%23.39%
Qwen3-Omni1,03669.40%30.60%
OmniVinci1,21068.02%31.98%
Gemma 41,15356.98%43.02%
Video-SALMONN 2+1,32959.74%40.26%
Overall5,50665.18%34.82%

Among 5,506 Full Video error instances, 65.18% are corrected by Gold Evidence Clips, indicating that evidence finding and long-context utilization constitute the dominant recoverable bottleneck for current OmniLLMs' multihop audio-visual reasoning capability. The remaining 34.82% persist despite access to all evidence clips, reflecting difficulties potentially involving audio-visual perception and cross-hop composition.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.