- Rating
- 3
- Confidence
- 4(+1)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces TraceAV-Bench, a benchmark dataset designed to evaluate long-term audio-visual understanding and false premise verification, which employs evidence trails that explicitly annotate temporal positions and source modalities. The dataset comprises 578 video clips and 2,200 multiple-choice questions, covering 4 dimensions and 15 subtasks. On average, each question requires 3.68 evidence jumps and spans 15.1 minutes. The data was generated through processes such as visual caption generation, audiovisual fusion, and trace-driven question generation, and was subsequently filtered automatically and reviewed manually.
Critical strengths
S1. A clearly defined evaluation objective. The evidence trajectoires simultaneously annotates the temporal location and source modality of each jump, which supports a more granular analysis than simply measuring the accuracy of the final answer. Distinguishing between AVR, VR, AR, and MH also avoids treating every question indiscriminately as joint audiovisual reasoning.
S2. The experiments are relatively comprehensive. The Evidence-hop masking reduced the average general-task accuracy of the four models from 43.84% to 28.87% (Mask-1) and 22.02% (Mask-All). The score of an independent text-only model was also close to or below the reported random baseline. Combined with monomodal control experiments and analysis of active methods, this evidence supports the argument that the benchmark dataset contains non-trivial problems that rely on source video.
Critical weaknesses
W1. The description of the actual input configuration for long-video reasoning is insufficient. To my knowledge, current models vary significantly in how they sample video inputs, and the context is insufficient to support the full information contained in such long videos. For long video, I would like to see more details regarding the actual input configurations. These details determine whether low scores result from information loss during the input stage or from failed reasoning after observing the evidence. They also determine whether others can reproduce the reported ranking results.
W2. The criteria for comparing related work are not clearly defined. In Table 1, all previous benchmarks in the “Multi-hop” column are marked with an “×,” but the paper does not define what constitutes “multi-hop.” To my knowledge, OmniVideoBench annotates step-by-step reasoning chains for each question, recording the modality and supporting evidence for each step. This is conceptually similar to the modality-annotated evidence trails proposed in this paper, and the authors themselves cite OmniVideoBench as one of their video data sources. LVOmniBench also includes a “Logic” category, which explicitly requires multi-step, cross-modal reasoning chains. I recall that some previous works have also referred to their tasks as “multi-hop”; I hope the authors will clarify the distinction between the definition of “multi-hop” in this paper and that used in previous works.
Questions for authors
Q1. Can the authors provide a table of actual long-video input configurations categorized by model?
Q2. Can the authors clearly define the “Multi-hop” column in Table 1 and explain why benchmarks such as OmniVideoBench and LVOmniBench do not meet this definition?
Q3. Can the authors report the accuracy of human responses or a small-scale human upper bound?
Q4. Could the authors add a matching control group to the masking analysis—specifically, non-evidence clips with the same total masking duration—and report what percentage of questions can still be answered when only a single step is retained?
5 comments
Response to Q3 (continued):
Human evaluators achieve an average accuracy of 89.50%, substantially exceeding the strongest evaluated OmniLLM (71.41%). Importantly, this high accuracy is achieved solely from the original videos and questions, without access to ground-truth answers or annotated evidence trajectories, demonstrating that the benchmark questions are largely solvable from the source videos alone. Nevertheless, questions requiring long-range evidence localization and multi-hop integration remain challenging even for humans.
The human advantage may stem from their ability to actively search for relevant evidence by seeking through the video timeline, adjusting playback speed, and integrating audio-visual information more effectively. Humans also demonstrate greater robustness on multimodal hallucination tasks, achieving 91.67%, suggesting a stronger ability to distinguish video-grounded evidence from misleading information.
Response to Q4:
Beyond the Mask-1 and Mask-All experiments reported in Appendix B, we conduct four complementary evaluations on five OmniLLMs. (a) Gold Evidence Clips concatenates all audio-visual evidence hops in temporal order, (b) Gold Evidence Text supplies the original captions corresponding to those hops, (c) Non-evidence Masking masks non-evidence segments with the duration-matched black screen and mute, and (d) Single-Hop Only evaluates each audio-visual hop separately, resulting in over 8,000 hop-level scenarios per model.
For the SL (Spatiotemporal Localization) task, we preserve the original source-video timestamps in the Gold Evidence Clips and Single-Hop Only prompts, and explicitly prefix timestamps to the Gold Evidence Text captions. Overall accuracy is summarized in Table 4, with task-level results reported in Tables 5–8.
Table 4. Full Video and four complementary evaluations (overall accuracy, %).
| Model | Full Video | Gold Clips | Gold Text | Non-evidence Mask | Single-Hop Only |
|---|---|---|---|---|---|
| Gemini 3 Flash | 64.64% | 86.00% | 92.45% | 67.91% | 30.82% |
| Qwen3-Omni | 52.91% | 80.00% | 90.27% | 52.45% | 28.85% |
| OmniVinci | 45.00% | 77.91% | 89.18% | 43.82% | 28.26% |
| Gemma 4 | 47.59% | 73.14% | 87.82% | 46.82% | 28.12% |
| Video-SALMONN 2+ | 39.59% | 71.95% | 86.27% | 38.91% | 29.17% |
Table 5. Gold Evidence Clips (accuracy, %).
| Model | IR | TS | ET | FCR | BCR | CMM | SL | SR | VC | SC | ES | BM | Gen. Avg | V2A | A2V | TSF | MH Avg | Overall |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemini 3 Flash | 85.71% | 83.51% | 77.42% | 89.04% | 84.27% | 88.24% | 95.59% | 81.82% | 89.38% | 88.46% | 82.95% | 80.15% | 86.29% | 89.57% | 82.97% | 82.53% | 85.28% | 86.00% |
| Qwen3-Omni | 80.71% | 78.35% | 70.16% | 84.93% | 77.53% | 80.00% | 93.83% | 74.55% | 82.30% | 83.08% | 72.73% | 74.05% | 80.38% | 82.61% | 75.55% | 78.92% | 79.04% | 80.00% |
| OmniVinci | 80.00% | 76.29% | 66.13% | 83.56% | 67.42% | 77.65% | 92.95% | 72.73% | 76.99% | 83.08% | 77.27% | 72.52% | 78.16% | 76.52% | 80.79% | 73.49% | 77.28% | 77.91% |
| Gemma 4 | 77.14% | 62.89% | 71.77% | 82.19% | 68.54% | 71.76% | 92.07% | 66.06% | 71.68% | 73.85% | 70.45% | 74.05% | 74.60% | 72.17% | 72.93% | 60.84% | 69.44% | 73.14% |
| Video-SALMONN 2+ | 71.43% | 58.76% | 62.90% | 82.19% | 76.40% | 68.24% | 93.83% | 71.52% | 79.20% | 63.85% | 75.00% | 68.70% | 74.29% | 64.35% | 67.69% | 66.27% | 66.08% | 71.95% |
Table 6. Gold Evidence Text (accuracy, %).
| Model | IR | TS | ET | FCR | BCR | CMM | SL | SR | VC | SC | ES | BM | Gen. Avg | V2A | A2V | TSF | MH Avg | Overall |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemini 3 Flash | 93.57% | 88.66% | 89.52% | 89.04% | 83.15% | 88.24% | 99.56% | 84.85% | 93.81% | 93.08% | 88.64% | 87.79% | 91.05% | 94.78% | 97.38% | 95.78% | 96.00% | 92.45% |
| Qwen3-Omni | 94.29% | 87.63% | 84.68% | 89.04% | 79.78% | 84.71% | 99.12% | 82.42% | 89.38% | 90.00% | 86.36% | 86.26% | 88.83% | 94.35% | 95.63% | 90.96% | 93.92% | 90.27% |
| OmniVinci | 90.71% | 84.54% | 85.48% | 87.67% | 78.65% | 82.35% | 98.68% | 78.79% | 89.38% | 93.85% | 81.82% | 86.26% | 87.75% | 90.00% | 94.76% | 93.98% | 92.80% | 89.18% |
| Gemma 4 | 90.71% | 86.60% | 79.84% | 86.30% | 73.03% | 81.18% | 99.12% | 73.94% | 89.82% | 82.31% | 82.95% | 83.97% | 85.52% | 90.87% | 97.38% | 92.17% | 93.60% | 87.82% |
| Video-SALMONN 2+ | 88.57% | 82.47% | 79.03% | 78.08% | 74.16% | 80.00% | 99.12% | 70.91% | 90.27% | 73.85% | 77.27% | 85.50% | 83.49% | 92.17% | 96.51% | 90.36% | 93.28% | 86.27% |
Response to Q4 (continued):
Table 7. Non-evidence Masking (accuracy, %).
| Model | IR | TS | ET | FCR | BCR | CMM | SL | SR | VC | SC | ES | BM | Gen. Avg | V2A | A2V | TSF | MH Avg | Overall |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemini 3 Flash | 80.71% | 52.58% | 63.71% | 82.19% | 56.18% | 50.59% | 70.93% | 72.12% | 34.96% | 86.92% | 61.36% | 62.60% | 63.75% | 76.96% | 73.36% | 87.35% | 78.40% | 67.91% |
| Qwen3-Omni | 45.00% | 53.61% | 37.10% | 43.84% | 48.31% | 41.18% | 29.96% | 56.36% | 39.38% | 75.38% | 57.95% | 61.83% | 47.68% | 63.04% | 65.07% | 65.66% | 64.48% | 52.45% |
| OmniVinci | 47.14% | 42.27% | 35.48% | 58.90% | 37.08% | 35.29% | 29.96% | 53.94% | 32.30% | 63.08% | 63.64% | 52.67% | 44.06% | 43.91% | 43.23% | 42.17% | 43.20% | 43.82% |
| Gemma 4 | 41.43% | 29.90% | 36.29% | 35.62% | 30.34% | 35.29% | 15.42% | 54.55% | 33.63% | 50.77% | 53.41% | 57.25% | 38.35% | 75.65% | 71.18% | 53.61% | 68.16% | 46.82% |
| Video-SALMONN 2+ | 42.14% | 40.21% | 25.81% | 27.40% | 28.09% | 29.41% | 31.72% | 47.88% | 38.94% | 49.23% | 48.86% | 46.56% | 38.54% | 44.78% | 37.99% | 35.54% | 39.84% | 38.91% |
Table 8. Single-Hop Only (accuracy, %).
| Model | IR | TS | ET | FCR | BCR | CMM | SL | SR | VC | SC | ES | BM | Gen. Avg | V2A | A2V | TSF | MH Avg | Overall |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemini 3 Flash | 32.59% | 31.01% | 25.05% | 24.23% | 23.25% | 22.91% | 25.68% | 48.83% | 23.65% | 45.69% | 34.33% | 43.57% | 31.52% | 30.76% | 25.43% | 32.34% | 29.21% | 30.82% |
| Qwen3-Omni | 28.86% | 30.49% | 20.42% | 21.15% | 24.84% | 19.50% | 23.99% | 44.88% | 19.06% | 44.26% | 34.33% | 43.98% | 29.10% | 29.31% | 24.89% | 31.48% | 28.27% | 28.85% |
| OmniVinci | 32.09% | 27.13% | 22.93% | 28.63% | 23.57% | 21.05% | 21.92% | 40.93% | 20.94% | 42.82% | 32.67% | 35.48% | 28.37% | 30.28% | 24.57% | 29.77% | 27.99% | 28.26% |
| Gemma 4 | 25.87% | 31.78% | 28.90% | 16.30% | 19.75% | 25.39% | 23.74% | 43.27% | 22.19% | 43.30% | 31.67% | 41.08% | 29.49% | 26.90% | 24.46% | 23.36% | 24.97% | 28.12% |
| Video-SALMONN 2+ | 30.10% | 32.30% | 26.01% | 20.26% | 16.88% | 29.41% | 22.31% | 43.99% | 18.85% | 44.50% | 30.00% | 41.91% | 29.17% | 32.00% | 23.80% | 32.91% | 29.18% | 29.17% |
The Overall columns in Tables 4–8 use question-weighted accuracy; the main Table 1 in the paper reports the macro-average across task categories. The matched Non-evidence Masking control largely preserves Full Video performance (38.91–67.91%), contrasting with the substantial degradation observed under evidence masking in Appendix B. This indicates that the performance drops under Mask-1/Mask-All are primarily associated with removing annotated evidence. Furthermore, Single-Hop Only achieves just 28.12–30.82% accuracy across individual hop scenarios, substantially below Gold Evidence Clips (71.95–86.00%). Together, these results provide complementary support for the importance of integrating evidence distributed across multiple hops.
We thank the reviewer for the thoughtful feedback and constructive suggestions. We address each concern below.
Response to W1 & Q1:
Our evaluation adopts model-specific maximum practical input configurations. Table 1 summarizes the settings.
Table 1. Model-specific audio-visual input configurations.
Over-length Handling. We uniformly sample (Nm) frames across the complete video timeline. If the input exceeds model limits, we progressively reduce image resolution (1280 → 960 → 640 → 480 px on the long edge), followed by frame count (Nm→⌊Nm/2⌋→⌊Nm/4⌋→⋯), always preserving full-timeline coverage. Audio is initially submitted in full where supported. If audio must be reduced, we halve its budget and select 30-second windows distributed across the original timeline, synchronized with the corresponding video segments. The implementation details are available in the
/evaldirectory of our anonymous git repository, and we have updated Appendix H with more detailed evaluation settings to facilitate reproducibility.Response to W2 & Q2:
We thank the reviewer for this insightful comment. We recognize that OmniVideoBench and LVOmniBench are excellent contributions to audio-visual reasoning evaluation.
We emphasize that Multi-hop in Table 1 of the paper refers to multi-hop evidence dependency, rather than merely multi-step reasoning. Specifically, our criterion requires questions to integrate evidence from multiple temporally separated events, with explicit temporal and modality annotations for each hop. Crucially, the evidence hops should provide complementary information needed to determine the answer.
OmniVideoBench provides valuable modality-tagged atomic reasoning steps, including supporting evidence and inference. However, its construction protocol does not explicitly establish the joint necessity of temporally separated evidence hops as a benchmark-wide requirement. LVOmniBench includes challenging multi-step cross-modal reasoning, particularly in its Logical category, but does not specify explicit, timestamped multi-hop evidence trajectories as a benchmark-wide annotation requirement.
In contrast, TraceAV-Bench enforces this multi-hop dependency throughout its construction pipeline, with an average of 3.68 hops spanning 15.1 minutes. Furthermore, our Mask-1/Mask-All and four supplementary experiments (see response to Q4) empirically demonstrate substantial performance degradation when evidence hops are removed or isolated, supporting the necessity of integrating the distributed evidence.
We will explicitly state this operational definition in Table 1 of the paper to distinguish trajectory-grounded multi-hop reasoning from general multi-step reasoning.
Response to Q3:
We conduct an additional human evaluation in which two PhD students independently answer the same 200 questions sampled from TraceAV-Bench. The questions are selected using stratified sampling across task categories and difficulty levels to ensure broad coverage. Their category counts are summarized in Table 2.
Table 2. Human evaluation subset.
We develop a dedicated evaluation interface (see Appendix I), where participants could access only the original videos, questions, and answer options, without ground-truth answers or evidence annotations. The interface supported video seeking, pausing, and playback-speed adjustment. Human and model performance are compared in Table 3.
Table 3. Human performance compared with representative OmniLLMs (accuracy, %).