- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces AgentVidBench, a benchmark for evaluating multimodal large language model (MLLM) agents on multi-hop video question answering. It contains 100 questions across 71 videos. Beyond final-answer accuracy, the benchmark uses tool-agnostic evidence milestones and an LLM judge to evaluate evidence acquisition and reasoning in agent execution traces. The paper also compares multiple MLLMs and agent workflows and introduces Video-ReAct as a baseline agent strategy.
Critical strengths
1.The paper extends video-agent evaluation beyond final-answer correctness to intermediate evidence acquisition and use, providing more detailed diagnostic information. The evidence milestones specify what evidence is needed without prescribing particular tools or search sequences. This supports comparisons across different agent workflows and helps identify missing evidence, observation errors, and inconsistencies in reasoning.
2.The dataset construction and scoring procedure include substantial quality-control measures. Questions underwent independent blind review and reconciliation, with high reported inter-annotator agreement. Each question includes 25 distractors, reducing the probability of random guessing. Cross-judge comparisons, adversarial trajectory tests, and human-alignment evaluations provide further support for the reliability of the trajectory scores.
Critical weaknesses
1.The dataset’s size and skill distribution limit its representativeness. It contains only 100 questions across 71 videos, and Figure 3 shows substantial differences in the number of questions associated with different skills: spatial relations and spatial search account for four and seven questions, respectively, while causal reasoning is not reported separately. The statistical tests do not replace an assessment of skill coverage. The paper should clarify which video-understanding abilities the overall score represents.
2.The judge evaluates evidence from textual tool outputs without directly checking the video frames. Although the authors validate the judge’s sensitivity to missing evidence, the paper does not report an independent check of whether tool descriptions match the corresponding video segments. The extent to which the accuracy of the tool observations themselves has been verified therefore remains unclear.
Questions for authors
1.Could the authors clarify the criteria, question counts, and performance breakdowns for reasoning types such as spatial and causal reasoning, to better explain the benchmark’s coverage and the meaning of its overall score?
2.Have the authors independently checked a sample of key tool descriptions to confirm that they are supported by the video segments inspected by the agents? If so, could they report the checking procedure and results?
5 comments
W2, Q2. Evidence grounding and observation validity
A2. To address this concern, we manually check whether tool-generated observations from Video-ReAct with Gemini 2.5 Pro are supported by the corresponding video evidence. Most observations are consistent with the video, while unsupported observations are concentrated in questions where the agent's final answer is incorrect.
We sampled 40 questions stratified by difficulty and randomly selected up to five timestamped observations from each video-analysis tool call, yielding 212 observations across 57 calls. One author checked each observation against the video at the corresponding timestamp. The judge outputs and the correctness of the agent's final answer were not shown to the rater.
Of the 212 observations, 170 (80.2%) were fully supported, 25 (11.8%) were partially supported, 16 (7.5%) were contradicted by or absent from the video at the stated timestamp, and one (0.5%) could not be verified because it required background knowledge that the rater could not independently confirm (i.e., identifying an aircraft model). Partial support means that part of the description, such as an attribute or count, differs from the video, while the remaining parts are supported.
Of the 16 contradicted or absent observations, 15 occurred in questions where the agent's final answer was incorrect. In all 15 cases, the judge assigned an Evidence Grounding score of 0 or 1 out of 2, with the judge's rationale identifying a mismatch between a claimed value and the reference milestones. The remaining observation occurred in a question that the agent answered correctly, for which the judge assigned the maximum Evidence Grounding score (2/2). In this case, the tool correctly described the required retrieval scene but also incorrectly described a later lowering scene as a second retrieval. This additional error did not affect the final answer. This highlights that our milestone-based evaluation focuses on the acquisition and grounding of answer-relevant evidence, rather than the factual accuracy of every observation generated along the trajectory.
We will include this discussion in our paper, and we hope that our responses address your concerns. If you have any further concerns, we would be happy to discuss and address them.
Looking forward to hearing from you
We understand that this is a busy time for everyone. We would be grateful to know whether our response has addressed your concerns. Please feel free to let us know if you have any remaining questions.
Best regards,
Authors of Submission 51677 (AgentVidBench)
We appreciate your thoughtful comments and valuable suggestions, which have helped us improve our paper. We provide our responses to each of your points below.
A1. We acknowledge that AgentVidBench is relatively compact compared with large-scale video QA benchmarks. However, its scale should also be considered in the context of agentic benchmarks, which evaluate carefully curated tasks involving multi-step tool interactions. For example, AgentDojo [1] comprises 97 realistic user tasks, while AgentHarm [2] includes 110 unique agent behaviors. Although these benchmarks target different capabilities and employ different evaluation units, they illustrate that established agentic benchmarks can be built around approximately 100 carefully designed base tasks.
AgentVidBench similarly prioritizes task complexity and evaluation granularity over the number of questions. Its 100 questions span 71 videos, cover 20 fine-grained question-level skills, and draw from 12 video genres organized into four super-categories. More importantly, each question is accompanied by a detailed step-by-step solution trace and tool-agnostic evidence-acquisition milestones. Beyond evaluating final-answer accuracy, these annotations enable assessment of the evidence acquired through tool interactions, its grounding, and its completeness. Thus, AgentVidBench is designed as a targeted benchmark for evaluating multi-hop agentic video understanding, with an emphasis on compositional spatial, temporal, and causal reasoning.
To clarify the benchmark's coverage, we introduce a separate multi-label annotation of all 100 questions using three high-level reasoning types: spatial, temporal, and causal. These reasoning types characterize the broad reasoning demands emphasized in our benchmark and complement, rather than replace, the existing 20 fine-grained skill tags in Figure 3(c). In particular, the three reasoning types do not map one-to-one onto the existing skill tags. For example, the question in Figure 1 is tagged with conditional counting, but solving it also requires distinguishing two people within the frame (spatial), locating the first utterance of "one more" (temporal), and using this utterance to determine whose repetitions should be counted (causal).
Specifically, a question is classified as spatial if it requires inspecting localized regions of the video to identify, distinguish, or examine relevant entities and their spatial relationships. This broader category is not limited to the specific spatial-relation and spatial-search skills in the existing taxonomy, which account for 4 and 7 questions, respectively. A question is classified as temporal if it requires searching for, localizing, ordering, or comparing events across different timestamps. A question is classified as causal if it requires reasoning about dependencies between distinct events, such that one event explains, constrains, or helps disambiguate another event or the relevant entity. A question can involve multiple reasoning types.
Accuracy by reasoning type:
Trajectory scores by reasoning type:
Accuracy and trajectory scores are averaged across the 20 backbone configurations for each setting in Table 1, with questions weighted equally within each reasoning type. The overall scores are calculated as the unweighted means across all 100 questions.
Every question requires temporal reasoning, while 66 also require spatial reasoning and 31 require causal reasoning. Notably, 74 questions require at least two reasoning types, including 23 that require all three. Therefore, the overall scores reflect performance on a mixture of questions involving these reasoning demands, often requiring their combined application within a single question, rather than an isolated assessment of each reasoning type.
[1] Debenedetti et al., "AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents", NeurIPS 2024.
[2] Andriushchenko et al., "AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents", ICLR 2025.