acceptodds
Under review as a conference paper at ICLR 2027

AgentVidBench: A Multi-Hop Video Question Answering Benchmark for Evaluating MLLM Agents

Abstract

Comprehensive video understanding is crucial for advancing artificial intelligence toward the intricate dynamics of the physical world. While recent advances in Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in video understanding, existing benchmarks remain confined to simple scene-level queries or global summaries that require only single-step inference. Real-world video understanding involves more challenging tasks that require multi-hop multimodal reasoning, and there is a critical absence of video benchmarks equipped to rigorously evaluate these agentic capabilities. To bridge this gap, we introduce AgentVidBench, a multi-hop video question answering benchmark focused on evaluating the spatial, temporal, and causal reasoning capabilities of MLLM agents. Beyond standard question-answer pairs, AgentVidBench provides step-by-step solution traces to support trajectory evaluation that assesses whether agents explicitly acquire the evidence needed to justify their answers. Experiments with 12 proprietary and open-source MLLMs show that single-turn performance remains limited on AgentVidBench, while integrating these models into state-of-the-art agentic workflows generally improves performance with respect to both accuracy and trajectory scores. We further present a simple yet effective agentic strategy that serves as a competitive baseline on AgentVidBench, establishing our benchmark as a holistic testbed for future research on agentic video understanding. Code and datasets are available at https://anonymous.4open.science/r/agentvidbench-C8B9 and https://huggingface.co/datasets/AgentVidBench123/agentvidbench.

Primary area:
datasets and benchmarks
Keywords:
Video Question Answering, MLLMs, MLLM Agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_AtDQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AgentVidBench, a benchmark for evaluating multimodal large language model (MLLM) agents on multi-hop video question answering. It contains 100 questions across 71 videos. Beyond final-answer accuracy, the benchmark uses tool-agnostic evidence milestones and an LLM judge to evaluate evidence acquisition and reasoning in agent execution traces. The paper also compares multiple MLLMs and agent workflows and introduces Video-ReAct as a baseline agent strategy.

Critical strengths

1.The paper extends video-agent evaluation beyond final-answer correctness to intermediate evidence acquisition and use, providing more detailed diagnostic information. The evidence milestones specify what evidence is needed without prescribing particular tools or search sequences. This supports comparisons across different agent workflows and helps identify missing evidence, observation errors, and inconsistencies in reasoning.

2.The dataset construction and scoring procedure include substantial quality-control measures. Questions underwent independent blind review and reconciliation, with high reported inter-annotator agreement. Each question includes 25 distractors, reducing the probability of random guessing. Cross-judge comparisons, adversarial trajectory tests, and human-alignment evaluations provide further support for the reliability of the trajectory scores.

Critical weaknesses

1.The dataset’s size and skill distribution limit its representativeness. It contains only 100 questions across 71 videos, and Figure 3 shows substantial differences in the number of questions associated with different skills: spatial relations and spatial search account for four and seven questions, respectively, while causal reasoning is not reported separately. The statistical tests do not replace an assessment of skill coverage. The paper should clarify which video-understanding abilities the overall score represents.

2.The judge evaluates evidence from textual tool outputs without directly checking the video frames. Although the authors validate the judge’s sensitivity to missing evidence, the paper does not report an independent check of whether tool descriptions match the corresponding video segments. The extent to which the accuracy of the tool observations themselves has been verified therefore remains unclear.

Questions for authors

1.Could the authors clarify the criteria, question counts, and performance breakdowns for reasoning types such as spatial and causal reasoning, to better explain the benchmark’s coverage and the meaning of its overall score?

2.Have the authors independently checked a sample of key tool descriptions to confirm that they are supported by the video segments inspected by the agents? If so, could they report the checking procedure and results?

5 comments

We appreciate your thoughtful comments and valuable suggestions, which have helped us improve our paper. We provide our responses to each of your points below.

W1, Q1. Dataset representativeness and coverage

A1. We acknowledge that AgentVidBench is relatively compact compared with large-scale video QA benchmarks. However, its scale should also be considered in the context of agentic benchmarks, which evaluate carefully curated tasks involving multi-step tool interactions. For example, AgentDojo [1] comprises 97 realistic user tasks, while AgentHarm [2] includes 110 unique agent behaviors. Although these benchmarks target different capabilities and employ different evaluation units, they illustrate that established agentic benchmarks can be built around approximately 100 carefully designed base tasks.

AgentVidBench similarly prioritizes task complexity and evaluation granularity over the number of questions. Its 100 questions span 71 videos, cover 20 fine-grained question-level skills, and draw from 12 video genres organized into four super-categories. More importantly, each question is accompanied by a detailed step-by-step solution trace and tool-agnostic evidence-acquisition milestones. Beyond evaluating final-answer accuracy, these annotations enable assessment of the evidence acquired through tool interactions, its grounding, and its completeness. Thus, AgentVidBench is designed as a targeted benchmark for evaluating multi-hop agentic video understanding, with an emphasis on compositional spatial, temporal, and causal reasoning.

To clarify the benchmark's coverage, we introduce a separate multi-label annotation of all 100 questions using three high-level reasoning types: spatial, temporal, and causal. These reasoning types characterize the broad reasoning demands emphasized in our benchmark and complement, rather than replace, the existing 20 fine-grained skill tags in Figure 3(c). In particular, the three reasoning types do not map one-to-one onto the existing skill tags. For example, the question in Figure 1 is tagged with conditional counting, but solving it also requires distinguishing two people within the frame (spatial), locating the first utterance of "one more" (temporal), and using this utterance to determine whose repetitions should be counted (causal).

Specifically, a question is classified as spatial if it requires inspecting localized regions of the video to identify, distinguish, or examine relevant entities and their spatial relationships. This broader category is not limited to the specific spatial-relation and spatial-search skills in the existing taxonomy, which account for 4 and 7 questions, respectively. A question is classified as temporal if it requires searching for, localizing, ordering, or comparing events across different timestamps. A question is classified as causal if it requires reasoning about dependencies between distinct events, such that one event explains, constrains, or helps disambiguate another event or the relevant entity. A question can involve multiple reasoning types.

Accuracy by reasoning type:

Reasoning type# QuestionsSingle-TurnSTARDVDAVPVideo-ReAct
Spatial660.220.080.280.400.43
Temporal1000.240.090.290.450.46
Causal310.280.140.340.520.46
All1000.240.090.290.450.46

Trajectory scores by reasoning type:

Reasoning type# QuestionsSingle-TurnSTARDVDAVPVideo-ReAct
Spatial660.430.240.450.640.70
Temporal1000.450.240.460.680.72
Causal310.450.240.460.710.73
All1000.450.240.460.680.72

Accuracy and trajectory scores are averaged across the 20 backbone configurations for each setting in Table 1, with questions weighted equally within each reasoning type. The overall scores are calculated as the unweighted means across all 100 questions.

Every question requires temporal reasoning, while 66 also require spatial reasoning and 31 require causal reasoning. Notably, 74 questions require at least two reasoning types, including 23 that require all three. Therefore, the overall scores reflect performance on a mixture of questions involving these reasoning demands, often requiring their combined application within a single question, rather than an isolated assessment of each reasoning type.

[1] Debenedetti et al., "AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents", NeurIPS 2024.

[2] Andriushchenko et al., "AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents", ICLR 2025.

W2, Q2. Evidence grounding and observation validity

A2. To address this concern, we manually check whether tool-generated observations from Video-ReAct with Gemini 2.5 Pro are supported by the corresponding video evidence. Most observations are consistent with the video, while unsupported observations are concentrated in questions where the agent's final answer is incorrect.

We sampled 40 questions stratified by difficulty and randomly selected up to five timestamped observations from each video-analysis tool call, yielding 212 observations across 57 calls. One author checked each observation against the video at the corresponding timestamp. The judge outputs and the correctness of the agent's final answer were not shown to the rater.

Of the 212 observations, 170 (80.2%) were fully supported, 25 (11.8%) were partially supported, 16 (7.5%) were contradicted by or absent from the video at the stated timestamp, and one (0.5%) could not be verified because it required background knowledge that the rater could not independently confirm (i.e., identifying an aircraft model). Partial support means that part of the description, such as an attribute or count, differs from the video, while the remaining parts are supported.

Of the 16 contradicted or absent observations, 15 occurred in questions where the agent's final answer was incorrect. In all 15 cases, the judge assigned an Evidence Grounding score of 0 or 1 out of 2, with the judge's rationale identifying a mismatch between a claimed value and the reference milestones. The remaining observation occurred in a question that the agent answered correctly, for which the judge assigned the maximum Evidence Grounding score (2/2). In this case, the tool correctly described the required retrieval scene but also incorrectly described a later lowering scene as a second retrieval. This additional error did not affect the final answer. This highlights that our milestone-based evaluation focuses on the acquisition and grounding of answer-relevant evidence, rather than the factual accuracy of every observation generated along the trajectory.

We will include this discussion in our paper, and we hope that our responses address your concerns. If you have any further concerns, we would be happy to discuss and address them.

AuthorsOpenReview ↗

Looking forward to hearing from you

We understand that this is a busy time for everyone. We would be grateful to know whether our response has addressed your concerns. Please feel free to let us know if you have any remaining questions.

Best regards,

Authors of Submission 51677 (AgentVidBench)

Reviewer_mfLROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces AgentVidBench, designed to evaluate the reasoning capabilities of multimodal large language models (MLLMs) and their agents in complex video scenarios, with a focus on multi-hop reasoning and active evidence acquisition. Unlike traditional Video QA benchmarks that evaluate solely based on final answers, this dataset comprises 100 multiple-choice questions derived from 71 videos, and for each question, human-curated reasoning notes and tool-agnostic evidence milestones are constructed. The paper further design an LLM Judge-based trajectory evaluation method that assesses the reasoning process of agents across five dimensions: task understanding, evidence coverage, evidence grounding, evidence completeness, and reasoning faithfulness. By comparing multiple MLLMs with frameworks such as STAR, DVD, AVP, and the authors' proposed Video-ReAct, the paper finds that active video exploration generally improves performance on complex video question answering. Overall, this work seeks to establish an agent-centric video reasoning benchmark that simultaneously measures both the correctness of answers and the quality of the reasoning process.

Critical strengths
  1. The work extends evaluation from final results to the evidence-acquisition process, and the research motivation is sound. A correct final answer does not necessarily mean the model actually observed and integrated the necessary video evidence. This issue is particularly important for video agents. The paper adopts tool-agnostic evidence milestones, decoupling the evidence to be acquired from the specific tool operations used to obtain it. This design can, in principle, avoid trajectory-matching biases caused by differing tool APIs or search paths across agents.

  2. The dataset construction involves clear human quality control, and the data annotation process is more rigorous than approaches relying solely on automatically generated questions.

  3. The paper tests multiple model families and different types of video-agent frameworks, reporting answer accuracy, trajectory scores, ablation results, and distributions of error types. These evaluations carry a certain diagnostic value.

Critical weaknesses
  1. Prior work [1] has already proposed a complex multi-step video reasoning dataset, human-written reasoning trajectories, and process-level reasoning evaluation. The design of AgentVidBench appears somewhat similar, although it places greater emphasis on real tool-call trajectories, evidence milestones, and evaluation fairness across tool interfaces. However, whether the newly added milestone-based evaluation constitutes a substantive methodological contribution over MINERVA sufficient to support a new benchmark paper remains unclear.

  2. The independent measurement validity of the trajectory score is not fully verified. The trajectory score relies on milestones abstracted from human solutions and on an LLM Judge performing semantic assessment of model trajectories. This raises a critical question: does the metric actually measure genuine evidence acquisition, or does it merely measure semantic consistency between the generated text and the reference evidence?

  3. How can it be verified that the 100 questions necessarily require dependent multi-hop reasoning, rather than merely information aggregation across temporal segments? Please provide an annotated definition and distribution of the actual number of reasoning hops, and distinguish between questions whose evidence can be acquired in parallel versus those that must be acquired sequentially.

[1]. Minerva: Evaluating complex video reasoning[C]//2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025

Questions for authors
  1. When final answer correctness is controlled for, does the Trajectory Score still provide information significantly correlated with independent human judgments of evidence authenticity? Can the Trajectory Score truly determine whether an agent actually acquired video evidence, rather than merely determining whether the agent wrote out the correct evidence—for instance, an agent might fabricate the evidence-acquisition process without genuinely obtaining that evidence through tools?

  2. If the external vision model, number of tool calls, input token budget, and total inference cost are all fixed, and Video-ReAct, AVP, and DVD are re-compared under strictly equal budgets, does Video-ReAct still retain a statistically significant advantage?

  3. To what extent do milestones depend on a specific annotator or a specific human solution path? Do independently created milestones from different annotators yield consistent trajectory scores? Do system rankings remain stable across different but equally valid evidence decompositions?

6 comments

We appreciate your thoughtful comments and valuable suggestions, which have helped us improve our paper. We provide our responses to each of your points below.

W1. Methodological novelty

A1. We clarify that AgentVidBench uniquely evaluates active, tool-aware agent trajectories, distinguishing it from existing benchmarks. While MINERVA is restricted to the static reasoning of passive, single-pass MLLMs, AgentVidBench specifically measures an agent's ability to dynamically invoke tools and explicitly gather evidence using tool-agnostic evidence milestones.

Our controlled experiments demonstrate that MINERVA's evaluation metric (MiRA) is unsuitable for diverse agentic workflows, as it unfairly penalizes valid reasoning trajectories simply for utilizing different tool log formats (e.g., frame indices instead of MM:SS timestamps). In contrast, our tool-agnostic milestone judge robustly evaluates the actual evidence acquired regardless of the specific tool interface, while still strictly penalizing genuine evidence omission. This empirical robustness proves that AgentVidBench provides a fairer and more accurate evaluation protocol for assessing MLLM agents. We note that this comparison of AgentVidBench with existing benchmarks is provided in Appendix A.

W2, Q1. Trajectory score validity and evidence authenticity

A2. To address concerns that Traj may only measure semantic consistency with the reference, we conduct a counterfactual analysis on 58 correctly answered Video-ReAct trajectories and 56 correctly answered AVP trajectories generated with Gemini 2.5 Pro. For each trajectory, we keep the reasoning text and final answer unchanged while either removing the tool outputs or replacing them with outputs from another video. This allows us to isolate the effect of the recorded evidence on Traj.

The original judge does not reliably penalize claims that lack corresponding tool outputs. To address this limitation, we introduce an additional grounding check that, for tool-using methods, credits only observations supported by the recorded tool outputs. Applying this check changes the Traj scores of the four methods evaluated with Gemini 2.5 Pro in Table 1 by at most 0.01. We report Traj with and without this check, using the paper's original judge prompt with Gemini 3.1 Pro as the judge model.

Counterfactual setting (correct answers retained)Video-ReActVideo-ReActAVPAVP
Paper judge+ Grounding checkPaper judge+ Grounding check
Original0.970.960.970.97
Tool outputs removed0.900.240.960.25
Tool outputs from another video0.270.240.380.22

These results reveal that the original judge penalizes inconsistent tool outputs from another video but is insufficiently sensitive to missing tool outputs. The additional grounding check addresses this limitation by explicitly verifying whether claimed observations are supported by the recorded tool outputs. Consequently, even when the written reasoning and final answer remain unchanged, trajectories with missing or mismatched evidence receive substantially lower scores. An agent therefore cannot receive credit merely by describing an evidence-acquisition process that is not supported by the tool outputs logged by the evaluation harness.

To further assess whether the recorded tool outputs reflect the actual video content, one author, blinded to the judge outputs and answer correctness, manually checked 212 tool observations from 40 Video-ReAct questions against the video at the corresponding timestamps. We found that 80.2% of the observations were fully supported and another 11.8% were partially supported. Among correctly answered trajectories, Traj also shows a modest but statistically significant positive association with the fraction of observations independently confirmed against the video by a separate MLLM verifier (Spearman's ρ = 0.31, p = 0.001).

Finally, we extend the human-alignment study in Appendix D.4 to all 100 questions and the five methods evaluated with Gemini 2.5 Pro. Two raters, blinded to the system identities, independently scored the trajectories using the judge's evaluation rubric and inputs. After controlling for answer correctness, the human-assigned and judge-assigned trajectory scores remain positively correlated (partial correlation = 0.64, 95% CI: [0.56, 0.70], permutation p < 0.001).

W3. Multi-hop reasoning validity

A3. We agree that dependent reasoning and aggregation across temporal segments should be distinguished, and we annotated all 100 questions accordingly. 30 questions are dependent, so their evidence must be acquired sequentially, and 70 are independent, so their evidence can be acquired in parallel.

(1) Section 3.1 defines a hop as "a single dependent reasoning step or evidence-update operation". We use the milestone as the unit of a hop, because a milestone is "an essential evidence-acquisition or reasoning step required to justify the correct answer". For reporting hop counts, we count each required evidence-acquisition or reasoning milestone as one hop. This counts required steps, rather than sequential dependency depth. Table 8 gives the distribution: 3 milestones for 14 questions, 4 for 32, 5 for 40, 6 for 12, and 7 for 2 (mean 4.56). The milestones are ordered as reasoning steps, but their evidence does not always have to be acquired in that order. (2) separates these two cases.

(2) A question is dependent if a result obtained at one moment of the video decides which evidence is relevant at another moment, so the later evidence cannot be identified from the question alone. Typical cases are identifying a person in one scene and finding that person's name in another, and counting events up to a moment that must first be located. A question is independent if every piece of evidence is specified by the question itself, as in counting all occurrences of an event that the question describes. ​​Here, parallel means that the required evidence targets can be specified independently from the question, not that tool calls execute concurrently.

CategoryEvidence acquisition#Questions
DependentSequential30
IndependentParallel70

We assigned the labels from the question, the human-written reasoning note, and the milestones. An author checked the 39 dependent or borderline questions against the video.

(3) The independent questions follow from the design stated in Section 3.2, which asks for "revisiting specific moments or aggregating evidence across distant timestamps". The aggregation in these questions is exhaustive: in 46 of the 70, the answer is determined only after the whole relevant scope has been checked, and the paper evaluates this requirement as a separate dimension, Evidence Completeness (ECom). In the revision, we will revise the phrase "multiple steps of dependent reasoning" in Section 3.1 to state that the benchmark contains both dependent and aggregation questions, report the split, and release the per-question labels.

Q2. Fair comparison and budget control

A4. We acknowledge that strictly matching the actual number of tool calls, input tokens, and inference costs across agentic frameworks is challenging because each agent dynamically determines when and how often to invoke tools and when to terminate. To ensure a controlled comparison, we use Gemini 2.5 Pro as both the planning/answering model and the external vision model for all three methods, and impose the same tool-call limit, which is sufficiently high to avoid truncating exploration. While the actual resource consumption still varies across methods, we additionally evaluate a low-resolution variant of Video-ReAct to assess its performance under a reduced inference budget.

MethodAccVideo calls / questionInput tokens / questionInference cost / question
Video-ReAct0.521.6138k$0.22
Video-ReAct (low resolution)0.491.773k$0.14
AVP0.481.095k$0.18
DVD0.302.185k$0.22

Video-ReAct achieves higher accuracy than both AVP and DVD under the same planning and vision models. Notably, the low-resolution Video-ReAct variant achieves comparable accuracy to AVP (0.49 vs 0.48) while consuming approximately 23% fewer input tokens and 22% less inference cost. It also outperforms DVD while using fewer video calls, input tokens, and inference costs. These results suggest that Video-ReAct provides a favorable accuracy-efficiency trade-off, although the actual resource consumption is not strictly identical across methods.

Q3. Milestone robustness and annotation dependency

A5. We clarify that milestones capture the evidence necessary to answer a question rather than a specific solution path. Therefore, the extracted milestones remain largely consistent across different versions of the solution notes. Specifically, none of the milestones prescribes a particular tool or search order, and the judge only checks whether the evidence specified by each milestone has been acquired. This design is intentional, as multiple valid solutions can acquire the same underlying evidence using different tools and in different orders.

To examine the robustness of milestone extraction, we compare the milestones extracted from two different versions of the solution notes. We find that 89% of the original milestones also appear in the alternative version, while the average number of milestones per question remains similar (4.6 vs 4.1). These minor differences have minimal effects on the trajectory scores of the five methods, with changes of at most 0.03, as shown in the table below.

MethodsOriginal notes (Traj.)Alternative notes (Traj.)
Single-Turn0.780.78
STAR0.360.34
DVD0.470.50
AVP0.820.82
Video-ReAct0.840.82

We will include this discussion in our paper, and we hope that our responses address your concerns. If you have any further concerns, we would be happy to discuss and address them.

Reviewer_qpAXOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

AgentVidBench addresses a gap in video benchmarks for MLLM agents, which are often evaluated on single-step questions or global summaries rather than on active, multi-step evidence gathering. It introduces 100 26-way multiple-choice questions over 71 public videos, spanning 12 fine-grained genres and requiring spatial, temporal, causal, and other compositional evidence; the questions are intended not to be solvable from a single frame or localized timestamp. Each question is paired with 3–7 tool-agnostic evidence milestones derived from human solution traces. An evaluated agent produces both a final answer and a tool-use trajectory, which are scored separately by answer accuracy and an LLM-based trajectory evaluator covering task understanding, evidence coverage, evidence grounding, evidence completeness, and reasoning faithfulness. Experiments with 12 proprietary and open-source MLLMs and several agentic workflows show that single-turn models struggle, while agentic workflows generally improve both answer and trajectory scores; the paper also reports that adaptive inspection across time, space, and causal relations is useful and proposes a simple competitive baseline. The benchmark design is potentially useful and the milestone representation is more interpretable than scoring unconstrained reasoning alone, but the current evidence does not yet establish that the measurements are separated from language shortcuts or that the conclusions are stable beyond the small, video-clustered sample. The question-level resampling and limited human-alignment check are insufficient to resolve these concerns, so I currently lean toward Weak Reject.

Critical strengths
  1. The benchmark addresses an important gap by requiring multi-step evidence acquisition across spatial, temporal, and causal dimensions; its 100 questions, 71 videos, and 3–7 milestones provide a more agentic testbed than standard single-step video QA.
  2. Tool-agnostic milestones make trajectory evaluation interpretable across interfaces, and the paper includes judge-robustness checks plus a limited human-alignment study.
Critical weaknesses
  1. The benchmark’s measurement validity is not yet established. It provides no video-free or option-order controls, while its trajectory score relies on milestones extracted from gold-answer-containing solution notes and judged against that sole reference. The limited judge-robustness and human-alignment checks (20 questions, two author raters) show consistency but not completeness or non-prescriptiveness across valid evidence paths; the 26-way format may reduce guessing without ruling out linguistic or reference-induced shortcuts.
  2. With 100 questions over 71 videos, the analysis is mainly at the question level. Shared videos make question-level bootstrap insufficient for assessing unseen-video generalization, while video-level or leave-one-video-out uncertainty and held-out-video stability remain unreported.
  3. The benchmark is small. With only 100 questions across 71 videos, coverage of 12 genres and multiple skill types is limited, so model comparisons and conclusions about agentic video reasoning may be sensitive to a small set of items rather than reflect stable capability differences.
Questions for authors
  1. Please add question-only and question-plus-options-only controls (i.e., video-free conditions), together with answer-option permutations or other counterfactual tests, to quantify how much performance can be obtained without visual evidence and whether the trajectory score is affected by language cues.

  2. Please recompute uncertainty at the video level, report leave-one-video-out or held-out-video results, and provide per-video, genre, and skill counts or stratified intervals to assess both unseen-video stability and whether the 100-question sample supports the claimed comparisons.

7 comments

We appreciate your thoughtful comments and valuable suggestions, which have helped us improve our paper. We provide our responses to each of your points below.

W1, Q1. Measurement validity and shortcut susceptibility

A1. (1) Video-free and option controls

To address concerns about potential shortcuts arising from language cues, we add option-permuted, video-free, and question-only controls in the Single-Turn setting as shown in the table below. We use Gemini 3.1 Pro as the judge model to evaluate trajectory scores (Traj.).

ControlsGemini 2.5 Pro (Acc.)Gemini 2.5 Pro (Traj.)Qwen3.5-27B (Acc.)Qwen3.5-27B (Traj.)
[1] Video, original option order0.540.780.290.64
[2] Video, permuted option order0.510.780.280.63
[3] No video, original option order0.160.160.120.20
[4] No video, no options0.050.130.040.18

First, comparing the first and second rows shows that changing the option order has minimal effects on both accuracy and trajectory scores. Next, removing the video (third row) leads to a substantial drop in accuracy and trajectory scores, indicating that visual evidence is essential for achieving competitive performance. These results suggest that language cues alone do not provide a meaningful shortcut. Lastly, the question-only setting (fourth row) yields similarly low accuracy and trajectory scores to the video-free setting with 26 options, suggesting that the answer options provide limited additional information in the absence of visual evidence.

(2) Milestones and judge robustness

We clarify that milestones capture the evidence necessary to answer a question rather than a specific solution path. Therefore, the extracted milestones remain largely consistent across different versions of the solution notes. Specifically, none of the milestones prescribes a particular tool or search order, and the judge only checks whether the evidence specified by each milestone has been acquired. This design is intentional, as multiple valid solutions can acquire the same underlying evidence using different tools and in different orders.

To examine the robustness of milestone extraction, we compare the milestones extracted from two different versions of the solution notes. We find that 89% of the original milestones also appear in the alternative version, while the average number of milestones per question remains similar (4.6 vs 4.1). These minor differences have minimal effects on the trajectory scores of the five methods, with changes of at most 0.03, as shown in the table below.

MethodsOriginal notes (Traj.)Alternative notes (Traj.)
Single-Turn0.780.78
STAR0.360.34
DVD0.470.50
AVP0.820.82
Video-ReAct0.840.82

(3) Human-alignment checks

We extend the human-alignment study in Appendix D.4 from 20 to all 100 questions, using the same two author raters. The Pearson correlation between the human-assigned and judge-assigned overall trajectory scores remains high at 0.80 (95% CI: [0.77, 0.84]), as shown in the table below.

TUECovEGEComRFOverall
20 questions0.390.710.750.600.580.77
100 questions0.380.690.770.530.580.80

W2, Q2. Statistical reliability and generalization

A2. We reassess uncertainty using video-level resampling, conduct leave-one-video-out analyses, and stratify comparisons by genre and skill group for both accuracy and trajectory scores. The overall patterns of the main comparisons remain stable across these analyses, with only a few borderline individual comparisons showing changes in statistical significance or in the direction of the estimated differences.

(1) Video-level uncertainty

We perform a paired cluster bootstrap by resampling the 71 videos with replacement (10,000 bootstrap replicates), retaining all questions associated with each sampled video and using the same resampled video sets across all framework and Single-Turn comparisons. Each row of the table below reports the difference between a framework and Single-Turn on the 0–1 scale used in Table 1, averaged across the 20 backbones, along with 95% confidence intervals computed using question-level resampling (as in the paper) and video-level resampling.

MetricFrameworkDifference from Single-TurnQuestion-level 95% CIVideo-level 95% CI
AccuracySTAR-0.15[-0.19, -0.11][-0.20, -0.10]
AccuracyDVD+0.05[+0.01, +0.09][+0.01, +0.09]
AccuracyAVP+0.21[+0.16, +0.26][+0.16, +0.26]
AccuracyVideo-ReAct+0.22[+0.16, +0.27][+0.16, +0.27]
TrajectorySTAR-0.21[-0.24, -0.18][-0.25, -0.18]
TrajectoryDVD+0.01[-0.02, +0.03][-0.02, +0.03]
TrajectoryAVP+0.23[+0.20, +0.26][+0.20, +0.26]
TrajectoryVideo-ReAct+0.27[+0.24, +0.30][+0.24, +0.30]

Across the 80 individual comparisons (20 backbones and 4 frameworks), the question-level and video-level confidence intervals lead to the same conclusion regarding whether zero is excluded in 77 of 80 accuracy comparisons and all 80 trajectory comparisons. In all three accuracy cases where the conclusions differ, the interval endpoint closest to zero is within 0.01 of zero.

(2) Leave-one-video-out analysis

Removing each of the 71 videos in turn changes the framework–Single-Turn differences averaged across backbones by at most 0.015 for accuracy and 0.010 for trajectory score. Across the 80 individual comparisons, the maximum change is 0.032 for accuracy and 0.021 for trajectory score. The sign flips in two comparisons for each metric; in all four cases, the original difference is close to zero.

(3) Counts and stratified confidence intervals

We clarify that, among the 71 videos, 51 contain one question, 13 contain two, 6 contain three, and 1 contains five. The tables below report the number of questions and videos in each genre category and skill group, along with the differences from Single-Turn averaged across the 20 backbones within each group (with video-level 95% confidence intervals). For each genre or skill group, we independently resample the videos containing questions in that group, retaining all corresponding questions when computing the confidence intervals. A question can belong to more than one skill group, so the skill-group counts sum to more than 100.

Accuracy by genre category:

GroupQuestionsVideosSTARDVDAVPVideo-ReAct
Daily Lifestyle2923-0.18 [-0.29, -0.09]+0.06 [-0.00, +0.14]+0.24 [+0.16, +0.33]+0.24 [+0.15, +0.34]
Knowledge & Documentary3421-0.11 [-0.14, -0.06]+0.04 [-0.02, +0.11]+0.21 [+0.14, +0.29]+0.19 [+0.10, +0.29]
Performance1815-0.16 [-0.28, -0.05]+0.08 [+0.01, +0.15]+0.26 [+0.14, +0.38]+0.29 [+0.16, +0.41]
Media Arts1912-0.17 [-0.34, -0.04]+0.01 [-0.08, +0.10]+0.11 [-0.04, +0.20]+0.16 [+0.04, +0.29]

Accuracy by skill group:

GroupQuestionsVideosSTARDVDAVPVideo-ReAct
Temporal9366-0.15 [-0.21, -0.10]+0.05 [+0.01, +0.09]+0.22 [+0.17, +0.26]+0.22 [+0.16, +0.27]
Perception7050-0.12 [-0.17, -0.07]+0.02 [-0.01, +0.06]+0.17 [+0.11, +0.22]+0.19 [+0.14, +0.25]
Counting5645-0.12 [-0.20, -0.05]+0.01 [-0.03, +0.06]+0.15 [+0.09, +0.21]+0.16 [+0.09, +0.23]
Agentic reasoning4842-0.12 [-0.18, -0.07]+0.07 [+0.02, +0.11]+0.18 [+0.10, +0.25]+0.25 [+0.17, +0.32]
OCR/Text4135-0.21 [-0.29, -0.14]+0.10 [+0.04, +0.17]+0.27 [+0.20, +0.35]+0.28 [+0.20, +0.37]
Audio-visual1310-0.22 [-0.39, -0.10]+0.13 [-0.03, +0.27]+0.34 [+0.17, +0.50]+0.19 [-0.01, +0.37]

Trajectory score by genre category:

GroupQuestionsVideosSTARDVDAVPVideo-ReAct
Daily Lifestyle2923-0.23 [-0.29, -0.17]-0.02 [-0.07, +0.03]+0.24 [+0.20, +0.29]+0.27 [+0.22, +0.32]
Knowledge & Documentary3421-0.21 [-0.24, -0.18]+0.01 [-0.03, +0.05]+0.23 [+0.18, +0.28]+0.27 [+0.22, +0.32]
Performance1815-0.20 [-0.30, -0.11]+0.05 [+0.02, +0.08]+0.29 [+0.22, +0.36]+0.33 [+0.28, +0.39]
Media Arts1912-0.20 [-0.33, -0.10]-0.01 [-0.08, +0.07]+0.16 [+0.06, +0.25]+0.22 [+0.11, +0.32]

Trajectory score by skill group:

GroupQuestionsVideosSTARDVDAVPVideo-ReAct
Temporal9366-0.21 [-0.25, -0.17]+0.01 [-0.01, +0.04]+0.24 [+0.21, +0.27]+0.27 [+0.24, +0.31]
Perception7050-0.20 [-0.23, -0.17]-0.01 [-0.04, +0.01]+0.20 [+0.17, +0.23]+0.25 [+0.23, +0.28]
Counting5645-0.21 [-0.26, -0.16]-0.02 [-0.05, +0.02]+0.21 [+0.16, +0.25]+0.26 [+0.21, +0.30]
Agentic reasoning4842-0.19 [-0.23, -0.16]+0.00 [-0.03, +0.04]+0.22 [+0.17, +0.27]+0.30 [+0.25, +0.34]
OCR/Text4135-0.24 [-0.30, -0.19]+0.03 [-0.01, +0.06]+0.24 [+0.20, +0.29]+0.27 [+0.23, +0.32]
Audio-visual1310-0.23 [-0.34, -0.13]+0.08 [+0.02, +0.14]+0.33 [+0.21, +0.44]+0.31 [+0.16, +0.43]

AVP and Video-ReAct outperform Single-Turn in every group on both metrics. For accuracy, their individual 95% confidence intervals exclude zero in 9 of the 10 groups for each method. The exceptions are Media Arts for AVP and Audio-Visual for Video-ReAct, which are also the two groups with the fewest videos. For trajectory scores, the confidence intervals exclude zero in all 10 groups for both methods. STAR underperforms Single-Turn in every group on both metrics. For DVD, which exhibits relatively small average differences, the confidence intervals exclude zero in 4 of the 10 groups for accuracy and 2 of the 10 groups for trajectory scores.

Furthermore, removing each of the four genre categories in turn changes the framework–Single-Turn differences averaged across the 20 backbones by at most 0.023 for accuracy and 0.015 for trajectory scores. Overall, these analyses indicate that the main comparisons remain stable when accounting for video-level clustering within our benchmark.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Revisions

We thank the reviewers for their constructive feedback and thoughtful discussions. In response, we conducted additional experiments and analyses and incorporated the corresponding results and clarifications into the revised manuscript. Our main revisions are summarized as follows.

1. Measurement Validity and Language-Based Shortcuts.
We added video-free, question-only, and option-permutation controls, confirming that competitive performance relies on visual evidence rather than linguistic shortcuts. We also demonstrated milestone robustness across alternative solution notes (89% overlap) and extended the human-alignment study to all questions.

2. Evidence Grounding and Trajectory Score Validity.
We introduced an explicit grounding check and counterfactual evaluations, demonstrating that trajectory scores penalize evidence claims unsupported by recorded tool outputs. We further validated tool observations against video content and confirmed alignment with human assessments after controlling for answer correctness.

3. Benchmark Coverage and Multi-Hop Reasoning.
We added spatial, temporal, and causal reasoning annotations and clarified the distinction between dependent reasoning and independent evidence aggregation. We also report the distribution of reasoning requirements to better characterize the benchmark's coverage.

4. Statistical Reliability and Dataset Scope.
We conducted video-level bootstrap, leave-one-video-out, and genre- and skill-level analyses, confirming that the main performance trends remain stable. We also clarified the scope of AgentVidBench as a focused agentic video understanding benchmark comprising 100 carefully curated questions, rather than an exhaustive video understanding benchmark. As agentic benchmarks are generally more compact and task-focused (AgentDojo [1]: 97 realistic user tasks, AgentHarm [2]: 110 unique agent behaviors), we believe that our carefully curated videos, questions, and milestones provide a reasonable basis for meaningful empirical comparisons of both answer accuracy and trajectory quality.

5. Methodological Novelty and Experimental Fairness.
We clarified how AgentVidBench differs from prior work, particularly MINERVA, by evaluating active tool use and evidence acquisition through tool-agnostic milestones. We also standardized the planning and vision models and tool-call limits across agentic frameworks, and added efficiency comparisons to further evaluate Video-ReAct.

Overall, we have incorporated these additional experiments, analyses, and clarifications into the revised manuscript and supplementary material to strengthen the validity, reliability, and clarity of AgentVidBench.

[1] Debenedetti et al., "AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents", NeurIPS 2024.

[2] Andriushchenko et al., "AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents", ICLR 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.