- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents GroundedVDR, an evidence-based evaluation benchmark for deep video research.
Beyond focusing solely on final answers, it requires models to submit both video clips and web evidence to verify whether temporal information within videos truly guides subsequent retrieval. Combining manual review and multi-model testing, the benchmark filters out questions solvable merely through model memorization, single-frame recognition, or unimodal reasoning. It additionally annotates video states, web facts, and the relationships between them.
Experiments on eight multimodal agents reveal that even when models produce correct answers, they frequently suffer from erroneous video localization, incomplete web evidence, and broken cross-stage constraints. Accordingly, the paper advocates expanding evaluation beyond answer accuracy to assess the complete chain of evidence.
Critical strengths
-
This work extends the evaluation focus from "whether the answer is correct" to "whether the evidence chain is valid". GroundedVDR requires models not only to output the final answer but also to submit corresponding video clips and web evidence, represented as a tuple of "video state – web fact – dependency between the two". Meanwhile, it separately assesses video localization, web fact retrieval, and final answer prediction.
-
The dataset construction pipeline is rigorous. Through unimodal ablations on videos and webpages, repeated testing across multiple models, and independent manual review, the authors filter out questions that can be answered solely via model memorization, single-frame recognition, OCR, transcribed speech, or random guessing.
Furthermore, the paper introduces hard negatives that are factually true yet associated with incorrect candidates, which examine whether the model binds valid evidence to the correct entities. This brings the benchmark closer to the real-world video-grounded web research task.
Critical weaknesses
- Most importantly, The appendix B is:
B SUBMISSION CHECKLIST FOR THE FINAL STUDY
Before converting this draft into a submission, we require:
• a frozen accepted set with train/development/test separation and immutable manifests;
• independent double review, adjudication, agreement, compensation, and time reporting;
• source-known, single-frame, OCR/title, state-deletion, web-only, and video-only tests;
• verified snapshots or fetch-time entailment for every required web claim;
• group-aware strict temporal scoring and rejection of contradictory answer strings;
• at least two model families, natural and controlled budgets, repeated runs, confidence intervals, and paired tests;
• complete provenance, licensing, privacy, web-drift, and takedown documentation;
• removal or resolution of every [TBD] marker in the main paper.
Evidently, this paper is still a work in progress.
-
GroundedVDR has a final set of merely 100 questions, and statistical significance and randomness are not reported. For this reason, the work cannot be considered valid.
-
Evaluation of the evidence chain relies on LLM judges and evidence self-submitted by models, making it difficult to fully reflect the underlying reasoning process. The paper employs a fixed GPT-5.6-Terra to judge whether video clips and web materials support the corresponding facts.
The authors also acknowledge that models may fail to submit the evidence they actually used, or adopt differing semantic granularity, web sources, and valid reasoning paths. Consequently, low evidence scores can sometimes stem from "incomplete evidence reporting" or discrepancies in judge assessments, rather than necessarily indicating that the model did not properly leverage the evidence. Moreover, this evaluation cannot directly prove that the final answer is causally derived from the submitted evidence.
Questions for authors
Refer to Critical Weaknesses.
GroundedVDR ultimately consists of only 100 items. For a complex task covering diverse video types, retrieval scenarios, and evidence relationships, its representativeness may be limited.
More importantly, the appendix still lists dataset freezing and splitting, repeated experiments, confidence intervals, and paired significance tests as items to be completed prior to formal submission.
Therefore, the current results are best treated as preliminary evidence and are insufficient to fully demonstrate that the model rankings can generalize stably to larger scales or other domains.