acceptodds
Withdrawn from ICLR 2027

GroundedVDR: Grounding Video-to-Web Evidence Dependencies in Video Deep Research

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video large multimodal models have improved at identifying cross-frame visual evidence and conducting open-web retrieval, as reflected in recent video deep research (VDR) benchmarks. However, existing VDR benchmarks primarily evaluate tool-use trajectories and final responses without verifying whether models use temporal video information or retrieved web evidence. We introduce GroundedVDR, an evidence-grounded benchmark with three features. First, it combines human review and model testing to exclude questions that can be answered using parametric knowledge alone. Second, it provides structured video-to-web evidence annotation that links related video frames with their corresponding web evidence, while recording candidate-linked hard negatives for diagnostic analysis. Third, it measures video-state localization, evidence retrieval, video-to-web evidence alignment, and final answer synthesis separately. Together, these features allow GroundedVDR to evaluate both answer correctness and support from the required video and web evidence.

Primary area:
datasets and benchmarks
Keywords:
Video Understanding, Deep Research Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_9RoDOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents GroundedVDR, an evidence-based evaluation benchmark for deep video research.

Beyond focusing solely on final answers, it requires models to submit both video clips and web evidence to verify whether temporal information within videos truly guides subsequent retrieval. Combining manual review and multi-model testing, the benchmark filters out questions solvable merely through model memorization, single-frame recognition, or unimodal reasoning. It additionally annotates video states, web facts, and the relationships between them.

Experiments on eight multimodal agents reveal that even when models produce correct answers, they frequently suffer from erroneous video localization, incomplete web evidence, and broken cross-stage constraints. Accordingly, the paper advocates expanding evaluation beyond answer accuracy to assess the complete chain of evidence.

Critical strengths
  1. This work extends the evaluation focus from "whether the answer is correct" to "whether the evidence chain is valid". GroundedVDR requires models not only to output the final answer but also to submit corresponding video clips and web evidence, represented as a tuple of "video state – web fact – dependency between the two". Meanwhile, it separately assesses video localization, web fact retrieval, and final answer prediction.

  2. The dataset construction pipeline is rigorous. Through unimodal ablations on videos and webpages, repeated testing across multiple models, and independent manual review, the authors filter out questions that can be answered solely via model memorization, single-frame recognition, OCR, transcribed speech, or random guessing.

Furthermore, the paper introduces hard negatives that are factually true yet associated with incorrect candidates, which examine whether the model binds valid evidence to the correct entities. This brings the benchmark closer to the real-world video-grounded web research task.

Critical weaknesses
  1. Most importantly, The appendix B is:

B SUBMISSION CHECKLIST FOR THE FINAL STUDY

Before converting this draft into a submission, we require:

• a frozen accepted set with train/development/test separation and immutable manifests;

• independent double review, adjudication, agreement, compensation, and time reporting;

• source-known, single-frame, OCR/title, state-deletion, web-only, and video-only tests;

• verified snapshots or fetch-time entailment for every required web claim;

• group-aware strict temporal scoring and rejection of contradictory answer strings;

• at least two model families, natural and controlled budgets, repeated runs, confidence intervals, and paired tests;

• complete provenance, licensing, privacy, web-drift, and takedown documentation;

• removal or resolution of every [TBD] marker in the main paper.

Evidently, this paper is still a work in progress.

  1. GroundedVDR has a final set of merely 100 questions, and statistical significance and randomness are not reported. For this reason, the work cannot be considered valid.

  2. Evaluation of the evidence chain relies on LLM judges and evidence self-submitted by models, making it difficult to fully reflect the underlying reasoning process. The paper employs a fixed GPT-5.6-Terra to judge whether video clips and web materials support the corresponding facts.

The authors also acknowledge that models may fail to submit the evidence they actually used, or adopt differing semantic granularity, web sources, and valid reasoning paths. Consequently, low evidence scores can sometimes stem from "incomplete evidence reporting" or discrepancies in judge assessments, rather than necessarily indicating that the model did not properly leverage the evidence. Moreover, this evaluation cannot directly prove that the final answer is causally derived from the submitted evidence.

Questions for authors

Refer to Critical Weaknesses.

GroundedVDR ultimately consists of only 100 items. For a complex task covering diverse video types, retrieval scenarios, and evidence relationships, its representativeness may be limited.

More importantly, the appendix still lists dataset freezing and splitting, repeated experiments, confidence intervals, and paired significance tests as items to be completed prior to formal submission.

Therefore, the current results are best treated as preliminary evidence and are insufficient to fully demonstrate that the model rankings can generalize stably to larger scales or other domains.

Reviewer_ros5OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose GroundedVDR, a video deep research benchmark. Its core goal is to assess not only whether models arrive at the correct final answer, but also whether they genuinely ground their answers in the rich evidence within the videos and in web pages, rather than relying on internal knowledge or guessing. During benchmark construction, the authors strictly excluded tasks solvable from the internal knowledge of models alone, provide structured video-web evidence chains, and incorporate human review to safeguard dataset quality. They further offer process-level evaluation that examines the quality of the evidence chains produced during a model's answering process.

Critical strengths
  • S1. The dataset and process-evidence annotation pipeline is well developed. The paper discusses the complete annotation pipeline in detail, including video selection, evidence mining, data annotation, and quality inspection. It also clearly presents the video-web evidence-chain annotation scheme, including Video States and Web Claims.
  • S2. The paper identifies a key issue in video deep research. Namely, completing the final task is not sufficient. A clear, rigorous, and correct process-level evidence chain must also be ensured. The paper accordingly provides both annotations and evaluation metrics for process evidence chains.
Critical weaknesses
  • W1. The main experiments are thin and do not provide sufficient support. They evaluate only Claude, GPT, Grok, and Qwen. For a benchmark paper, more models should be evaluated to offer a comprehensive conclusion and perspective; Gemini, GLM, and relevant open-source models are not comprehensively assessed.
  • W2. The paper uses GPT-5.6 Terra as the unified judge, but does not validate the reliability and rigor of LLM Judger, nor measure its agreement with human. Moreover, GPT-series models themselves are among the evaluated systems, raising the possibility of same-family preference, which could undermine the credibility of the experimental results and model rankings.
  • W3. The dataset annotation lacks details such as the annotation protocol, the number of annotators, agreement statistics, and the final quality results. Notably, the appendix still lists part of the quality-control work as TODO, which leaves me doubtful about the quality and reliability of the annotations.
Questions for authors
  • Is the paper still at a draft stage, including the main text and appendix? The "Submission Checklist for the Final Study" in Appendix B reads more like a TODO list of unfinished work than supplementary material that enriches the study.
Reviewer_QTtDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces GroundedVDR, arguing that existing VDR (Video-based Document Retrieval/Reasoning) benchmarks focus solely on final answers and tool invocations. Consequently, they fail to distinguish whether a model truly retrieves evidence from videos and web sources or simply relies on its parametric memory to "recall" the correct answer. To address this, the authors decompose the information required for each correct answer into video-based states and web-based claims, creating structured annotations. They further implement a "necessity check"—combining human review with unimodal ablation studies—to filter out questions that can be answered correctly via memorization alone. The evaluation framework then assesses performance in stages: video state recall, web claim recall, and final answer accuracy. Experiments conducted on 100 questions—evaluating eight multimodal agents alongside a human baseline—reveal that while the top-performing models often arrive at the correct answers, they frequently fail to actually ground those answers in the required evidence (particularly from web sources).

Critical strengths
  1. The problem is well-identified, and the study offers a finding that is genuinely useful to the community. Currently, Video-based Document Retrieval leaderboards largely reward reliance on memory and guesswork, rather than "open-corpus retrieval guided by sequential video evidence." This paper quantifies the issue through staged metrics—specifically, the decoupling of high answer scores from low evidence scores. This insight is set to shift how VDR is approached and provides a clear rationale for future agent design. It is far more valuable than simply creating yet another benchmark of the same type.
  2. The design of the annotation and scoring process is quite sophisticated—particularly the "semantic-level, source-agnostic" approach. Since different agents may follow vastly different retrieval paths, relying on exact URL matches or specific search sequences would effectively turn the evaluation into a test of how well an agent mimics the original annotator. Allowing any reliable source to be considered valid is the right way to go. Furthermore, the annotation schema provided in Appendix A is a resuable resource.
Critical weaknesses
  1. The paper’s primary contribution is its evaluation methodology—a method that has not itself been validated. All four metrics rely on a single, fixed LLM "judge" for scoring; the rubric remains undisclosed, and there is no data on the judge’s alignment with human annotations. Furthermore, the fact that the judge is also one of the models being evaluated raises significant concerns regarding fairness. With a sample size of only 100 queries and a complete absence of repeated runs, confidence intervals, or paired statistical tests, the authors draw sweeping conclusions, such as "no single model leads across the board" and "models with similar answer accuracy exhibit vast differences in grounding capability." These claims rest on marginal score differences. More problematic is that each model was evaluated on its respective vendor’s native platform, with no standardization or disclosure of the tools and budgets employed; consequently, the purported comparison between models is, in reality, a comparison of "model plus infrastructure" hybrids.
  2. The metrics themselves suffer from two flaws: they focus exclusively on recall without penalizing redundancy. Since the submission format allows for an unlimited inclusion of citations and excerpts, the system effectively rewards "searching and pasting more", which likely explains why models with higher answer accuracy also achieve higher "partial" scores for web-based grounding. Moreover, the judge cannot distinguish between a task that was not performed and a task that was performed but not explicitly reported. It is therefore unclear whether these metrics are measuring the quality of the research itself or merely the completeness of the final report.
Questions for authors
  1. Could you provide the rubric used by the judge model? Have you measured the consistency between the judge model and human annotators? How do the results change if a different judge model is used?
  2. Is it possible to re-run several models using a standardized framework and a fixed budget, and then report confidence intervals for these repeated runs? Given a sample size of only 100 questions, a difference of a few points is not statistically robust.
  3. Since the approach relies on pure retrieval combined with the ability to include an unlimited amount of evidence, how do you prevent a "more evidence equals a higher score" bias? What is the correlation between the number of retrieved items submitted and the final score?
  4. For "Video State" and "Web Claim," the evaluation focuses solely on whether the evidence supports the claim, without verifying whether the web search was actually constrained by the video content. Could you require the agent to output its search queries, and then check whether those queries incorporate constraints derived from the video? Alternatively, could you perform an ablation study—such as providing only the text-based query or masking the video input—to observe the impact on the final answers?
  5. How many annotators were involved, and what was the level of inter-annotator agreement? Furthermore, out of the 100 test questions, what proportion were flagged by the probe, sent for human review, and subsequently rejected?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.