acceptodds
Under review as a conference paper at ICLR 2027

UNIVERSE: A Unified Benchmark for Video Deep Research

Abstract

Video deep research requires agents to coordinate video understanding with external information seeking, yet existing benchmarks cover only fragmented aspects of this capability and remain difficult to scale. Some provide a video and require research grounded in its content, while others require agents to discover relevant videos on the open web. Moreover, existing benchmarks largely emphasize using visual observations to initiate research, leaving the reverse process of using external knowledge to determine what should be observed underexplored. We introduce UNIVERSE, a unified benchmark that jointly evaluates supplied-video research and open-web video discovery. Beyond observation-to-research, UNIVERSE introduces knowledge-guided observation, where agents first resolve a target from indirect external clues and then locate it in video to obtain the requested visual evidence, together with image-based query clues that couple visual references with relational research. To enable scalable construction, we combine automated instance generation with multi-stage verification and shortcut filtering, yielding 800 high-quality questions across ten domains, substantially exceeding the scale of prior video deep research benchmarks. Evaluations of representative proprietary and open-weight multimodal models under a shared tool-enabled framework reveal substantial challenges in coordinating external research, video discovery, and visual observation, highlighting considerable room for progress in video deep research.

Primary area:
datasets and benchmarks
Keywords:
Video Deep Research, Multimodal Agents, Video Understanding, Open-Web Video Discovery, Knowledge-Guided Observation, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GjSgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

UNIVERSE introduces 800 video deep-research questions covering observation-to-research, knowledge-guided observation, and open-web video discovery, with additional image-based relational clues. It combines automated construction, evidence verification, shortcut filtering, and human review. Evaluations of eight models under a shared tool environment reveal substantial performance gaps, while paired text-oracle experiments examine the difficulty introduced by image clues.

Critical strengths
  1. Meaningful task formulation. Knowledge-guided observation requires external research to determine what should be examined in a video, complementing observation-driven research and open-web discovery. This provides useful coverage of video research capabilities.

  2. Informative paired evaluation. Replacing image clues with their textual identities improves performance across all evaluated models while preserving the remaining task. This offers concrete evidence that resolving visual references is an important bottleneck.

Critical weaknesses
  1. Benchmark quality and judging lack independent calibration. Appendix B describes quality-control stages but omits verification configurations, retention counts, and quantitative annotation-quality estimates. Gemini-3.6-Flash is both an evaluated model and the correctness/failure judge, without reported independent calibration. This does not establish bias, but limits confidence in the benchmark’s reliability and reported rankings.

  2. Evaluation details and failure attribution need clarification. Tool implementations, video sampling, judging criteria, and handling of changing or unavailable web evidence are insufficiently specified. The shared VideoSummary backend is a reasonable system-evaluation choice, but pooled visual-understanding failures cannot clearly distinguish errors by the evaluated model from errors in backend observations. This limits reproducibility and interpretation of the diagnostic results.

Questions for authors
  1. Can you provide the key generation/verification settings, stage-level retention counts, and an independent sample audit of final questions and judge decisions? How were the shortcut probes implemented, and what residual bypasses remain?

  2. Please specify the search and video-tool configurations, judging criteria, evaluation dates, and handling of unavailable evidence. Will supporting evidence and evaluation logs be released?

  3. Can the failure analysis distinguish direct visual interpretation, shared-backend errors, and downstream reasoning errors, or can its conclusions be restricted to system-level failures?

7 comments
AuthorsOpenReview ↗

Response to Reviewer (1/4)

Thank you very much for your time, thoughtful review, and constructive feedback. We are encouraged that you recognized the meaningful task formulation of UNIVERSE, particularly Knowledge-Guided Observation and the paired Text Oracle evaluation of image-based clues. Your comments on benchmark quality, judge calibration, evaluation reproducibility, and failure attribution helped us substantially strengthen the paper.

Because W1 and W2 are restated more specifically in Q1–Q3, we organize our response around the three questions to avoid duplicating the same evidence. Q1 addresses benchmark quality, shortcut filtering, and judge validation; Q2 covers evaluation configuration and reproducibility; and Q3 addresses failure attribution. Together, these responses cover all concerns raised in W1 and W2.

Q1: Construction, Quality Control, Shortcut Filtering, and Independent Validation

We have substantially expanded Sec. 3.2 and Appendix B to document the construction and quality-control pipeline, including model configurations, verification criteria, shortcut probes, human-review procedures, stage-level filtering statistics, and independent audits.

Generation and verification. The new Table B.1 specifies 11 model-assisted operations spanning generation (G1–G6), evidence verification (V1–V3), and counterfactual shortcut probing (S1–S2), together with their eligible models and input/output roles. The model pool includes GPT-5, GPT-5.1, GPT-5.2, GPT-5.4, Gemini-2.5-Pro, Gemini-3-Pro, and Gemini-3.1-Pro. Operations that inspect video frames (G1, G2, and V3) use Gemini-family models, while text- and image-based operations draw from the corresponding eligible pools. Generation and verification are conducted in isolated sessions, and verification samples a different model from the generator whenever multiple models are eligible.

Verification is decomposed into three complementary checks: V3 verifies visual grounding and answer correctness against the supporting video or images; V1 checks factual and relational consistency against graph paths and web evidence; and V2 checks target specificity and searches for plausible competing entities or events satisfying the same clues. Each applicable verification check is executed twice independently, and a candidate advances only if both judgments are confirmed; any refuted or inconclusive judgment causes rejection. These configurations and acceptance rules are now detailed in Appendix B.1–B.3.

Shortcut filtering. Appendix B.4 and Table B.2 now specify task-specific counterfactual probes that test whether the intended information dependency can be bypassed. For O2R, we test whether the anchor or answer can be recovered without the supplied video or without the video-derived anchor reference. For KGO, we remove the anchor-identifying external clues and test whether the visual answer can be obtained directly from the supplied video. For OVD, we reveal the intended anchor/event but disable video access, separately testing whether the answer can be recovered from textual web evidence or static images/thumbnails. Image-based clues additionally undergo a direct-recognition probe with retrieval disabled. Each applicable probe permits up to two independent attempts, and a single successful bypass is sufficient to reject the candidate. These probes eliminated 466 candidates in total.

These checks target the principal bypasses relevant to the intended task dependencies, including linguistic leakage, text-only recovery, video-only guessing, static-image shortcuts, and direct recognition of image clues. In addition, the independent audit of 80 final questions explicitly checked for residual shortcuts, and none were identified in the audited sample.

AuthorsOpenReview ↗

Response to Reviewer (2/4)

Q1: Construction, Quality Control, Shortcut Filtering, and Independent Validation (continued)

Human review and filtering statistics. Every candidate surviving automated verification and shortcut filtering is independently reviewed by two annotators drawn from three graduate students against five criteria: question clarity, factual consistency, target unambiguity, visual-evidence sufficiency, and answer validity. A candidate is retained only if both annotators approve it. Across all 1,135 human-reviewed candidates, inter-annotator agreement is 90.1% (Cohen's κ=0.64\kappa=0.64), with within-task agreement rates of 91.2% for O2R, 89.6% for KGO, and 90.0% for OVD. Human-review pass rates are 77.8% (168/216), 79.6% (191/240), and 78.9% (536/679), respectively.

The revised Appendix B.5 and Table B.3 report the complete stage-level filtering statistics. Each rejected candidate is counted only at the first stage where it fails; removal percentages use the initial candidate count for the corresponding task as the denominator.

Construction & Quality-Control StageO2RKGOOVDTotal
Initial candidates4507001,5002,650
Removed: Programmatic consistency checks29 (6.4%)54 (7.7%)96 (6.4%)179 (6.8%)
Removed: Visual grounding verification (V3)62 (13.8%)117 (16.7%)308 (20.5%)487 (18.4%)
Removed: Factual and ambiguity verification (V1–V2)42 (9.3%)163 (23.3%)178 (11.9%)383 (14.5%)
Removed: Counterfactual shortcut filtering (S1–S2)101 (22.4%)126 (18.0%)239 (15.9%)466 (17.6%)
Removed: Human review48 (10.7%)49 (7.0%)143 (9.5%)240 (9.1%)
Passed quality control168 (37.3%)191 (27.3%)536 (35.7%)895 (33.8%)
Excluded during final random sampling18413695
Final dataset150150500800

Among the 240 candidates rejected during human review, the primary reasons were insufficient or occluded visual evidence (35.8%), ambiguous clues or competing valid answers (34.2%), factual or relational inaccuracies (19.6%), and unnatural phrasing or residual shortcuts (10.4%). The final 800 questions are sampled from the 895 candidates that passed all quality-control stages, so this final downsampling is not counted as a quality-control rejection.

Independent audit of final questions. Independent auditors who were not involved in the original construction examined 80 stratified random final questions (15 O2R, 15 KGO, and 50 OVD, covering both text-only and image-clue instances). Blinded to prior verification decisions, they checked factual correctness, visual-evidence sufficiency, answer validity, ambiguity, and residual shortcuts. All 80 questions passed, with no factual errors, ambiguous answers, insufficient visual evidence, or residual shortcuts identified. We have added this audit to Sec. 3.2 and Appendix B.5.

Independent validation of answer judging. To address the concern that Gemini-3.6-Flash is both an evaluated model and the primary correctness judge, we re-evaluated all 38,400 model predictions using GPT-5.6-Sol as an independent cross-family judge under the same semantic-equivalence criterion. The two judges agree on 97.83% of trajectory-level decisions on average. Across all 16 model–setting combinations, the mean absolute change in Overall Accuracy is only 0.62 percentage points (maximum: 1.17 pp), the mean absolute change in Overall Pass@3 is 0.53 pp (maximum: 0.88 pp), and model rankings under both metrics remain identical in both tool settings (Spearman's ρ=1.0\rho=1.0).

We further validated both judges against blinded human annotations on 160 stratified random predictions and 50 additional cases where the two automated judges disagreed. On the random sample, human adjudicators agreed with Gemini-3.6-Flash on 96.9% (155/160) and with GPT-5.6-Sol on 98.1% (157/160). Among the 50 disagreement cases, humans sided with Gemini-3.6-Flash in 19 cases and with GPT-5.6-Sol in 31. We have added the full cross-judge re-evaluation, per-model breakdown, and human validation of both automated judges to Appendix C.4 and Table 4.

For reproducibility, the revised Appendix B.1–B.4 now makes the operational content of the generation, verification, and shortcut-probing stages explicit, including their required inputs and outputs, evidence requirements, negative constraints, and acceptance criteria. We will additionally release the complete verbatim construction prompt templates, structured output schemas, construction code, and available supporting evidence records together with the benchmark.

AuthorsOpenReview ↗

Response to Reviewer (3/4)

Q2: Evaluation Configuration, Evidence Availability, and Reproducibility

We have substantially expanded Appendix C to specify the evaluation prompts, tool implementations, video preprocessing, judging protocol, evaluation period, and handling of unavailable web evidence.

Tool and retrieval configurations. All models share the same seven tool interfaces and backend configurations, with no model-specific retrieval settings, personalization, or reranking. The revised Appendix C.2 and Table 3 specify:

ToolBackendConfiguration and Returned Information
WebSearchGoogle Search via SerpApiTop 20 results per query in Google's native relevance order; returns titles, URLs, and snippets.
ImageSearchGoogle Images via SerpApiTop 5 images for a text query, with previews, titles, and source URLs.
ReverseImageSearchGoogle Lens via SerpApi5 visually matched web images with titles and source URLs for a supplied query image.
ReverseFrameSearchFrame extraction + Google Lens via SerpApiExtracts the nearest sampled frame to an agent-selected timestamp and returns 5 reverse-image matches.
LinkSummaryWeb crawler + Qwen3.5-35B-A3BFetches agent-selected webpage URLs and returns a query-focused summary.
ChannelVideosYouTube Data API v3Supports title-keyword, duration, publication-date, and result-limit filters; results ordered by publication date; default limit of 50 videos. Returns URLs, titles, descriptions, channel metadata, upload dates, and durations.
VideoSummaryGemini-3-ProAnswers a targeted question about an agent-selected online video and returns a concise answer with supporting visual/audio description.

Agents decide which tools to invoke, how to formulate queries or filters, and which returned sources to inspect further.

Video processing and execution. For O2R and KGO, supplied videos are initially sampled at 1 fps. To retain full temporal coverage within a 64K video-token budget, we progressively reduce frame resolution when necessary; only if the budget still cannot be satisfied at approximately 360p do we reduce the temporal sampling rate uniformly across the recording rather than truncating it. OVD provides no video directly, and online videos are inspected through the shared VideoSummary backend. Each tool-enabled trajectory permits at most 50 model turns and one tool call per turn, and we sample three independent trajectories per question.

Judging criteria. The primary judge evaluates semantic equivalence between the model prediction and the reference answer, accepting valid aliases, equivalent paraphrases, and alternative numerical forms rather than requiring exact string matching. It receives the question, reference answer, and model prediction, together with supporting visual context when needed for disambiguation, while remaining blinded to the evaluated model's identity. The independent GPT-5.6-Sol re-evaluation and human validation described in Q1 provide additional validation of this protocol. The complete evaluation prompts and judging procedure are now provided in Appendix C.1 and C.4.

Evaluation dates and unavailable evidence. Evaluations were conducted during August–September 2026. If a webpage or video is unavailable, or a tool call fails, the resulting error is returned directly to the agent, which may retry, reformulate its request, or seek an alternative source within its remaining interaction budget. We provide no manual intervention or ground-truth-based fallback.

Because web indexes and rankings evolve over time, exact retrieval results may change across evaluation runs. We therefore report the evaluation period and use the same retrieval backends, ranking procedures, and result limits for all evaluated models. OVD also does not require retrieval of a unique construction-time video: the supporting video serves as a verified witness, and any alternative footage containing sufficient evidence is a valid research path.

Release plan. We will release the benchmark questions and reference answers, complete evaluation prompts and tool schemas, all model predictions, judging outputs, and tool-interaction trajectories/logs, together with construction-time supporting evidence records that can be distributed. Third-party web and video content will be referenced rather than redistributed where redistribution rights are unavailable. Together with the configurations now documented in Appendix C, these materials support independent auditing and reproduction of the evaluation protocol.

Reviewer_WDnNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces UNIVERSE, a unified benchmark for Video Deep Research, aiming to address the limitations of existing benchmarks in terms of incomplete task coverage, insufficient evaluation of knowledge-guided video observation, and limited scalability in dataset construction. The benchmark unifies three task types: Observation-to-Research (O2R), Knowledge-Guided Observation (KGO), and Open-Web Video Discovery (OVD). It also introduces image-based query clues to evaluate agents' ability to integrate video understanding, external knowledge retrieval, and cross-modal reasoning. The paper proposes a scalable data construction pipeline that combines automated instance generation, multi-stage verification, and shortcut filtering, resulting in a benchmark comprising 800 questions and 782 videos across 10 domains. Through evaluations of eight representative multimodal models, the paper reveals significant limitations in current agents' abilities to perform external knowledge retrieval, target localization, video discovery, and visual evidence integration, providing a more comprehensive and scalable evaluation framework for future research on video deep research.

Critical strengths
  1. Comprehensive and Innovative Benchmark Task Design The paper introduces UNIVERSE, which integrates three types of tasks—Observation-to-Research (O2R), Knowledge-Guided Observation (KGO), and Open-Web Video Discovery (OVD)—into a unified evaluation framework. In particular, the introduction of knowledge-guided observation and image-based query clues addresses the limitations of existing benchmarks in evaluating the ability to use external knowledge to guide video observation and perform cross-modal reasoning. This design enables a more comprehensive assessment of video deep research capabilities.
  2. Large-Scale Dataset and Scalable Construction Pipeline The paper develops a data construction pipeline that combines automated instance generation, multi-stage evidence verification, and shortcut filtering, reducing reliance on manual annotation while improving data reliability and evaluation validity. The resulting benchmark comprises 800 questions and 782 videos across 10 domains, surpassing the scale of existing benchmarks in this area and providing a valuable resource for future research.
  3. Comprehensive Experimental Evaluation and Valuable Research Insights The paper evaluates eight representative proprietary and open-weight multimodal models under a unified tool-enabled framework, analyzing their performance across different task types, tool-access conditions, and input modalities. This systematic evaluation helps identify the limitations of existing agents in external knowledge retrieval, video discovery, target localization, and visual evidence integration, providing valuable insights for improving future video deep research methods.
Critical weaknesses
  1. Insufficient Validation of Data Quality and Annotation Reliability UNIVERSE relies on automated question generation, multi-stage verification, and human review to ensure data quality. However, the paper does not sufficiently clarify the annotation criteria, the number of human reviewers, or inter-annotator agreement. In particular, automatically generated questions in KGO and OVD may contain ambiguities, insufficient evidence, or multiple valid answers. The authors are encouraged to report annotation agreement, human verification pass rates, and error rates across different question types to better demonstrate the reliability of the benchmark.
  2. Insufficient Verification of Shortcut Filtering Effectiveness Although the paper introduces Shortcut Filtering to prevent models from bypassing the intended research and video observation processes, it remains unclear whether this mechanism effectively eliminates linguistic biases, prior knowledge shortcuts, or potential data leakage. Some questions may still be answerable through text-based web searches without actually watching videos. The authors should conduct additional controlled experiments, such as question-only, text-only research, video-only, and full-research settings, to verify that the benchmark genuinely requires the integration of external research and video understanding.
  3. Lack of Fine-Grained Evaluation of Different Capabilities UNIVERSE integrates O2R, KGO, and OVD into a unified benchmark, but these tasks involve substantially different capabilities, including visual clue identification, external knowledge retrieval, target resolution, and video discovery. Evaluating models primarily through final answer accuracy makes it difficult to determine which stages are responsible for failures. The authors are encouraged to introduce more fine-grained metrics, such as research anchor identification accuracy, external information retrieval success rate, target video recall, and temporal localization accuracy, to better characterize the strengths and limitations of current video deep research agents.
Questions for authors

1.Do the tasks genuinely require deep research? If a substantial number of questions can be answered through a single simple search or by directly watching a video, the distinction between these tasks and traditional video question answering or visual retrieval tasks becomes less clear. 2.Does the benchmark truly evaluate deep research capabilities? If evaluation relies solely on final answer accuracy, it may fail to adequately capture models' actual capabilities in research planning, cross-source evidence integration, iterative search, and multimodal reasoning.

5 comments
AuthorsOpenReview ↗

Response to Reviewer (1/3)

Thank you very much for your time, thoughtful review, and constructive feedback. We are encouraged that you recognized the core strengths of UNIVERSE, including its unified task design spanning O2R, KGO, and OVD, the introduction of knowledge-guided observation and image-based query clues, the scalable construction pipeline, and the systematic evaluation of eight representative multimodal models. Your comments on data quality, shortcut filtering, fine-grained capability analysis, and the depth of research required helped us substantially strengthen the paper. We address each point below and indicate where the corresponding changes appear in the revised manuscript.

W1: Validation of Data Quality and Annotation Reliability

We have substantially expanded Sec. 3.2 and Appendix B to clarify our quality-control criteria, verification procedure, human-review protocol, and stage-level filtering statistics.

Automated verification. Candidates first undergo programmatic consistency checks for missing fields, inconsistent references, disconnected relational paths, and answer leakage. They are then verified through three complementary operations: V3 checks visual grounding and answer correctness against the supporting video or images; V1 checks factual and relational consistency against the underlying graph paths and web evidence; and V2 checks target specificity and searches for plausible competing entities or events satisfying the same clues. For image-based questions, the depicted entity and its relational connection to the research anchor are additionally verified. Generation and verification are conducted in isolated sessions, and verification samples a different model from the generator whenever multiple models are eligible. Each applicable verification check is executed twice independently, and a candidate advances only if both judgments are confirmed; any refuted or inconclusive judgment causes rejection. These procedures are now documented in Appendix B.1–B.3.

Human review and annotation agreement. Every candidate surviving automated verification and shortcut filtering is independently reviewed by two annotators drawn from three graduate students against five criteria: question clarity, factual consistency, target unambiguity, visual-evidence sufficiency, and answer validity. A candidate is retained only if both annotators approve it. Across all 1,135 human-reviewed candidates, inter-annotator agreement is 90.1% (Cohen's κ=0.64\kappa=0.64), with agreement rates of 91.2% for O2R, 89.6% for KGO, and 90.0% for OVD. Human-review pass rates are 77.8% (168/216), 79.6% (191/240), and 78.9% (536/679), respectively.

The revised Appendix B.5 and Table B.3 report the complete stage-level filtering statistics. Each rejected candidate is counted only at the first stage where it fails; removal percentages use the initial candidate count for the corresponding task as the denominator.

Construction & Quality-Control StageO2RKGOOVDTotal
Initial candidates4507001,5002,650
Removed: Programmatic consistency checks29 (6.4%)54 (7.7%)96 (6.4%)179 (6.8%)
Removed: Visual grounding verification (V3)62 (13.8%)117 (16.7%)308 (20.5%)487 (18.4%)
Removed: Factual and ambiguity verification (V1–V2)42 (9.3%)163 (23.3%)178 (11.9%)383 (14.5%)
Removed: Counterfactual shortcut filtering (S1–S2)101 (22.4%)126 (18.0%)239 (15.9%)466 (17.6%)
Removed: Human review48 (10.7%)49 (7.0%)143 (9.5%)240 (9.1%)
Passed quality control168 (37.3%)191 (27.3%)536 (35.7%)895 (33.8%)
Excluded during final random sampling18413695
Final dataset150150500800

Among the 240 candidates rejected during human review, the primary reasons were insufficient or occluded visual evidence (35.8%), ambiguous clues or competing valid answers (34.2%), factual or relational inaccuracies (19.6%), and unnatural phrasing or residual shortcuts (10.4%). The final 800 questions are sampled from the 895 candidates that passed all quality-control stages, so this final downsampling is not counted as a quality-control rejection.

Finally, independent auditors who were not involved in the original construction examined 80 stratified random final questions (15 O2R, 15 KGO, and 50 OVD, covering both text-only and image-clue instances). Blinded to prior verification decisions, they checked factual correctness, answer unambiguity, visual-evidence sufficiency, and residual shortcuts. All 80 passed inspection. We have added this audit to Sec. 3.2 and Appendix B.5.

AuthorsOpenReview ↗

Response to Reviewer (2/3)

W2: Verification of Shortcut Filtering Effectiveness

Our shortcut filtering is designed to test, at the individual-instance level, whether a question can be solved while bypassing the information that its intended solution requires. Because O2R, KGO, and OVD rely on different combinations of external research and video evidence, we use task-specific counterfactual probes rather than applying the same ablation to all three tasks.

Specifically, the revised Appendix B.4 and Table B.2 now document the following probes:

  • O2R: A no-video probe withholds the supplied video while retaining text web search, testing whether the anchor or answer can be recovered without video observation. A complementary relation-tail probe removes the anchor-identifying visual reference, testing whether the answer can be inferred from the downstream relational structure alone.

  • KGO: A no-anchor-clues probe removes all external anchor-identifying clues and provides only the supplied video and visual request, testing whether the answer can be obtained directly from the video without first resolving the intended target through external information.

  • OVD: Because no video is supplied at test time, the relevant shortcut is whether the requested visual answer can be recovered without discovering and inspecting video footage. We therefore reveal the intended anchor/event and test two non-video routes separately: a textual-evidence probe using web search and page reading, and a still-image probe that additionally permits static web images and video thumbnails.

  • Image-based clues: A direct-recognition probe provides only the query image and a coarse entity type with retrieval disabled, filtering cases where the depicted entity can be recognized immediately from parametric knowledge without research.

For every applicable probe, we sample up to two independent solving attempts, and a single successful bypass is sufficient to reject the candidate. These probes removed 101/450 O2R candidates (22.4%), 126/700 KGO candidates (18.0%), and 239/1,500 OVD candidates (15.9%), indicating that shortcut filtering removes a substantial fraction of generated candidates.

The controlled settings you suggest, such as question-only, text-only research, video-only, and full research, serve the same general purpose as our shortcut tests, but the relevant shortcut differs across the three task configurations. For example, a video-only test is meaningful for O2R and KGO, where a video is supplied, but not for OVD, where the agent must first discover a video on the web. For OVD, the more relevant question is whether the visual answer can already be recovered from text or static images without inspecting any video. We therefore design the shortcut tests separately for each task, based on the information that the task is intended to require, and reject any candidate that can still be solved after that information is removed.


W3: Fine-Grained Evaluation of Different Capabilities

Our evaluation uses end-to-end Accuracy together with trajectory-level failure attribution to distinguish overall task success from the specific stage at which a research attempt breaks down. In Sec. 4.5, we inspect the complete interaction trace of each failed trajectory and assign its primary failure to the earliest consequential unrecovered error among six categories: Anchor Resolution, Evidence Retrieval, Visual Understanding, Reasoning, Research Control, and Budget Exhaustion.

This analysis directly diagnoses several of the capabilities highlighted in your suggested metrics. Failures in research-anchor identification are categorized as Anchor Resolution, while failures to acquire the required external facts, sources, or video evidence are categorized as Evidence Retrieval. The resulting profiles reveal substantial differences that aggregate Accuracy obscures. For example, Visual Understanding accounts for 66.9% of Gemini-3.6-Flash's failures, whereas Qwen3-VL is dominated by Anchor Resolution (27.7%), Evidence Retrieval (30.2%), and Research Control (32.9%). We have clarified the category definitions and attribution rule in the revised Sec. 4.5.

The other two suggested metrics are less directly applicable to UNIVERSE. In OVD, the construction-time supporting video v⋆v^\star is a verified witness rather than a unique ground-truth retrieval target: an agent may correctly discover a different upload or recording containing the required evidence, so recall against a single reference video could mark a successful retrieval as incorrect.

AuthorsOpenReview ↗

Response to Reviewer (3/3)

W3: Fine-Grained Evaluation of Different Capabilities (continued)

Likewise, temporal localization is not an explicit prediction target in our evaluation: agents must obtain the relevant visual evidence to answer the question, but are not required to output timestamp intervals. Introducing timestamp-level accuracy would therefore evaluate an additional task beyond the current benchmark interface.

We therefore use end-to-end Accuracy to measure overall research success and trajectory-level failure attribution to identify where that process breaks down, rather than imposing stage-wise metrics that require a unique retrieval target or additional annotations not inherent to the task.


Q1: Do the Tasks Genuinely Require Deep Research?

We address this concern by examining whether the benchmark construction enforces the intended information dependencies and whether solving agents exhibit substantial iterative research behavior in practice.

As detailed in W2, every retained instance has passed task-specific counterfactual probes designed to remove the critical information source for that task, followed by dual-annotator human review. We additionally conducted an independent stratified audit of 80 final questions, all of which passed inspection. These tests directly target the main ways in which a question could collapse into conventional video QA, standalone retrieval, or a static-image/text shortcut.

The actual agent trajectories provide further evidence of substantial iterative research behavior rather than one-shot retrieval. As shown in Fig. 2(b–d), even our strongest model repeatedly interacts with the tool environment across all three tasks, with particularly intensive tool use on OVD. Tool access improves Gemini-3.6-Flash by +10.7, +7.3, and +24.3 pp on O2R, KGO, and OVD, respectively, yet its overall Accuracy remains only 56.46%. The combination of construction-time counterfactual checks and observed multi-step tool use shows that UNIVERSE is not reducible to conventional video QA or single-step retrieval and instead requires coordination between external information seeking and visual evidence.


Q2: Does the Benchmark Truly Evaluate Deep Research Capabilities?

We believe UNIVERSE genuinely evaluates deep research capabilities. Final-answer Accuracy measures end-to-end research success, while complete interaction trajectories allow us to analyze the intermediate research process that produces that outcome.

Deep research does not have a unique correct trajectory. Different agents may use different queries, sources, videos, and tool sequences while still arriving at the same well-supported answer. We therefore use Accuracy to measure whether the complete research process ultimately succeeds, rather than scoring intermediate steps against a single prescribed trajectory.

Beyond final-answer correctness, we analyze the complete interaction trajectories to characterize the capabilities that lead to success or failure. In Sec. 4.5, failed trajectories are attributed to the earliest unrecovered error among Anchor Resolution, Evidence Retrieval, Visual Understanding, Reasoning, Research Control, and Budget Exhaustion. These categories correspond closely to the capabilities you highlight. Research Control captures ineffective planning, repeated unproductive actions, insufficient verification, and premature termination; Evidence Retrieval captures failures to acquire the required external facts, sources, or video evidence; Reasoning captures failures to correctly combine or use acquired evidence; and Visual Understanding captures errors in interpreting image or video evidence. We also analyze tool-use frequency and tool composition across tasks, which reveals how models conduct iterative search, and use the paired Text Oracle experiment to isolate the effect of multimodal grounding on the downstream research process.

Thus, UNIVERSE evaluates deep research at two complementary levels: Accuracy measures whether the end-to-end research process succeeds, while trajectory-level analyses reveal how models plan, retrieve, integrate, and interpret evidence along the way. This allows us to assess deep research capability without assuming that all successful agents must follow the same research path.


Thank you again for your encouraging assessment and constructive suggestions. We hope our responses and revisions have addressed your concerns and further clarified the contributions and reliability of UNIVERSE. We would be happy to discuss any remaining questions and appreciate your time and consideration.

Reviewer_C6qwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces UNIVERSE, an 800-question benchmark covering observation-to-research, knowledge-guided observation, and open-web video discovery, with optional image-based clues. It presents a construction and verification pipeline and evaluates eight multimodal models, highlighting challenges in coordinating external research, video discovery, and visual understanding.

Critical strengths
  1. Clear decomposition of video deep research. This paper organizes video deep research into three complementary task configurations, distinguishing supplied-video research from open-web video discovery and separating observation-to-research from knowledge-guided observation. This organization clearly captures different dependencies between external research and video observation, providing a coherent structure for evaluating complementary capabilities within a unified benchmark (Sec. 3.1).

  2. Well-motivated KGO task. This paper introduces knowledge-guided observation (KGO), where agents resolve a target from external clues before locating it in a supplied video to answer a visual question. This task complements observation-to-research by evaluating how external knowledge guides what to observe in a video (Sec. 3.2.2, p. 5, lines 216–232).

  3. Active acquisition of video evidence. This paper includes open-web video discovery (OVD), requiring agents to actively retrieve, select, and inspect relevant videos. This broadens the benchmark's coverage to video evidence acquisition and selection, complementing research over supplied videos (Sec. 3.2.2, p. 5, lines 234–248).

Critical weaknesses
  1. Limited validation of the answer judge and failure annotator. This paper uses Gemini-3.6-Flash as both the final-answer judge and the failure-mode annotator, while also including it among the evaluated models (Secs. 4.1 and 4.5). The sensitivity of the reported findings to these choices remains unclear. Please discuss potential judge/annotator biases and, ideally, assess how using alternative models in each role affects answer accuracies and model rankings.

  2. Insufficient detail on evaluation prompts. This paper describes a shared tool-enabled framework (Sec. 4.1 and Appendix C) but does not provide the full system/instruction prompts or clarify whether agents are explicitly informed of the task type (O2R/KGO/OVD) or given task-specific workflow guidance. These details are needed to interpret how much planning is performed autonomously and to reproduce the evaluation. Please provide the complete evaluation prompts and clarify any differences across tasks or models.

  3. Unexamined sensitivity to the video backend. This paper uses Gemini-3-Pro as the shared VideoSummary backend (Sec. 4.1), which provides a common tool environment. However, OVD results may depend on the backend's reliability and its compatibility with each agent's queries: a valid video URL and well-posed question could still yield an incorrect tool response. Please audit a sample of tool outputs, distinguish agent-side errors from backend visual errors in the failure analysis (Sec. 4.5), and assess whether the main findings are robust to an alternative video backend.

  4. Insufficient detail on data construction and verification. This paper outlines a multi-stage construction and quality-control pipeline (Sec. 3.2 and Appendix B), but key operational details are missing: e.g., generation/verification models and prompts, the number of confirming judgments required for acceptance, etc. Please provide detailed configurations and clarify the human-review protocol to support reproducibility. Reporting candidate counts and retention/correction rates at each stage would also help assess the effectiveness and cost of quality control.

  5. Potential effects of retrieval coverage and result ranking. This paper allows agents to discover videos through WebSearch and ChannelVideos (Appendix C, Table 3), but does not specify the search providers/APIs, ranking and personalization settings. These factors may affect which relevant videos agents can discover and thus influence OVD performance. Please clarify the retrieval configuration and discuss potential coverage and ranking effects.

Questions for authors

See weakness.

6 comments
AuthorsOpenReview ↗

Response to Reviewer (1/4)

Thank you very much for your time, thoughtful review, and constructive feedback. We are encouraged that you recognized several core strengths of UNIVERSE, including its decomposition of video deep research into three complementary task configurations (O2R, KGO, and OVD), the motivation for Knowledge-Guided Observation (KGO), and the inclusion of Open-Web Video Discovery (OVD) to evaluate active video evidence acquisition and selection. Your comments on the reliability of our answer judging and failure attribution, prompt transparency, video-backend sensitivity, construction reproducibility, and retrieval configuration helped us substantially strengthen the paper. We address each point below and indicate where the corresponding changes appear in the revised manuscript.

W1: Validation of the Answer Judge and Failure Annotator

Our answer-judging setup is designed to minimize potential self-preference effects. The judge is blinded to the identity of the evaluated model and does not perform an open-ended assessment of response quality. Given the question, reference answer, and model prediction (with supporting visual context when needed for disambiguation), it determines only whether the prediction is semantically equivalent to the reference answer. Moreover, UNIVERSE uses highly concise factual answers (median length: 2 words; 88.4% contain at most 5 words), leaving little room for stylistic preference to influence correctness judgments.

To directly test sensitivity to the choice of judge, we re-evaluated all 38,400 model predictions using GPT-5.6-Sol as an independent cross-family judge under the same semantic-equivalence criterion. Across the 16 model–setting combinations, the two judges agree on 97.83% of trajectory-level decisions on average (96.62–98.71%). The resulting scores are highly stable: the mean absolute change is only 0.62 percentage points (pp) for Overall Accuracy (maximum: 1.17 pp) and 0.53 pp for Pass@3 (maximum: 0.88 pp). Crucially, model rankings under both metrics remain identical under the two judges in both tool settings (Spearman's ρ=1.0\rho=1.0). Gemini-3.6-Flash itself scores slightly higher under GPT-5.6-Sol than under its own judging—38.62% vs. 37.88% without tools and 56.79% vs. 56.46% with tools—providing no evidence of favorable self-bias.

We further validated both automated judges against blinded human annotations. On 160 stratified random predictions, human adjudicators agreed with Gemini-3.6-Flash on 96.9% (155/160) and with GPT-5.6-Sol on 98.1% (157/160). We additionally examined 50 cases where the two automated judges disagreed; human judgment sided with Gemini-3.6-Flash in 19 cases and with GPT-5.6-Sol in 31. We have added the full cross-judge re-evaluation, the per-model breakdown, and human validation of both automated judges to Appendix C.4 and Table 4 of the revised manuscript.

The failure-mode annotator serves a different purpose from the answer judge. It is applied only after a trajectory has already been judged incorrect and is used solely for post-hoc diagnosis; it therefore has no effect on Accuracy, Pass@3, or model rankings. Blinded to model identity, the annotator assigns each failed trajectory to the earliest consequential unrecovered error among six predefined categories with explicit operational definitions. We validate the interpretation of these labels through their consistency with observed tool-use behavior and through the targeted manual OVD audit described in W3, which directly examines the main attribution ambiguity introduced by the shared video backend. For example, InternVL3.5's 71.7% Research Control failure share is consistent with its near-zero tool usage, whereas stronger tool-using models concentrate more of their remaining failures in Visual Understanding. We have clarified the annotator's post-hoc role, category definitions, and failure-attribution rule in the revised Sec. 4.5.

AuthorsOpenReview ↗

Response to Reviewer (2/4)

W2: Evaluation Prompts and Autonomous Planning

We have added the complete verbatim evaluation prompts and detailed tool interfaces to Appendix C.1. Importantly, agents are never informed whether an instance is O2R, KGO, or OVD, and receive no task-specific workflow guidance. Within each evaluation setting, all models and task categories share the same task-agnostic instructions. In the tool-enabled setting, the system prompt specifies only the general multimodal research-agent role, the available tool primitives, interaction constraints, evidence verification, and the final-answer format. It explicitly leaves the agent to determine what to observe, what to search for, and in what order to use the available information sources and tools.

User messages vary only according to the modalities actually present in the instance—question text, optional query images, and an optional supplied video. In particular, O2R and KGO instances without image clues have the same input structure. The no-tool setting preserves the same user input while disabling external retrieval and online-video inspection. Across all eight evaluated models, the prompt instructions and tool semantics are identical; only API-specific message formatting and tool-call serialization are adapted to the corresponding interfaces. Thus, the ordering and coordination of external research and video observation are determined autonomously by each evaluated agent rather than prescribed by the task label or prompt. We have made this task-agnostic evaluation protocol explicit in the revised Sec. 4.1 and provide the complete tool-enabled and no-tool prompts in Appendix C.1.


W3: Video Backend Reliability and Failure Attribution

Our evaluation deliberately holds Gemini-3-Pro fixed as the shared VideoSummary backend across all agents, providing a common video-inspection environment. Under this setup, the evaluated agents remain responsible for discovering relevant videos, selecting which candidate URLs to inspect, formulating targeted questions, and integrating the returned observations into their subsequent reasoning.

To directly assess backend reliability, we manually audited 150 VideoSummary calls sampled from OVD trajectories across all eight agents. We first checked whether the selected video contained the requested evidence and whether the agent's query was sufficiently specific and answerable. 54/150 calls failed this prerequisite check because the selected video did not contain the required evidence or the query could not be reliably answered from it. Among the remaining 96 valid calls, 89 produced correct visual observations and 7 contained an incorrect backend observation, corresponding to 92.7% accuracy under appropriate calls.

We further audited 80 failed OVD trajectories end-to-end to determine whether a backend error was the primary cause of failure. Only 5/80 (6.3%) were primarily attributable to an incorrect VideoSummary observation; the remaining failures predominantly arose from unsuccessful video retrieval, ineffective query formulation, or incorrect downstream use of acquired evidence. Thus, backend visual errors are not a dominant source of observed OVD failures in this sample.

Because the evaluation intentionally fixes a common video backend, we do not interpret the reported OVD scores as backend-invariant estimates. Rather, they measure performance under a fixed shared agent–tool environment. The audits above quantify the reliability of that environment and show that backend errors account for only 5/80 sampled failed OVD trajectories. Moreover, OVD is consistently the most challenging task even in the no-tool setting, where VideoSummary is not involved. This suggests that its difficulty extends beyond the use of the shared video backend. We have incorporated the call-level reliability audit and trajectory-level attribution audit into the new Appendix D.4, and revised Sec. 4.5 to make the system-level interpretation and remaining backend dependence explicit.

AuthorsOpenReview ↗

Response to Reviewer (3/4)

W4: Data Construction, Verification Configurations, and Filtering Statistics

Thank you for pointing out the missing operational details. We have substantially expanded Appendix B to provide a detailed account of the construction and quality-control pipeline, including model configurations, verification procedures, shortcut probes, human-review protocols, and stage-level filtering statistics.

Model configurations and separation of duties. The new Table B.1 specifies 11 model-assisted operations spanning generation (G1–G6), evidence verification (V1–V3), and counterfactual shortcut probing (S1–S2), together with the eligible models and the input/output role of each operation. The model pool comprises GPT-5, GPT-5.1, GPT-5.2, GPT-5.4, Gemini-2.5-Pro, Gemini-3-Pro, and Gemini-3.1-Pro. Operations that inspect video frames (G1, G2, and V3) use Gemini-family models, while text- and image-based operations draw from the corresponding eligible pools. Generation, verification, and shortcut probing are conducted in isolated sessions with dedicated prompts, and verification samples a different model from the generator whenever multiple models are eligible, reducing self-confirmation effects.

Evidence verification. Candidates pass three complementary verification operations. V3 checks visual grounding and answer correctness against the supporting images or video; V1 verifies factual and relational consistency against the underlying graph paths and archived web evidence; and V2 checks target specificity and ambiguity by verifying consistency with the intended anchor and searching for plausible competing entities or events satisfying the same clues. Each applicable verification check requires two independently sampled confirmed judgments; a refuted or inconclusive verdict causes rejection. We have made these verification roles and the dual-confirmation acceptance rule explicit in Appendix B.3.

Counterfactual shortcut filtering. Candidates that pass evidence verification are then subjected to task-specific probes that test whether the intended information dependency can be bypassed. For O2R, we test whether the anchor or answer can be recovered without video access or without the anchor-identifying visual reference. For KGO, we remove the external anchor clues and test whether the requested visual answer can be obtained directly from the supplied video without resolving the target. For OVD, we provide the intended anchor or event while disabling video access, testing separately whether the answer can instead be recovered from textual web evidence or static images/thumbnails. Image-based clues additionally undergo a direct-recognition probe with retrieval disabled, filtering cases where the depicted entity can be identified immediately without research. Each applicable probe allows up to two independent solving attempts, and a single successful bypass is sufficient to reject the candidate. The revised Appendix B.4 and Table B.2 specify the counterfactual inputs, permitted tools, shortcut being tested, and rejection rule for each probe.

Human review and filtering statistics. Every candidate surviving automated verification and shortcut filtering is independently reviewed by two annotators drawn from three graduate students, and both must approve it based on question clarity, factual consistency, target unambiguity, visual-evidence sufficiency, and answer validity. Across all human-reviewed candidates, inter-annotator agreement reaches 90.1% (Cohen's κ=0.64\kappa=0.64).

The revised Appendix B.5 and Table B.3 report the complete stage-level filtering statistics. Each rejected candidate is counted only at the first stage where it fails; removal percentages use the initial candidate count for the corresponding task as the denominator.

Construction & Quality-Control StageO2RKGOOVDTotal
Initial candidates4507001,5002,650
Removed: Programmatic consistency checks29 (6.4%)54 (7.7%)96 (6.4%)179 (6.8%)
Removed: Visual grounding verification (V3)62 (13.8%)117 (16.7%)308 (20.5%)487 (18.4%)
Removed: Factual and ambiguity verification (V1–V2)42 (9.3%)163 (23.3%)178 (11.9%)383 (14.5%)
Removed: Counterfactual shortcut filtering (S1–S2)101 (22.4%)126 (18.0%)239 (15.9%)466 (17.6%)
Removed: Human review48 (10.7%)49 (7.0%)143 (9.5%)240 (9.1%)
Passed quality control168 (37.3%)191 (27.3%)536 (35.7%)895 (33.8%)
Excluded during final random sampling18413695
Final dataset150150500800

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Manuscript Revisions

We sincerely thank all reviewers for their thoughtful and constructive feedback. We have carefully considered the comments and revised our manuscript accordingly. To facilitate the review of the updated paper, we summarize the main revisions below.

  1. Expanded benchmark construction details (Appendix B.1–B.3). We substantially expanded the documentation of our automated construction pipeline, including the models and operations involved in question generation, the separation of generation and verification, and the multi-stage evidence verification procedures.

  2. Strengthened benchmark quality validation (Section 3.2, Appendix B.4–B.5). We detailed the task-specific counterfactual shortcut probes used to verify the intended dependencies on external research and video observation. We also reported the human review protocol, stage-level filtering statistics, and inter-annotator agreement (90.1%, Cohen's κ = 0.64). An additional independent audit of 80 final questions confirmed that all sampled instances satisfied our quality criteria.

  3. Clarified evaluation protocols and tool configurations (Section 4.1, Appendix C.1–C.3). We provided the complete evaluation prompts, tool schemas, implementation details, and retrieval configurations. We also clarified that evaluation is task-agnostic, with no task-specific workflow instructions, and that all models share the same tool interfaces, retrieval backends, and configurations.

  4. Validated answer-judging reliability (Appendix C.4). We independently re-evaluated all 38,400 model predictions using GPT-5.6-Sol. The two judges achieved 97.83% agreement, with only minor changes in aggregate metrics and unchanged model rankings. We further conducted blinded human adjudication to assess the reliability of both judges.

  5. Audited video-backend reliability and failure attribution (Section 4.5, Appendix D.4). We manually examined 150 VideoSummary calls and 80 failed OVD trajectories to distinguish backend observation errors from failures in the agents' research processes. Only 5 of the 80 failed trajectories (6.3%) were primarily attributable to backend errors, providing additional evidence for interpreting our failure analysis.

We hope these revisions address the reviewers' key concerns and make the benchmark methodology, evaluation procedures, and empirical findings clearer and more transparent. We sincerely appreciate your time and consideration, and we would be happy to discuss any remaining questions or suggestions.

AuthorsOpenReview ↗

General Response

We sincerely thank the Program Chairs, Area Chair, and all reviewers for their time and thoughtful feedback. The discussion has helped strengthen the validation and transparency of UNIVERSE. Before the discussion period concludes, we would like to briefly summarize the main points raised across the reviews and the corresponding evidence and revisions.

Across the three reviews, we are encouraged that the core contributions of UNIVERSE were consistently recognized. Reviewer C6qw highlighted the clear decomposition of video deep research into complementary O2R, KGO, and OVD settings, the motivation for Knowledge-Guided Observation, and the inclusion of active open-web video evidence acquisition. Reviewer WDnN further recognized the comprehensive task design, the scale and scalability of the construction pipeline, and the systematic evaluation across eight representative models. Reviewer GjSg highlighted the meaningful KGO formulation and the paired Text Oracle evaluation as useful evidence for the difficulty introduced by image-based research clues.

The main concerns raised across the reviews centered on benchmark reliability, shortcut prevention, evaluation reproducibility, judge reliability, and failure attribution. We substantially revised the manuscript and added new validation to address these points:

  1. Benchmark quality and required research dependencies. We now fully specify the generation and verification pipeline, stage-level retention statistics, dual-confirmation verification rules, task-specific counterfactual shortcut probes, and human-review protocol. Human annotators achieved 90.1% agreement (Cohen's κ = 0.64), and an additional stratified audit of 80 final questions found all 80 to satisfy the quality criteria. The task-specific counterfactual probes explicitly test whether O2R, KGO, and OVD can be solved while bypassing their intended information dependencies, complementing the observed multi-step research behavior of evaluated agents.

  2. Independent validation of answer judging. We re-evaluated all 38,400 predictions using GPT-5.6-Sol as an independent cross-family judge. The two judges agree on 97.83% of trajectory-level decisions on average; the mean absolute change in Overall Accuracy is only 0.62 pp, and model rankings remain unchanged in both evaluation settings (Spearman's ρ = 1.0). We further conducted blinded human adjudication on both a stratified random sample and judge-disagreement cases, with both automated judges showing high agreement with human judgments.

  3. Evaluation transparency and reproducibility. The revised manuscript now provides the complete tool-enabled and no-tool prompts, explicitly clarifies that agents are not informed of the O2R/KGO/OVD task category and receive no task-specific workflow guidance, and documents the shared tool implementations, retrieval providers, ranking procedures, result limits, video preprocessing, evaluation period, and handling of unavailable evidence. These additions make clear that research planning and tool use are determined autonomously by the evaluated agents under a common environment.

  4. Video-backend reliability and failure attribution. We manually audited 150 VideoSummary calls and found that, among calls with an appropriate video and answerable query, 89/96 (92.7%) returned correct visual observations. We separately examined 80 failed OVD trajectories, of which only 5/80 (6.3%) were primarily attributable to an incorrect backend observation. We accordingly clarified that the failure analysis should be interpreted as an end-to-end diagnosis of the shared agent–tool system and explicitly separated backend observation errors from downstream evidence use.

We greatly appreciate the reviewers' engagement with these additions. In the subsequent discussion, Reviewer WDnN stated that our response adequately addressed the main concerns and maintained the positive evaluation. Reviewer GjSg specifically noted that the cross-judge evaluation and blinded human validation substantially alleviated the concerns about judging reliability, and that the benchmark-quality audit and clarified failure attribution strengthened the paper, subsequently raising the evaluation from 2 to 3. We have also addressed the concerns raised by Reviewer C6qw through the additional judge validation, complete evaluation prompts and tool configurations, video-backend audit, expanded construction and verification details, and retrieval-configuration analysis described above.

Thank you again for the constructive reviews and discussion. We hope that the additional experiments, audits, and detailed revisions have clarified the reliability, reproducibility, and intended research requirements of UNIVERSE, while preserving its central contribution as a unified benchmark for video deep research. We hope the revised manuscript and discussion provide a clear basis for the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.