acceptodds
Under review as a conference paper at ICLR 2027

Beyond Watching and Searching: Benchmarking Agents for Video-centered Deep Research Reports

Abstract

Deep research agents synthesize open-web evidence into long reports, but current benchmarks do not evaluate research centered on a source video. We introduce Video-Centered Deep Research Report Generation. Given a video and user request, an agent identifies research questions, retrieves external multimodal evidence, and produces an interactive report linked to relevant video moments. This task has direct practical value because it helps users locate relevant moments, verify claims, reproduce demonstrated procedures, and make informed decisions from video content. We present a skill-based workflow and OmniDR-Bench for this task. For evaluation, we propose Omni Atomic Rubric Trees (Omni-ARTs). Each tree contains three dimensions, 23 shared directions, and task-specific atomic binary criteria. The directions test audiovisual grounding, multimodal evidence use, and interactive presentation. Audited Rubric Construction (ARC) builds these trees through independent evidence collection, automated checks, behavioral calibration, and iterative human review. OmniDR-Bench contains 50 Chinese and English tasks across 19 real-world subcategories. It includes 3,047 evidence records and 6,414 criteria. Across 18 agent configurations, the best system scores 63.3 out of 100. Current agents still need broader content coverage, stronger causal and logical analysis across modalities, and more accurate citation support. We further analyze models, harnesses, skill configurations, video categories, and research trajectories.

Primary area:
datasets and benchmarks
Keywords:
Deep Research, Video Agent

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mZckOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes video-centered deep research report generation: an agent gathers multimodal evidence from a source video and user request, then produces an interactive HTML report linked to video moments. OmniDR-Bench has 50 Chinese/English tasks, 3,047 evidence records, and 6,414 binary criteria (content, citation, presentation), built via Audited Rubric Construction independent of evaluated reports. Across 18 model–harness configurations, presentation quality substantially exceeds audited evidence coverage, reasoning, and claim support. A 15-task ablation shows the full workflow beats atomic-skill access, mainly via citation quality, at higher cost.

Critical strengths
  1. The task is useful and goes beyond video summarization: agents need to connect the video to external evidence and produce a report users can inspect. The evaluation also helps distinguish finding sources from actually using them to support claims.

  2. The authors put considerable effort into checking the benchmark and its evaluation. They include repeated human review, comparisons with human report ratings, and checks on scoring choices. The workflow experiments also show where quality improves and how much extra cost it takes.

Critical weaknesses
  1. The N/A guard may not fully prevent a report from scoring higher by omitting a fact. When an omitted fact makes a failed citation check inapplicable, removing that check changes both the citation denominator and the relative weights of the citation directions. The guard appears to miss part of this effect. As a simplified example, suppose one citation direction has two failed checks and another has one passing check. With weights proportional to check counts, the citation score is 1/3. Removing one failed check raises it to 1/2. Although the first direction still scores zero, the passing direction now carries more weight. If this gain exceeds the content penalty for omitting the fact, the less informative report receives a higher total score.The authors should clarify whether the implementation checks the exact final-score change and whether this issue affects the reported results.

  2. The experiments do not clearly show how much the video’s visual information adds beyond a transcript. The video capability bundles transcription, frame extraction, and video reading together. A comparison using only transcripts with the same web access would help establish what the video itself contributes.

Questions for authors
  1. Could you clarify how the implementation handles changes in citation-direction weights when failed checks are marked N/A? Does the guard check the exact final score to ensure that omitting a fact cannot score higher than stating it correctly with insufficient citation? If a correction is needed, please report its effect on scores, rankings, and the main conclusions.

  2. Could you add a transcript-only comparison on the existing 15-task subset, keeping web access and report requirements unchanged? This would help show how much visual and temporal information from the source video contributes. If additional runs are not feasible, an audit of criteria that require information unavailable in the transcript would also help.

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for recognizing the value of video-centered research reports, human verification, and the proposed workflow. Below, we provide detailed responses to the issues you raise.

W1&Q1: More Details about N/A Guarding

Our goal is to avoid penalizing the same omitted fact in both content and citation, while preventing omission from yielding a higher score. Before marking citation criteria for omitted facts as N/A, Equation (5) estimates two gains: the content gain from stating the fact and the citation gain from removing citation checks. N/A is retained only when the former is larger. Otherwise, failed citation checks are restored. We further test its practical effect by comparing the final overall scores:

Practical effect of Equation (5): We use the reports in the main table and select 4,770 cases of fact omission with citation criteria eligible for N/A before applying the guard. Keeping criterion-level judgments fixed, we compare N/A removal with and without the Equation (5) check. We measure the fraction of cases in which omitting a fact scores no higher than stating it without citation. Applying Equation (5) increases this fraction from 74.19% to 86.58%, a gain of 12.39 percentage points. This shows that the guard helps preserve the intended score ordering.

Equation (5) estimates the gains from the current scoring state (the na-order check verifies whether Equation (5) holds). The final overall score also depends on changes in citation weights in Equation (3), so this check cannot strictly guarantee the final score ordering. To address your concerns about scores, rankings, and the main conclusions, we conduct a single-fact sensitivity analysis of the 640 remaining reversals (640/4770). For each report, we subtract the largest observed single-fact reversal gap from its original overall score. The mean scores of the 18 configurations decrease by 0–0.63 points. The top three systems, Claude Opus 5, GLM-5.3, and Claude Opus 4.8, change from 63.35, 62.05, and 60.72 to 63.25, 61.57, and 60.10, respectively. All rankings remain unchanged, indicating stability under this single-fact sensitivity analysis.

We will clarify the scope of the Equation (5) check and include these results in the revised paper.

W2&Q2: About the Contribution of Visual Information

We add a transcript-only comparison on the existing 15 ablation tasks, using the full workflow in Table 5 (configuration v) as the baseline. Both conditions use GPT-5.6 Sol and Claude Code, with the same retrieval permissions, report requirements, and resource limits. The transcript-only condition reuses the timestamped speech transcripts saved from the full audiovisual experiment, without frame descriptions or OCR results. We remove the original video to control information access during generation. If the agent still receives the video file, it can recover frames through frame extraction or multimodal tools. The condition would then no longer be transcript-only. We therefore remove the source video and block known access points to it during generation, while retaining web access for external research. After the report text and images are finalized, we reconnect the same source video to check playback and interaction. The difference is “transcript-only − full audiovisual.”

MetricFull audiovisualTranscript-onlyDifference
Content40.9334.29−6.64
Citation55.3649.81−5.55
Presentation83.9151.22−32.69
Overall53.8642.33−11.52

The overall score drops by 11.52 points. Part of this comes from presentation, such as the availability of report images. However, the content score also drops by 6.64 points, showing that the lack of visual understanding reduces report quality. Concrete examples further show how visual information affects report content:

  1. In the CNC keyway task, the full audiovisual report uses machine-tool footage to explain the current Z coordinate, target Z coordinate, and designed groove depth shown on the screen. It warns readers against directly copying the live coordinates. The transcript-only report gives the groove-depth formula but does not verify the values on the machine interface.
  2. In the under-display camera task, the full audiovisual report analyzes image quality using enlarged selfie images shown in the video and cross-checks them against external research. The transcript-only report mainly relies on spoken content and lacks direct analysis of the actual image quality shown in the source video.

We hope these responses can address your concerns. We are committed to incorporating all suggested improvements in the revised version and welcome any further feedback.

Thank you for the careful response and the additional experiments. The sensitivity analysis makes me less concerned that the remaining scoring reversals affect the main conclusions, and the transcript-only comparison gives clearer evidence that access to the video improves report content.

I’ll keep my weak accept rating. The findings aren’t especially surprising, but the experiments are solid, and the benchmark is useful for understanding where current systems fall short.

Dear Reviewer mZck:

We sincerely appreciate your thoughtful and constructive feedback. We are glad that our rebuttal addresses your concerns, and we appreciate your positive assessment of the paper.

Reviewer_FHewOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This is a benchmark paper that focuses on deep research report generation, particularly for video. The agent is asked to view a video and read a user query, then generate a report based on evidence from the source video and external research. The proposed benchmark, OmniDR-bench, contains 50 Chinese and English tasks. To collect the dataset, the authors establish a genetic workflow with humans to select and refine possible user queries, evidence from the source video, and external knowledge. The authors also establish corresponding metrics covering various dimensions. Experiments compare 18 model-harness configurations and identify gaps from different perspectives, such as content coverage, evidence integration, and citation support.

Critical strengths
  1. I think the task is quite practical. Users want to quickly summarize a video, get correct supporting evidence from it, and cross-verify the information with external knowledge. This benchmark tries to make this process quantifiable.

  2. The evidence records and criteria undergo human review and correction, with human agreement checks.

Critical weaknesses
  1. It is unclear to me whether the proposed rubric can objectively evaluate different but valid research outcomes. A report may identify useful findings outside the constructed library. How can such a report be evaluated in this case?

  2. (Major) The comparisons rely heavily on an LLM as a judge, GPT-5.6 sol, but GPT-5.6 is also evaluated in Table 2. This raises a concern about possible preferences and bias within GPT-5.6. The human validation on 30 reports is helpful, but it does not establish an automatic evaluation pipeline. If the benchmark is to be used in different scenarios, it is difficult to have humans evaluate every case.

Questions for authors
  1. In the evaluation metric, can the authors explain why the saturation threshold is 0.18? Why is it appropriate?

  2. In fact, GPT-6 Astra and Claude Opus 5.5 are the most advanced models. I understand that they were released very close to the ICLR deadline, so the authors may not have had time to include them in the paper. But I think the authors should report the performance of these most advanced models in the rebuttal.

5 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for recognizing the practical value of the task and the human verification of the evidence and evaluation rubric. Below, we provide detailed responses to the issues you raise.

W1: About Valid Research Findings beyond the Evidence Library

Our current annotation design tries to ensure, as far as possible, broad coverage of facts and findings relevant to the user query. For the video, we record speech, visual content, procedures, and key parameters segment by segment. External retrieval covers related entities, original sources, concepts and standards, relevant people, similar cases, and event timelines. We follow new leads over multiple rounds. Retrieval stops when two consecutive rounds yield few new relevant facts and sources, or when the preset round limit is reached. This reduces omissions caused by a single search or search path. After collection, we merge duplicates and preserve disagreements across sources. Multiple reviewers then check each fact and criterion for every candidate task. Broad retrieval and human verification help cover important facts and make the scoring basis reliable.

Even when a report contains useful findings outside the library, we do not add fact-coverage credit specifically for that report. This keeps the scoring criteria consistent across reports. We have used multiple retrieval rounds and human annotation to minimize such omissions.

We also test whether these fixed scoring requirements reflect the practical value of reports. As reported in the paper, human raters assess the overall usefulness and trustworthiness of 30 reports. Automatic scores have a Spearman correlation of 0.64 with the holistic human ratings. This provides additional support for the validity of the current scoring.

W2: About LLM Judge Robustness

We add an evaluation of the same reports by three judges. We fix 45 reports generated by three models on the same 15 tasks. GPT-5.6 Sol, Qwen3.8-Max, and Claude Opus 4.8 each score these reports, yielding 135 evaluations.

Report-generating modelGPT-5.6 Sol judgeQwen3.8-Max judgeClaude Opus 4.8 judge
GPT-5.6 Sol52.0051.6453.74
Qwen3.8-Max59.0358.4660.96
Claude Opus 4.858.0657.1059.40

All three judges give the same ranking of the three models by mean score. The ranking does not change when the report generator and judge are from the same provider. Compared with GPT-5.6 Sol, the alternative judges have report-level Spearman correlations of 0.721–0.932, agreement rates of 94.47%–97.53% on model-judged criteria, and Cohen's κ of 0.807–0.897. These results show stability across judge choices.

Q1: More Details about the Saturation Threshold

The reference fact library undergoes extensive repeated retrieval, correction, and supplementation. It usually contains more information than an agent can fit into one report. We therefore set a threshold for full coverage credit, so a report does not need to include every reference fact to receive full credit. The current saturation threshold is estimated from report length. During construction, we randomly selected 12 examples. The median length of their key-fact text is about 21,700 characters, while the factual statements in an agent-generated report occupy about 4,000 characters. The ratio is approximately 18%. We use this ratio to set the estimated threshold for full coverage credit at 0.18.

Appendix F.2 further tests nearby thresholds:

Saturation thresholdClaude Opus 5GLM-5.3Claude Opus 4.8Maximum rank change across all 18 configurations
0.1265.965.163.60
0.1863.362.060.70
0.2460.559.457.91

The top three systems and their order remain unchanged. No configuration moves by more than one rank. The main rankings thus do not depend heavily on the exact value of 0.18, indicating robustness in this hyperparameter choice.

Q2: About the Results of GPT-6 Astra and Claude Opus 5.5

GPT-6 Astra and Claude Opus 5.5 were released in September 2026, around the ICLR deadline. Testing these latest frontier models is therefore valuable. We evaluate both with Claude Code under the same protocol. The results are as follows:

ModelReportsOverall95% CIContentCitationPresentation
GPT-6 Astra5063.4[61.1, 65.7]51.173.080.0
Claude Opus 5.55070.4[66.5, 74.9]65.964.390.7

GPT-6 Astra achieves a mean score of 63.4, clearly outperforming the GPT-5.6 Sol (54.9). Claude Opus 5.5 achieves 70.4, the highest score among the tested models.

We hope these responses can address your concerns. We are committed to incorporating all suggested improvements in the revised version and welcome any further feedback.

AuthorsOpenReview ↗

Further Discussion or Feedback

Dear Reviewer FHew,

As the discussion period is approaching its end, we would greatly appreciate any further feedback on our responses, especially regarding whether our clarifications have addressed your main concerns.

Thank you again for your time and consideration!

Thank you for the clarifications and additional experiments. The cross-judge evaluation provides useful evidence of scoring stability, and the additional frontier-model results address my request. However, I remain concerned about the validity of the LLM-based evaluation. Agreement among judges on the sampled reports does not establish that the scores reliably distinguish substantive research quality from shared model preferences. The existing human validation offers some support, but together these results do not sufficiently resolve this concern for me. I therefore maintain my rating.

Reviewer_NG1cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces video-centered deep research report generation. Given a video and a user request, an agent gathers external multimodal evidence and produces an interactive report linked to video moments and sources. OmniDR-Bench contains 50 Chinese and English tasks, 3,047 evidence records, and 6,414 atomic criteria. Omni-ART evaluates content, citation, and presentation; ARC constructs and audits the evidence and criteria. Experiments compare 18 agent configurations. Under a shared report template, systems satisfy presentation requirements more consistently than research coverage and claim-support requirements. Full orchestration mainly improves citation quality while increasing realized computational cost.

Critical strengths
  1. The task addresses a useful evaluation gap. Producing a complete research report requires choosing what to investigate, integrating external evidence, and making findings inspectable. Linking those findings to a source video provides a meaningful setting for studying research agents beyond short-answer accuracy.
  2. The benchmark supports detailed diagnosis. Independent evidence acquisition, evidence-bound criteria, programmatic checks, and human review make individual judgments easier to inspect. The human validation in Appendix A.6 and sensitivity analysis in F.2 provide useful support. The experiments distinguish source acquisition from claim support and show how a well-presented report can still omit important research content.
Critical weaknesses
  1. The contribution of visual information beyond transcripts is not directly established. Appendix B.2 and Tables 10 and 25 provide reasonable curation rules and examples. However, B.2 explicitly acknowledges that Table 5 does not separate frames from transcripts, and F.5 bundles transcription, frame extraction, and multimodal reading. The experiments therefore do not show how much important research content depends on visual access rather than timestamped text and retrieval. This limits the evidence for the benchmark as a measure of visual research ability. A content-focused comparison would be more informative than a change in the overall score, which also includes video-interface checks.
  2. The N/A guard needs clarification under the final aggregation rule. Appendix A.4 explicitly aims to prevent omissions from benefiting when failed citation criteria become inapplicable. Equation (3), however, weights directions by their final applicable criterion counts. Equation (5) estimates changes in direction pass rates but does not appear to fully account for the resulting reweighting of other directions. It remains unclear whether the guard preserves the intended score ordering after those weights are recomputed. I have not verified the evaluator implementation, so this is a concern about the published rule rather than an established error in the reported results.
Questions for authors
  1. Additional validation. Could you compare timestamped transcripts with full audiovisual input on a representative subset, keeping retrieval permissions, report requirements, and resource limits fixed? Please prevent the transcript condition from recovering the original frames. Report the change on criteria identified in advance as visually dependent, separately from playback and navigation checks. Which important findings are lost without visual access?
  2. Technical clarification. How does Equation (5) account for direction-weight changes after N/A removal? Does the implementation directly check that omitting a fact cannot produce a higher final score than stating it correctly without adequate citation? Please clarify whether the offline na-order check verifies this final-score ordering or only the guard inequality. If the two checks can differ, does this affect the reported results?
3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for recognizing the practical value of the task and the diagnostic value of the evaluation. Below, we provide detailed responses to the issues you raise.

W1&Q1: About the Contribution of Visual Information

We add a transcript-only comparison on the existing 15 ablation tasks, using the full workflow in Table 5 (configuration v) as the baseline. Both conditions use GPT-5.6 Sol and Claude Code, with the same retrieval permissions, report requirements, and resource limits. The transcript-only condition uses timestamped speech transcripts, without frame descriptions or OCR results. We remove the original video to control information access during generation. If the agent still receives the video file, it can recover frames through frame extraction or multimodal tools. The condition would then no longer be transcript-only. We therefore remove the source video and block known access points to it during generation, while retaining web access for external research. After the report text and images are finalized, we reconnect the same source video to check playback and interaction. The difference is “transcript-only − full audiovisual.”

MetricFull audiovisualTranscript-onlyDifference
Content40.9334.29−6.64
Citation55.3649.81−5.55
Presentation83.9151.22−32.69
Overall53.8642.33−11.52

The overall score drops by 11.52 points. Part of this comes from presentation, such as the availability of report images. However, the content score also drops by 6.64 points, showing that the lack of visual understanding reduces report quality. We also inspect examples from these tasks:

  1. In the CNC keyway task, the full audiovisual report uses machine-tool footage to explain the current Z coordinate, target Z coordinate, and designed groove depth shown on the screen. It warns readers against directly copying the live coordinates. The transcript-only report gives the groove-depth formula but does not verify the values on the machine interface.
  2. In the under-display camera task, the full audiovisual report analyzes image quality using enlarged selfie images shown in the video and cross-checks them against external research. The transcript-only report mainly relies on spoken content and lacks direct analysis of the actual image quality shown in the source video.

We will add this experiment to the revised paper.

W2&Q2: More Details about N/A Guarding

Our goal is to avoid penalizing the same omitted fact in both content and citation, while preventing omission from yielding a higher score. Before marking citation criteria for omitted facts as N/A, Equation (5) estimates two gains: the content gain from stating the fact and the citation gain from removing citation checks. N/A is retained only when the former is larger. Otherwise, failed citation checks are restored. We further test its practical effect by comparing the final overall scores:

Practical effect of Equation (5): We use the reports in the main table and select 4,770 cases of fact omission with citation criteria eligible for N/A before applying the guard. Keeping criterion-level judgments fixed, we compare N/A removal with and without the Equation (5) check. We measure the fraction of cases in which omitting a fact scores no higher than stating it without citation. Applying Equation (5) increases this fraction from 74.19% to 86.58%, a gain of 12.39 percentage points. This shows that the guard helps preserve the intended score ordering.

Equation (5) does not provide an exact estimate of the final score (the na-order check verifies whether Equation (5) holds). The final score is affected by the weighted aggregation in Equation (3), so some score-order reversals remain. To assess the impact of the 640 remaining reversals (640/4770), we conduct a sensitivity analysis. For each report, we subtract the largest observed single-fact reversal gap from its original overall score. The mean scores of the 18 configurations decrease by 0–0.63 points. The top three systems, Claude Opus 5, GLM-5.3, and Claude Opus 4.8, change from 63.35, 62.05, and 60.72 to 63.25, 61.57, and 60.10, respectively. All rankings remain unchanged, indicating stability under this single-fact sensitivity analysis.

We hope these responses can address your concerns. We are committed to incorporating all suggested improvements in the revised version and welcome any further feedback.

AuthorsOpenReview ↗

Further Discussion or Feedback

Dear Reviewer NG1c,

As the discussion period is approaching its end, we would greatly appreciate any further feedback on our responses, especially regarding whether our clarifications have addressed your main concerns.

Thank you again for your time and consideration!

AuthorsOpenReview ↗

Follow-up on Our Responses

Dear Reviewer NG1c,

As the discussion period will end in less than 8 hours, we would greatly appreciate it if you could share any further feedback on our responses. Thanks again for your time and consideration!

Comments on the paper

1 comment

Dear Reviewers,

We are approaching the deadline of the discussion period (Nov. 18). Please go through authors' rebuttal at your earliest convenience, and engage in a discussion or leave any follow-up comments.

Thank you for your time and participation.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.