- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes video-centered deep research report generation: an agent gathers multimodal evidence from a source video and user request, then produces an interactive HTML report linked to video moments. OmniDR-Bench has 50 Chinese/English tasks, 3,047 evidence records, and 6,414 binary criteria (content, citation, presentation), built via Audited Rubric Construction independent of evaluated reports. Across 18 model–harness configurations, presentation quality substantially exceeds audited evidence coverage, reasoning, and claim support. A 15-task ablation shows the full workflow beats atomic-skill access, mainly via citation quality, at higher cost.
Critical strengths
-
The task is useful and goes beyond video summarization: agents need to connect the video to external evidence and produce a report users can inspect. The evaluation also helps distinguish finding sources from actually using them to support claims.
-
The authors put considerable effort into checking the benchmark and its evaluation. They include repeated human review, comparisons with human report ratings, and checks on scoring choices. The workflow experiments also show where quality improves and how much extra cost it takes.
Critical weaknesses
-
The N/A guard may not fully prevent a report from scoring higher by omitting a fact. When an omitted fact makes a failed citation check inapplicable, removing that check changes both the citation denominator and the relative weights of the citation directions. The guard appears to miss part of this effect. As a simplified example, suppose one citation direction has two failed checks and another has one passing check. With weights proportional to check counts, the citation score is 1/3. Removing one failed check raises it to 1/2. Although the first direction still scores zero, the passing direction now carries more weight. If this gain exceeds the content penalty for omitting the fact, the less informative report receives a higher total score.The authors should clarify whether the implementation checks the exact final-score change and whether this issue affects the reported results.
-
The experiments do not clearly show how much the video’s visual information adds beyond a transcript. The video capability bundles transcription, frame extraction, and video reading together. A comparison using only transcripts with the same web access would help establish what the video itself contributes.
Questions for authors
-
Could you clarify how the implementation handles changes in citation-direction weights when failed checks are marked N/A? Does the guard check the exact final score to ensure that omitting a fact cannot score higher than stating it correctly with insufficient citation? If a correction is needed, please report its effect on scores, rankings, and the main conclusions.
-
Could you add a transcript-only comparison on the existing 15-task subset, keeping web access and report requirements unchanged? This would help show how much visual and temporal information from the source video contributes. If additional runs are not feasible, an audit of criteria that require information unavailable in the transcript would also help.
3 comments
Thank you for the careful response and the additional experiments. The sensitivity analysis makes me less concerned that the remaining scoring reversals affect the main conclusions, and the transcript-only comparison gives clearer evidence that access to the video improves report content.
I’ll keep my weak accept rating. The findings aren’t especially surprising, but the experiments are solid, and the benchmark is useful for understanding where current systems fall short.
Dear Reviewer mZck:
We sincerely appreciate your thoughtful and constructive feedback. We are glad that our rebuttal addresses your concerns, and we appreciate your positive assessment of the paper.
Rebuttal by Authors
Thank you for recognizing the value of video-centered research reports, human verification, and the proposed workflow. Below, we provide detailed responses to the issues you raise.
W1&Q1: More Details about N/A Guarding
Our goal is to avoid penalizing the same omitted fact in both content and citation, while preventing omission from yielding a higher score. Before marking citation criteria for omitted facts as N/A, Equation (5) estimates two gains: the content gain from stating the fact and the citation gain from removing citation checks. N/A is retained only when the former is larger. Otherwise, failed citation checks are restored. We further test its practical effect by comparing the final overall scores:
Practical effect of Equation (5): We use the reports in the main table and select 4,770 cases of fact omission with citation criteria eligible for N/A before applying the guard. Keeping criterion-level judgments fixed, we compare N/A removal with and without the Equation (5) check. We measure the fraction of cases in which omitting a fact scores no higher than stating it without citation. Applying Equation (5) increases this fraction from 74.19% to 86.58%, a gain of 12.39 percentage points. This shows that the guard helps preserve the intended score ordering.
Equation (5) estimates the gains from the current scoring state (the na-order check verifies whether Equation (5) holds). The final overall score also depends on changes in citation weights in Equation (3), so this check cannot strictly guarantee the final score ordering. To address your concerns about scores, rankings, and the main conclusions, we conduct a single-fact sensitivity analysis of the 640 remaining reversals (640/4770). For each report, we subtract the largest observed single-fact reversal gap from its original overall score. The mean scores of the 18 configurations decrease by 0–0.63 points. The top three systems, Claude Opus 5, GLM-5.3, and Claude Opus 4.8, change from 63.35, 62.05, and 60.72 to 63.25, 61.57, and 60.10, respectively. All rankings remain unchanged, indicating stability under this single-fact sensitivity analysis.
We will clarify the scope of the Equation (5) check and include these results in the revised paper.
W2&Q2: About the Contribution of Visual Information
We add a transcript-only comparison on the existing 15 ablation tasks, using the full workflow in Table 5 (configuration v) as the baseline. Both conditions use GPT-5.6 Sol and Claude Code, with the same retrieval permissions, report requirements, and resource limits. The transcript-only condition reuses the timestamped speech transcripts saved from the full audiovisual experiment, without frame descriptions or OCR results. We remove the original video to control information access during generation. If the agent still receives the video file, it can recover frames through frame extraction or multimodal tools. The condition would then no longer be transcript-only. We therefore remove the source video and block known access points to it during generation, while retaining web access for external research. After the report text and images are finalized, we reconnect the same source video to check playback and interaction. The difference is “transcript-only − full audiovisual.”
The overall score drops by 11.52 points. Part of this comes from presentation, such as the availability of report images. However, the content score also drops by 6.64 points, showing that the lack of visual understanding reduces report quality. Concrete examples further show how visual information affects report content:
We hope these responses can address your concerns. We are committed to incorporating all suggested improvements in the revised version and welcome any further feedback.