- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
UNIVERSE introduces 800 video deep-research questions covering observation-to-research, knowledge-guided observation, and open-web video discovery, with additional image-based relational clues. It combines automated construction, evidence verification, shortcut filtering, and human review. Evaluations of eight models under a shared tool environment reveal substantial performance gaps, while paired text-oracle experiments examine the difficulty introduced by image clues.
Critical strengths
-
Meaningful task formulation. Knowledge-guided observation requires external research to determine what should be examined in a video, complementing observation-driven research and open-web discovery. This provides useful coverage of video research capabilities.
-
Informative paired evaluation. Replacing image clues with their textual identities improves performance across all evaluated models while preserving the remaining task. This offers concrete evidence that resolving visual references is an important bottleneck.
Critical weaknesses
-
Benchmark quality and judging lack independent calibration. Appendix B describes quality-control stages but omits verification configurations, retention counts, and quantitative annotation-quality estimates. Gemini-3.6-Flash is both an evaluated model and the correctness/failure judge, without reported independent calibration. This does not establish bias, but limits confidence in the benchmark’s reliability and reported rankings.
-
Evaluation details and failure attribution need clarification. Tool implementations, video sampling, judging criteria, and handling of changing or unavailable web evidence are insufficiently specified. The shared VideoSummary backend is a reasonable system-evaluation choice, but pooled visual-understanding failures cannot clearly distinguish errors by the evaluated model from errors in backend observations. This limits reproducibility and interpretation of the diagnostic results.
Questions for authors
-
Can you provide the key generation/verification settings, stage-level retention counts, and an independent sample audit of final questions and judge decisions? How were the shortcut probes implemented, and what residual bypasses remain?
-
Please specify the search and video-tool configurations, judging criteria, evaluation dates, and handling of unavailable evidence. Will supporting evidence and evaluation logs be released?
-
Can the failure analysis distinguish direct visual interpretation, shared-backend errors, and downstream reasoning errors, or can its conclusions be restricted to system-level failures?
7 comments
Response to Reviewer (2/4)
Q1: Construction, Quality Control, Shortcut Filtering, and Independent Validation (continued)
Human review and filtering statistics. Every candidate surviving automated verification and shortcut filtering is independently reviewed by two annotators drawn from three graduate students against five criteria: question clarity, factual consistency, target unambiguity, visual-evidence sufficiency, and answer validity. A candidate is retained only if both annotators approve it. Across all 1,135 human-reviewed candidates, inter-annotator agreement is 90.1% (Cohen's ), with within-task agreement rates of 91.2% for O2R, 89.6% for KGO, and 90.0% for OVD. Human-review pass rates are 77.8% (168/216), 79.6% (191/240), and 78.9% (536/679), respectively.
The revised Appendix B.5 and Table B.3 report the complete stage-level filtering statistics. Each rejected candidate is counted only at the first stage where it fails; removal percentages use the initial candidate count for the corresponding task as the denominator.
| Construction & Quality-Control Stage | O2R | KGO | OVD | Total |
|---|---|---|---|---|
| Initial candidates | 450 | 700 | 1,500 | 2,650 |
| Removed: Programmatic consistency checks | 29 (6.4%) | 54 (7.7%) | 96 (6.4%) | 179 (6.8%) |
| Removed: Visual grounding verification (V3) | 62 (13.8%) | 117 (16.7%) | 308 (20.5%) | 487 (18.4%) |
| Removed: Factual and ambiguity verification (V1–V2) | 42 (9.3%) | 163 (23.3%) | 178 (11.9%) | 383 (14.5%) |
| Removed: Counterfactual shortcut filtering (S1–S2) | 101 (22.4%) | 126 (18.0%) | 239 (15.9%) | 466 (17.6%) |
| Removed: Human review | 48 (10.7%) | 49 (7.0%) | 143 (9.5%) | 240 (9.1%) |
| Passed quality control | 168 (37.3%) | 191 (27.3%) | 536 (35.7%) | 895 (33.8%) |
| Excluded during final random sampling | 18 | 41 | 36 | 95 |
| Final dataset | 150 | 150 | 500 | 800 |
Among the 240 candidates rejected during human review, the primary reasons were insufficient or occluded visual evidence (35.8%), ambiguous clues or competing valid answers (34.2%), factual or relational inaccuracies (19.6%), and unnatural phrasing or residual shortcuts (10.4%). The final 800 questions are sampled from the 895 candidates that passed all quality-control stages, so this final downsampling is not counted as a quality-control rejection.
Independent audit of final questions. Independent auditors who were not involved in the original construction examined 80 stratified random final questions (15 O2R, 15 KGO, and 50 OVD, covering both text-only and image-clue instances). Blinded to prior verification decisions, they checked factual correctness, visual-evidence sufficiency, answer validity, ambiguity, and residual shortcuts. All 80 questions passed, with no factual errors, ambiguous answers, insufficient visual evidence, or residual shortcuts identified. We have added this audit to Sec. 3.2 and Appendix B.5.
Independent validation of answer judging. To address the concern that Gemini-3.6-Flash is both an evaluated model and the primary correctness judge, we re-evaluated all 38,400 model predictions using GPT-5.6-Sol as an independent cross-family judge under the same semantic-equivalence criterion. The two judges agree on 97.83% of trajectory-level decisions on average. Across all 16 model–setting combinations, the mean absolute change in Overall Accuracy is only 0.62 percentage points (maximum: 1.17 pp), the mean absolute change in Overall Pass@3 is 0.53 pp (maximum: 0.88 pp), and model rankings under both metrics remain identical in both tool settings (Spearman's ).
We further validated both judges against blinded human annotations on 160 stratified random predictions and 50 additional cases where the two automated judges disagreed. On the random sample, human adjudicators agreed with Gemini-3.6-Flash on 96.9% (155/160) and with GPT-5.6-Sol on 98.1% (157/160). Among the 50 disagreement cases, humans sided with Gemini-3.6-Flash in 19 cases and with GPT-5.6-Sol in 31. We have added the full cross-judge re-evaluation, per-model breakdown, and human validation of both automated judges to Appendix C.4 and Table 4.
For reproducibility, the revised Appendix B.1–B.4 now makes the operational content of the generation, verification, and shortcut-probing stages explicit, including their required inputs and outputs, evidence requirements, negative constraints, and acceptance criteria. We will additionally release the complete verbatim construction prompt templates, structured output schemas, construction code, and available supporting evidence records together with the benchmark.
Response to Reviewer (3/4)
Q2: Evaluation Configuration, Evidence Availability, and Reproducibility
We have substantially expanded Appendix C to specify the evaluation prompts, tool implementations, video preprocessing, judging protocol, evaluation period, and handling of unavailable web evidence.
Tool and retrieval configurations. All models share the same seven tool interfaces and backend configurations, with no model-specific retrieval settings, personalization, or reranking. The revised Appendix C.2 and Table 3 specify:
| Tool | Backend | Configuration and Returned Information |
|---|---|---|
| WebSearch | Google Search via SerpApi | Top 20 results per query in Google's native relevance order; returns titles, URLs, and snippets. |
| ImageSearch | Google Images via SerpApi | Top 5 images for a text query, with previews, titles, and source URLs. |
| ReverseImageSearch | Google Lens via SerpApi | 5 visually matched web images with titles and source URLs for a supplied query image. |
| ReverseFrameSearch | Frame extraction + Google Lens via SerpApi | Extracts the nearest sampled frame to an agent-selected timestamp and returns 5 reverse-image matches. |
| LinkSummary | Web crawler + Qwen3.5-35B-A3B | Fetches agent-selected webpage URLs and returns a query-focused summary. |
| ChannelVideos | YouTube Data API v3 | Supports title-keyword, duration, publication-date, and result-limit filters; results ordered by publication date; default limit of 50 videos. Returns URLs, titles, descriptions, channel metadata, upload dates, and durations. |
| VideoSummary | Gemini-3-Pro | Answers a targeted question about an agent-selected online video and returns a concise answer with supporting visual/audio description. |
Agents decide which tools to invoke, how to formulate queries or filters, and which returned sources to inspect further.
Video processing and execution. For O2R and KGO, supplied videos are initially sampled at 1 fps. To retain full temporal coverage within a 64K video-token budget, we progressively reduce frame resolution when necessary; only if the budget still cannot be satisfied at approximately 360p do we reduce the temporal sampling rate uniformly across the recording rather than truncating it. OVD provides no video directly, and online videos are inspected through the shared VideoSummary backend. Each tool-enabled trajectory permits at most 50 model turns and one tool call per turn, and we sample three independent trajectories per question.
Judging criteria. The primary judge evaluates semantic equivalence between the model prediction and the reference answer, accepting valid aliases, equivalent paraphrases, and alternative numerical forms rather than requiring exact string matching. It receives the question, reference answer, and model prediction, together with supporting visual context when needed for disambiguation, while remaining blinded to the evaluated model's identity. The independent GPT-5.6-Sol re-evaluation and human validation described in Q1 provide additional validation of this protocol. The complete evaluation prompts and judging procedure are now provided in Appendix C.1 and C.4.
Evaluation dates and unavailable evidence. Evaluations were conducted during August–September 2026. If a webpage or video is unavailable, or a tool call fails, the resulting error is returned directly to the agent, which may retry, reformulate its request, or seek an alternative source within its remaining interaction budget. We provide no manual intervention or ground-truth-based fallback.
Because web indexes and rankings evolve over time, exact retrieval results may change across evaluation runs. We therefore report the evaluation period and use the same retrieval backends, ranking procedures, and result limits for all evaluated models. OVD also does not require retrieval of a unique construction-time video: the supporting video serves as a verified witness, and any alternative footage containing sufficient evidence is a valid research path.
Release plan. We will release the benchmark questions and reference answers, complete evaluation prompts and tool schemas, all model predictions, judging outputs, and tool-interaction trajectories/logs, together with construction-time supporting evidence records that can be distributed. Third-party web and video content will be referenced rather than redistributed where redistribution rights are unavailable. Together with the configurations now documented in Appendix C, these materials support independent auditing and reproduction of the evaluation protocol.
Response to Reviewer (1/4)
Thank you very much for your time, thoughtful review, and constructive feedback. We are encouraged that you recognized the meaningful task formulation of UNIVERSE, particularly Knowledge-Guided Observation and the paired Text Oracle evaluation of image-based clues. Your comments on benchmark quality, judge calibration, evaluation reproducibility, and failure attribution helped us substantially strengthen the paper.
Because W1 and W2 are restated more specifically in Q1–Q3, we organize our response around the three questions to avoid duplicating the same evidence. Q1 addresses benchmark quality, shortcut filtering, and judge validation; Q2 covers evaluation configuration and reproducibility; and Q3 addresses failure attribution. Together, these responses cover all concerns raised in W1 and W2.
Q1: Construction, Quality Control, Shortcut Filtering, and Independent Validation
We have substantially expanded Sec. 3.2 and Appendix B to document the construction and quality-control pipeline, including model configurations, verification criteria, shortcut probes, human-review procedures, stage-level filtering statistics, and independent audits.
Generation and verification. The new Table B.1 specifies 11 model-assisted operations spanning generation (
G1–G6), evidence verification (V1–V3), and counterfactual shortcut probing (S1–S2), together with their eligible models and input/output roles. The model pool includesGPT-5,GPT-5.1,GPT-5.2,GPT-5.4,Gemini-2.5-Pro,Gemini-3-Pro, andGemini-3.1-Pro. Operations that inspect video frames (G1,G2, andV3) use Gemini-family models, while text- and image-based operations draw from the corresponding eligible pools. Generation and verification are conducted in isolated sessions, and verification samples a different model from the generator whenever multiple models are eligible.Verification is decomposed into three complementary checks: V3 verifies visual grounding and answer correctness against the supporting video or images; V1 checks factual and relational consistency against graph paths and web evidence; and V2 checks target specificity and searches for plausible competing entities or events satisfying the same clues. Each applicable verification check is executed twice independently, and a candidate advances only if both judgments are
confirmed; anyrefutedorinconclusivejudgment causes rejection. These configurations and acceptance rules are now detailed in Appendix B.1–B.3.Shortcut filtering. Appendix B.4 and Table B.2 now specify task-specific counterfactual probes that test whether the intended information dependency can be bypassed. For O2R, we test whether the anchor or answer can be recovered without the supplied video or without the video-derived anchor reference. For KGO, we remove the anchor-identifying external clues and test whether the visual answer can be obtained directly from the supplied video. For OVD, we reveal the intended anchor/event but disable video access, separately testing whether the answer can be recovered from textual web evidence or static images/thumbnails. Image-based clues additionally undergo a direct-recognition probe with retrieval disabled. Each applicable probe permits up to two independent attempts, and a single successful bypass is sufficient to reject the candidate. These probes eliminated 466 candidates in total.
These checks target the principal bypasses relevant to the intended task dependencies, including linguistic leakage, text-only recovery, video-only guessing, static-image shortcuts, and direct recognition of image clues. In addition, the independent audit of 80 final questions explicitly checked for residual shortcuts, and none were identified in the audited sample.