acceptodds
Under review as a conference paper at ICLR 2027

Video-Browser: Towards Open-web Agentic Video Browsing

Abstract

Autonomous agents are shifting information seeking from passive retrieval toward proactive, open-ended research, yet this progress has largely bypassed video, one of the web's richest sources of evidence. We formalize Agentic Video Browsing, in which an agent must query video search engines, decide which returned videos to watch, and answer questions grounded in their visual content. To study this setting reproducibly, we introduce Video-BrowseComp, a benchmark and infrastructure suite with 660 human-annotated questions at three difficulty levels, paired online and offline evaluation tracks, and a fixed video search stack of roughly 12,000 videos spanning over 2,000 hours for evaluation and training-data construction. Our analysis reveals that the main bottleneck lies before video understanding: agents often fail to find the answer-bearing video at all. This bottleneck stems from two coupled challenges. Search engines index textual metadata such as titles and tags, whereas questions often target implicit visual or cross-source evidence absent from these surface forms; and exhaustively inspecting the many returned videos is prohibitively expensive. We propose Video-Browser, an evidence-surfacing agent that improves both sides of search. A Translation Memory helps the Planner phrase queries by analogy to past question-to-query transformations that successfully surfaced their target videos, and a Watcher performs sparse visual screening over returned candidates, retaining only videos that plausibly contain answer-bearing evidence for later dense verification. On the offline track, Video-Browser raises overall accuracy from 11.6% to 27.3% with Qwen3-VL-8B and from 39.6% to 45.1% with Gemini-3.1-Pro over the corresponding ReAct search agents, showing that better query formulation and post-search candidate selection improve end-to-end browsing performance. We open-source the benchmark and search system at [https://anonymous.4open.science/r/videobrowser_iclr-F340](https://anonymous.4open.science/r/videobrowser_iclr-F340).

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video DeepResearch, Video Browsecomp

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ThKEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a benchmark to evaluate capabilities in Agentic Video Browsing: a task in which an agent queries video search engines, has to decide which of the retrieved videos to watch, and answers questions grounded in such videos. The benchmark is composed of 660 human-annotated questions at different difficulty levels. They introduce a new agent system, VideoBrowser, that aims to improve the performance on such a task. VideoBrowser leverages a translation memory that allows the Planner, responsible for finding the relevant videos for the question, to use past successful searches to phrase new queries. The retrieved videos are then passed to the Watcher, which efficiently screens part of them to obtain the evidence needed to answer a question. A Checker decides when to interrupt the browsing, and, finally, an Analyst formulates the response based on the evidence gathered. The authors compare themselves to another search agent framework (ReAct), showing that their formulation improves results on such a task.

Critical strengths
  • S1: Consistent gains over ReAct agents, and each component is tested on its own.. The evaluation highlights improvements on both Qwen3-VL-8B, and Gemini-3.1-Pro. Additionally, they ablate the impact of every component like Translation Memory and Watcher to the evaluation results.
  • S2: Valuable open benchmark. The benchmark includes 660 human-annotated, divided on multiple difficulty levels and a large offline video library for improved reproducibility.
Critical weaknesses
  • W1: No baseline pairs an existing search agent with a video agent. The video agents receive the top-3 candidates of a single retrieval. Since the offline index is text-only a text search agent (like Search-o1 - EMNLP 2025) can drive the research and hand its results to the video agents. I believe this would actually show the advantages of a single unified agent structure.
  • W2: Video-Browser is not evaluated with GPT-5.4.
    Table 3 has a GPT-5.4 ReAct row but no Video-Browser row, no reason is given.
  • W3: Translation Memory is only shown to work offline, while the paper targets the open web.
    All memory evaluations results are offline (Tables 4,6). The arguments about memory improving the retrieval are hard to extend to online scenarios without further experiments. The live-web rankings, in fact, "change over time" (as stated in the Introduction), an online memory/no memory ablation would actually show the impact of such component.
Questions for authors
  • Q1: Do the author think the search agent + video agent baseline could be applicable to their evaluations? Can they give some clarifications about this?
  • Q2: Why was Video-Browser not evaluated with GPT 5.4?
  • Q3: Can the authors address the ablation proposed in W3?
9 comments

W1: Can a search agent be coupled with an existing video agent? (Q1)

We would like to thank for your suggestions, this approach is feasible. Initially, as far as we know, there was no dedicated model or agent designed for the task of open-web video search. Therefore, we selected several agents designed for video understanding tasks to evaluate, with the aim of demonstrating that open-web video search requires not only video understanding capabilities but also the ability to locate the target video.

Now we added some of the models and agents for text and image search tasks for evaluation. We reproduced four search agents: Video-DeepResearch-30B, WebWatcher-7B, Search-o1 (Since search-o1 is a framework itself, we were unable to find any publicly available model weights, so we use Qwen3-VL-8B both for the planner and video reader), and MMSearch-R1-7B.

SystemOA (%)Recall (%)L1L2L3CE (%)DocEduFilmGameMusicSportTVVidTarget in Top-3 (%)
Video-Browser27.346.234.620.124.656.816.231.426.021.138.519.435.526.239.3
Video-DeepResearch-30B12.226.215.69.88.267.713.523.512.05.316.911.98.18.221.3
WebWatcher-7B11.622.416.66.59.869.413.57.814.07.023.110.46.59.817.8
Search-o1 (Qwen3-VL-8B)10.022.012.75.414.864.813.57.816.010.512.37.53.211.513.6
MMSearch-R1-7B6.49.88.33.88.269.010.87.88.07.09.27.53.20.08.0

W2: Why is the GPT-5.4 result missing? (Q2)

We have added the results for GPT 5.4 on the Video Browser framework.

ModelOA (%)Recall (%)L1L2L3CE (%)DocEduFilmGameMusicSportTVVid
GPT-5.447.147.855.642.931.134.948.660.836.045.655.437.350.044.3

W3: Does Translation Memory help in live-web browsing? (Q3)

Unlike the offline track, we do not cache all videos in the online track. Conducting end-to-end evaluation would require downloading a large number of videos from online platforms, which risks triggering platform access restrictions and makes reproducibility difficult. This is also a key motivation why we constructed a fixed, reproducible offline track for open-web video browsing evaluation.

Instead, we evaluate Memory's effectiveness on live-web search using YouTube's search interface, with 100 training examples to build an online Memory bank on the same search stack. R@10 measures whether the Top-10 results contain the seed video selected during annotation, while Judged@10 measures whether an LLM (GPT-5.1-mini) identifies at least one relevant video among them.

ModelMemoryR@10Judged@10
Qwen3-VL-8BNone30.024.4
Retrieved36.425.8
Gemini-3.1-ProNone37.943.3
Retrieved39.746.2

For Qwen3-VL-8B, Memory raises R@10 from 30.0% to 36.4% (+6.4 pp) and Judged@10 from 24.4% to 25.8% (+1.4 pp). For Gemini-3.1-Pro, the corresponding changes are 37.9% to 39.7% (+1.8 pp) and 43.3% to 46.2% (+2.9 pp). Both planners benefit on the live search metrics, although the gains differ.

Reviewer_7iktOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Video-Browser, an autonomous agent framework for open-web video search and question answering, aiming to address the challenges of discovering videos containing answer-relevant evidence and the high computational cost of examining numerous candidate videos. The proposed method employs Translation Memory to improve query formulation by leveraging previously successful search experiences and a Watcher module to perform sparse visual screening of candidate videos, thereby improving the efficiency of video retrieval and evidence localization. Additionally, the paper introduces Video-BrowseComp, a benchmark comprising 660 human-annotated questions and approximately 12,000 videos. Experimental results demonstrate that Video-Browser outperforms the corresponding ReAct baselines with both Qwen3-VL-8B and Gemini-3.1-Pro, validating its effectiveness in open-ended video search and question-answering tasks.

Critical strengths
  1. The research problem is novel and practically significant. The paper extends autonomous agents' information-seeking capabilities from traditional text and image-based scenarios to open-web video environments by introducing the Agentic Video Browsing task. Unlike conventional video question answering, this task requires agents not only to understand video content but also to autonomously discover, select, and verify relevant video evidence. The paper identifies a critical bottleneck in video discovery, addressing a meaningful research problem with substantial practical relevance.
  2. The paper introduces a reasonably large-scale and reproducible evaluation benchmark. The proposed Video-BrowseComp benchmark contains 660 human-annotated questions across three difficulty levels and approximately 12,000 videos, with both online and offline evaluation settings. The fixed offline video search environment helps mitigate the variability caused by changing web search results, enabling more consistent comparisons across different methods. This benchmark provides a valuable evaluation resource for future research on agentic video browsing.
Critical weaknesses
  1. Insufficient Comparisons with Existing Methods and Lack of Fairness Analysis The paper compares Video-Browser with ReAct search agents and existing video-understanding methods, including VideoAgent, VideoTree, and DVD. However, traditional video-understanding agents primarily focus on question answering over given videos, whereas Video-Browser is specifically designed for open-ended video retrieval. These methods differ in their task settings and optimization objectives.

  2. Limited Performance Gains and Insufficient Generalization Analysis of Translation Memory Table 4 shows that Translation Memory improves Recall@10 by only 1.9–2.5 percentage points with Qwen3-VL-8B, while the improvements are substantially larger, ranging from 7.7 to 8.3 percentage points, with Gemini-3.1-Pro. This discrepancy suggests that the effectiveness of Translation Memory may be strongly influenced by the capabilities of the underlying model.

  3. Limited Practical Gains of the Watcher and Potential Loss of Critical Visual Evidence Table 5 shows that the Watcher improves the overall Top-1 Hit rate from 12.0% to 15.6% compared with the metadata-only LLM Reranker, representing a gain of 3.6 percentage points. Although the conditional Top-1 accuracy increases substantially from 57.4% to 75.3%, this metric is calculated only for cases where the target video is already present in the BM25 Top-10 candidate pool. Therefore, it does not fully reflect the end-to-end retrieval capability of the system. Furthermore, the Watcher relies on sparsely sampled video frames for candidate screening, which may cause it to miss critical events that appear only briefly or dynamic information that requires observing consecutive frames.

Questions for authors

1.The authors should consider conducting additional experiments on cross-domain and out-of-distribution questions, and analyzing the effects of memory bank size, the number of retrieved examples, and example similarity on retrieval performance. 2.The authors are encouraged to further investigate how different frame sampling densities affect candidate video recall and final question-answering accuracy, and to provide representative failure cases in which the Watcher incorrectly filters out videos containing the correct evidence. 3.The authors are encouraged to conduct more comprehensive comparisons with advanced agentic search strategies and video retrieval methods under consistent search budgets and computational resources to better demonstrate the advantages of the proposed approach.

6 comments

W1: Fairer retrieval-plus-video-agent comparison (Q3)

Initially, as far as we know, there was no dedicated model or agent designed for the task of open-web video search. Therefore, we selected several agents designed for video understanding tasks to evaluate, with the aim of demonstrating that open-web video search requires not only video understanding capabilities but also the ability to locate the target video.

Now we added some of the models and agents for text and image search tasks for evaluation. We reproduced four search agents: Video-DeepResearch-30B, WebWatcher-7B, Search-o1 (Qwen3-VL-8B), and MMSearch-R1-7B.

SystemOA (%)Recall (%)L1L2L3CE (%)DocEduFilmGameMusicSportTVVidTarget in Top-3 (%)
Video-Browser27.346.234.620.124.656.816.231.426.021.138.519.435.526.239.3
Video-DeepResearch-30B12.226.215.69.88.267.713.523.512.05.316.911.98.18.221.3
WebWatcher-7B11.622.416.66.59.869.413.57.814.07.023.110.46.59.817.8
Search-o1 (Qwen3-VL-8B)10.022.012.75.414.864.813.57.816.010.512.37.53.211.513.6
MMSearch-R1-7B6.49.88.33.88.269.010.87.88.07.09.27.53.20.08.0

In above table, Video-Browser reaches 27.3% OA, 46.2% recall, and 39.3% correct-video Top-3. The strongest added baseline, Video-DeepResearch-30B, reaches 12.2% OA and 21.3% Top-3.

W2: Translation Memory: cross-model effectiveness and robustness (Q1)

We added Qwen3-VL-30B-A3B and Qwen3.5-35B-A3B to the one-shot query-rewriting study. With BM25, Memory increases R@10 by 2.4 pp for Qwen3-VL-8B, 3.7 pp for the 30B model, 5.3 pp for the 35B model, and 8.3 pp for Gemini-3.1-Pro. Both larger Qwen planners also improve with BGE-M3 and Qwen3-Embedding-8B.

RewriterMemory VariantBM25BGE-M3Qwen3-E
Qwen3-VL-8BNone15.519.237.5
Retrieved M17.9 (+2.4)21.7 (+2.5)39.4 (+1.9)
Qwen3-VL-30B-A3BNone17.823.344.1
Retrieved M21.5 (+3.7)26.2 (+2.9)47.2 (+3.1)
Qwen3.5-35B-A3BNone19.423.944.7
Retrieved M24.7 (+5.3)26.8 (+2.9)48.2 (+3.5)
Gemini-3.1-ProNone29.932.549.6
Retrieved M38.2 (+8.3)40.2 (+7.7)57.8 (+8.2)

One possible reason for the larger gains with stronger planners is that they understand the injected Memory examples better. They may be better at inferring the query-writing pattern from past questions and reusing it on new ones. We present this as a possible explanation, not a causal conclusion.

Generalization and memory settings

We used Qwen3.5-35B-A3B to test cross-category and unseen-difficulty transfer, then varied bank size, the number of retrieved cards, and example similarity.
(a) Cross-category and unseen-difficulty transfer.

For each question, we also tried retrieving Memory only from the other seven video categories. R@10 falls from 25.6% with the full bank to 22.6% (−3.0 pp), but remains above the no-Memory baseline of 19.4% (+3.1 pp as reported in the table). Relevant examples from the same category help more, though cross-category examples still offer some benefit.

MemoryR@1R@3R@5R@10
None9.414.816.919.4
Retrieved (full bank)12.3 (+3.0)18.5 (+3.7)21.5 (+4.6)25.6 (+6.2)
Retrieved from other categories only11.4 (+2.1)16.2 (+1.4)19.5 (+2.6)22.6 (+3.1)

The existing Memory bank contains only L1 and L2 questions. We tested it directly on L3, without adding any L3 examples. R@10 increases from 25.1% to 28.1% (+3.0 pp), as shown below.

MemoryR@1R@3R@5R@10
None11.720.823.025.1
Retrieved12.3 (+0.5)20.5 (−0.3)24.0 (+1.1)28.1 (+3.0)

The cross-category gain is smaller than the full-bank gain, while the L3 result shows that examples from easier questions can still help with a difficulty level absent from the bank.

(b) Memory bank size.

BankR@1R@3R@5R@10
None9.414.816.919.4
10% (85 cards)11.4 (+2.1)18.0 (+3.1)21.1 (+4.3)24.8 (+5.4)
100% (846)12.3 (+3.0)18.5 (+3.7)21.5 (+4.6)25.6 (+6.2)

Bank size has a small effect beyond the first 10%. With 85 cards (10% of the bank), R@10 is 24.8%; with all 846, it is 25.6%, a gain of just 0.8 pp for a tenfold increase in bank size. A small bank may already contain enough relevant examples to guide the query. The 25.6% result here and the 24.7% Qwen3.5-35B-A3B result in the earlier cross-model table come from independent runs.

(c) Number of retrieved examples (K).

MemoryR@1R@3R@5R@10
None9.414.816.919.4
K = 111.0 (+1.6)16.7 (+1.9)19.0 (+2.1)23.7 (+4.3)
K = 3 (default)12.3 (+3.0)18.5 (+3.7)21.5 (+4.6)25.6 (+6.2)
K = 511.4 (+2.1)16.9 (+2.1)21.3 (+4.4)24.8 (+5.4)
K = 1011.6 (+2.2)17.7 (+2.9)20.7 (+3.8)24.2 (+4.8)

Adding more examples to the prompt also gives diminishing returns. One card raises R@10 from 19.4% to 23.7% (+4.3 pp). K=3 performs best at 25.6%, using K=5 or K=10 instead gives 24.8% and 24.2%. More context does not necessarily produce a better search query.

(d) Example similarity. We compared the default three cards (average similarity 0.58) with less similar cards (0.25).

Similarity of the injected examplesR@1R@3R@5R@10
No memory9.414.816.919.4
0.58 (default)12.3 (+3.0)18.5 (+3.7)21.5 (+4.6)25.6 (+6.2)
0.2511.5 (+2.1)15.6 (+0.7)18.4 (+1.6)22.6 (+3.1)

The higher-similarity cards yield 25.6% R@10, versus 22.6% for less similar ones. Both exceed the no-Memory result of 19.4%. Retrieval quality matters more here than simply supplying additional examples.

Reviewer_uQCmOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper formalizes Agentic Video Browsing: an agent queries a video search engine, decides which returned videos to watch, and answers from their visual content. It introduces Video-BrowseComp, with 660 human-annotated questions (450 offline / 210 online, three difficulty levels, eight genres), 1,000 auto-generated training questions, and an offline BM25 search stack of about 12.5k videos. A search/reasoning decomposition identifies video discovery as the main bottleneck. The proposed agent, Video-Browser, combines:

  • a Planner guided by Translation Memory, i.e., retrieved past questions with BM25-verified queries;
  • a Watcher that reranks BM25 candidates from metadata plus 2×3×3 frame grids;
  • a Checker that decides when to stop browsing;
  • an Analyst that answers from 128 frames plus the transcript of the accumulated videos.

On the offline track, Video-Browser raises overall accuracy from 11.6% to 27.3% (Qwen3-VL-8B) and from 39.8% to 45.1% (Gemini-3.1-Pro) over ReAct agents.

Critical strengths

Timely and under-explored setting. Video is largely missing from browsing benchmarks.

Critical weaknesses

W1. Does the benchmark require watching (or finding) the video? No closed-book baseline (no search, no video) or text-web-search-only baseline is reported. Several showcased items appear answerable from world knowledge:

  • Fig. 1: the nose color of Chopper in the live-action One Piece.
  • Fig. 4, L3: the item DiCaprio's character hands over in Blood Diamond, with answer "Diamond".
  • Fig. 4, L2: CS231n's "An image is a tensor of integers", which is also widely available as slide text.

Please report closed-book accuracy for the evaluated LLMs, accuracy with text-only web search, and per-item filtering, and remove or flag items answerable without video evidence. Please also report inter-annotator agreement, the discard rate in cross-validation, and the answer-type distribution.

W2. The offline stack is under-documented and conditioned on the evaluation questions.

  • §3.3 promises "full annotation and search-stack construction details" in App. A.3, but A.3 covers only annotation. How the ~11.9k base videos were collected is never described.
  • The hard negatives (App. A.5) were mined with the query rewrites of the very agents being evaluated (Qwen3-VL-8B, GPT-5.4, Gemini-3.1-Pro). They were then verified by Qwen3-VL-8B looking at 16 sparse frames, and only candidates it judged "NO" were kept.
  • The Watcher itself is Qwen3-VL-8B looking at sparse frame grids. Table 5 therefore compares a sparse-visual Qwen judge against a metadata-only reranker on a pool whose negatives were selected to be rejected by a sparse-visual Qwen judge. This plausibly inflates the Watcher's gain (conditional Top-1 57.4 → 75.3).

Please verify hard negatives with an independent model or with humans, report Watcher results on negatives not filtered by Qwen, and describe the base-pool construction.

W3. Possible leakage through Translation Memory. The training questions are generated from seed videos drawn from the same offline pool, which is itself built around the evaluation questions. The splits are disjoint only at the video-ID level, and the online split shares two videos (App. A.4). Memory cards contain entity anchors and verified queries, possibly from the same channels, series, or events as evaluation targets. Please report overlap at the channel, entity, and event level between memory cards and evaluation targets, and ablate with a memory built from a disjoint corpus. Which LLM produced the memory cards?

W4. Attribution of the gains.

  • In Table 6, the pipeline without Translation Memory and Watcher already reaches 22.2% OA vs. 11.6% for ReAct. Translation Memory adds +2.9 and the Watcher +2.2, single runs on 450 questions (a 95% CI is roughly ±4 points). The abstract and introduction attribute the full +15.7 to "better query formulation and post-search candidate selection".
  • Component ablations are given for Qwen3-VL-8B only.
  • With Gemini-3.1-Pro, the gain is +5.3 overall, but L2 drops (35.9 → 35.3) and so do Documentary, Education, and Music.
  • Calibration error worsens substantially for Gemini (21.3 → 39.0), which is not discussed.

W5. Baselines and cost.

  • ReAct agents get 7 tool calls, but their token cost is not reported, while Video-Browser uses 100.9k (Qwen) / 500.7k (Gemini) tokens per question. A cost-matched comparison is needed.
  • The adapted video agents (VideoAgent, VideoTree, DVD) receive the top-3 BM25 results for the raw question, a weak setting that does not reflect their design.
  • Systems the paper itself cites (Video-DeepResearch, AdaVDR, VideoSearch-R1, WebWatcher, MMSearch-R1) are not compared.
  • The online track (App. A.8) uses older backbones (Gemini-2.5, GPT-5.1) than the offline track and only 23 L3 questions, which limits its value as evidence.

W6. BM25 design choice. With the raw question, a dense retriever reaches 48.5 R@10 vs. 7.2 for BM25 (Table 1). The "black-box keyword search" motivation for BM25 is reasonable, but the bottleneck conclusion should be qualified as specific to lexical search, and main results with a dense backend would show whether Translation Memory and the Watcher still help.

W7. Method novelty. Exemplar-retrieval few-shot query rewriting, VLM reranking from frame thumbnails, and planner/checker/analyst loops are standard agent components. The main contribution is the benchmark, which strengthens the case for W1–W3.

Questions for authors
  1. What is the closed-book accuracy (no search, no video) of GPT-5.4 and Gemini-3.1-Pro on the offline track?
  2. How was the base pool (~11.9k videos) collected? Were any queries derived from the evaluation questions?
  3. What is the channel- and entity-level overlap between memory cards and evaluation targets?
  4. What are the Watcher's results with hard negatives verified by a different model or by humans?
  5. Why does Gemini calibration error increase from 21.3 to 39.0?
10 comments

W1: Video necessity, text-only shortcuts, and annotation quality

W1.1: Clarification of the three showcased questions

We want to clarify the names One Piece, Chopper, CS231n, Leonardo DiCaprio, The Great Gatsby, and Blood Diamond appear in the ground-truth Evidence Chain or Checklist, not in the questions given to the agent. The agent must identify the source from an indirect description before it can verify the answer. The L3 example in Figure 4 also requires connecting two works through a shared actor. We acknowledge that familiar works may sometimes be answerable from text or parametric knowledge, but those names are not provided in the questions.

W1.2: Text-only baselines and contribution of visual evidence (Q1)

We evaluated all 450 offline questions with closed-book and text-only web-search baselines. Both perform well below the video-grounded settings. We also compare text-only input with text plus video frames for the retrieved videos, to measure the contribution of visual evidence after the source has been located. The gains hold for Qwen3-VL-8B, GPT-5.4, and Gemini-3.1-Pro.

BackboneClosed-book OAText-web OAOracle video OACorrect video retrieved: text onlyCorrect video retrieved: text + frames
Qwen3-VL-8B5.82.461.612.351.9
GPT-5.48.418.276.029.174.1
Gemini-3.1-Pro13.124.982.732.582.9

Our annotation criteria require verifiable visual evidence for each answer. They cannot guarantee that every pretrained model lacks the relevant knowledge, a model may have encountered the same scene during training. Removing every question answered correctly by a particular closed-book model would also make the benchmark depend on that model.

W1.3: Annotation agreement, discard rate, and answer types

We used two rounds of human annotation with independent cross-validation. In round one, annotators agreed on 201 of 300 candidates (67.0%). They disagreed on 99, further checks rejected 79, giving a discard rate of 26.3% (79/300). We then revised the task objectives and annotation guidelines. In round two, agreement rose to 430/500 (86.0%). Of the 70 disagreements, 61 were rejected after verification, a discard rate of 12.2% (61/500). We discarded 140 of 800 candidates in total and retained 660 evaluation questions.

The table below breaks down the answer types for the 450 offline questions into 10 categories.

Answer typeCount
Number109 (24.2%)
Color100 (22.2%)
Object66 (14.7%)
Person53 (11.8%)
Text or code35 (7.8%)
Organization or brand22 (4.9%)
Location19 (4.2%)
Concept or term18 (4.0%)
Date or time16 (3.6%)
Title of work12 (2.7%)
Total450 (100%)

W2: Base-pool provenance and hard-negative selection bias

W2.1: Base-pool provenance (Q2)

We froze the base pool in April 2026, before running the offline-track experiments. It contains (i) 1,254 annotated videos, including all 821 videos associated with the 450 offline questions (source videos, positives, and annotated negatives), the remaining videos came from training and online-track questions. (ii) 10,679 distractors downloaded in earlier live-web browsing runs. Crucially, the queries that produced these distractors were neither derived from nor conditioned on the 450 offline evaluation questions. In the earlier runs, agents searched YouTube and opened two or three of the top results per query. We retained downloaded videos with usable Whisper transcripts.

W2.2: Independent verification of hard negatives (Q4)

We checked the Qwen3-VL-8B hard-negative filtering with an independent model, InternVL3.5-8B. InternVL returned a non-YES answer-presence verdict for 519 of the 522 selected negatives (99.4%). It agreed with Qwen on 541/564 retain-or-discard decisions (95.9%). The 564 candidates include both accepted and rejected samples. InternVL also rated 500/522 selected negatives (95.8%) as HIGH or MID in topical relevance. These checks indicate that the selected negatives are generally relevant without containing the answer.

MetricInternVL3.5-8B result
Selected hard negatives judged non-YES519/522 (99.4%)
Retain/discard decision agreement541/564 (95.9%)
Selected negatives judged HIGH/MID relevance500/522 (95.8%; HIGH: 277, MID: 223)

W2.3: Watcher evaluation without mined hard negatives

We repeated the reranking experiment on the clean base pool, without the added hard negatives. With Qwen3-VL-8B, Watcher raises overall Top-1 Hit from 15.6% to 18.2% (+2.6 pp), conditional Top-1 from 70.7% to 82.8% (+12.1 pp), and conditional MRR@10 from 79.6% to 88.2% (+8.6 pp) relative to the metadata-only LLM reranker. The gain therefore persists without the Qwen-filtered negatives.

MethodTop-1 Hit (%)Cond. Top-1 (%)Cond. MRR@10 (%)
LLM reranker (metadata only)15.670.779.6
Watcher (metadata + frames)18.282.888.2
Gain (Watcher − reranker)+2.6+12.1+8.6

W3: Translation Memory leakage, overlap, and transfer

W3.1: Cross-split overlap and leakage audit (Q3)

Training and offline evaluation use disjoint video IDs. We also checked the Memory cards actually injected for the 450 offline questions. The target video shared a channel with a Memory card for 21 questions (4.7%), an entity for 44 (9.8%), and an event for 10 (2.2%). In total, 65 questions (14.4%) had at least one type of overlap.

Overlap categoryOffline questions
Same channel21 (4.7%)
Shared entity44 (9.8%)
Shared event10 (2.2%)
Any overlap65 (14.4%)

W3.2: Overlap-purged Memory ablation

We removed the 65 potentially overlapping Memory cards and reran Qwen3-VL-8B Video-Browser on the same 450 offline questions. OA changed from 27.3% to 27.1%, and cumulative recall from 46.2% to 45.6%. Removing the identified overlaps had little effect on either metric.

RunOACum. RecallL1L2L3CEDocEduFilmGameMusicSportTVVid
Original Memory27.346.234.620.124.656.816.231.426.021.138.519.435.526.2
Overlap-purged Memory (−65 cards)27.145.635.119.622.951.924.339.230.019.335.428.422.618.0

W3.3: Memory-card generator

We construct Memory with the corresponding agent backbone: Qwen3-VL-8B generates the cards for Qwen3-VL-8B experiments, and Gemini-3.1-Pro generates those for Gemini-3.1-Pro experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.