- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a benchmark to evaluate capabilities in Agentic Video Browsing: a task in which an agent queries video search engines, has to decide which of the retrieved videos to watch, and answers questions grounded in such videos. The benchmark is composed of 660 human-annotated questions at different difficulty levels. They introduce a new agent system, VideoBrowser, that aims to improve the performance on such a task. VideoBrowser leverages a translation memory that allows the Planner, responsible for finding the relevant videos for the question, to use past successful searches to phrase new queries. The retrieved videos are then passed to the Watcher, which efficiently screens part of them to obtain the evidence needed to answer a question. A Checker decides when to interrupt the browsing, and, finally, an Analyst formulates the response based on the evidence gathered. The authors compare themselves to another search agent framework (ReAct), showing that their formulation improves results on such a task.
Critical strengths
- S1: Consistent gains over ReAct agents, and each component is tested on its own.. The evaluation highlights improvements on both Qwen3-VL-8B, and Gemini-3.1-Pro. Additionally, they ablate the impact of every component like Translation Memory and Watcher to the evaluation results.
- S2: Valuable open benchmark. The benchmark includes 660 human-annotated, divided on multiple difficulty levels and a large offline video library for improved reproducibility.
Critical weaknesses
- W1: No baseline pairs an existing search agent with a video agent. The video agents receive the top-3 candidates of a single retrieval. Since the offline index is text-only a text search agent (like Search-o1 - EMNLP 2025) can drive the research and hand its results to the video agents. I believe this would actually show the advantages of a single unified agent structure.
- W2: Video-Browser is not evaluated with GPT-5.4.
Table 3 has a GPT-5.4 ReAct row but no Video-Browser row, no reason is given. - W3: Translation Memory is only shown to work offline, while the paper targets the open web.
All memory evaluations results are offline (Tables 4,6). The arguments about memory improving the retrieval are hard to extend to online scenarios without further experiments. The live-web rankings, in fact, "change over time" (as stated in the Introduction), an online memory/no memory ablation would actually show the impact of such component.
Questions for authors
- Q1: Do the author think the search agent + video agent baseline could be applicable to their evaluations? Can they give some clarifications about this?
- Q2: Why was Video-Browser not evaluated with GPT 5.4?
- Q3: Can the authors address the ablation proposed in W3?
9 comments
W2: Why is the GPT-5.4 result missing? (Q2)
We have added the results for GPT 5.4 on the Video Browser framework.
| Model | OA (%) | Recall (%) | L1 | L2 | L3 | CE (%) | Doc | Edu | Film | Game | Music | Sport | TV | Vid |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-5.4 | 47.1 | 47.8 | 55.6 | 42.9 | 31.1 | 34.9 | 48.6 | 60.8 | 36.0 | 45.6 | 55.4 | 37.3 | 50.0 | 44.3 |
W3: Does Translation Memory help in live-web browsing? (Q3)
Unlike the offline track, we do not cache all videos in the online track. Conducting end-to-end evaluation would require downloading a large number of videos from online platforms, which risks triggering platform access restrictions and makes reproducibility difficult. This is also a key motivation why we constructed a fixed, reproducible offline track for open-web video browsing evaluation.
Instead, we evaluate Memory's effectiveness on live-web search using YouTube's search interface, with 100 training examples to build an online Memory bank on the same search stack. R@10 measures whether the Top-10 results contain the seed video selected during annotation, while Judged@10 measures whether an LLM (GPT-5.1-mini) identifies at least one relevant video among them.
| Model | Memory | R@10 | Judged@10 |
|---|---|---|---|
| Qwen3-VL-8B | None | 30.0 | 24.4 |
| Retrieved | 36.4 | 25.8 | |
| Gemini-3.1-Pro | None | 37.9 | 43.3 |
| Retrieved | 39.7 | 46.2 |
For Qwen3-VL-8B, Memory raises R@10 from 30.0% to 36.4% (+6.4 pp) and Judged@10 from 24.4% to 25.8% (+1.4 pp). For Gemini-3.1-Pro, the corresponding changes are 37.9% to 39.7% (+1.8 pp) and 43.3% to 46.2% (+2.9 pp). Both planners benefit on the live search metrics, although the gains differ.
W1: Can a search agent be coupled with an existing video agent? (Q1)
We would like to thank for your suggestions, this approach is feasible. Initially, as far as we know, there was no dedicated model or agent designed for the task of open-web video search. Therefore, we selected several agents designed for video understanding tasks to evaluate, with the aim of demonstrating that open-web video search requires not only video understanding capabilities but also the ability to locate the target video.
Now we added some of the models and agents for text and image search tasks for evaluation. We reproduced four search agents: Video-DeepResearch-30B, WebWatcher-7B, Search-o1 (Since search-o1 is a framework itself, we were unable to find any publicly available model weights, so we use Qwen3-VL-8B both for the planner and video reader), and MMSearch-R1-7B.