- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes VideoResearcher, a framework that analyzes failure trajectories in long-video question answering to automatically design, implement, and select video-understanding tools. The Worker and perception model remain fixed while the toolbox is optimized on 300 development questions from ALLVB, expanding from one initial tool to three tools. The frozen toolbox is then evaluated on LVBench, LongVideoBench, and Video-MME, improving average accuracy from 72.1% to 74.5%.
Critical strengths
-
The paper demonstrates the feasibility of automatically developing video-understanding tools from failure trajectories. After selection on the development set, the tools are frozen and evaluated on three other benchmarks, where the system outperforms the initial agent in each case. This suggests that the improvements extend to questions beyond the development set.
-
The paper provides tool implementations, examples of accepted and rejected candidates, and detailed execution trajectories. These materials help readers understand what the automatically generated tools do and how they change the way video evidence is acquired.
Critical weaknesses
-
I would like to better understand how the two new tools are used during testing. The paper provides comparisons across evolution rounds and execution examples, but does not report which questions invoke each tool or how many different videos each tool is reused on. This information would help clarify which tasks each tool is suited to and which use cases account for the performance gains.
-
The benefits of further evolution would be worth discussing. In Table 2(b), Round 7 improves development-set accuracy over Round 2, but accuracy decreases by 0.5 percentage points on both LVBench and LongVideoBench and increases by 0.2 points on Video-MME. This does not negate the overall improvement over the initial system, but it suggests that a better toolbox on the development set may not perform better on other datasets. I would appreciate an explanation of this result in relation to how the newly added tool is used.
Questions for authors
-
How frequently are the two new tools invoked on each evaluation benchmark, and how many questions and videos do they cover? Which question types benefit most, and under what circumstances might the tools hurt performance?
-
Which development-set questions benefit most from the tool added in Round 7? How do these questions differ from those in LVBench and LongVideoBench, and could these differences help explain the improvement on the development set alongside the decreases on some test benchmarks?
4 comments
Response to Reviewer vPRo (2)
#W2/Q2: Why Round 7 Gains on the Development Set but Not on LVBench and LongVideoBench.
We truly appreciate this insightful question. To answer it, we re-evaluate the Round-2 and Round-7 toolboxes on 300 newly sampled ALLVB development questions and all 1,549 LVBench questions, and explain the result from three aspects below. Note: based on the 9 task types of ALLVB, we broadly divide its questions into fine-grained and coarse-grained tasks.
| Toolbox | ALLVB (development) | ALLVB (development) | LVBench | LVBench |
| Fine-grained tasks | Coarse-grained tasks | Key-information retrieval | Summarization | |
| Share of questions | 54.0% | 46.0% | 18.8% | 3.7% |
| Round 2 | 71.0% (115/162) | 77.5% (107/138) | 73.9% (215/291) | 75.9% (44/58) |
| Round 7 | 76.5% (124/162) ↑ | 69.6% (96/138) ↓ | 77.7% (226/291) ↑ | 65.5% (38/58) ↓ |
1) Development questions that benefit most: the first table's 1st column shows that Round 7 mainly helps fine-grained development tasks (71.0% → 76.5%), led by object detection and tracking (+8.1%), emotion recognition (+6.5%), temporal action localization (+5.4%) and action recognition (+4.9%). These questions only name the characters and events of a moment and ask about a brief detail, e.g., "What is the color of Estella's ball-gown in the scene on the balcony?". The atomic scan suits them, as it checks fine details near a candidate moment.
2) Different distributions between development and test sets: the scan mainly helps fine-grained questions, but their share differs greatly across the three benchmarks. In our development set, fine-grained tasks, which cover 5 of the 9 ALLVB task types, make up 54.0% of the questions. In LVBench, only 18.8% of the questions are fine-grained key-information retrieval. In LongVideoBench, 88.3% of the questions first describe a specific scene in detail and then ask about it, so they are hard to divide into fine-grained and coarse-grained ones, and Round 7 brings no significant change on any of its 17 official question types. Hence, the questions that the scan helps are common in our development set but much less common in the two test benchmarks.
| Toolbox | ALLVB coarse-grained tasks | ALLVB coarse-grained tasks |
| Round 7 calls the scan | Round 7 does not call the scan | |
| Round 2 | 77.9% (81/104) | 76.5% (26/34) |
| Round 7 | 66.3% (69/104) ↓ | 79.4% (27/34) ↑ |
3) Tool misuse: besides the distribution difference, the Worker may also misuse the new tool, which further widens the gap. Why does Round 7 lose accuracy on coarse-grained tasks? The second table splits these questions by whether Round 7 calls the scan. When Round 7 calls the scan, accuracy drops from 77.9% to 66.3%, but when it does not, accuracy rises from 76.5% to 79.4%. Hence, the Worker sometimes misuses the local scan on questions that need the whole video. This misuse also explains the drop of LVBench summarization in the first table (75.9% → 65.5%). We think that such misuse may be reduced by a stronger Worker. As Tab. 2(c) shows, with the same toolbox, a stronger Worker uses the tools much better, raising LVBench accuracy from 45.6% with o4-mini to 68.4% with GPT-5.6-SOL.
Overall, we think two factors explain why Round 7 gains on the development set but not on the test benchmarks: their different question distributions, and the Worker's misuse of the new tool. As you noted, a better toolbox on the development set may not perform better on other benchmarks, although this "does not negate the overall improvement". This analysis will be added to the paper.
Thanks for authors response. I will maintain my positive score.
Appreciate your continued support!
Response to Reviewer vPRo (1)
#W1/Q1: How the Two New Tools Are Used During Testing.
Thank you for this helpful suggestion. Following it, we log every tool call of the final toolbox on all 1,549 LVBench questions and all 1,337 LongVideoBench validation questions, and report how the two new tools are used from three aspects:
1) Invocation frequency and coverage: the first table shows that both tools are reused on most questions and videos, e.g., the landmark index is called on 80.6% of LVBench questions and 99.0% of its videos, rather than fitting a few cases. They also play different roles: the Worker usually calls the landmark index first to locate events coarsely, and calls the atomic scan around its fifth tool call to check details near a candidate moment. Moreover, the atomic scan is called more on longer videos, e.g., on 56.6% of LongVideoBench questions with 15–60-min videos, but on only 15.7% of those with 15–60-s videos.
2) Question types that benefit most: the second table breaks LVBench down by question type, with the tool calls of our re-run and the accuracy of Tab. 2(d). The landmark index (Round 2) mainly helps summarization, lifting it from 60.3% to 77.6%, and the atomic scan (Round 7) helps key-information retrieval, lifting it from 72.9% to 74.6%. Overall, summarization and temporal grounding gain the most over Base (+8.7% and +4.6%). Notably, the Worker calls each tool only when needed: it calls the landmark index on 75.9%–84.2% of the other types, but on only 7.3% of temporal-grounding questions, as 91.8% of them already give the time, e.g., "What happens from 04:45-05:25?". When no time is given, it calls the tool on 10 of the 18 questions.
3) When the tools hurt: each tool hurts the type that the other one helps. After the landmark index is added, key-information retrieval drops from 77.3% to 72.9%, since each landmark segment summarizes about 13.6 minutes of video and may lead the Worker to miss a brief detail. After the atomic scan is added, summarization drops from 77.6% to 69.0%, as the Worker may misuse the local scan on questions that need the whole video (see W2/Q2).
Overall, both new tools are reused on most test videos and are called as needed. The landmark index helps questions that need an overview, while the atomic scan helps those that need a specific detail, but the current Worker may not always use them wisely. This analysis will be added to the paper.