acceptodds
Under review as a conference paper at ICLR 2027

VideoResearcher: Self-Improving Tool Design for Long-Video Understanding

12 views

Abstract

Video agents have made substantial progress in long-video Understanding. Yet effective video-agent systems require costly, time-consuming manual design and trial and error. Current self-improvement methods either refine low-impact prompts, recombine predefined micro-tools, or pursue poorly convergent harness optimization. To bridge this gap, we present VideoResearcher, a training-free multi-agent framework that autonomously designs, tests, and refines tools for video understanding, like a human researcher. VideoResearcher operates through dual Solving and Evolving loops: it analyzes tool-use trajectories to identify capability gaps, coordinates specialized agents to develop and validate executable tools, and reuses evolved tools to strengthen evidence acquisition in subsequent video reasoning. Through iterative tool refinement and validation, it progressively strengthens evidence acquisition without updating model parameters. VideoResearcher achieves state-of-the-art performance among self-improving agents and approaches the human-designed upper bound, demonstrating a training-free paradigm for long-video understanding that expands agent capabilities through autonomous tool development while reducing costly manual engineering.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long Video Understanding, Self-improving Agent System

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vPRoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VideoResearcher, a framework that analyzes failure trajectories in long-video question answering to automatically design, implement, and select video-understanding tools. The Worker and perception model remain fixed while the toolbox is optimized on 300 development questions from ALLVB, expanding from one initial tool to three tools. The frozen toolbox is then evaluated on LVBench, LongVideoBench, and Video-MME, improving average accuracy from 72.1% to 74.5%.

Critical strengths
  1. The paper demonstrates the feasibility of automatically developing video-understanding tools from failure trajectories. After selection on the development set, the tools are frozen and evaluated on three other benchmarks, where the system outperforms the initial agent in each case. This suggests that the improvements extend to questions beyond the development set.

  2. The paper provides tool implementations, examples of accepted and rejected candidates, and detailed execution trajectories. These materials help readers understand what the automatically generated tools do and how they change the way video evidence is acquired.

Critical weaknesses
  1. I would like to better understand how the two new tools are used during testing. The paper provides comparisons across evolution rounds and execution examples, but does not report which questions invoke each tool or how many different videos each tool is reused on. This information would help clarify which tasks each tool is suited to and which use cases account for the performance gains.

  2. The benefits of further evolution would be worth discussing. In Table 2(b), Round 7 improves development-set accuracy over Round 2, but accuracy decreases by 0.5 percentage points on both LVBench and LongVideoBench and increases by 0.2 points on Video-MME. This does not negate the overall improvement over the initial system, but it suggests that a better toolbox on the development set may not perform better on other datasets. I would appreciate an explanation of this result in relation to how the newly added tool is used.

Questions for authors
  1. How frequently are the two new tools invoked on each evaluation benchmark, and how many questions and videos do they cover? Which question types benefit most, and under what circumstances might the tools hurt performance?

  2. Which development-set questions benefit most from the tool added in Round 7? How do these questions differ from those in LVBench and LongVideoBench, and could these differences help explain the improvement on the development set alongside the decreases on some test benchmarks?

4 comments
AuthorsOpenReview ↗

Response to Reviewer vPRo (1)

#W1/Q1: How the Two New Tools Are Used During Testing.

Thank you for this helpful suggestion. Following it, we log every tool call of the final toolbox on all 1,549 LVBench questions and all 1,337 LongVideoBench validation questions, and report how the two new tools are used from three aspects:

ToolLVBenchLVBenchLongVideoBenchLongVideoBench
Questions calling the toolVideos coveredQuestions calling the toolVideos covered
Landmark index (Round 2)80.6% (1,248/1,549)99.0% (102/103)89.0% (1,190/1,337)95.5% (719/753)
Atomic scan (Round 7)66.0% (1,023/1,549)100% (103/103)43.8% (586/1,337)49.8% (375/753)

1) Invocation frequency and coverage: the first table shows that both tools are reused on most questions and videos, e.g., the landmark index is called on 80.6% of LVBench questions and 99.0% of its videos, rather than fitting a few cases. They also play different roles: the Worker usually calls the landmark index first to locate events coarsely, and calls the atomic scan around its fifth tool call to check details near a candidate moment. Moreover, the atomic scan is called more on longer videos, e.g., on 56.6% of LongVideoBench questions with 15–60-min videos, but on only 15.7% of those with 15–60-s videos.

LVBench question typeQuestions calling the toolQuestions calling the toolAccuracy in Tab. 2(d) ↑Accuracy in Tab. 2(d) ↑Accuracy in Tab. 2(d) ↑
Landmark indexAtomic scanBaseRound 2Round 7
Entity recognition83.8% (567/677)69.6% (471/677)66.2%66.2%66.6%
Event understanding83.2% (538/647)70.0% (453/647)65.2%68.3%67.4%
Key-information retrieval84.2% (245/291)58.8% (171/291)77.3%72.9%74.6%
Temporal grounding7.3% (16/220)44.1% (97/220)67.7%71.4%72.3%
Reasoning78.6% (158/201)63.7% (128/201)61.2%65.7%64.7%
Summarization75.9% (44/58)58.6% (34/58)60.3%77.6%69.0%

2) Question types that benefit most: the second table breaks LVBench down by question type, with the tool calls of our re-run and the accuracy of Tab. 2(d). The landmark index (Round 2) mainly helps summarization, lifting it from 60.3% to 77.6%, and the atomic scan (Round 7) helps key-information retrieval, lifting it from 72.9% to 74.6%. Overall, summarization and temporal grounding gain the most over Base (+8.7% and +4.6%). Notably, the Worker calls each tool only when needed: it calls the landmark index on 75.9%–84.2% of the other types, but on only 7.3% of temporal-grounding questions, as 91.8% of them already give the time, e.g., "What happens from 04:45-05:25?". When no time is given, it calls the tool on 10 of the 18 questions.

3) When the tools hurt: each tool hurts the type that the other one helps. After the landmark index is added, key-information retrieval drops from 77.3% to 72.9%, since each landmark segment summarizes about 13.6 minutes of video and may lead the Worker to miss a brief detail. After the atomic scan is added, summarization drops from 77.6% to 69.0%, as the Worker may misuse the local scan on questions that need the whole video (see W2/Q2).

Overall, both new tools are reused on most test videos and are called as needed. The landmark index helps questions that need an overview, while the atomic scan helps those that need a specific detail, but the current Worker may not always use them wisely. This analysis will be added to the paper.

AuthorsOpenReview ↗

Response to Reviewer vPRo (2)

#W2/Q2: Why Round 7 Gains on the Development Set but Not on LVBench and LongVideoBench.

We truly appreciate this insightful question. To answer it, we re-evaluate the Round-2 and Round-7 toolboxes on 300 newly sampled ALLVB development questions and all 1,549 LVBench questions, and explain the result from three aspects below. Note: based on the 9 task types of ALLVB, we broadly divide its questions into fine-grained and coarse-grained tasks.

ToolboxALLVB (development)ALLVB (development)LVBenchLVBench
Fine-grained tasksCoarse-grained tasksKey-information retrievalSummarization
Share of questions54.0%46.0%18.8%3.7%
Round 271.0% (115/162)77.5% (107/138)73.9% (215/291)75.9% (44/58)
Round 776.5% (124/162) ↑69.6% (96/138) ↓77.7% (226/291) ↑65.5% (38/58) ↓

1) Development questions that benefit most: the first table's 1st column shows that Round 7 mainly helps fine-grained development tasks (71.0% → 76.5%), led by object detection and tracking (+8.1%), emotion recognition (+6.5%), temporal action localization (+5.4%) and action recognition (+4.9%). These questions only name the characters and events of a moment and ask about a brief detail, e.g., "What is the color of Estella's ball-gown in the scene on the balcony?". The atomic scan suits them, as it checks fine details near a candidate moment.

2) Different distributions between development and test sets: the scan mainly helps fine-grained questions, but their share differs greatly across the three benchmarks. In our development set, fine-grained tasks, which cover 5 of the 9 ALLVB task types, make up 54.0% of the questions. In LVBench, only 18.8% of the questions are fine-grained key-information retrieval. In LongVideoBench, 88.3% of the questions first describe a specific scene in detail and then ask about it, so they are hard to divide into fine-grained and coarse-grained ones, and Round 7 brings no significant change on any of its 17 official question types. Hence, the questions that the scan helps are common in our development set but much less common in the two test benchmarks.

ToolboxALLVB coarse-grained tasksALLVB coarse-grained tasks
Round 7 calls the scanRound 7 does not call the scan
Round 277.9% (81/104)76.5% (26/34)
Round 766.3% (69/104) ↓79.4% (27/34) ↑

3) Tool misuse: besides the distribution difference, the Worker may also misuse the new tool, which further widens the gap. Why does Round 7 lose accuracy on coarse-grained tasks? The second table splits these questions by whether Round 7 calls the scan. When Round 7 calls the scan, accuracy drops from 77.9% to 66.3%, but when it does not, accuracy rises from 76.5% to 79.4%. Hence, the Worker sometimes misuses the local scan on questions that need the whole video. This misuse also explains the drop of LVBench summarization in the first table (75.9% → 65.5%). We think that such misuse may be reduced by a stronger Worker. As Tab. 2(c) shows, with the same toolbox, a stronger Worker uses the tools much better, raising LVBench accuracy from 45.6% with o4-mini to 68.4% with GPT-5.6-SOL.

Overall, we think two factors explain why Round 7 gains on the development set but not on the test benchmarks: their different question distributions, and the Worker's misuse of the new tool. As you noted, a better toolbox on the development set may not perform better on other benchmarks, although this "does not negate the overall improvement". This analysis will be added to the paper.

Thanks for authors response. I will maintain my positive score.

Appreciate your continued support!

Reviewer_C6ZVOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces VideoResearcher, a training-free, multi-agent framework designed to automatically develop and refine tools for long-video understanding. The framework operates through two loops: a "Solving Loop," which collects traces of tool usage, and an "Evolving Loop," which identifies failure cases, designs new tools, and evaluates their effectiveness. Throughout this process, the agent responsible for executing tasks and the underlying perception model remain fixed. Experimental results on LVBench, LongVideoBench, and Video-MME demonstrate that this framework outperforms both baseline agents and various existing self-improvement methods, increasing average accuracy from 72.1% to 74.5%.

Critical strengths
  • Interesting paper with clear writing, easy to follow.
  • The paper addresses a practical challenge in the development of video-based agents. By automatically refining tools based on observed failure cases, it offers the potential to reduce the need for manual tool engineering.
  • The distinction between the "Solving" and "Evolving" loops is clear and well-motivated. Keeping the worker and the perception model constant makes it easier to isolate and evaluate the impact of changes made to the tools.
  • The experiments cover three long-video benchmarks, providing comparisons against various self-improvement methods under a unified experimental setup.
  • The paper provides examples of the evolved tools and their specific implementations. Qualitative case studies illustrate how improved evidence-retrieval capabilities contribute to arriving at correct answers.
Critical weaknesses
  • The overall performance improvement is relatively modest, with average accuracy increasing from 72.1% to 74.5%. Moreover, the Round-2 toolbox performs slightly better than the final Round-7 toolbox on LVBench and LongVideoBench, raising questions about the benefits of continued evolution.
  • The framework employs multiple specialized agents for diagnosis, design, debugging, and verification; however, the paper does not evaluate the individual contribution of each component. It remains unclear whether a simpler process for tool generation and selection might yield similar results.
  • The paper lacks a detailed analysis of the computational costs incurred during the tool-evolution process. Given the frequent agent invocations, code generation, and evaluation processes involved across multiple iterations, it is difficult to determine whether this approach truly offers an efficiency advantage over manual development.
  • Throughout the entire evolutionary process, only two tools were ultimately adopted, both of which focus primarily on temporal indexing and localized video inspection. This is insufficient to demonstrate that the framework is capable of discovering a diverse range of novel tool functionalities. - The tool selection process relies repeatedly on the same set of 300 development problems, which may lead to overfitting. Although the final benchmarking is conducted independently, performing additional experiments with different development sets or varying random seeds would help assess the robustness of the evolved tools.
Questions for authors

Please see the Weaknesses part.

8 comments
AuthorsOpenReview ↗

Response to Reviewer C6ZV (1)

#W1: Modest Improvement, and Round 2 vs. Round 7.

We will clarify it from two aspects:

1) Modest improvement: existing self-improving methods struggle to improve the strong Base (72.1%) they all evolve from (Tab. 1): SkillOpt and AutoResearch even reduce its average accuracy (−2.3% and −0.4%), and Meta-Harness only gains +1.4%. In contrast, VideoResearcher achieves the best results on all three benchmarks (+2.4% on average), without updating any model parameter. This shows that existing self-improving methods are less reliable, while VideoResearcher steadily improves long-video understanding by evolving tools.

2) Round 2 vs. Round 7: to compare the two toolboxes, we evaluate them again on the full LVBench (1,549 questions) and on 300 freshly drawn ALLVB development questions, which we group into fine-grained tasks (e.g., action recognition) and coarse-grained tasks (e.g., video captioning). Round 7 differs from Round 2 only by the added atomic scan.

ToolboxLVBenchLVBenchALLVBALLVBLVBenchALLVB coarse-grained tasksALLVB coarse-grained tasks
Key-information retrievalSummarizationFine-grained tasksCoarse-grained tasksScan covers answer timeScan calledScan not called
Round 273.9% (215/291)75.9% (44/58)71.0% (115/162)77.5% (107/138)72.4% (76/105)77.9% (81/104)76.5% (26/34)
Round 777.7% (226/291) ↑65.5% (38/58) ↓76.5% (124/162) ↑69.6% (96/138) ↓82.9% (87/105) ↑66.3% (69/104) ↓79.4% (27/34) ↑

  • (a) From the task aspect: the table's 1st–4th columns show the same pattern on both sets: Round 7 is stronger on fine-grained tasks and weaker on coarse-grained ones. For example, it raises key-information retrieval to 77.7% (from 73.9%) but lowers summarization to 65.5% (from 75.9%), which agrees with Tab. 2(d) (+1.7% and −8.6%).

  • (b) From the tool-gain aspect: the table's 5th column restricts key-information retrieval to the 105 questions whose annotated answer moment falls inside the window inspected by the atomic scan. Here, Round 7 answers 87 of them and Round 2 only 76 (82.9% vs. 72.4%), confirming that the new scan truly benefits fine-grained tasks.

  • (c) From the tool-misuse aspect: the table's 6th and 7th columns split the coarse-grained ALLVB questions by whether Round 7 calls the atomic scan. With the scan, its accuracy falls to 66.3%; without it, it reaches 79.4%. Thus, the drop on coarse-grained tasks is due to the Worker wrongly applying a local tool to questions that require the whole video. We believe this misuse can be alleviated by a stronger Worker. In Tab. 2(c), with the same toolbox, a stronger LLM greatly raises LVBench accuracy (45.6% → 68.4%), showing that it uses the tools more wisely.

Overall, VideoResearcher's gain is more reliable than that of existing self-improving methods on long videos. A new tool adds a missing ability, but its test gain is not guaranteed, as the Worker may misuse it. This analysis will be added to the paper.

AuthorsOpenReview ↗

Response to Reviewer C6ZV (2)

#W2: Contribution of Each Component.

We sincerely appreciate this valuable comment. To check whether a simpler process for tool generation and selection yields similar results, we remove all specialized agents at once, leaving the simplest process, i.e., plain Codex. It receives the same failure trajectories and evaluation feedback and, like VideoResearcher, runs 10 rounds on the same 300 development questions with one candidate tool per round. Under this same setting, we also reproduce VideoResearcher, and evaluate both final toolboxes on all 1,549 LVBench questions.


MethodNewly designed tools ↑Fully tested candidates ↓Least-used tested tool ↑Accepted newly designed toolsDevelopment accuracy ↑ (Base: 74.7%)LVBench accuracy ↑ (Base: 66.9%)
Plain Codex20% (2/10)100% (10/10)1% (3/300)076.3% (+1.7%)65.1% (−1.8%)
VideoResearcher100% (10/10)40% (4/10)40% (119/300)180.3% (+5.7%)68.8% (+1.9%)

The table above shows what these agents contribute from four aspects:

1) From the design aspect: the table's 1st column shows that the agents turn every candidate into a new tool (100%), whereas only 20% of plain-Codex candidates are new tools. Without diagnosis, plain Codex spends 8 of its 10 rounds rewriting the prompt of ask_video, much like the ineffective SkillOpt (Tab. 1). With them, the Capability Auditor and Trajectory Analyzer first pinpoint what is missing, e.g., failed trajectories see only 3% of the video, and the Tool Designer then specifies a tool that differs from existing ones and past attempts (Lines 258–269), e.g., whole-video indexing, scene-change detection, and fine-detail verification.

2) From the testing aspect: as the 2nd column shows, plain Codex sends all of its candidates to the full 300-question test, the most costly step (Lines 298–303), while only 40% of ours reach it. The reason is our Debugger, which first checks on a few questions whether the Worker calls the tool and whether its output is valid (Lines 295–297), removing 6 flawed candidates, e.g., hallucinated event locations and mostly empty outputs. The costly test is therefore reserved for valid tools.

3) From the usage aspect: the 3rd column shows that plain Codex never checks whether the Worker calls its tools: one of them is called on just 1% of the 300 questions, yet still goes through the full test. Since our Debugger demands Worker invocation, even our least-used tested tool is called on 40% of questions, so no tested tool sits idle.

4) From the outcome aspect: in the 4th column, none of the updates accepted by plain Codex is a new tool, as both only revise the prompt and settings of ask_video, which proved ineffective for SkillOpt (Tab. 1). Ours is a genuinely new tool that checks a candidate event with five overlapping clips. This single tool lifts accuracy more on both the development set (+5.7% vs. +1.7%, 5th column) and the unseen LVBench (+1.9% vs. −1.8%, 6th column), while the prompt revisions of plain Codex bring no gain on LVBench.

Overall, a simpler process cannot match VideoResearcher. Plain Codex is built for coding and only checks whether code runs. Our agents also use Codex (Lines 255–256), but each adds one engineering prior of human researchers: diagnosing failures, designing callable tools, or verifying their effect. These engineering priors are crucial for designing new tools for long-video understanding. This analysis will be added to the paper.

AuthorsOpenReview ↗

Response to Reviewer C6ZV (3)

#W3: Computational Cost of Tool Evolution.

Thank you for raising this concern. To show whether our tool evolution is truly more efficient than manual development, we also add plain Codex as a baseline, which removes all our specialized agents and only uses the original Codex to develop tools. We run both methods for 10 rounds on the same 300 ALLVB development questions, record every agent invocation, code generation, and evaluation run with its time, and report the development cost from three views:

  • (a) Cost of each development step: the API calls, runs, and cost of the three paid steps (Lines 267–303) are listed below. Note: \$67.45/h is the US median wage of computer and information research scientists (BLS, May 2025).


    One 10-round runPlain CodexVideoResearcherEstimated human (US AI researcher, \$67.45/h)
    1) Design and build13 API calls (\$8.6)106 API calls (\$42.3)20 h (\$1,349)
    2) Smoke tests on 5 questions13 runs (\$2.5)40 runs (\$11.5)5 h (\$337)
    3) Accuracy Backtest on 300 questions10 runs (\$147.5)4 runs (\$93.0)15 h (\$1,012)
    Total cost\$158.5\$146.840 h (\$2,698)

    As shown in the first table, the costly backtest takes most of the budget of plain Codex (\147.5of147.5 of \\158.5), since it backtests all 10 candidates without checks, including a tool called on only 3 of 300 questions. In contrast, our Debugger screens each candidate with more API calls and smoke-test runs (Lines 295–297), and rejects 6 flawed ones in advance. Thus, only 4 candidates reach the backtest, and our total cost is \11.7lower.Notably,manualdevelopmentbyahumanresearcherwouldcostabout18timesmore(11.7 lower. Notably, manual development by a human researcher would cost about 18 times more (\\2,698).

  • (b) Cost of each token type: as the API charges different prices for different types of tokens, we break down the tokens and their cost by type below.


    Token type (price per 1M)Plain CodexVideoResearcher
    Uncached input (\$4)20.13M (\$80.5)20.69M (\$82.8)
    Cache write (\$5)3.63M (\$18.1)4.73M (\$23.6)
    Cache hit (\$0.40)4.95M (\$2.0)17.79M (\$7.1)
    Output (\$20)2.89M (\$57.9)1.66M (\$33.2)
    Total31.61M (\$158.5)44.88M (\$146.8)

    As shown in the second table, the uncached input is about 20M for both methods, while plain Codex spends more tokens on the costly side: its 10 backtests let the Worker generate 2.89M output tokens, the most expensive type (\20per1M).Incontrast,mostofourextratokensgototooldesign,whereourrolesre−readthesametrajectories,code,andreports,yielding17.79Mcachehitsatonlyonetenthoftheinputprice(20 per 1M). In contrast, most of our extra tokens go to tool design, where our roles re-read the same trajectories, code, and reports, yielding 17.79M cache hits at only one tenth of the input price (\\0.40 per 1M). Hence, despite more tokens (44.88M vs. 31.61M), our total cost remains lower (\146.8vs.146.8 vs. \\158.5).

  • (c) Time of each development step: the wall-clock time of the three steps is listed below.


    One 10-round runPlain CodexVideoResearcherEstimated human
    Design and build0.38 h2.69 h20 h (2 h per round)
    Smoke tests1.64 h8.50 h5 h (0.5 h per round)
    Accuracy Backtest21.52 h12.42 h15 h (1.5 h per round)
    Total wall-clock time23.54 h23.62 h40 h (one week)

    As shown in the third table, plain Codex spends 21.52 h of its 23.54 h on backtests. In contrast, we spend more time on design (2.69 h) and smoke tests (8.50 h), but only 12.42 h on backtests. Thus, both runs take nearly the same total time, far below the 40 h of manual development.

Overall, VideoResearcher rejects flawed candidates before the costly backtest and spends its extra tokens on tool design, mostly as cheap cache hits. Thus, its total cost is similar to that of plain Codex (\146.8vs.146.8 vs. \\158.5) and about 18 times lower than manual development (\$2,698). As you kindly noted, it "offers the potential to reduce the need for manual tool engineering". This analysis will be added to the paper.

Reviewer_QY4MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VideoResearcher automates tool development for long-video question answering without updating model parameters. A fixed reasoning Worker interacts with a fixed video perception model through an evolving toolbox. A separate development loop analyzes failed trajectories, proposes and implements tools, tests them, and retains changes that improve accuracy on 300 development questions from ALLVB. The resulting toolbox is frozen before evaluation on LVBench, LongVideoBench, and Video-MME. Two accepted tools provide coarse scene indexing and detailed inspection of candidate temporal windows. The framework improves average accuracy from 72.1% to 74.5% and outperforms the evaluated self-improvement baselines initialized from the same agent.

Critical strengths
  1. The main comparison makes a meaningful effort to control the starting conditions. SkillOpt, Meta-Harness, AutoResearch, and VideoResearcher share the initial agent, Worker, perception model, development questions, and evolution-round budget. The toolbox is then frozen for evaluation on separate benchmarks. This is a useful design for studying whether development experience produces reusable improvements, rather than merely correcting the questions used during evolution. The authors also appropriately distinguish these comparisons from the published results of human-engineered systems.

  2. The paper makes the output of tool evolution concrete. Table 3 shows what was implemented, why it was proposed, and whether it helped. The landmark index is a simple but useful observation procedure, and its improvements extend beyond the development set. Including rejected tools and the weaker transfer of the later update is valuable: these results give a more informative picture of tool development than reporting only the final accuracy.

Critical weaknesses
  1. The experiments do not establish what the specialized development workflow contributes beyond directing a coding agent to improve video tools. The framework separates auditing, trajectory analysis, design, coding, debugging, verification, and memory, but Table 2(a) only compares evolution against no evolution. A simpler coding agent with the same trajectories, evaluation feedback, and tool-editing scope is missing. This matters because the paper attributes its advantage to targeted tool development, while the broader baselines reportedly spend much of their search on prompts. The current results do not separate the benefit of choosing the tool-editing scope from the benefit of the proposed multi-agent process.

  2. The efficiency claim is not supported by an actual cost comparison. The introduction motivates tool-level search as a more efficient alternative to broader system optimization, but matching 20 evolution rounds does not necessarily match expenditure. A round can contain several agent calls, repairs, and development-set evaluations. The paper reports neither development time nor token/API cost, and it does not report realized inference costs after tool evolution. Since an evolved tool can make several perception calls within one Worker turn, equal turn limits also do not establish equal inference cost. The accuracy results are useful, but they do not yet substantiate the claimed efficiency advantage.

Questions for authors
  1. How does VideoResearcher compare with a single coding agent restricted to editing the toolbox, given the same failure trajectories, development evaluator, and computational budget? This would help determine whether the specialized workflow is necessary.

  2. What are the total development costs for each self-improvement method—model calls or tokens, evaluator runs, and wall-clock time—and the average perception calls and sampled frames per benchmark question? Does the advantage remain under a matched total-cost budget?

  3. How reproducible are the evolution outcomes across independent runs or different development samples? In Table 2(b), Round 2 already achieves 68.9/75.5/80.0, compared with 68.4/75.0/80.2 after Round 7. The later accepted update improves development accuracy but lowers two test scores. Is this representative of the selection procedure’s behavior or specific to this run?

7 comments
AuthorsOpenReview ↗

Response to Reviewer QY4M (1)

#W1/Q1: Specialized Workflow vs. a Single Coding Agent.

Appreciate this precise and valuable suggestion. Following your advice, we run a single coding agent, i.e., plain Codex, restricted to editing the toolbox, with the same failure trajectories and evaluation feedback. For a controlled comparison, we reproduce VideoResearcher alongside plain Codex under the same setting: both run 10 rounds on the same 300 development questions, each round proposes one candidate tool, and both final toolboxes are tested on all 1,549 LVBench questions.


MethodNewly designed tools ↑Fully tested candidates ↓Least-used tested tool ↑Accepted newly designed toolsDevelopment accuracy ↑ (Base: 74.7%)LVBench accuracy ↑ (Base: 66.9%)
Plain Codex20% (2/10)100% (10/10)1% (3/300)076.3% (+1.7%)65.1% (−1.8%)
VideoResearcher100% (10/10)40% (4/10)40% (119/300)180.3% (+5.7%)68.8% (+1.9%)

We would like to clarify the contribution of the workflow from four aspects:

1) From the design aspect: in the table's 1st column, only 20% of plain-Codex candidates are newly designed tools, versus 100% for ours. Although restricted to editing tools, plain Codex still prefers rewriting the prompt of ask_video (8 of 10), degenerating into SkillOpt-style prompt refinement (Tab. 1). In contrast, our Capability Auditor and Trajectory Analyzer first locate the missing capability, e.g., failed trajectories cover only 3% of the video, and the Tool Designer then specifies a tool unlike existing ones and past attempts (Lines 258–269). Hence, all 10 of our rounds propose new tools, e.g., whole-video indexing, scene-change detection, and fine-detail verification.

2) From the testing aspect: in the table's 2nd column, plain Codex fully tests 100% of its candidates on all 300 questions, the most costly step (Lines 298–303), versus only 40% for ours. This is because plain Codex only checks whether the code runs, while our Debugger first checks Worker invocation and output validity on a few questions with the Debug Checklist (Lines 295–297), and stops 6 flawed candidates early, e.g., hallucinated event locations and mostly empty outputs. This ensures that the most costly step is spent only on valid tools rather than blind trials, keeping our development cost low.

3) From the usage aspect: in the table's 3rd column, plain Codex cannot ensure that its designed tools are actually called by the Worker, e.g., one of them is called on only 1% of the 300 questions, yet it still enters the costly full test. In contrast, our Debugger checks Worker invocation before this test (Lines 295–297), so even our least-used tested tool is called on 40% of questions. This ensures that every tested tool is truly used, rather than wasting the costly test on idle tools.

4) From the outcome aspect: in the table's 4th column, plain Codex fails to accept any newly designed tool, as its two accepted updates only revise the prompt and settings of ask_video, similar to the ineffective SkillOpt (Tab. 1). In contrast, our accepted update is a genuinely new tool that verifies a candidate event with five overlapping clips, accepted only after passing both the Debugger and the full test (Fig. 2). With this single new tool, our toolbox gains more on both the development set (+5.7% vs. +1.7%, 5th column) and the unseen LVBench (+1.9% vs. −1.8%, 6th column). Hence, our toolbox truly gains a new capability, whereas the prompt revisions of plain Codex do not help on LVBench.

Overall, plain Codex merely writes runnable code, lacking the engineering experience of human researchers that VideoResearcher injects, e.g., diagnosing failures, designing callable tools, and verifying their evidence. These engineering priors are crucial for designing new long-video tools. This analysis will be added to the paper.

AuthorsOpenReview ↗

Response to Reviewer QY4M (2)

#W2/Q2: Development and Inference Costs.

Following your suggestion, we measure both the development and inference costs. Our cost mainly comes from two parts (Lines 347–355):

  • (a) Video perception: all video frames are processed by Qwen3.5-4B, which runs locally on eight B200 GPUs and thus has no API cost. We measure it by the number of sampled frames.

  • (b) Text reasoning: only text, i.e., questions, tool outputs, and code, is sent to the paid GPT-5.6-SOL API, which serves as the Worker and drives all evolving roles through the Codex SDK (Lines 255–256). We measure it by API calls and their cost at the public list price.

Below, we first report the development cost and then the inference cost.

1) Development cost: we run both methods for 10 rounds on the same 300 ALLVB development questions, log every API request and its time, and report this cost from three views. Note: the cost logs of previous self-improving methods were not kept. To save time, we fairly compare cost with your suggested plain Codex.

  • (a) Cost of each development step: we count the API calls, runs, and cost of the three paid steps (Lines 267–303), as shown below.


    One 10-round runPlain CodexVideoResearcherEstimated human (US AI researcher, \$67.45/h)
    1) Design and build13 API calls (\$8.6)106 API calls (\$42.3)20 h (\$1,349)
    2) Smoke tests on 5 questions13 runs (\$2.5)40 runs (\$11.5)5 h (\$337)
    3) Accuracy Backtest on 300 questions10 runs (\$147.5)4 runs (\$93.0)15 h (\$1,012)
    Total cost\$158.5\$146.840 h (\$2,698)

    In the first table, plain Codex spends most of its budget on the costly backtest (\147.5of147.5 of \\158.5), as it blindly backtests all 10 candidates, even a tool that the Worker calls on only 3 of 300 questions. In contrast, our Debugger strictly checks each candidate with more API calls and smoke-test runs (Lines 295–297), and stops 6 flawed ones early. Hence, only 4 candidates are backtested, making our total cost \11.7lower.Notably,anestimatedhumancostsabout18timesmore(11.7 lower. Notably, an estimated human costs about 18 times more (\\2,698).

  • (b) Cost of each token type: since the API bills each type of token at a different price, we split the tokens and their cost by type, as shown below.


    Token type (price per 1M)Plain CodexVideoResearcher
    Uncached input (\$4)20.13M (\$80.5)20.69M (\$82.8)
    Cache write (\$5)3.63M (\$18.1)4.73M (\$23.6)
    Cache hit (\$0.40)4.95M (\$2.0)17.79M (\$7.1)
    Output (\$20)2.89M (\$57.9)1.66M (\$33.2)
    Total31.61M (\$158.5)44.88M (\$146.8)

    In the second table, with nearly the same uncached input (about 20M each), plain Codex spends its tokens on the costly side: its 10 backtests make the Worker produce 2.89M output tokens, the most expensive type (\20per1M).Incontrast,wespendmostextratokensondesigningtools,whereourrolesrepeatedlyreadthesametrajectories,code,andreports.Thus,17.79Mofourtokensarecachehits,billedatonlyonetenthoftheinputprice(20 per 1M). In contrast, we spend most extra tokens on designing tools, where our roles repeatedly read the same trajectories, code, and reports. Thus, 17.79M of our tokens are cache hits, billed at only one tenth of the input price (\\0.40 per 1M). Hence, although we use more tokens (44.88M vs. 31.61M), our total cost is still lower (\146.8vs.146.8 vs. \\158.5).

  • (c) Time of each development step: we measure the wall-clock time of the three steps above, as shown below.


    One 10-round runPlain CodexVideoResearcherEstimated human
    Design and build0.38 h2.69 h20 h (2 h per round)
    Smoke tests1.64 h8.50 h5 h (0.5 h per round)
    Accuracy Backtest21.52 h12.42 h15 h (1.5 h per round)
    Total wall-clock time23.54 h23.62 h40 h (one week)

    In the third table, plain Codex spends most of its time on backtests (21.52 h of 23.54 h). In contrast, we spend more time on design (2.69 h) and smoke tests (8.50 h), but only 12.42 h on backtests. Hence, both runs take nearly the same total time, far less than the 40 h of an estimated human.

(The inference cost is reported in the next comment.)

AuthorsOpenReview ↗

Response to Reviewer QY4M (3)

#W2/Q2: Development and Inference Costs (Continued).

2) Inference cost: we count the perception calls and sampled frames per test question, as shown below. Note: the 10-turn limit (Line 349) counts Worker turns, not perception calls.


MethodPerception callsSampled framesFrames per call
VideoResearcher (w/o Evolution)4.03,108771
VideoResearcher11.92,959249

In the table, VideoResearcher calls Qwen3.5-4B more often than VideoResearcher (w/o Evolution) (11.9 vs. 4.0 calls), but reads even fewer frames per question (2,959 vs. 3,108). This is because we deliberately set a frame budget when developing tools: no matter how many perception calls a new tool makes (at most five), its total frames are capped at 512 (Lines 352–353), only a quarter of the 2,048 frames of the default ask_video (Lines 349–350). Thus, each call reads only a few targeted frames (249 vs. 771 frames per call). This budget strictly ensures that our gain comes from better tools, rather than from sampling more frames.

Overall, for development cost, VideoResearcher costs less than plain Codex by stopping flawed tools before the costly backtest. For inference cost, our strict frame budget ensures that the gain comes from better tools rather than more frames. This analysis will be added to the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.