acceptodds
Under review as a conference paper at ICLR 2027

VideoAgentBoost: Evolving Video Agents through Additive Capability Improvements

Abstract

Changing video sources and question demands create a recurring need for video agents to evolve. Multimodal perception, tool use, and long-horizon reasoning make execution feedback costly to obtain, motivating targeted capability improvements from limited execution evidence. We introduce VideoAgentBoost, a framework for additive video-agent evolution guided by capability residuals: reusable descriptions of missing or insufficient behaviors identified by comparing execution trajectories with reference solutions grounded in annotated answers and supporting video evidence. Each round develops an increment for one residual, expanding or refining the relevant capabilities within the current agent. Targeted research and design guide code generation to implement the increment in the affected modules, including the coordination needed to use the improved capability. Before integration, targeted probing examines whether the increment contributes to resolving the selected deficiency through improved evidence acquisition or reasoning. Subsequent executions guide further improvements. We independently evolve DVD, VideoARM, and VideoAgent for five rounds on a shared evolution set. All three agents improve over their starting points on LVBench, LongVideoBench, and Video-MME. DVD gains 6.59 percentage points on LVBench over five evolution rounds, and component ablations further support the effectiveness of the proposed method. Our code and evolution-set annotations will be made publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Agents, Automated Agent Evolution, Video Understanding, Additive Capability Improvements, Capability Residuals

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xyUPOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes VideoAgentBoost. This is a system helps to recursively improve video agents (DVD, VideoARM and VideoAgent). Based on a training set of 139 questions and 12 videos, qwen3.8-max is used to generate gold answer. Then the main video agent (qwen3.7-max for thinking, qwen3.7-plus for vision) rollout some trajectories. Codex with gpt-5.6-sol is used to review the trajectories and diagnosize shortcomings in five semantic modules (structure, localization, perception, memory, and thinking). Then codex adds new tools or edits prompt for the main video agents, with codex auditing these changes. Experiments are conducted on LVBench, LongVideoBench and Video-MME, which shows improvement.

Critical strengths
  1. Lightweight in-context learning. The paper only uses 139 questions and 12 videos as training set for in-context learning, while not training any weights. This methodology can help quickly transfer and generalize to other application and deployment environments.
  2. Good interpretability. This paper dedicatedly designs a road map to improve video agents. Each stages is interpretable and controllable. Humans can study this self-improvement process comfortably.
Critical weaknesses
  1. This paper doesn't compare its meta evolving system with other evolution method. Table 1 only compares the proposed method with other end-to-end video models (raw model without complicated harness) and raw video agents themselves, but doesn't compare with codex + gpt-5.6-sol and other sota (e.g. MetaVideoAgent in this paper's reference).
  2. No budget is disclosed for experiments. So it's not sure whether this improvement comes from the dedicatedly designed system or simply test-time scaling.
  3. For long-horizon agentic tasks, like trajectory reviewing and auditing of codex, the code evolution is very stochastic. But the main and ablation experiments are all conducted only once. It's not sure whether this improvement is stable, or it's luck, or simply cherry picking.
Questions for authors
  1. Can you run experiments of codex itself as video agent, with the same token budget as the addition of your VideoAgentBoost system's cost and video agent's cost. Although gpt doesn't support video directly, it can view images, and it can use python, ffmpeg or other tools to help analyze videos. Just the raw codex agent is ok. You may run gpt-6.1-sol, it's better and cheaper than 5.6 sol.
  2. Can you report the budget of each experiment.
  3. Can you compare your evolving method with related works like MetaVideoAgent?
  4. When using the codex for reviewing or improving, can the codex agent access the websearch tool to get ideas like LVBench/Video-MME leaderboard tricks, or other works optimized on these benchmarks?
6 comments
AuthorsOpenReview ↗

Additional Comparisons, Cost Reporting, and Evaluation Details

Thank you for your feedback on the experimental comparisons, computational costs, and evaluation.

W1 & Q1 & Q3 — Comparisons with evolution methods and direct Codex evaluation

We have added an experiment in which Codex with GPT-5.6 Sol directly answers LVBench questions, with an inference budget capped at 1.5 times that of the DVD agent evolved by VideoAgentBoost. Its accuracy after evidence auditing of the answering trajectories is 64.75% (VideoAgentBoost: 81.19%).

We further compare evolution methods using the same initial DVD, evolution set, model configuration, and evolution budget. After five evolution rounds, each final agent is evaluated on LVBench three times:

E1 (DVD evolution results under the same configuration): Initial DVD 74.56%, simple feedback loop 74.40%, ADAS 73.34%, Meta-Harness 75.56%, MetaVideoAgent 78.24%, VideoAgentBoost 81.19%.

Code optimization for all methods uses Codex with GPT-5.6 Sol. We observed that simple feedback loops and general-purpose evolution methods lack in-depth analysis of video-evidence acquisition and use, making it difficult to attribute a mature agent's failures to specific missing capabilities. Their modifications also tend to cover a broad scope. Repeatedly screening such changes through full-set evaluation incurs substantial overhead in long-video tasks.

MetaVideoAgent can analyze video-understanding capability needs, but its modifications do not follow residual-guided additive correction, making effective improvement of mature systems challenging. VideoAgentBoost instead compares Gold Paths with student trajectories to identify actual capability deficiencies, develops targeted capability increments, and uses Probe to check whether the improvements take effect. The results demonstrate the value of this design for automatically improving mature video agents.

W2 & Q2 — Evolution and inference costs

Each starting agent has an evolution-budget cap of 300M tokens and undergoes five evolution rounds. The estimated total consumption across the three experiments is approximately 650M tokens. DVD consumes an estimated average of 38.8M tokens per round, broken down below:

E2 (Average tokens per DVD evolution round): Full Evaluation on the evolution set 18.4M, Gold Path construction 9.4M, other evolution stages 5.4M, Probe 5.6M.

Other evolution stages include trajectory comparison in Review, Diagnosis, Research, Design, Codegen, and Smoke.

We additionally report average inference tokens and inspected frames per question before and after evolution:

E3 (Average inference tokens per question, before → after evolution): VideoAgent 24K → 45K, DVD 111K → 158K, VideoARM 103K → 129K.

E4 (Average inspected frames per question, before → after evolution): VideoAgent 28.59 → 34.26, DVD 86.88 → 52.67, VideoARM 164.37 → 154.38.

Evolution adds some prompt and tool-use overhead, but overall inference costs do not grow dramatically. Together with the comparisons under the same evolution budget, these results support targeted capability improvement as the source of the gains, rather than simply increasing test-time compute.

W3 — Repeated evaluation and result selection

We evaluated each of the three final evolved agents three times. The results consistently retain the improvement trend, with mean accuracies reported below:

Final evolved agentLVBenchLongVideoBench (Val), OverallLongVideoBench (Val), LongVideo-MME, OverallVideo-MME, Long
VideoAgent54.1268.0166.7971.3064.34
DVD81.1980.6576.7282.0976.19
VideoARM84.7981.8578.6784.1780.56

VideoAgentBoost uses Probe to ground improvement targets and admission decisions in execution evidence. Across the three video-agent evolution experiments, 15 capability increments were admitted through 68 Probe attempts: 15 passed, 42 failed and triggered repair, and 11 failed and returned to Diagnosis.

Once Probe verifies the target capability, the verified capability is admitted; if it does not pass verification, the process returns to Diagnosis within the same round to determine a revised improvement direction. This process grounds each round's admission in execution evidence of the targeted capability improvement, thereby achieving stable gains.

Q4 — Web access and evaluation separation

We audited all Research search records. They concern reusable mechanisms such as coarse-to-fine temporal localization, conflicting-evidence verification, timestamped ASR/OCR, failure recovery, and cross-segment comparison. We found no leaderboard tricks or test-question answers. We have added the specific Research content for the evolution of all three agents in Appendix A.2. The evolution procedure was not informed of the eventual benchmark evaluation, and test questions, answers, and evaluation results did not participate in evolution.

Thank you for the detailed response and the revised paper. The new same-budget comparison with a simple feedback loop, ADAS, Meta-Harness and MetaVideoAgent (Table 2), the cost accounting (Sec. 6.4, App. D.3), and the w/o Probe ablation address my main concerns, and I will raise my score. Remaining points: (1) According to App. D.2, correct Codex answers are discarded when the trajectory lacks evidence from the supporting intervals, but the same audit does not seem to be applied to VideoAgentBoost. Please report Codex's raw accuracy, or apply the same audit to both. (2) The repeated runs re-evaluate a single evolved agent. Independent evolution runs (at least for DVD, with std, and ideally for the baselines) are still needed to assess the stochasticity of the evolution process; please at least acknowledge this in the Limitations. (3) Inference tokens increase by 25–88%, so a compute-matched initial-agent baseline would further strengthen the claim.

AuthorsOpenReview ↗

Clarifications on Codex Evaluation, Budget Matching, and Evolution Reproducibility

Thank you for acknowledging our additional experiments and manuscript revisions, and for raising these specific follow-up questions.

1 With a reasoning budget cap of 1.5 times that of the evolved DVD, raw Codex achieves an accuracy of 77.53%. In our previous response and manuscript revisions, we did not clearly explain the cases in which execution trajectories lacked evidence from the supporting evidence intervals. When inspecting the answer trajectories, we repeatedly observed that Codex could produce correct answers without examining any information within the questions' supporting evidence intervals. We therefore suspect that Codex may have been trained on these benchmarks and memorized relevant information, and excluded these correct responses accordingly. However, we would like to emphasize that, in the evolution experiments involving the three video agents in our paper, Codex was not informed that the final evolved model would be evaluated on the benchmarks, and the benchmarks were never exposed to the evolution process.

3 We increased the budget caps of the initial DVD and evaluated this budget-matched initial agent. The main adjustments were as follows:

E5 (adjustments to the initial DVD settings): reasoning steps 15 → 20, frame inspect 50 → 100, clip search 16 → 32, global browse 300 → 500.

Across three runs with the expanded budget, the average results were 123.32 observed frames per question, 157K tokens per question, and 77.45% accuracy.

2 Regarding independent repetitions of the evolution process, we will provide results from rerunning the complete evolution pipeline as soon as possible over the next few days and revise the corresponding content in the paper.

Reviewer_5GjyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes VideoAgentBoost, which identifies reusable capability deficiencies by comparing agent execution trajectories with reference solutions constructed from annotated answers and supporting video evidence, and iteratively generates targeted code changes. The framework includes diagnosis, research, design, code generation, and targeted probing to expand or refine different functional modules of video agents. The authors independently optimize DVD, VideoARM, and VideoAgent for five rounds on a fixed development set and demonstrate performance improvements on three independent video-understanding benchmarks.

Critical strengths
  1. The improvement targets are grounded in execution evidence. The method combines reference solutions and execution trajectories to identify capability deficiencies, and checks whether newly introduced capabilities actually contribute to evidence acquisition or reasoning. The case studies help explain how code changes affect agent behavior.

  2. The method is evaluated across different starting agents and independent benchmarks. All three agents improve on all three test benchmarks after optimization with the same framework, supporting its applicability across architectures and the transfer of the resulting improvements. The component ablations also provide evidence for the contributions of Review and Research.

Critical weaknesses
  1. Comparisons with other agent optimization methods remain limited. The paper demonstrates clear improvements across multiple starting agents and test benchmarks. However, it is unclear why methods discussed in the related work, such as Gödel Agent and the Darwin Gödel Machine, are not included in the experimental comparison. Could these methods be adapted to optimize the same video agents? The before-and-after comparisons and component ablations support the effectiveness of the proposed method, but without comparisons against similar optimization methods, its relative advantages remain difficult to assess. This is my main concern about the current experiments.

  2. The targeted verification and capability-preservation mechanisms require further clarification. The appendix describes trajectory-based Probe auditing, but the sample selection and acceptance criteria are not sufficiently explicit. In addition, a modification is accepted after passing the targeted Probe, with full development-set evaluation performed afterward. It remains unclear how the framework handles a modification that improves the targeted capability but degrades performance on other tasks. (Appendices A.3–A.4)

Questions for authors
  1. Have the authors considered adapting methods such as Gödel Agent or the Darwin Gödel Machine to optimize the same starting agents? If there are barriers to adaptation, please clarify them. If feasible, comparing performance gains under the same feedback conditions and comparable development budgets would help establish the advantages of the proposed method.
  2. How are Probe questions selected, and how is a passing result determined in each round? If a modification passes the targeted Probe but reduces performance on the full development set, is there a mechanism to reject or roll back the change?
  3. Regarding the motivation, I suggest clarifying the connection between changes in video sources and user requirements and the offline capability-optimization setting studied here, so that the abstract and introduction more closely reflect the objectives evaluated in the experiments.
7 comments
AuthorsOpenReview ↗

Evolution Comparisons, Probe Criteria, and Revised Motivation

Thank you for recognizing the evidence-grounded improvement targets and the transfer of improvements across agents and benchmarks.

W1 & Q1 — Comparisons with other optimization methods

VideoAgentBoost uses an external development model to analyze feedback and improve a target video agent, following an autonomous-evolution setting, whereas Gödel Agent and Darwin Gödel Machine use self-evolution. We have revised Related Work in Section 2 to focus on automated agent evolution and the positioning of our work.

We have added comparisons with the simple feedback loop, ADAS, and other autonomous-evolution methods under the same budget and configuration. Starting from the initial DVD, each method performs five evolution rounds, followed by three LVBench evaluations to obtain mean accuracy. The experimental setup and results are presented in Section 5.3 and Table 2 in Section 6.2, respectively.

E1 (DVD evolution results under the same configuration): Initial DVD 74.56%, simple feedback loop 74.40%, ADAS 73.34%, Meta-Harness 75.56%, MetaVideoAgent 78.24%, VideoAgentBoost 81.19%.

All methods use Codex to modify the student agent. We observed that simple feedback loops and general-purpose evolution methods lack sufficiently clear diagnoses of video-understanding capability deficiencies and improvement targets, limiting sustained improvement. Failed trajectories commonly involve intertwined localization, perception, and reasoning issues, while costly long-video execution and validation make these changes harder to analyze and screen.

VideoAgentBoost addresses these difficulties by comparing reference solutions with student trajectories, identifying specific missing video-understanding capabilities, and implementing additive improvements. Probe verifies the targeted capability before the candidate proceeds to Full Evaluation on the evolution set.

W2 & Q2 — Probe selection, acceptance, and capability preservation

We have expanded Section 4 and Appendices A.3–A.4 to describe Probe question selection, capability-admission criteria, and candidate revision, and to explain how Full Evaluation on the evolution set provides feedback for subsequent evolution.

Review compares each failed trajectory with its Gold Path to identify video-understanding capabilities missing from the student agent. Diagnosis consolidates these findings into reusable capability residuals, selects one residual for the current round, and constructs a small Probe set from the questions that exposed it. During Probe Audit, Codex examines execution trajectories against the improvement target. A candidate passes when the target capability is exercised, at least one instance demonstrates the expected improvement, and the trajectory shows how the relevant modules address the capability gap. For example, a candidate may correctly recognize a local visual detail while its final answer remains affected by a separate temporal-localization deficiency. Probe can recognize the demonstrated contribution of local recognition as an effective capability improvement.

We retain capability increments that pass Probe. Failures exposed by subsequent Full Evaluation, including misuse or overuse of new tools, enter the next Review and Diagnosis to guide further improvements in capability use and module coordination. An unsuccessful Probe returns the candidate to Design for plan revision, followed by further implementation and validation. If the current improvement plan fails three consecutive Probe attempts, the process returns to Diagnosis within the same round to reassess the improvement direction.

Probe precedes Full Evaluation to establish whether the round's target capability has actually improved. In the DVD experiment, one Probe costs approximately 1.4M tokens, whereas Full Evaluation on the evolution set costs approximately 18.4M. Candidates that have not achieved the targeted improvement are therefore revised before full-set evaluation, reducing the substantial evaluation cost of ineffective candidates.

Q3 — Motivation and the offline evaluation setting

VideoAgentBoost aims to automatically improve existing, mature, human-designed video agents using execution feedback. We evaluate this objective through capability optimization on an offline evolution set and performance evaluation on independent public benchmarks.

We have revised the abstract, Introduction in Section 1, and Related Work in Section 2 to clarify the objective of automatically improving mature, human-designed video agents and align the motivation with offline capability optimization and independent benchmark evaluation. In brief, the original motivation was that “changing video sources and user requirements call for continuous video-agent evolution.” The revised motivation is to “automatically improve mature, human-designed video agents by addressing capability deficiencies exposed during execution, thereby strengthening their video-understanding capabilities.”

AuthorsOpenReview ↗

Kind Reminder

We hope you are doing well. We sincerely appreciate the thoughtful feedback and constructive suggestions you have provided.

With the discussion deadline approaching, we would be grateful if you could review our responses when you have a chance and let us know whether any concerns remain. Your feedback would allow us sufficient time to address additional questions and provide any further information that would support the discussion.

If our rebuttal and manuscript revisions have resolved your concerns, we would sincerely appreciate your consideration of an increase in your rating.

Thank you again for your time and consideration. We look forward to continuing the discussion with you.

Thank you for the additional experiments and clarifications. The new comparisons and the revised motivation substantially improve the paper. However, two issues remain. First, the distinction between external-model evolution and self-evolution does not fully explain why Godel Agent and Darwin Godel Machine cannot be adapted to the same setting. Second, please clarify the exact Probe selection procedure, the number of Probe questions, and whether one successful trajectory is sufficient for admission.

Reviewer_kQkdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VideoAgentBoost improves video agents through iterative code changes guided by recurring capability gaps. It compares execution traces with reference solutions grounded in annotated answers and video evidence, develops one targeted improvement per round, and checks it through focused probes. After five rounds on a shared evolution set, all three starting agents improve on LVBench, LongVideoBench, and Video-MME.

Critical strengths
  1. The improvements are consistent across three starting agents and three held-out benchmarks. DVD improves from 74.56% to 81.15% on LVBench, while VideoARM improves from 78.57% to 84.25%. The stated separation between evolution and benchmark evaluation makes these results meaningful evidence of transfer.

  2. The trajectory examples help explain what changed, such as recovering brief visual details through focused inspection or resolving ambiguity through speech evidence. The Review and Research ablations also provide useful evidence that these stages contribute to the final performance.

Critical weaknesses
  1. The experiments do not sufficiently isolate the benefit of the proposed evolution strategy. Table 2 tests Review and Research, but retains residual diagnosis and the broader development loop. A simpler coding-agent baseline, given the same feedback, tools, and budget, would help establish whether targeting and accumulating capability residuals offers an advantage over ordinary iterative code improvement. This matters because the evolution procedure is the main contribution.

  2. The cost of obtaining the improvements is unclear. Appendix B reports approximately 47M teacher-model tokens for reference-solution construction during DVD evolution, but the paper does not report the full evolution cost, candidate/retry budgets, or inference costs before and after evolution. Since the method is motivated by expensive multimodal feedback, these measurements are needed to assess its practical tradeoff

Questions for authors
  1. How does the method compare with a simpler coding-agent loop under the same supervision, tool access, and total evolution budget, without explicit capability-residual targeting?

  2. Could you report the total evolution cost and candidate attempts, together with inference cost per question before and after evolution? In particular, how much additional video inspection and tool use accompanies the accuracy gains?

3 comments
AuthorsOpenReview ↗

Evolution Baselines and Computational Cost Analysis

Thank you for recognizing the improvements across starting agents on held-out benchmarks. We address the evolution baselines and costs below.

W1 & Q1 — Comparison with ordinary iterative code improvement

We have added comparisons with other evolution methods under the same budget and configuration. Section 5.3 describes the experimental setup, Table 2 in Section 6.2 reports the results, and Appendix D.2 introduces the comparison methods. All methods start from the same initial DVD and use Codex with GPT-5.6 Sol for five evolution rounds. The final agents are evaluated on LVBench three times, yielding the following mean accuracies:

E1 (DVD evolution results under the same configuration): Initial DVD 74.56%, simple feedback loop 74.40%, ADAS 73.34%, Meta-Harness 75.56%, MetaVideoAgent 78.24%, VideoAgentBoost 81.19%.

The simple feedback loop directly analyzes failed execution trajectories and modifies the code. We observed that, in complex video-understanding tasks, simple feedback loops and general-purpose evolution methods struggle to identify the specific missing capabilities. They tend to make broad, insufficiently targeted changes, making sustained improvement difficult.

VideoAgentBoost organizes this process around residual-guided additive improvements. It first constructs reference solutions describing correct reasoning and compares them step by step with student trajectories to identify missing video-understanding capabilities, then develops functionality addressing the selected deficiency. Probe checks whether the minimal capability improvement is effective and guides further revision of candidates that have not achieved the target capability. MetaVideoAgent also analyzes video-understanding capability needs, but its modifications are not additive improvements, making effective modification of mature systems challenging.

We have included the additional w/o Probe result in Table 3 in Section 6.3: w/o Probe achieves 75.58% (VideoAgentBoost: 81.19%), further supporting the role of minimal capability verification.

W2 & Q2 — Evolution cost, attempts, and inference cost

We have added the evolution-budget cap of 300M tokens per starting agent in Section 5.2, reported evolution costs and inference tokens and inspected frames before and after evolution in Section 6.4, and provided candidate-attempt statistics in Appendix D.3, with DVD's stage-wise costs in Table 6.

Each of the three starting video agents completes five evolution rounds, yielding 15 admitted capability increments. The estimated total consumption is 650M tokens. Across these runs, 68 Probe attempts were performed: 15 passed, 42 failed and triggered repair, and 11 failed and returned to Diagnosis.

For DVD, the estimated average cost per evolution round is 38.8M tokens, broken down as follows:

E2 (Average tokens per DVD evolution round): Full Evaluation on the evolution set 18.4M, Gold Path construction 9.4M, other evolution stages 5.4M, Probe 5.6M.

Other evolution stages include trajectory comparison in Review, Diagnosis, Research, Design, Codegen, and Smoke. DVD's five rounds involved 20 Probe attempts: 5 passed, 12 failed and triggered repair, and 3 failed and returned to Diagnosis. A Probe costs approximately 1.4M tokens, compared with 18.4M for Full Evaluation. Probe thus avoids expensive full-set evaluation of candidates that have not achieved the intended capability improvement.

We also recorded average inference tokens and inspected frames per question before and after evolution:

E3 (Average inference tokens per question, before → after evolution): VideoAgent 24K → 45K, DVD 111K → 158K, VideoARM 103K → 129K.

E4 (Average inspected frames per question, before → after evolution): VideoAgent 28.59 → 34.26, DVD 86.88 → 52.67, VideoARM 164.37 → 154.38.

As functionality increases, longer system prompts and tool use introduce additional inference overhead, but post-evolution costs do not increase dramatically. The performance gains primarily come from targeted corrections to capability deficiencies, rather than simply allocating more test-time compute.

AuthorsOpenReview ↗

Kind Reminder

We hope this message finds you well. Thank you again for your constructive comments and insightful suggestions.

As the discussion deadline approaches, we would greatly appreciate it if you could take a moment to review our rebuttal and share any remaining questions or concerns. Hearing from you soon would give us more time to provide further clarifications or supplementary evidence and continue the discussion.

If you find that our responses and revisions have addressed your concerns, would you kindly consider raising your rating for our paper?

Thank you very much for your time and consideration. We look forward to hearing from you.

AuthorsOpenReview ↗

Final Revision: Feedback Baseline and Compute Costs

Thank you for your suggestions. We have updated the final version of the paper to include a direct comparison with a simple feedback loop and to report evolution and inference costs.

  • Comparison with a simple feedback loop. In Table 2 of Section 6.2, we report the results of one five-round evolution run per method under the same settings. We evaluated each final agent three times on LVBench and obtained mean accuracies of 74.40% for the simple feedback loop and 81.19% for VideoAgentBoost. We also added the individual results from three independent evolution runs to Figure 3, with mean accuracies of 73.79% and 81.34%, respectively. In Table 3 of Section 6.3, we report an accuracy of 75.58% without Probe.
  • Evolution and inference costs. In Section 5.2, we specify an evolution-budget cap of 300M tokens per starting agent. In Section 6.4, we report the total evolution cost for the three five-round runs, along with the average inference tokens and inspected frames per question before and after evolution. We also report the outcomes of the Probe attempts, including passes and revisions: 68 attempts in total, including 20 for DVD. In Appendix D.3, we provide DVD's stage-wise cost breakdown.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Final Summary of Rebuttal and Revisions (Part 1/2)

Dear Reviewers, ACs, SACs, and PCs,

We sincerely thank all three reviewers for reviewing VideoAgentBoost and offering valuable suggestions. After the rebuttal discussion, two reviewers gave positive evaluations and one reviewer did not participate in the discussion.

Our main contribution is VideoAgentBoost, a method that addresses the difficulty existing evolution methods have in effectively improving agents for long-video understanding. Inspired by boosting idea of progressively addressing residuals, VideoAgentBoost achieves consistent and effective evolution of state-of-the-art video agents in an additive manner. By evolving different video agents and obtaining consistent gains across multiple video-understanding benchmarks, we demonstrate the potential of automated video-agent evolution, inspiring the community to shift from completely open-ended code modifications to orthogonality-aware additive evolution.

Strengths noted by the reviewers

  • Applicability across video agents and transfer of improvements. kQkd and 5Gjy noted that all three starting agents improved across three public benchmarks after evolution with VideoAgentBoost. kQkd pointed out that separating evolution from benchmark evaluation provides evidence that the improvements transfer. 5Gjy also highlighted the method works across different video-agent architectures and the transferability of the resulting gains.
  • Lightweight evolution with traceable improvements. xyUP highlighted that VideoAgentBoost improves video agents using a small evolution set without updating model weights. 5Gjy noted that we use reference solutions and execution trajectories to identify what to improve and then check whether the improvement is effective on a minimal validation set. xyUP also appreciated that we organize the process into interpretable, controllable stages, while kQkd and 5Gjy found the trajectory case studies helpful for understanding how the agents improved.
  • Consistent gains across independent evolution runs. In the discussion, xyUP confirmed that the results of three independent runs addressed their concern about stochasticity in evolution. By using a small set of trajectories to identify where to improve, VideoAgentBoost makes targeted capability additions that improve mature, high-performing video agents more consistently than the open-ended code modifications made by several general-purpose methods. xyUP also noted the inference-budget comparison: at similar token usage per question, the evolved DVD still outperformed the initial DVD with an increased inference budget, indicating that the gains do not come from additional inference compute alone.
AuthorsOpenReview ↗

Final Summary of Rebuttal and Revisions (Part 2/2)

How we addressed the reviewers' concerns

  • Motivation for a video-agent-specific evolution method and comparisons with general-purpose methods (kQkd, xyUP, and 5Gjy). We added LVBench results for several evolution methods starting from the same initial DVD and using the same budget and configuration: initial DVD 74.56%, simple feedback loop 74.40%, ADAS 73.34%, Meta-Harness 75.56%, MetaVideoAgent 78.24%, GA 73.82%, DGM 71.31%, and VideoAgentBoost 81.19% (see our replies to kQkd's W1 & Q1, xyUP's W1 & Q1 & Q3, and 5Gjy's follow-up, item 1, E6). These results show that the additive improvements made by VideoAgentBoost can effectively improve mature video agents. These are precisely the video-understanding challenges that our proposed method aims to solve: (1) the high cost of long-video execution and evaluation limits how many examples can be used to validate candidates in each round, leaving too little evidence to assess the effects of broad code changes; (2) trajectory-based localization and analysis are difficult, as incorrect answers and actual execution logs alone make it hard to determine what capabilities the agent lacks or should improve (see our reply to 5Gjy's W1 & Q1 and item 1 of our follow-up). 5Gjy confirmed that the GA/DGM comparisons addressed their concerns and raised their Soundness score. xyUP also welcomed our additions and raised their rating.
  • Independent evolution runs (xyUP). To address stochasticity in the evolution process, we added a comparison across three complete DVD evolution runs: simple feedback loop 73.79%, ADAS 72.36%, Meta-Harness 73.75%, MetaVideoAgent 77.15%, and VideoAgentBoost 81.34% (see our follow-up to xyUP, “Independent Evolution Results and Updated Comparisons”). The results show that the open-ended code modifications made by several methods did not yield consistent gains. VideoAgentBoost, in contrast, used a small set of trajectories to identify what to improve and achieved consistent gains through targeted capability increments. xyUP acknowledged these results.
  • Evolution and inference costs (kQkd and xyUP). We reported the costs of the evolution stages and estimated that one Probe uses about 1.4M tokens, compared with 18.4M tokens for Full Evaluation on the evolution set. In total, 53 ineffective improvements were rejected before entering evolution-set evaluation, effectively avoiding wasted compute on unnecessary evaluations (see our replies to kQkd's W2 & Q2 and xyUP's W3; for the stage-wise cost breakdown, see our reply to xyUP's W2 & Q2). We also evaluated the initial DVD with an increased inference budget. Its accuracy was still 3.74 percentage points below that of the DVD evolved with VideoAgentBoost, supporting the conclusion that the gains do not come from increased inference compute alone (see our follow-up to xyUP, item 3, E5). xyUP welcomed this comparison.
  • Research motivation and experimental details (5Gjy and xyUP). We clarified the paper's goal: to use execution feedback to automatically and effectively improve mature, manually designed video agents (see our reply to 5Gjy's Q3). The additional experiments show that, for the high-cost and hard-to-validate characteristics of long-video understanding, VideoAgentBoost can improve video agents more effectively than general-purpose evolution methods. We also described how we select Probe questions and admit capability increments, and what we investigated during Research (see our reply to 5Gjy's W2 & Q2 and item 2 of our follow-up, and our reply to xyUP's Q4, respectively). 5Gjy noted that the revised motivation substantially improved the paper and confirmed that the Probe clarification addressed their concerns.
  • Concerns from Reviewer kQkd. kQkd has not yet responded to our rebuttal. Their questions primarily concern comparisons with other evolution methods (see our reply to kQkd's W1 & Q1) and evolution costs (see our reply to kQkd's W2 & Q2). The new results above directly address both points. We hope the reviewer will have an opportunity to continue the discussion at a later stage.

We thank all the reviewers and chairs again for their time and feedback. Their questions helped us improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.