- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper proposes VideoAgentBoost. This is a system helps to recursively improve video agents (DVD, VideoARM and VideoAgent). Based on a training set of 139 questions and 12 videos, qwen3.8-max is used to generate gold answer. Then the main video agent (qwen3.7-max for thinking, qwen3.7-plus for vision) rollout some trajectories. Codex with gpt-5.6-sol is used to review the trajectories and diagnosize shortcomings in five semantic modules (structure, localization, perception, memory, and thinking). Then codex adds new tools or edits prompt for the main video agents, with codex auditing these changes. Experiments are conducted on LVBench, LongVideoBench and Video-MME, which shows improvement.
Critical strengths
- Lightweight in-context learning. The paper only uses 139 questions and 12 videos as training set for in-context learning, while not training any weights. This methodology can help quickly transfer and generalize to other application and deployment environments.
- Good interpretability. This paper dedicatedly designs a road map to improve video agents. Each stages is interpretable and controllable. Humans can study this self-improvement process comfortably.
Critical weaknesses
- This paper doesn't compare its meta evolving system with other evolution method. Table 1 only compares the proposed method with other end-to-end video models (raw model without complicated harness) and raw video agents themselves, but doesn't compare with codex + gpt-5.6-sol and other sota (e.g. MetaVideoAgent in this paper's reference).
- No budget is disclosed for experiments. So it's not sure whether this improvement comes from the dedicatedly designed system or simply test-time scaling.
- For long-horizon agentic tasks, like trajectory reviewing and auditing of codex, the code evolution is very stochastic. But the main and ablation experiments are all conducted only once. It's not sure whether this improvement is stable, or it's luck, or simply cherry picking.
Questions for authors
- Can you run experiments of codex itself as video agent, with the same token budget as the addition of your VideoAgentBoost system's cost and video agent's cost. Although gpt doesn't support video directly, it can view images, and it can use python, ffmpeg or other tools to help analyze videos. Just the raw codex agent is ok. You may run gpt-6.1-sol, it's better and cheaper than 5.6 sol.
- Can you report the budget of each experiment.
- Can you compare your evolving method with related works like MetaVideoAgent?
- When using the codex for reviewing or improving, can the codex agent access the websearch tool to get ideas like LVBench/Video-MME leaderboard tricks, or other works optimized on these benchmarks?
6 comments
Thank you for the detailed response and the revised paper. The new same-budget comparison with a simple feedback loop, ADAS, Meta-Harness and MetaVideoAgent (Table 2), the cost accounting (Sec. 6.4, App. D.3), and the w/o Probe ablation address my main concerns, and I will raise my score. Remaining points: (1) According to App. D.2, correct Codex answers are discarded when the trajectory lacks evidence from the supporting intervals, but the same audit does not seem to be applied to VideoAgentBoost. Please report Codex's raw accuracy, or apply the same audit to both. (2) The repeated runs re-evaluate a single evolved agent. Independent evolution runs (at least for DVD, with std, and ideally for the baselines) are still needed to assess the stochasticity of the evolution process; please at least acknowledge this in the Limitations. (3) Inference tokens increase by 25–88%, so a compute-matched initial-agent baseline would further strengthen the claim.
Clarifications on Codex Evaluation, Budget Matching, and Evolution Reproducibility
Thank you for acknowledging our additional experiments and manuscript revisions, and for raising these specific follow-up questions.
1 With a reasoning budget cap of 1.5 times that of the evolved DVD, raw Codex achieves an accuracy of 77.53%. In our previous response and manuscript revisions, we did not clearly explain the cases in which execution trajectories lacked evidence from the supporting evidence intervals. When inspecting the answer trajectories, we repeatedly observed that Codex could produce correct answers without examining any information within the questions' supporting evidence intervals. We therefore suspect that Codex may have been trained on these benchmarks and memorized relevant information, and excluded these correct responses accordingly. However, we would like to emphasize that, in the evolution experiments involving the three video agents in our paper, Codex was not informed that the final evolved model would be evaluated on the benchmarks, and the benchmarks were never exposed to the evolution process.
3 We increased the budget caps of the initial DVD and evaluated this budget-matched initial agent. The main adjustments were as follows:
E5 (adjustments to the initial DVD settings): reasoning steps 15 → 20, frame inspect 50 → 100, clip search 16 → 32, global browse 300 → 500.
Across three runs with the expanded budget, the average results were 123.32 observed frames per question, 157K tokens per question, and 77.45% accuracy.
2 Regarding independent repetitions of the evolution process, we will provide results from rerunning the complete evolution pipeline as soon as possible over the next few days and revise the corresponding content in the paper.
Additional Comparisons, Cost Reporting, and Evaluation Details
Thank you for your feedback on the experimental comparisons, computational costs, and evaluation.
W1 & Q1 & Q3 — Comparisons with evolution methods and direct Codex evaluation
We have added an experiment in which Codex with GPT-5.6 Sol directly answers LVBench questions, with an inference budget capped at 1.5 times that of the DVD agent evolved by VideoAgentBoost. Its accuracy after evidence auditing of the answering trajectories is 64.75% (VideoAgentBoost: 81.19%).
We further compare evolution methods using the same initial DVD, evolution set, model configuration, and evolution budget. After five evolution rounds, each final agent is evaluated on LVBench three times:
E1 (DVD evolution results under the same configuration): Initial DVD 74.56%, simple feedback loop 74.40%, ADAS 73.34%, Meta-Harness 75.56%, MetaVideoAgent 78.24%, VideoAgentBoost 81.19%.
Code optimization for all methods uses Codex with GPT-5.6 Sol. We observed that simple feedback loops and general-purpose evolution methods lack in-depth analysis of video-evidence acquisition and use, making it difficult to attribute a mature agent's failures to specific missing capabilities. Their modifications also tend to cover a broad scope. Repeatedly screening such changes through full-set evaluation incurs substantial overhead in long-video tasks.
MetaVideoAgent can analyze video-understanding capability needs, but its modifications do not follow residual-guided additive correction, making effective improvement of mature systems challenging. VideoAgentBoost instead compares Gold Paths with student trajectories to identify actual capability deficiencies, develops targeted capability increments, and uses Probe to check whether the improvements take effect. The results demonstrate the value of this design for automatically improving mature video agents.
W2 & Q2 — Evolution and inference costs
Each starting agent has an evolution-budget cap of 300M tokens and undergoes five evolution rounds. The estimated total consumption across the three experiments is approximately 650M tokens. DVD consumes an estimated average of 38.8M tokens per round, broken down below:
E2 (Average tokens per DVD evolution round): Full Evaluation on the evolution set 18.4M, Gold Path construction 9.4M, other evolution stages 5.4M, Probe 5.6M.
Other evolution stages include trajectory comparison in Review, Diagnosis, Research, Design, Codegen, and Smoke.
We additionally report average inference tokens and inspected frames per question before and after evolution:
E3 (Average inference tokens per question, before → after evolution): VideoAgent 24K → 45K, DVD 111K → 158K, VideoARM 103K → 129K.
E4 (Average inspected frames per question, before → after evolution): VideoAgent 28.59 → 34.26, DVD 86.88 → 52.67, VideoARM 164.37 → 154.38.
Evolution adds some prompt and tool-use overhead, but overall inference costs do not grow dramatically. Together with the comparisons under the same evolution budget, these results support targeted capability improvement as the source of the gains, rather than simply increasing test-time compute.
W3 — Repeated evaluation and result selection
We evaluated each of the three final evolved agents three times. The results consistently retain the improvement trend, with mean accuracies reported below:
VideoAgentBoost uses Probe to ground improvement targets and admission decisions in execution evidence. Across the three video-agent evolution experiments, 15 capability increments were admitted through 68 Probe attempts: 15 passed, 42 failed and triggered repair, and 11 failed and returned to Diagnosis.
Once Probe verifies the target capability, the verified capability is admitted; if it does not pass verification, the process returns to Diagnosis within the same round to determine a revised improvement direction. This process grounds each round's admission in execution evidence of the targeted capability improvement, thereby achieving stable gains.
Q4 — Web access and evaluation separation
We audited all Research search records. They concern reusable mechanisms such as coarse-to-fine temporal localization, conflicting-evidence verification, timestamped ASR/OCR, failure recovery, and cross-segment comparison. We found no leaderboard tricks or test-question answers. We have added the specific Research content for the evolution of all three agents in Appendix A.2. The evolution procedure was not informed of the eventual benchmark evaluation, and test questions, answers, and evaluation results did not participate in evolution.