- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents VideoEvoHarness, a self-evolving agent framework for video understanding that improves the agent harness using unlabeled videos without updating the underlying model parameters. The framework jointly evolves a Questioner and a Solver. The Questioner generates questions and reference answers from structured video memory and refines its generation policy based on question quality and Solver feedback. The Solver evolves its execution harness, including textual skills, executable strategies, hyperparameters, and tools, through failure analysis and validation-based selection. Experiments are conducted on LongVideoBench, MMVU, and Video-Holmes using two frozen backbones, Qwen3.7 Plus and Gemini 3.8 Flash. The results demonstrate improvements over the initial harness, while additional ablations investigate the effects of question filtering, candidate validation, and harness component evolution.
Critical strengths
- The research topic is interesting. The paper explores a meaningful research direction by enabling video agents to evolve their execution harnesses directly from unlabeled videos without updating model parameters. The joint evolution of the Questioner and Solver provides an interesting perspective on how automatically generated tasks and execution feedback can support iterative agent improvement.
- A reasonably comprehensive framework with useful ablation studies. The framework integrates question generation, quality filtering, failure diagnosis, and harness validation into a unified evolution pipeline. The experiments cover multiple video understanding benchmarks and two different model backbones. The ablation studies further demonstrate the importance of question filtering and candidate validation, supporting the effectiveness of these components.
Critical weaknesses
- Limited and inconsistent performance improvements. Although VideoEvoHarness improves overall accuracy, the gains are relatively modest in some settings. For instance, Gemini 3.8 Flash achieves only 0.6, 1.5, and 0.5 percentage-point improvements on LongVideoBench, MMVU, and Video-Holmes, respectively. Moreover, the improvements are not always consistent across evolution rounds, and reasoning accuracy drops from 72.2% to 67.8% in Table 5. These results raise concerns about the stability and general effectiveness of the evolution process.
- Limited validation of comprehensive harness evolution. Although the framework supports evolving skills, strategies, hyperparameters, and tools, the actual updates are limited to strategies and hyperparameters, as stated in Section 4.4 and Appendix D. Therefore, the experiments do not fully demonstrate the effectiveness of evolving all four components. Additional comparisons with simpler alternatives, such as hyperparameter optimization or manually designed strategies, would further strengthen the contribution.
Questions for authors
- Regarding the relatively modest performance gains: The improvements on several benchmarks, particularly with Gemini 3.8 Flash, are relatively small. Could the authors explain why the proposed evolution framework provides limited gains with stronger backbones? Are the improvements consistent across different evolution runs or random seeds? Reporting variance or confidence intervals would help assess their reliability.
- Regarding performance degradation during evolution: Table 5 shows a noticeable decrease in reasoning accuracy after evolution, and some benchmark results decline between R1 and R3. What causes these regressions? Could the proposed validation mechanism be extended to preserve previously acquired capabilities rather than optimizing only overall accuracy?
- Regarding the effectiveness of different harness components: Since the retained updates involve only strategies and hyperparameters, could the authors provide further evidence of the effectiveness of skill and tool evolution? Additionally, how does the proposed approach compare with conventional hyperparameter optimization or manually designed retrieval strategies under comparable computational budgets?