acceptodds
Withdrawn from ICLR 2027

VideoEvoHarness: Self Evolving Agent Harness from Unlabeled Videos for Video Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video understanding agents rely on a harness to acquire evidence, use tools, and reason, but improving it requires substantial effort. Recent self evolution methods seek to optimize harnesses through execution feedback to reduce human intervention. However, existing methods often rely on annotated video questions and answers or focus on updating model parameters, leaving harness evolution from unlabeled videos underexplored. VideoEvoHarness jointly evolves a Questioner and a Solver harness from unlabeled videos with frozen model parameters. The Questioner constructs questions and computes reference answers directly from video memory records. Question quality assessments and Solver execution trajectories guide question generation policy revisions. The Solver supports evolving skills, strategies, hyperparameters, and tools using execution feedback. Executable strategies define workflows in code, reducing repeated interpretation of natural language instructions. Each round, candidate harnesses are compared on identical validation questions, with validated improvements retained for later use. Experiments on benchmarks and a dedicated benchmark question set, using videos disjoint from evolution videos, show higher overall accuracy than initially and transfer gains across backbones. Overall accuracy on this set also improves progressively across rounds. This supports our foundational framework for harness evolution from unlabeled videos, opening a promising direction for self evolving video agents.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video understanding, video agent, agent harness, self-evolution, unlabeled video

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_Z3PnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents VideoEvoHarness, a self-evolving agent framework for video understanding that improves the agent harness using unlabeled videos without updating the underlying model parameters. The framework jointly evolves a Questioner and a Solver. The Questioner generates questions and reference answers from structured video memory and refines its generation policy based on question quality and Solver feedback. The Solver evolves its execution harness, including textual skills, executable strategies, hyperparameters, and tools, through failure analysis and validation-based selection. Experiments are conducted on LongVideoBench, MMVU, and Video-Holmes using two frozen backbones, Qwen3.7 Plus and Gemini 3.8 Flash. The results demonstrate improvements over the initial harness, while additional ablations investigate the effects of question filtering, candidate validation, and harness component evolution.

Critical strengths
  1. The research topic is interesting. The paper explores a meaningful research direction by enabling video agents to evolve their execution harnesses directly from unlabeled videos without updating model parameters. The joint evolution of the Questioner and Solver provides an interesting perspective on how automatically generated tasks and execution feedback can support iterative agent improvement.
  2. A reasonably comprehensive framework with useful ablation studies. The framework integrates question generation, quality filtering, failure diagnosis, and harness validation into a unified evolution pipeline. The experiments cover multiple video understanding benchmarks and two different model backbones. The ablation studies further demonstrate the importance of question filtering and candidate validation, supporting the effectiveness of these components.
Critical weaknesses
  1. Limited and inconsistent performance improvements. Although VideoEvoHarness improves overall accuracy, the gains are relatively modest in some settings. For instance, Gemini 3.8 Flash achieves only 0.6, 1.5, and 0.5 percentage-point improvements on LongVideoBench, MMVU, and Video-Holmes, respectively. Moreover, the improvements are not always consistent across evolution rounds, and reasoning accuracy drops from 72.2% to 67.8% in Table 5. These results raise concerns about the stability and general effectiveness of the evolution process.
  2. Limited validation of comprehensive harness evolution. Although the framework supports evolving skills, strategies, hyperparameters, and tools, the actual updates are limited to strategies and hyperparameters, as stated in Section 4.4 and Appendix D. Therefore, the experiments do not fully demonstrate the effectiveness of evolving all four components. Additional comparisons with simpler alternatives, such as hyperparameter optimization or manually designed strategies, would further strengthen the contribution.
Questions for authors
  1. Regarding the relatively modest performance gains: The improvements on several benchmarks, particularly with Gemini 3.8 Flash, are relatively small. Could the authors explain why the proposed evolution framework provides limited gains with stronger backbones? Are the improvements consistent across different evolution runs or random seeds? Reporting variance or confidence intervals would help assess their reliability.
  2. Regarding performance degradation during evolution: Table 5 shows a noticeable decrease in reasoning accuracy after evolution, and some benchmark results decline between R1 and R3. What causes these regressions? Could the proposed validation mechanism be extended to preserve previously acquired capabilities rather than optimizing only overall accuracy?
  3. Regarding the effectiveness of different harness components: Since the retained updates involve only strategies and hyperparameters, could the authors provide further evidence of the effectiveness of skill and tool evolution? Additionally, how does the proposed approach compare with conventional hyperparameter optimization or manually designed retrieval strategies under comparable computational budgets?
Reviewer_iuPmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VideoEvoHarness constructs structured memories from unlabeled videos while freezing model parameters. The Questioner generates questions and reference answers, which are then used to modify the Solver’s harness based on execution feedback. Candidate modifications are compared on the same set of generated validation questions, and those that pass are retained. Its main contribution lies in combining the construction of unlabeled tasks with the evolution of video agent harnesses.

Critical strengths
  1. The setting is well chosen and the pipeline is carefully guarded. Evolving a frozen video agent's harness from unlabeled videos fills a real gap between MetaVideoAgent, which needs labeled QA, and Video-Zero/EvoVid, which update weights. Blind/Oracle filtering, paired validation and disjoint evaluation videos are sensible safeguards, and the no-validation ablation is convincing. This makes the framework a credible foundation.

  2. I like that the evolved artifacts are readable and that the paper is candid about failures. Executable strategy code and config diffs (Figs. 8–9) let a reader see exactly what changed, and the authors openly report the drop in reasoning accuracy and that skill and tool updates were never accepted. That honesty raises my trust in the numbers, though it also weakens some headline claims.

Critical weaknesses
  1. My main concern is that the improvement is modest and unstable. The gains are only 0.5–2.8 points on public benchmarks from what seems to be a single run, and several results get worse from R1 to R3, sometimes below the initial harness, with little analysis of why. With only two closed API backbones and no reported inference cost, I'd want a few more runs, an open-weight backbone and simple baselines before trusting the gains.

  2. I'm also not convinced the label-free supervision is trustworthy. Reference answers come from model-built memory, and the Oracle probe only checks them against that same memory, so nothing confirms they're true of the video. With Solver accuracy near chance in Fig. 4, label noise explains the decline as well as "harder questions" does, and a small human audit of generated questions would go a long way here.

Questions for authors
  1. The results seem to come from a single run. Could you repeat the evolution a few times and show how much the R3 gains vary?
  2. Both backbones are closed API models, and the gains are smaller on the stronger one (+0.5 to +1.5 for Gemini vs. +1.5 to +2.8 for Qwen). Could you add at least one open-weight backbone, ideally a weaker one, and test transfer in both directions? Since the stronger backbone gains less, I wonder whether the benefit of harness evolution shrinks as the base model improves.
  3. Which updates were actually accepted, and which evidence path does each one affect? MMVU and Video-Holmes videos are mostly short and go through the sampling path, while the changes shown in Figs. 8–9 only apply to the memory path for videos of 30 minutes or more, so I'm not sure what explains the +2.4 gain on MMVU (+5.8 on Healthcare). A breakdown by video duration would help.
  4. How many of the generated reference answers are actually correct and unambiguous? Could you manually check a random sample of questions from each round against the video? This would help show whether the generated questions can be trusted as a supervision signal.
  5. Could you run an ablation where the Questioner policy stays fixed at its initial version, while the Blind and Oracle filtering is still applied? This would show how much the evolving Questioner itself contributes to downstream accuracy.
  6. How much does evolution cost in model calls, tokens and wall-clock time, and how does the per-question inference cost (frames, calls, tokens) of the R3 harness compare with the initial one on each benchmark?
  7. Some results get worse from R1 to R3, in some cases falling below the initial harness. Could you add some analysis of why this happens?
Reviewer_tHUcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tries to improve a video question answering agent without touching the model's weights: what it changes is the layer around the model, the prompts, the code that decides how frames are sampled and evidence is retrieved, the sampling settings, and the tool library, and the only input is unlabelled video. A question generator builds a structured memory of each video, writes multiple-choice questions from that memory, and computes their answers from the records rather than from a model's guess. A solver answers those questions; its failures are diagnosed, candidate changes to the surrounding layer are proposed, and a change is kept only if it improves accuracy on a shared validation set. Experiments run three rounds and evaluate on three video benchmarks with two frozen backbones.

Critical strengths
  • The framing is useful and not much explored: improving the layer around the model rather than the model itself, starting from raw video, with parameters frozen throughout.
  • The quality of the self-generated questions is measured rather than assumed, questions answerable without the video, or unanswerable even with the supporting memory, are removed before training, which is the most transferable idea in the paper.
  • The cross-backbone result is the most interesting part: a setup evolved with one model still helps a different one (66.0 to 67.8 overall), which suggests it captures something beyond one model's habits.
Critical weaknesses
  • There is no comparison with any published method. The only reference point is the authors' own starting setup, so I cannot tell whether this design is better than the alternatives the paper itself discusses.
  • I cannot tell whether the gain comes from a better strategy or from simply gathering more evidence. The settings that are allowed to change include sampling density, the replay window and the breadth of retrieval, yet the paper never reports how many frames or tokens each question costs, so the two explanations cannot be separated.
  • The evidence is thinner than the claims. Over three rounds only three changes were kept, two of the four advertised components (skills and tools) never changed at all, and one capability clearly drops, reasoning from 72.2 to 67.8, while the acceptance rule only looks at overall accuracy.
Questions for authors

See the weaknesses above.

Reviewer_5aJ6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VideoEvoHarness, a framework that jointly updates a Questioner and a video-understanding Solver harness while keeping model parameters frozen. The Questioner generates questions and reference answers from structured video memory. Solver execution traces guide candidate changes, which are retained based on validation accuracy. Although the framework supports updates to skills, strategy code, hyperparameters, and tools, the reported experiments retain only strategy and hyperparameter changes.

Critical strengths
  1. Constructing video-grounded evaluation tasks without human-authored question–answer annotations is a useful practical direction.

  2. Comparing candidate harnesses on the same validation questions is a sensible design. The paper also explicitly acknowledges that the retained updates do not cover all four proposed component types.

Critical weaknesses
  1. The methodological novelty is limited and insufficiently distinguished from existing prompt and agent-program optimization.

    The central procedure is straightforward: generate tasks, inspect failures, ask an LLM to propose modifications, evaluate candidates, and retain improvements. This follows the familiar feedback-driven search pattern underlying automated prompt design and agent-program optimization. The paper does cite relevant work, including GEPA, Promptbreeder, DSPy, and Automated Design of Agentic Systems. However, it does not clearly establish a new optimization mechanism beyond combining these ideas with memory-derived video questions.

    The unlabeled-video setting is a potentially useful application, but the paper needs to explain what methodological contribution makes this more than an incremental adaptation of existing approaches.

  2. The demonstrated evolution is technically naive relative to the breadth of the framework.

    The concrete strategy example in Appendix C.3 introduces keyword-conditioned OCR retrieval, event retrieval, and a fallback procedure. The configuration example enables the generated strategy and reduces the replay-window cap from 60 to 15 seconds. These are elementary retrieval heuristics and configuration adjustments.

    Furthermore, Table 5 explicitly states that skills and tools remain unchanged throughout the three-round experiment. Thus, the empirical evidence supports a much narrower contribution than the proposed four-component evolution framework. Encoding a workflow in executable code is useful engineering, but does not by itself establish a novel self-evolution method. More substantial examples are needed to demonstrate that the system discovers strategies beyond routine prompt and pipeline design.

  3. The experiments do not establish an advantage over existing optimization methods.

    The primary comparison is against the initial harness. This establishes that some modifications help, but does not show that the proposed approach is better than existing prompt optimization, agent-code search, or straightforward hyperparameter tuning under comparable budgets. Despite discussing closely related methods, the paper does not provide such comparisons.

    There is also no controlled comparison between a fixed Questioner and an evolving Questioner under the same Solver optimization procedure. Consequently, the specific benefit of coevolution remains unclear.

    The reported gains are modest, and the main result tables provide no uncertainty estimates. On the dedicated analysis set, reasoning accuracy decreases from 72.2% to 67.8% despite an increase in overall accuracy. The evidence therefore supports limited improvements in evidence acquisition, rather than a convincing demonstration of broadly improving video-agent capabilities.

Questions for authors
  1. What specific algorithmic contribution distinguishes this framework from applying existing prompt or agent-program optimization to automatically generated video questions?

  2. Can the authors provide budget-matched comparisons against established optimization methods and a simple hyperparameter-search baseline?

  3. Can the authors isolate the contribution of Questioner evolution by comparing fixed and evolving Questioners under otherwise identical conditions?

  4. Can the authors provide examples of retained updates beyond keyword-based retrieval and configuration changes, particularly successful skill or tool evolution?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.