acceptodds
Withdrawn from ICLR 2027

EvoVE: Evolving Video Evaluation Through A Self-Improving Harness

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Modern video generation models are improving at a rapid rate, and they are quickly spreading into domains such as film production, advertising, education, and robotics simulation. Each new generation brings new capabilities and, with them, new failure modes: Seedance 2.5 accepts new audio references and emits longer clips than Seedance 2.0, and its most consequential errors (e.g., Audio-Visual Speech Misalignment) simply did not exist for its predecessor. Benchmarks, however, do not move with the models. Therefore, we introduce EvoVE (Evolving Video Evaluation), a model-agnostic, recursively self-improving evaluation harness for AI-generated video and multimodal understanding. EvoVE jointly evolves an evaluation plan, a trace-analysis textbook that serves as memory, and a dynamic tool bank. On error taxonomy benchmark, EvoVE achieves a macro-F1 of 88% at 5-shot, compared to the 67% VLM baseline. We formulate harness evolution as meta-learning, enabling harness self-improving. With only 5-shot target examples, EvoVE self-improves by 11.9% F1 score in-distribution and 7.8% F1 out-of-distribution. Finally, with only a task-adapter change, EvoVE generalizes from video evaluation to image benchmarks, achieving 82%–93% held-out accuracy on TextVQA, MMBench, and MMMU.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Recursive self-improvement, self-evolving agents, agentic evaluation harness, video generation evaluation, meta-learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ynTbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces EvoVE, a self-improving harness for video evaluation. It keeps the judge frozen and jointly evolves the evaluation plan, error-analysis textbook, tool bank, and global context. Labeled execution traces guide the analyst in identifying judgment errors and missing evidence. A tool factory acquires and verifies tools, and a planner updates the evaluation plans and decision rules. The authors also study adaptation to unseen error categories using a small labeled support set, with the tool bank frozen. In the development comparison across five source categories, EvoVE reports a macro F1 of 88.6% at 5-shot.

Critical strengths
  1. The research question is important. New error categories emerge as video-generation models and application requirements change. Most existing video benchmarks fix their evaluation dimensions and procedures at release, making it difficult to cover newly specified defects directly. EvoVE addresses this practical problem by updating the evaluation procedure from labeled feedback while keeping the judge frozen.

  2. The method design is reasonable. Its trace analysis, tool acquisition, and updates to the textbook and evaluation plans offer a concrete approach to adapting evaluators to new error categories.

Critical weaknesses
  1. The technical contribution of coordinated evolution has not been fully validated. The authors explicitly identify coordination as a distinguishing feature and describe the information flow between memory, tools, and plans in the algorithm. However, the main component ablations in Table 3 remove components after evolution without reoptimizing the remaining parts. As the authors themselves note, these results show that the harness “depends on all three.” They do not answer how EvoVE would compare with optimizing memory, tools, and plans separately using the same initial failure traces and then combining them. The current experiments do not distinguish the gains from optimizing more components from those due to coordination. The baselines focus on updating one or several components, while EvoVE evolves memory, tools, plans, and global instructions. Its performance advantage may therefore come from optimizing more components and does not, by itself, establish the novelty of the proposed method.

  2. The experiments do not fully establish the reported performance advantage and adaptation gains. Source-category evaluation rows in Table 2 guided several methods during development, but exposure was not quantified separately by method. The ranking therefore still needs confirmation on the same set of untouched source-category queries. Table 4 changes two factors at once: 0-shot provides no target support, while 5-shot supplies ten labeled target examples and updates the textbook and evaluation plans. The judge may improve simply from having these support examples in context, so the reported gains do not separate the contribution of in-context support from that of harness updates. The appendix includes related controls and case studies, but it does not report matching controls for each setting in Table 4.

Questions for authors
  1. The paper states that the evaluation rows in Tables 2, 3, and 10 guided some methods during development. At which stages did each method use these rows and their labels, and why does the main comparison report only development results? Are there results on held-out queries from the same categories that were not used for harness updates or checkpoint selection?

  2. The benchmark contains 27 error categories across seven error families, but the main comparison in Table 2 covers only five source categories from three families. Why were these five selected, and what criteria were used? Was the selection made before method development and comparison?

  3. Why are the 10-shot result for MCE and the 5-shot and 10-shot results for Meta-Harness not reported in Table 2?

  4. Table 2 lists Meta-Harness’s 74.0% under the 0-shot setting, while Table 10 reports the same score in the 5-shot setting. Is this merely a coincidence? What is the underlying reason?

  5. The related-work section identifies coordination as a distinguishing feature, but Table 3 mainly measures dependence on components removed from the final harness. Could the authors compare EvoVE with separate optimization followed by combination, or with restricted feedback between components, while matching the editable components, data access, and compute budget?

  6. In Table 4, moving from 0-shot to 5-shot introduces ten labeled target examples and updates the harness. While the appendix includes relevant controls, it does not have results matching all six configurations in Table 4. Could the authors evaluate the learned harness with the same support examples in context but no harness updates, and target adaptation starting from the seed harness? These controls would help separate the contribution of harness updates from that of source-category learning.

  7. Table 10 defines Macro as the mean F1 across the five categories. However, the EvoVE scores of 0.781, 0.800, 0.966, 0.963, and 0.960 average to 0.894, rather than the 0.886 reported in Tables 10 and 2. Which needs correction: the per-category scores, the macro F1, or the description of the experimental configuration?

Reviewer_Zs2GOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Video generation can fail when matching modalities, wrong pronunciation, lips not matching what's being said, etc. Fixed benchmarks cannot cover new error types when model adds new function and they cannot evaluate and measure these kind of failures. In "EvoVE: Evolving Video Evaluation Through A Self-Improving Harness", the authors proposed harness for multimodal judge for detecting defects in AI generated video. Agents improve harness: its evaluation plan, error analysis memory and tool bank. It is evaluated on error categories, such as pronunciation, speech, face reference... In addition, harness is built on some categories and then adapted on unseen categories on 5 defective and 5 not defective examples, which authors claim was inspired by meta-learning.

They report an improvement on F1 score on their benchmark and limited improvement on TextVQA, MMBench and MMMU. New benchmark that is in scope of the paper is having 1886 videos across 27 categories. They themselves pointed out that most comparisons are on development, not held out data.

Critical strengths

The topic of this paper is a relevant and important problem that the authors are trying to solve. The narrative is transparent, with the authors pointing out some limitations.

Critical weaknesses

The choice of evaluation metric makes the reported improvements difficult to interpret. The primary evaluation metric is the defective-class F1 score. However, for a balanced binary classification dataset, a trivial classifier that always predicts the defective class achieves an F1 score of approximately 66.6%. Here, the direct VLM judge achieves F1 scores in the range of 65.8% and 67.9% in Tab. 2, which are comparable to this trivial baseline. This raises concerns about the informativeness of the reported metric and the actual discriminative ability of the baseline.
Although F1 is useful for evaluating defect detection, reporting it alone provides an incomplete picture, particularly when false positives and false negatives may have different practical implications.
I therefore believe that the main experimental tables should additionally report accuracy, precision, and recall, alongside an always-defective baseline, to enable a more meaningful assessment of the proposed framework.

The strongest experimental results are obtained on development data rather than held-out test sets. The most substantial improvements reported in the paper, particularly those presented in Tab. 2 and 3, are measured on data used during the optimization process. Since these data influence both harness evolution and checkpoint selection, the reported improvements cannot be interpreted as evidence of generalization to unseen examples. This is a significant limitation, given that these results constitute a central part of the paper's contribution.
In contrast, the experiments conducted on held-out data provide less conclusive evidence. Table 4 evaluates adaptation to unseen error categories but does not include a direct-judge baseline under the same few-shot setting, making it difficult to isolate the benefit of harness adaptation. Similarly, the cross-generator experiments in Table 11 lack comparable baselines, while Table 5 reports relatively modest and inconsistent improvements on established multimodal benchmarks.

Overall, I find that the current experimental evidence is insufficient to establish the claimed advantages of the proposed approach on genuinely unseen data.

The benefits of harness adaptation are not clearly disentangled from those of in context learning. In the few-shot experiments, the paper uses labeled examples both to adapt the harness and to provide in-context demonstrations to the judge. Consequently the reported improvements may result partly from conventional in-context learning rather than from the proposed self-improving mechanism. Without a direct comparison against a frozen judge receiving exactly the same few-shot examples, it is difficult to understand how much of the observed improvement can be attributed specifically to harness adaptation.

Questions for authors

Additional evaluation metrics. Could the authors report accuracy, precision, and recall for the main comparison in Table 2, ideally on a separate validation set? Including an always-defective baseline would also help contextualize the reported F1 scores.

Few-shot direct-judge baseline. For the unseen-category experiments in Table 4, how does the direct multimodal judge perform when provided with the same five defective and five non-defective examples per category, without harness adaptation? This comparison would help in clarifying if the improvements come from the proposed framework or from the additional labeled examples.

Disentangling harness adaptation from in-context learning. Since the few-shot setting involves both updating the harness and providing labeled examples in context, could the authors include an ablation that isolates these two contributions?

This paper was very far away from my domain of expertise. If the authors manage to bring enough evidence in support of their method, I will consider changing score.

Reviewer_3KVKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Evolving Video Evaluation (EvoVE), a model-agnostic self-improving evaluation method compared to static existing video generation benchmarks. EvoVE keeps a multimodal judge (VLM) frozen and jointly optimizes the system by co-evolving four distinct components: a task-specific evaluation skill, a trace-analysis textbook, a dynamic tool bank, and global instructions. Evaluated across 27 error categories, EvoVE recursively adapts to unseen error families well, and also generalizes well to standard visual question answering benchmarks.

Critical strengths
  1. The EvoVE framework coordinates the acquisition of specialized tools with updated planning and memory, providing a co-optimized system-level solution. It ability to adapt to unseen error categories using a 5-shot support set is also good.
  2. The inclusion of Harbor's ATIF to log and audit 5,093 verdicts and 13,828 probes with an adversarial verifier is good for diagnostics.
Critical weaknesses
  1. Running the fully evolved harness requires an average of 2.1 VLM judge calls and 274 seconds per query (Table 7), which severely limits its feasibility for high-throughput or real-time video evaluation.
  2. The optimization loop relies on a complex, expensive orchestrator utilizing multiple Claude 5.5 agents making local or resource-constrained execution highly impractical.
  3. Designing complex, multi-clause evaluation plans and custom code tools based on only a 5-shot support set poses a high risk of overfitting specific artifacts or narrow edge-cases.
  4. Key quantitative results (like Table 2, Table 3, and the trajectories in figure 2) are reported from single runs without standard deviation boundaries, paired confidence intervals, or statistical significance tests.
Questions for authors
  1. Can the 2.1 VLM judge calls and tool-execution steps be executed in parallel during inference to reduce the 274-second per-query latency bottleneck?
  2. Have you tested replacing the Claude 5.5 agents in the optimization loop with SOTA open-weight models (like Qwen-2.5-72B)? What is the resulting drop in tool-writing and trace-analysis success?
  3. As the tool bank grows across many categories, how does the system prevent tool redundancy or conflicts where a tool admitted for one task generates misleading evidence for another?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.