- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces EvoVE, a self-improving harness for video evaluation. It keeps the judge frozen and jointly evolves the evaluation plan, error-analysis textbook, tool bank, and global context. Labeled execution traces guide the analyst in identifying judgment errors and missing evidence. A tool factory acquires and verifies tools, and a planner updates the evaluation plans and decision rules. The authors also study adaptation to unseen error categories using a small labeled support set, with the tool bank frozen. In the development comparison across five source categories, EvoVE reports a macro F1 of 88.6% at 5-shot.
Critical strengths
-
The research question is important. New error categories emerge as video-generation models and application requirements change. Most existing video benchmarks fix their evaluation dimensions and procedures at release, making it difficult to cover newly specified defects directly. EvoVE addresses this practical problem by updating the evaluation procedure from labeled feedback while keeping the judge frozen.
-
The method design is reasonable. Its trace analysis, tool acquisition, and updates to the textbook and evaluation plans offer a concrete approach to adapting evaluators to new error categories.
Critical weaknesses
-
The technical contribution of coordinated evolution has not been fully validated. The authors explicitly identify coordination as a distinguishing feature and describe the information flow between memory, tools, and plans in the algorithm. However, the main component ablations in Table 3 remove components after evolution without reoptimizing the remaining parts. As the authors themselves note, these results show that the harness “depends on all three.” They do not answer how EvoVE would compare with optimizing memory, tools, and plans separately using the same initial failure traces and then combining them. The current experiments do not distinguish the gains from optimizing more components from those due to coordination. The baselines focus on updating one or several components, while EvoVE evolves memory, tools, plans, and global instructions. Its performance advantage may therefore come from optimizing more components and does not, by itself, establish the novelty of the proposed method.
-
The experiments do not fully establish the reported performance advantage and adaptation gains. Source-category evaluation rows in Table 2 guided several methods during development, but exposure was not quantified separately by method. The ranking therefore still needs confirmation on the same set of untouched source-category queries. Table 4 changes two factors at once: 0-shot provides no target support, while 5-shot supplies ten labeled target examples and updates the textbook and evaluation plans. The judge may improve simply from having these support examples in context, so the reported gains do not separate the contribution of in-context support from that of harness updates. The appendix includes related controls and case studies, but it does not report matching controls for each setting in Table 4.
Questions for authors
-
The paper states that the evaluation rows in Tables 2, 3, and 10 guided some methods during development. At which stages did each method use these rows and their labels, and why does the main comparison report only development results? Are there results on held-out queries from the same categories that were not used for harness updates or checkpoint selection?
-
The benchmark contains 27 error categories across seven error families, but the main comparison in Table 2 covers only five source categories from three families. Why were these five selected, and what criteria were used? Was the selection made before method development and comparison?
-
Why are the 10-shot result for MCE and the 5-shot and 10-shot results for Meta-Harness not reported in Table 2?
-
Table 2 lists Meta-Harness’s 74.0% under the 0-shot setting, while Table 10 reports the same score in the 5-shot setting. Is this merely a coincidence? What is the underlying reason?
-
The related-work section identifies coordination as a distinguishing feature, but Table 3 mainly measures dependence on components removed from the final harness. Could the authors compare EvoVE with separate optimization followed by combination, or with restricted feedback between components, while matching the editable components, data access, and compute budget?
-
In Table 4, moving from 0-shot to 5-shot introduces ten labeled target examples and updates the harness. While the appendix includes relevant controls, it does not have results matching all six configurations in Table 4. Could the authors evaluate the learned harness with the same support examples in context but no harness updates, and target adaptation starting from the seed harness? These controls would help separate the contribution of harness updates from that of source-category learning.
-
Table 10 defines Macro as the mean F1 across the five categories. However, the EvoVE scores of 0.781, 0.800, 0.966, 0.963, and 0.960 average to 0.894, rather than the 0.886 reported in Tables 10 and 2. Which needs correction: the per-category scores, the macro F1, or the description of the experimental configuration?