acceptodds
Withdrawn from ICLR 2027

VideoForge: Toward Recursive Self-Improvement Environments for Agentic Video Evaluation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Improving video evaluation agents requires identifying why they miss or misdescribe defects and determining which harness changes correct these failures. Existing methods typically optimize the evaluation harness with a fixed outer-loop procedure, leaving the improvement process itself unchanged. We introduce VideoForge, a recursive self-improvement environment that adapts both the video-evaluation harness and the procedure used to improve it. VideoForge exposes task execution, evidence inspection, harness revision, and self-reflection as interaction primitives, allowing improvement trajectories to adapt to observed failures and accumulated experience. To evaluate these capabilities, we introduce VidDiag, a benchmark comprising 1,087 real-world video generation and editing cases with 2,932 human-annotated defects spanning seven dimensions and 27 fine-grained failure types. The benchmark evaluates both defect-category identification and instance-level defect descriptions. Across two model–agent configurations, VideoForge improves overall category F1 by 3.6%–4.9% and strict semantic matching by 2.9%–4.6% over the corresponding initial harnesses, achieving the highest overall scores among the compared optimization methods, including SkillOpt and AHE. Ablations under the same task-execution budget further support the benefit of adapting the improvement procedure. We further conduct a separate human-evaluated video-debugging study on 100 cases to examine whether accurate diagnosis can support downstream repair. Providing ground-truth defect descriptions improves repair success by 11%–12%, yet the highest success rate remains only 32%. These results show that adapting the improvement process can strengthen fine-grained video diagnosis, while translating accurate diagnosis into effective repair remains challenging.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Recursive Self-Improvement, Autonomous Agents, Agent Harness Evolution, Multimodal Reasoning, Video Generation Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Vc6tOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors present a self-improving agent framework that improves video evaluation harness. And also they provide a video benchmark with annotated defects for evaluating fine-grained video error detection and descriptions.

Critical strengths
  1. Generated video evaluation is a practical problem. The authors explore this by using recent concept, Recursive Self-Improvement.

  2. This paper introduces a useful benchmark, VidDiag, which includes representative errors in AI-generated videos. The human annotation process appears reasonably rigorous.

  3. The proposed metrics in benchmark (Category-level F1, Strict Semantic Matching) seems well-motivated for video error evaluation.

Critical weaknesses
  1. The novelty seems limited, which mainly combines recursive self-improvement with harness optimization methods through a test-time optimization. And the writing need to be improved.

  2. I concern that such a complex pipeline is truly need? It seems stronger vlms with better prompts and frame sampling might detect many errors. In table 2, shows stronger models tend to perform better, whether the improvements mainly come from the VLM itself, with only some additional guidance from the framework.

  3. It shows in table.3, such complex system only show relative small improvements. And this pipeline's token cost is high.

Questions for authors
  1. The writing needs improvement. The abstract is a bit complicated, and there are several typos (e.g., Lines 153, 286, 313, and 319). Line 282 refers to details in Appendix B, but the information seems missing. I also suggest moving the method overview figure earlier in the paper.

  2. Is such a complex pipeline really needed? The improvements seem relatively small, while the token cost is much higher than SkillOpt. Some errors shown in Figure 1 seem relatively easy to detect. Could the authors compare with stronger VLMs (GPT-5.6-Sol / GPT-6.1-Sol) using simple prompts covering all seven evaluation dimensions?

  3. Could the authors test VideoForge with a self-hosted open-source VLM (e.g. Qwen3.5-35B-A3B [1]) to verify our framework not heavily on the underlying VLM? It would be helpful to report the average runtime and computational cost per agent loop, and compare them with the fixed optimization pipeline in Figure 2.

  4. I question about the video frames sampled for evaluation? Because different sampling methods might miss certain defects and affect the evaluation and results.

  5. The "training" set and test cover the same error categories. It worth to show whether the framework generalizes to more ai-generated video's errors. During inference loop, it would be helpful to show the performance at each round and compare single-round and multi-round optimization (whether continue to improve) for in-domain / out-of-domain errors.

[1] Qwen3.5-35B-A3B: https://huggingface.co/Qwen/Qwen3.5-35B-A3B

Reviewer_YUVMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VideoForge proposes an environment for improving video-defect diagnosis agents by adapting both the task harness and the procedure used to improve it. It provides task execution, evidence inspection, harness revision, and self-reflection as interaction primitives. The authors also introduce VidDiag, containing 1087 video cases and 2932 human-annotated defects across 27 fine-grained types. On two model-agent configurations, VideoForge improves category-level F1 and strict semantic matching over the initial harness and compared optimization methods. A separate 100-case study evaluates the effect of ground-truth defect descriptions on video repair.

Critical strengths

VidDiag provides human annotated defects covering several aspects of video generation and editing. Its instance level descriptions make it useful for studying specific diagnosis failures rather than only overall video quality. VideoForge improves diagnosis results in tested configurations.

Critical weaknesses
  1. The main claimed novelty is not sufficiently explained. VideoForge distinguishes itself from previous harness evolution methods by evolving the improvement procedure π\pi as well as the task harness HH. It does not explain how π\pi is represented or updated. Appendix E.3 introduces an editable improvement harness, but its relationship to π\pi is unclear, and Algorithm 1 leaves the update as an abstract "Evolve" operation. Figure 3 shows memory guided interaction selection, but its relationship to π\pi is also unclear. A π\pi using interaction history could also achieve this selection. The writing could also be clearer, with unresolved figure references and redundant text in Sections 4.3 and 5.1.
  2. The experiments do not sufficiently establish the value of evolving π\pi. The paper does not specify what is disabled in VideoForge w/o Evo., making it difficult to tell whether the ablation isolates the evolution of π\pi. Table 4 reports improved update acceptance rates, but does not show how changes to π\pi affect later optimization. VideoForge also achieves only slightly better F1 than SkillOpt while using much more optimizer tokens. The benefit relative to the added optimization cost remains unclear.
Questions for authors
  1. The paper defines π\pi as the improvement procedure, but its detail remain unclear. Please clarify how π\pi is represented and updated, and how this differs from a fixed π\pi making decisions based on interaction history.
  2. The ablation does not clearly proof the contribution of evolving π\pi. What exactly is disabled in VideoForge w/o Evo., and does it retain memory, reflection, and adaptive case selection? Clarifying this is necessary to attribute the reported improvements to changes in π\pi.
  3. VideoForge relies on human annotated defect feedback during optimization, but its sensitivity to the amount of annotation is unclear. How does its performance change when more or fewer annotated training cases are available?
Reviewer_UmwGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VideoForge, a recursive self-improvement environment for agentic video evaluation. It improves evaluation by adapting both the task harness and the process that organizes improvement, while keeping model parameters fixed. It exposes task execution, evidence inspection, harness revision, and self-reflection as interaction primitives, so that accumulated experience can both guide later improvement sessions and change how subsequent improvement is organized (two-level evolution). The authors also build VidDiag, containing 1,087 video generation and editing cases across 7 dimensions and 27 fine-grained failure types, with 2,932 annotated defect instances.

Critical strengths
  1. VidDiag: a human-annotated and reviewed benchmark for video defect diagnosis, providing both a failure taxonomy and case-specific defect descriptions for fine-grained evaluation of video generation and editing failures.

  2. Joint adaptation of the evaluation harness and its improvement process: consistent gains across two model–agent configurations, with ablations comparing with and without the evolution mechanism under the same task-execution budget.

  3. A human-evaluated video debugging study: ground-truth defect guidance improves repair, yet a clear gap remains between accurate diagnosis and successful repair.

Critical weaknesses
  1. The independent contribution of the recursive improvement mechanism is not clearly isolated. The paper reports results for the full method versus a version without the evolutionary component, but does not specify which operations this ablation disables or whether other capabilities are kept identical. As a result, it is hard to tell whether the gains come from evolving the improvement process itself or from accompanying differences such as memory use, history access, or experiment selection. The contribution of the core mechanism therefore remains unclear.
  2. There is no comparison against agent self-analysis and reflection, so the added value of human-annotated feedback is unclear. The current optimization uses evaluation feedback based on human annotations, but the paper does not report results when optimization relies on the agent's own analysis, self-evaluation, and reflection under the same initial setup and comparable compute. It is therefore unclear how much extra gain the annotated feedback provides, and whether the method can still improve performance when such supervision is unavailable. This limits assessment of the method's supervision dependence and deployment cost.
  3. Current evaluation does not adequately capture instance-level false positives. Existing metrics measure defect-category recognition and coverage of true defects, but do not directly constrain extra, unsupported defect descriptions within an already predicted category. Current results are therefore insufficient to determine whether higher diagnostic coverage comes with more false positives, and output reliability cannot be fully assessed.
  4. Writing and manuscript consistency issues are numerous. The manuscript contains unfinished cross-references, repeated statements, and unclear correspondence between the main text and the appendix. These issues reduce clarity of the method description and make the experimental setup harder to understand and reproduce, requiring systematic proofreading and revision.
Questions for authors
  1. Which operations does the ablation without the evolutionary component freeze or remove? Is it identical to the full method in persistent memory, execution-history access, validation feedback, and autonomous interaction selection? Can you provide one actual example of an improvement-process update and show how it changed subsequent experiments or revision decisions?
  2. Under the same initial harness, model, tools, and comparable compute, can you compare optimization driven by annotated feedback with optimization driven by the agent's own analysis, self-evaluation, and reflection? Please state what supervision each setting keeps. To test optimization without human defect annotations, both the optimization and candidate selection in the self-reflection setting should avoid using those annotations and any scores derived from them, with final evaluation on the same independently annotated test set.
  3. Using existing predictions, can you add instance-level precision/F1, or run a small human check for false positives, to test whether the main performance advantage still holds once extra predicted defect descriptions are included in evaluation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.