- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors present a self-improving agent framework that improves video evaluation harness. And also they provide a video benchmark with annotated defects for evaluating fine-grained video error detection and descriptions.
Critical strengths
-
Generated video evaluation is a practical problem. The authors explore this by using recent concept, Recursive Self-Improvement.
-
This paper introduces a useful benchmark, VidDiag, which includes representative errors in AI-generated videos. The human annotation process appears reasonably rigorous.
-
The proposed metrics in benchmark (Category-level F1, Strict Semantic Matching) seems well-motivated for video error evaluation.
Critical weaknesses
-
The novelty seems limited, which mainly combines recursive self-improvement with harness optimization methods through a test-time optimization. And the writing need to be improved.
-
I concern that such a complex pipeline is truly need? It seems stronger vlms with better prompts and frame sampling might detect many errors. In table 2, shows stronger models tend to perform better, whether the improvements mainly come from the VLM itself, with only some additional guidance from the framework.
-
It shows in table.3, such complex system only show relative small improvements. And this pipeline's token cost is high.
Questions for authors
-
The writing needs improvement. The abstract is a bit complicated, and there are several typos (e.g., Lines 153, 286, 313, and 319). Line 282 refers to details in Appendix B, but the information seems missing. I also suggest moving the method overview figure earlier in the paper.
-
Is such a complex pipeline really needed? The improvements seem relatively small, while the token cost is much higher than SkillOpt. Some errors shown in Figure 1 seem relatively easy to detect. Could the authors compare with stronger VLMs (GPT-5.6-Sol / GPT-6.1-Sol) using simple prompts covering all seven evaluation dimensions?
-
Could the authors test VideoForge with a self-hosted open-source VLM (e.g. Qwen3.5-35B-A3B [1]) to verify our framework not heavily on the underlying VLM? It would be helpful to report the average runtime and computational cost per agent loop, and compare them with the fixed optimization pipeline in Figure 2.
-
I question about the video frames sampled for evaluation? Because different sampling methods might miss certain defects and affect the evaluation and results.
-
The "training" set and test cover the same error categories. It worth to show whether the framework generalizes to more ai-generated video's errors. During inference loop, it would be helpful to show the performance at each round and compare single-round and multi-round optimization (whether continue to improve) for in-domain / out-of-domain errors.
[1] Qwen3.5-35B-A3B: https://huggingface.co/Qwen/Qwen3.5-35B-A3B