- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces CV-Arena, a benchmark of 12K high-resolution image-instruction pairs across 16 editing tasks. It proposes Active Elo for human-AI collaborative evaluation and CV-Agent for planning and closed-loop editing. Experiments on 21 systems demonstrate the effectiveness of the evaluation framework and the potential of agentic editing.
Critical strengths
- Active Elo combines automated evaluation with selective human supervision, achieving improved human preference agreement and ranking stability under a fixed annotation budget.
- The evaluation of 21 editing systems provides valuable insights into model capabilities, while CV-Agent demonstrates the potential benefits of planning and closed-loop refinement.
Critical weaknesses
- I am somewhat concerned about the scalability and long-term maintainability of CV-Arena, as its Active Elo evaluation protocol relies on human annotators to resolve ambiguous comparisons. Given the rapid pace of image editing model development, continuously incorporating and evaluating new models may require substantial additional human annotation efforts. This could make the benchmark costly to maintain and less convenient for researchers seeking to evaluate new models independently.
- The generalizability of CV-Agent remains insufficiently validated, as it is evaluated exclusively on the proposed CV-Arena benchmark. While the reported results suggest improvements from planning and closed-loop refinement, evaluations on independent out-of-distribution benchmarks are needed to establish whether these gains transfer to other editing tasks and data distributions.
- The evaluation framework relies heavily on GPT-4o as the backbone of CV-Judge, without sufficient validation across different VLM judges. Evaluating alternative VLM judges and assessing ranking consistency would strengthen the robustness and reliability of the proposed protocol.
Questions for authors
Please refer to the weaknesses.
Thank you for your time and feedback - happy to discuss further
Dear Reviewer sMRb,
Thank you very much for your time and thoughtful feedback on our work. As the discussion period is nearing its end, we would be grateful if you could take a look at our rebuttal when you have a chance. Please let us know if there is anything we can clarify or elaborate on; we would be happy to discuss further.
Thank you again for your time and consideration.
Best regards, Authors of Submission #51993