acceptodds
Under review as a conference paper at ICLR 2027

CV-Arena: An Open Benchmark for Instructional Computer Vision Problem Solving with Human-AI Collaborative Preferences

Abstract

Instruction-guided image editing is becoming a general interface for visual work, yet existing benchmarks provide uneven coverage of the diverse real-image tasks encountered in professional workflows. Here, we define instructional computer vision problem solving as a broader formulation of image editing: given a real input image and a natural-language instruction, a system must produce an edited output that realizes the requested transformation while satisfying explicit preservation, geometric, physical, and usability constraints. We introduce CV-Arena, an open benchmark designed to evaluate this capability in professional editing tasks. CV-Arena contains 12K high-resolution real-image instruction pairs spanning 16 instruction-based visual task types, constructed using CogRetriever, a dual-track retrieval-and-curation pipeline that combines targeted web search, agentic query refinement, verification, and traceability. To evaluate models at scale with selective expert supervision, we propose Active Elo, a human-AI collaborative preference protocol that leverages CV-Judge, a logic-gated, multi-dimensional VLM evaluator, to identify clear failures and resolve high-confidence comparisons, while routing close, high-quality comparisons to expert raters. Mixed human and AI supervision is then aggregated through reliability-weighted Elo updates. Our comprehensive evaluation of 21 systems, including proprietary, open-source, and agentic models, on CV-Arena reveals persistent gaps in instruction adherence, physical reasoning, structural control, and fine-grained detail preservation. We further develop CV-Agent, a simple agentic baseline that combines planning, editing, and verification. Our results suggest that closed-loop reasoning is a promising direction for professional-grade instruction-following visual editing.

Primary area:
datasets and benchmarks
Keywords:
Image Editing, Agent, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_e9VYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CV-Arena, a benchmark of 12K high-resolution image-instruction pairs across 16 editing tasks. It proposes Active Elo for human-AI collaborative evaluation and CV-Agent for planning and closed-loop editing. Experiments on 21 systems demonstrate the effectiveness of the evaluation framework and the potential of agentic editing.

Critical strengths
  1. Active Elo combines automated evaluation with selective human supervision, achieving improved human preference agreement and ranking stability under a fixed annotation budget.
  2. The evaluation of 21 editing systems provides valuable insights into model capabilities, while CV-Agent demonstrates the potential benefits of planning and closed-loop refinement.
Critical weaknesses
  1. I am somewhat concerned about the scalability and long-term maintainability of CV-Arena, as its Active Elo evaluation protocol relies on human annotators to resolve ambiguous comparisons. Given the rapid pace of image editing model development, continuously incorporating and evaluating new models may require substantial additional human annotation efforts. This could make the benchmark costly to maintain and less convenient for researchers seeking to evaluate new models independently.
  2. The generalizability of CV-Agent remains insufficiently validated, as it is evaluated exclusively on the proposed CV-Arena benchmark. While the reported results suggest improvements from planning and closed-loop refinement, evaluations on independent out-of-distribution benchmarks are needed to establish whether these gains transfer to other editing tasks and data distributions.
  3. The evaluation framework relies heavily on GPT-4o as the backbone of CV-Judge, without sufficient validation across different VLM judges. Evaluating alternative VLM judges and assessing ranking consistency would strengthen the robustness and reliability of the proposed protocol.
Questions for authors

Please refer to the weaknesses.

Reviewer_sMRbOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CV-Arena, an open benchmark for "Instructional Computer Vision Problem Solving" (iCVPS) comprising 12,000 real, high-resolution image-instruction pairs across 4 progressive cognitive tiers (Pixel-Level Restoration, Semantic Object Editing, Geometric Warping & Text Editing, and Physical/Optical Reasoning) and 16 sub-tasks. To curate suitable real-world source images, the authors propose CogRetriever, a dual-track (manual + agentic VLM reflection) web retrieval and human verification pipeline. For scalable evaluation of 21 editing systems, the paper proposes Active Elo, a human-AI collaborative ranking protocol that combines an MLLM judge (CV-Judge, built on Gemini 2.5 Pro with hierarchical capping rules) and human annotations via a two-gate routing policy (routing pairs where both candidates pass a quality floor min⁡(si,A,si,B)≥τ\min(s_{i,A}, s_{i,B}) \ge \tau and have an ambiguous score gap ∣si,A−si,B∣<Δ|s_{i,A} - s_{i,B}| < \Delta to humans) and updates ratings using a reliability-weighted Elo rule. Finally, the authors present CV-Agent, a training-free 3-stage agentic baseline (understanding, planning, and closed-loop editing with nano banana) that ranks first under Active Elo.

Critical strengths
  1. High-resolution real-world benchmark covering under-explored physical, geometric, and restoration tasks: Unlike existing editing benchmarks that rely heavily on synthetic images or simple object swaps, CV-Arena curates 12,000 real high-resolution images across 16 diverse tasks—including typography/UI restoration, text-driven geometric warping, and physical/optical interactions (shadows, reflections, deformable surfaces)—with documented license and feasibility verification.
  2. Thoughtful human-AI hybrid evaluation design (Two-Gate Routing + Capped Rubric): The hierarchical capping rules in CV-Judge (preventing high background-consistency scores when the requested edit fails completely) and the two-gate routing principle (reserving human annotation budget for high-quality, low-margin pairs rather than obvious failures) are intuitive, practical ideas for scaling arena-style visual evaluation.
Critical weaknesses
  1. Severe cross-protocol ranking instability raises doubts about whether Active Elo favors CV-Agent via evaluator bias: In Tables 4 and 5, the leaderboard rankings change drastically across evaluation protocols. Specifically, the authors' proposed CV-Agent ranks #1 under Active Elo (1145), but drops to #5 under budget-matched Human-Only (1116) and #6 under CV-Judge Only (993). Conversely, Qwen Image Edit ranks #1 under Human-Only (1187) with a narrow CI, but plunges to #11 under Active Elo (965) and #15 under CV-Judge Only (821); similarly, Step1X Edit is #1 under CV-Judge Only (1095) but #13 under Active Elo (932). It is mathematically counterintuitive that a hybrid combination of CV-Judge (#6 for CV-Agent) and Human-Only (#5 for CV-Agent) elevates CV-Agent to #1 overall. Moreover, CV-Agent uses Gemini 2.5 Pro as its internal closed-loop verifier during inference to select/refine edits, while CV-Judge also uses Gemini 2.5 Pro as the benchmark evaluator—creating a direct self-preference / reward-hacking loop.
  2. Incomplete experimental transparency on benchmark evaluation scale and calibration: While the dataset contains 12,000 image-instruction pairs and evaluates 21 systems, running all 21 systems (including slow commercial agents like Manus and ChatGPT Agent and multi-round CV-Agent) on 12,000 high-resolution images would require 252,000 generations. The paper does not clearly state how many of the 12,000 instances and how many total pairwise comparisons were actually evaluated per model to construct Table 5, nor does it reconcile the mismatch noted in Figure 5's caption ("Model labels and comparison-pool definitions require reconciliation with the underlying evaluation records"). Furthermore, the routing validation study (Table 2) was conducted on only 4 models and 8 tasks excluding CV-Agent.
Questions for authors
  1. Explaining the ranking paradox of CV-Agent and Qwen Image Edit (Clarification — Critical): Why does CV-Agent rank #1 in Active Elo when it ranks #5 in Human-Only and #6 in CV-Judge Only, and why does Qwen Image Edit drop from #1 in Human-Only to #11 in Active Elo? Since Active Elo simply routes a subset of pairs to humans and uses CV-Judge for the rest, what mechanism in the pair sampling or reliability-weighted Elo update causes CV-Agent to leapfrog four models that beat it in both Human-Only and CV-Judge-Only protocols?
  2. Evaluator self-preference in CV-Agent (Clarification): CV-Agent uses Gemini 2.5 Pro in Stage 3 to verify and iteratively refine edits against instructions, and CV-Judge uses the exact same Gemini 2.5 Pro model to score the outputs. How much of CV-Agent's gain in Table 3 (+89 Active Elo over single-pass nano banana) is due to optimizing directly against the judge model, and what is CV-Agent's gain when evaluated purely by human raters?
  3. Exact number of evaluated instances and pairs in Table 5 (Clarification): Out of the 12,000 pairs in CV-Arena, exactly how many images were generated by each of the 21 systems, how many total pairwise battles were conducted for the main leaderboard (Table 5), and what percentage of those battles were routed to humans vs. resolved by CV-Judge?
1 comment
AuthorsOpenReview ↗

Thank you for your time and feedback - happy to discuss further

Dear Reviewer sMRb,

Thank you very much for your time and thoughtful feedback on our work. As the discussion period is nearing its end, we would be grateful if you could take a look at our rebuttal when you have a chance. Please let us know if there is anything we can clarify or elaborate on; we would be happy to discuss further.

Thank you again for your time and consideration.

Best regards, Authors of Submission #51993

Reviewer_sDUmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces CV-Arena, a benchmark containing 12K real-image–instruction pairs across 16 task types, targeting instruction-guided image editing under preservation, geometric, physical, and usability constraints. It proposes Active Elo, which combines a GPT-4o-based evaluator, CV-Judge, with selective human annotation and reliability-weighted Elo updates. The paper also presents CV-Agent, which uses Gemini 2.5 Pro for understanding, planning, and verification, and nano banana for editing. Experiments compare 21 systems and investigate the evaluation protocol and agent components.

Critical strengths
  • Relevant benchmark scope. The emphasis on real, high-resolution images and professional editing constraints addresses an important gap between producing visually plausible outputs and faithfully completing an editing request.

  • Practical human–AI evaluation design. Routing close, high-quality comparisons to human annotators is a reasonable way to allocate limited annotation resources. The budget-matched routing study provides initial evidence supporting this design.

  • Useful experimental coverage. The evaluation includes diverse editing systems, multiple evaluation protocols, and same-editor agent ablations. Reporting confidence intervals and protocol-dependent rankings helps expose uncertainty in the results.

Critical weaknesses
  • Insufficient detail about dataset construction. Although the retrieval pipeline is described, the paper does not report how many final samples come from the manual Base Track versus the Agentic Track. Counts before and after filtering, rejection rates, and the extent of manual intervention would improve transparency and reproducibility.

  • The instruction-generation process is unclear. CogRetriever assumes an editing instruction as input, but the paper does not adequately explain how these instructions are initially created. Were they written by humans, generated by an LLM, or produced through a hybrid process? Were they revised after image retrieval? Please describe how image-specific targets and preservation constraints were added, and provide the relevant prompts, templates, and quality-control procedures.

  • The agent comparison does not isolate the benefit of closed-loop reasoning. CV-Agent can perform up to three editing steps and select the highest-scoring candidate, whereas standalone editors use a single-pass configuration. Its gains may therefore reflect additional editing calls, prompt refinement, or candidate selection. An equal-budget baseline that generates multiple independent candidates and selects among them using the same verifier would help isolate the value of planning and feedback. Reporting editing-call counts, latency, and cost would also clarify the trade-off.

  • Statistical evidence for the reported improvements remains incomplete. The paper reports per-system 95% confidence intervals, but these do not directly establish the significance of pairwise Elo differences. In particular, the 18-point difference between CV-Agent and nano banana pro should be accompanied by a paired bootstrap confidence interval or an appropriate test. The cumulative module ablations likewise need uncertainty estimates for their differences. Overlapping marginal intervals alone neither establish nor rule out significance.

  • The main ranking requires stronger independent validation. CV-Agent ranks first under Active Elo, fifth under budget-matched human-only evaluation, and sixth under CV-Judge-only evaluation. Meanwhile, the routing-validation study covers only four models and excludes CV-Agent. Broader held-out human validation, including the leading systems, would strengthen confidence in the full leaderboard.

  • Minor presentation issue. On page 9, the caption of Fig. 4 is too close to the adjacent body text. Please increase the spacing or revise the layout to improve readability.

Questions for authors

None

1 comment
AuthorsOpenReview ↗

Thank you for your time and feedback - happy to discuss further

Dear Reviewer sDUm,

Thank you very much for your time and thoughtful feedback on our work. As the discussion period is nearing its end, we would be grateful if you could take a look at our rebuttal when you have a chance. Please let us know if there is anything we can clarify or elaborate on; we would be happy to discuss further.

Thank you again for your time and consideration.

Best regards, Authors of Submission #51993

Comments on the paper

1 comment

Thank you for the detailed response; while the authors have addressed part of my concerns, I still view GPT-4o as an outdated judge backbone and will maintain my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.