acceptodds
Under review as a conference paper at ICLR 2027

From Evaluation to Evolution: Benchmarking Image Editing and Evolving Image Models

Abstract

Real-world image editing requires coordinating multiple transformations with preservation and reference constraints, yet aggregate scores can obscure unmet requirements. Useful evaluation must therefore reveal local failures and translate them into actionable priorities for subsequent model improvement. We introduce WeGenEditBench, a benchmark of 2,740 instances and 148 task types spanning single-image, text-intensive, and multi-reference editing. Shared, pre-generated checklists decompose each request into atomic requirements and connect scores and deductions to visual evidence across edit execution, preservation, visual quality, and task-specific correctness. To translate diagnosis into training priorities, WeGenEvoStudio combines failure analysis, retrieval-grounded data construction, supervised adaptation, and re-evaluation in a unified multi-agent framework. Among 15 evaluated configurations, GPT-Image-2 leads single-image editing with an Overall score of 4.076, while Seedream-5.0-Pro leads text-intensive editing at 4.291. Across four generation and editing benchmarks, adaptation yields task-dependent gains; on their report-selected Top-5 targets in our single-image subset, FLUX.2 [dev] improves Overall from 1.083 to 3.517, while Qwen-Image-Edit-2511 gains 13.2%. Together, our benchmark and framework provide a foundation for developing more reliable image models through interpretable evaluation and targeted, feedback-driven learning. The benchmark and code will be publicly released.

Primary area:
datasets and benchmarks
Keywords:
Benchmarks, Image Editing, Agent, Harness

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_q7ZXOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces WeGenEditBench and WeGenEvoStudio. WeGenEditBench contains 2,740 instances and 148 subset-qualified task types covering single-image, text-intensive, and multi-reference editing. Each instance is associated with a pre-generated checklist that decomposes the request into requirements concerning Edit Goal, Preservation, Visual Quality, and Domain-specific Capability. A fixed VLM judge records rationales and item-level deductions before assigning dimensional scores, which are then aggregated according to subset-specific scoring rules. WeGenEvoStudio uses benchmark results to identify weak task categories, inspect representative failures, formulate data requirements, retrieve new source images or concepts, construct targeted training examples, adapt the evaluated model, and re-evaluate it. The paper benchmarks 15 open- and closed-source editing configurations and studies model adaptation across four generation and editing benchmarks with Qwen-Image, Qwen-Image-Edit, and FLUX.2. The results reveal scenario-dependent rankings and show substantial gains for FLUX.2 on selected weak categories, while the improvements for Qwen are smaller and occasionally accompanied by regressions.

Critical strengths
  1. Fine-grained evaluation across complementary image-editing settings. The benchmark covers three complementary settings—single-image, text-intensive, and multi-reference editing—and decomposes each request into sample-specific, inspectable requirements. Generating the checklists before evaluating model outputs and sharing them across models within the same evaluation version is a sensible design that improves consistency and reduces dependence on output-specific evaluation criteria. The resulting item-level deductions and rationales make failures such as omitted edits, preservation violations, and incorrect reference binding easier to trace than with aggregate scores alone.
  2. A useful bridge from evaluation to targeted model improvement. The paper goes beyond benchmark construction by using evaluation results to identify recurring weaknesses and translate them into task-level data requirements for subsequent training. WeGenEvoStudio connects failure diagnosis, retrieval-grounded data construction, supervised adaptation, and re-evaluation in a coherent pipeline, and the framework is tested across multiple models and benchmarks. The substantial gains obtained by FLUX.2 on selected weak categories suggest that benchmark-informed data construction can be practically effective, even though the current experiments do not fully disentangle the contribution of the individual components in the pipeline.
Critical weaknesses
  1. Evaluation reliability is not sufficiently validated. The 10% corrective audit does not measure judge accuracy or inter-rater agreement. The close leaderboard differences are also reported without ranking uncertainty estimates that account for generation and judging variability.
  2. The contribution of WEGENEVOSTUDIO is not isolated. The experiments lack equal-budget random and score-only baselines, while the procedure used to produce target supervision for the constructed tasks is insufficiently specified. The ImgEdit before/after comparison also uses different inference settings, and no sealed test set rules out benchmark-guided overfitting.
Questions for authors
  1. Can the authors report human agreement, judge accuracy, and ranking uncertainty across repeated generations and judge calls?
  2. The paper itself shows that 48.9% of GPT-Image-2 single-image outputs with Overall ≥ 4 still contain at least one checklist item scored at most 2. This motivates the checklist-based evaluation, but it also raises a question about how informative the Overall metric is for model comparison. Do model rankings remain stable when Overall is replaced by an aggregation derived directly from item-level checklist scores, or by a metric measuring complete requirement satisfaction?
  3. How is target supervision generated for the constructed tasks, and what filtering or quality-control procedures are applied to it? Under matched training and inference settings, how does the full method compare with equal-budget random and score-only selection on a sealed test set?

Convincing answers to these questions would make me consider raising my score.

2 comments

Thank you for the additional experiments and clarifications. Most of my main concerns have been addressed, and I will maintain my score .

Thank you for taking the time to review our rebuttal and for confirming that most of your main concerns have been addressed.

Given our focus on the human validation and controlled training experiments in our initial response, we did not fully elaborate on Question 2 regarding ranking stability when replacing Overall with alternative checklist aggregations. To address this point thoroughly, we report the direct rank correlations on the same evaluated tasks across all three subsets below. Here, Item Mean averages applicable checklist scores within each task and then across tasks, while CS₄ and CS₅ measure the share of tasks where every applicable item scores at least 4 and exactly 5, respectively. Each cell reports Spearman’s ρ\rho and Kendall’s τb\tau_b relative to Overall:

MetricSingle-imageTextMulti-reference
Item Mean0.9821 / 0.92380.9786 / 0.90481.0000 / 1.0000
CS₄0.9589 / 0.85580.9607 / 0.86670.9429 / 0.8667
CS₅0.9245 / 0.83370.9393 / 0.80950.9856 / 0.9661

Key Takeaways:

  • Consistent macro rankings: Broad model orderings hold strongly across all three alternatives, with Spearman's ρ\rho exceeding 0.92 across every subset, confirming that Overall reliably preserves global model comparisons.
  • Limitations of replacing Overall with Item Mean: Unweighted item averaging treats core edit execution and minor auxiliary checks equally, which introduces checklist-length bias and allows high scores on numerous preservation and quality items to dilute critical failures on the primary edit goal.
  • Limitations of replacing Overall with CS₄ or CS₅: All-or-nothing satisfaction thresholds collapse partial progress into binary failure, reducing discrimination across models on complex, multi-constraint tasks where minor localized imperfections are common.

Consequently, Overall provides balanced, continuous discrimination across capability tiers, while item-level scores and CS₄ or CS₅ serve as complementary diagnostic indicators rather than standalone replacements.

Thank you again for raising this insightful question. Please let us know if you have any further questions or deeper concerns—we would be very glad to discuss them before the discussion closes, and we hope these clarifications resolve your remaining reservation and support a more positive overall assessment.

Reviewer_hmT5OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The authors created a real-world image editing benchmark, and proposed an agent framework to perform retrieval-grounded data construction, supervised adaptation, and re-evaluation to improve editing result via fine-grained criteria, showing a great improvement in both own benchmark and prior benchmarks.

Critical strengths

S1: I think the retrieval part on synthetic data is actually pretty good idea, like the prior research has already shown synthetic data helps tasks on image models especially typography on image (Qwen Image), this forms a good motivation and its new to see it blend into agent framework.

Critical weaknesses

W1: If I understood correctly, same VLM (Gemini) was used in both agent framework and the evaluation framework. The authors need to prove that the method is not doing reward hacking by showing another evaluator and supporting it with traditional image editing metrics (e.g. LPIPS/TPIPS). It is also unclear whether the "agent" framing provides any measurable empirical gain over a standard automated prompt-expansion script.

W2: Because the training data is synthesized specifically to address the exact failures diagnosed by the benchmark evaluation, there could be a severe risk of overfitting. Authors should consider held-out test sets to prove that the model actually became a better general image editor, rather than simply learning to satisfy the specific linguistic checklists of the benchmark.

W3: The failure analysis from diagnosis agent is basically unverified in the study. Author should take account on the reliability on VLMs on whether it can really point out issues, perhaps with a small scale of human study.

W4: Missing core references on Re-Imagen (Google Research, 2022), RichHF (CVPR 2024), ImageDoctor, and ImagenWorld (ICLR 2026) which the idea of retrieval, localizing errors, and real-world edits are highly relevant.

W5: The aggregation weights at Equation 1 and 2 feels arbitrary. The authors should justify the choice and provide an ablation study to support, e.g. to show how sensitive it is with current setting and other potential settings.

Questions for authors

How is the computational cost and scalability of the agent framework?

Small suggestion: for chinese prompts shown in figure, maybe translate it to english and state it was originally chinese so readers can understand better on the prompt used. Figures can be simplified too for readability.

Reviewer_MkUbOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces WeGenEditBench, a benchmark containing 2,740 samples across single-image, text-intensive, and multi-reference editing tasks. It uses sample-specific checklists to evaluate editing performance at the requirement level, identifying failures that may be overlooked by aggregate scores. The paper also proposes WeGenEvoStudio, a multi-agent framework that diagnoses model weaknesses, constructs targeted training data, and improves image editing models through supervised fine-tuning. Experiments evaluate 15 image editing configurations and demonstrate performance improvements on Qwen-Image and FLUX.2, although the gains vary across tasks and models.

Critical strengths

This paper presents a comprehensive image editing benchmark and connects failure diagnosis with targeted data construction, leading to improvements in model performance. While the overall framework appears promising, I have several concerns about the quality, as detailed in the weaknesses below.

Critical weaknesses
  1. The key differences from existing benchmarks (ComplexBench-Edit, Qwen-Image-Bench, and Mind-Bench) are unclear. The main contribution seems to be broader task coverage, which feels somewhat incremental.

  2. The paper lacks complete checklist and judge evaluation examples. More importantly, human validation is insufficiently detailed, with no reported human–VLM agreement or checklist error rates.

  3. The reported improvements may simply result from additional training data. A baseline using randomly selected data under the same training budget is needed to demonstrate the effectiveness of WeGenEvoStudio.

Questions for authors

For data construction, how can you get the groundtruth edited image for supervision?

Comments on the paper

1 comment
Area_Chair_yHpROpenReview ↗

Reminder: Please Respond to Author Rebuttals

Dear Reviewers,

As the discussion period is coming to an end, please review the authors’ responses and provide a brief follow-up on whether your concerns have been addressed.

Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.