- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces WeGenEditBench and WeGenEvoStudio. WeGenEditBench contains 2,740 instances and 148 subset-qualified task types covering single-image, text-intensive, and multi-reference editing. Each instance is associated with a pre-generated checklist that decomposes the request into requirements concerning Edit Goal, Preservation, Visual Quality, and Domain-specific Capability. A fixed VLM judge records rationales and item-level deductions before assigning dimensional scores, which are then aggregated according to subset-specific scoring rules. WeGenEvoStudio uses benchmark results to identify weak task categories, inspect representative failures, formulate data requirements, retrieve new source images or concepts, construct targeted training examples, adapt the evaluated model, and re-evaluate it. The paper benchmarks 15 open- and closed-source editing configurations and studies model adaptation across four generation and editing benchmarks with Qwen-Image, Qwen-Image-Edit, and FLUX.2. The results reveal scenario-dependent rankings and show substantial gains for FLUX.2 on selected weak categories, while the improvements for Qwen are smaller and occasionally accompanied by regressions.
Critical strengths
- Fine-grained evaluation across complementary image-editing settings. The benchmark covers three complementary settings—single-image, text-intensive, and multi-reference editing—and decomposes each request into sample-specific, inspectable requirements. Generating the checklists before evaluating model outputs and sharing them across models within the same evaluation version is a sensible design that improves consistency and reduces dependence on output-specific evaluation criteria. The resulting item-level deductions and rationales make failures such as omitted edits, preservation violations, and incorrect reference binding easier to trace than with aggregate scores alone.
- A useful bridge from evaluation to targeted model improvement. The paper goes beyond benchmark construction by using evaluation results to identify recurring weaknesses and translate them into task-level data requirements for subsequent training. WeGenEvoStudio connects failure diagnosis, retrieval-grounded data construction, supervised adaptation, and re-evaluation in a coherent pipeline, and the framework is tested across multiple models and benchmarks. The substantial gains obtained by FLUX.2 on selected weak categories suggest that benchmark-informed data construction can be practically effective, even though the current experiments do not fully disentangle the contribution of the individual components in the pipeline.
Critical weaknesses
- Evaluation reliability is not sufficiently validated. The 10% corrective audit does not measure judge accuracy or inter-rater agreement. The close leaderboard differences are also reported without ranking uncertainty estimates that account for generation and judging variability.
- The contribution of WEGENEVOSTUDIO is not isolated. The experiments lack equal-budget random and score-only baselines, while the procedure used to produce target supervision for the constructed tasks is insufficiently specified. The ImgEdit before/after comparison also uses different inference settings, and no sealed test set rules out benchmark-guided overfitting.
Questions for authors
- Can the authors report human agreement, judge accuracy, and ranking uncertainty across repeated generations and judge calls?
- The paper itself shows that 48.9% of GPT-Image-2 single-image outputs with Overall ≥ 4 still contain at least one checklist item scored at most 2. This motivates the checklist-based evaluation, but it also raises a question about how informative the Overall metric is for model comparison. Do model rankings remain stable when Overall is replaced by an aggregation derived directly from item-level checklist scores, or by a metric measuring complete requirement satisfaction?
- How is target supervision generated for the constructed tasks, and what filtering or quality-control procedures are applied to it? Under matched training and inference settings, how does the full method compare with equal-budget random and score-only selection on a sealed test set?
Convincing answers to these questions would make me consider raising my score.
2 comments
Thank you for taking the time to review our rebuttal and for confirming that most of your main concerns have been addressed.
Given our focus on the human validation and controlled training experiments in our initial response, we did not fully elaborate on Question 2 regarding ranking stability when replacing Overall with alternative checklist aggregations. To address this point thoroughly, we report the direct rank correlations on the same evaluated tasks across all three subsets below. Here, Item Mean averages applicable checklist scores within each task and then across tasks, while CS₄ and CS₅ measure the share of tasks where every applicable item scores at least 4 and exactly 5, respectively. Each cell reports Spearman’s and Kendall’s relative to Overall:
| Metric | Single-image | Text | Multi-reference |
|---|---|---|---|
| Item Mean | 0.9821 / 0.9238 | 0.9786 / 0.9048 | 1.0000 / 1.0000 |
| CS₄ | 0.9589 / 0.8558 | 0.9607 / 0.8667 | 0.9429 / 0.8667 |
| CS₅ | 0.9245 / 0.8337 | 0.9393 / 0.8095 | 0.9856 / 0.9661 |
Key Takeaways:
- Consistent macro rankings: Broad model orderings hold strongly across all three alternatives, with Spearman's exceeding 0.92 across every subset, confirming that
Overallreliably preserves global model comparisons. - Limitations of replacing
OverallwithItem Mean: Unweighted item averaging treats core edit execution and minor auxiliary checks equally, which introduces checklist-length bias and allows high scores on numerous preservation and quality items to dilute critical failures on the primary edit goal. - Limitations of replacing
OverallwithCS₄orCS₅: All-or-nothing satisfaction thresholds collapse partial progress into binary failure, reducing discrimination across models on complex, multi-constraint tasks where minor localized imperfections are common.
Consequently, Overall provides balanced, continuous discrimination across capability tiers, while item-level scores and CS₄ or CS₅ serve as complementary diagnostic indicators rather than standalone replacements.
Thank you again for raising this insightful question. Please let us know if you have any further questions or deeper concerns—we would be very glad to discuss them before the discussion closes, and we hope these clarifications resolve your remaining reservation and support a more positive overall assessment.
Thank you for the additional experiments and clarifications. Most of my main concerns have been addressed, and I will maintain my score .