acceptodds
Under review as a conference paper at ICLR 2027

GEditBench v2: A Human-Aligned Benchmark for General Image Editing

Abstract

Recent advances in image editing have enabled models to handle complex instructions with impressive realism. However, existing evaluation frameworks lag behind: current benchmarks suffer from narrow task coverage, while standard metrics fail to adequately capture visual consistency, i.e., the preservation of identity, structure and semantic coherence between edited and original images. To address these limitations, we introduce GEditBench v2, a comprehensive benchmark with 1,200 real-world user queries spanning 23 tasks, including a dedicated open-set category for unconstrained, out-of-distribution editing instructions beyond predefined tasks. Furthermore, we propose PVC-Judge, an open-source pairwise assessment model for visual consistency, trained with two novel region-decoupled preference data synthesis pipelines. Besides, we construct VCReward-Bench using expert-annotated preference pairs to assess the alignment of PVC-Judge with human judgments on visual consistency evaluation. Experiments show that our PVC-Judge achieves state-of-the-art evaluation performance among open-source models and even surpasses GPT-5.1 on average. Finally, by benchmarking 16 frontier editing models, we show that GEditBench v2 enables more human-aligned evaluation, revealing critical limitations of current models, and providing a reliable foundation for advancing precise image editing.

Primary area:
datasets and benchmarks
Keywords:
Image-Editing, Real-World Benchmark, Assessment Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_B4C8OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces GEditBench v2, a benchmark for general instruction-guided image editing comprising 1,200 samples across 22 predefined editing tasks and an open-set category containing 100 real-world instructions. To address the difficulty of automatically evaluating visual consistency—the preservation of non-target content, identity, and structure—the authors propose PVC-Judge, a pairwise evaluator fine-tuned from Qwen3-VL-8B. Its training preferences are constructed using object-centric and human-centric region-decoupled metrics, supplemented by VLM-based annotations for global editing tasks. The authors also introduce VCReward-Bench, an expert-annotated preference benchmark for measuring agreement between visual consistency evaluators and human judgments. Experiments report that PVC-Judge outperforms existing open-source evaluators and achieves higher average accuracy than GPT-5.1 on this benchmark.

Critical strengths
  1. Important and timely evaluation problem. As image editing models become increasingly capable, distinguishing successful instruction execution from the preservation of content outside the intended editing scope becomes more important. Treating visual consistency as a separate evaluation dimension addresses a meaningful practical need.

  2. Broad and practically relevant task coverage. Beyond common local, global, and reference-guided edits, the benchmark includes chart editing, in-image text translation, enhancement, and hybrid editing. The open-set category further extends evaluation beyond a closed taxonomy and introduces more challenging, unconstrained instructions.

  3. Human-annotated meta-evaluation. In addition to developing an evaluator, the authors construct VCReward-Bench using expert preference annotations. Evaluating agreement with human judgments provides a more direct basis for assessing evaluator reliability than relying solely on automatically synthesized preferences.

Critical weaknesses
  1. Insufficient detail on the reliability of human annotations. Although Appendix D describes the annotation interface, evaluation dimensions, and available preference options, it does not adequately specify the number and qualifications of annotators, the number of independent annotations per pair, the label aggregation or disagreement-resolution procedure, or inter-annotator agreement statistics. Since VCReward-Bench provides central evidence for the claimed human alignment of PVC-Judge, these omissions make it difficult to assess the reliability of the benchmark labels and the credibility of the reported performance improvements.

  2. The filtering protocol does not sufficiently isolate visual consistency and may introduce cross-dimensional confounding. Appendix D requires the VC-preferred candidate to be no worse than the other candidate in IF, VQ, and Overall. However, this rule still permits pairs in which one candidate is preferred across all dimensions. An evaluator could therefore potentially predict the VC label using instruction following, visual quality, or overall preference alone. Consequently, the current results do not sufficiently substantiate the claim that PVC-Judge is specifically aligned with human judgments of visual consistency.

Questions for authors

See Weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer B4C8

We sincerely appreciate the reviewer for the detailed and constructive feedback. We are glad that you found the visual consistency evaluation problem important and timely, our task coverage broad, and our meta-evaluation practical.

We hope our responses address your concerns, and we would be grateful if you would reconsider your rating in light of them. Below, we provide detailed responses to all comments.


W1: Insufficient detail on the reliability of human annotations.

A1: Thanks for pointing this out. Our annotation team comprises 18 annotators, all trained using a detailed manual with annotation guidelines and representative examples. The manual is iteratively refined as disagreements arise. Each pair is independently labeled by 2 annotators on IF, VQ, VC, and Overall dimensions.

Labels agreed upon by both annotators are accepted directly. Disagreements are first adjudicated by a senior expert, with a second PhD-level expert reviewing unresolved edge-cases. Pairs are discarded if consensus cannot be reached.

For inter-annotator agreement, we spot-check agreement between the two annotators on a daily sample of pairs. Batches with agreement below 90% are relabeled. The average daily spot-check agreement was 93.6%. The annotated pool (covers 4,388 input prompts, ~65k pairs) is then filtered by the Pareto-style rule described in Appendix D, yielding ~21k valid pairs.

Finally, two experts with Master’s degrees curated VCReward-Bench from these valid pairs, selecting pairs whose quality differences are relatively small yet human-resolvable while ensuring diversity in image-instruction prompts. We will add these details to the revised manuscript.


W2: The filtering protocol for VCReward-Bench does not sufficiently isolate visual consistency.

A2: Thanks for the question. We agree that the Pareto-style rule in Appendix D alone does not rule out all-dimension dominance. Beyond it, the two experts conduct a second-stage curation (see A1) that prioritizes subtle yet human-resolvable differences, which further limits cross-dimensional confounding.

To address this concern further, we examine the distribution of preference labels in VCReward-Bench. Among its 3,506 pairs, 3,147 (89.8%) are tied in both IF and VQ, 323 (9.2%) have one preference in IF or VQ, and only 36 (1.0%) prefer the same candidate across all dimensions. All-dimension dominance thus accounts for a very small fraction of the benchmark. In the 3,147-pair subset, IF and VQ are both tied, so any Overall preference can only arise from VC.

We then evaluate the judges on the 3,147-pair subset. The table below shows that our PVC-Judge remains the strongest open-source evaluator, outperforming its base model, Qwen3-VL-8B-Instruct, by 9.79 points and GPT-5.1 by 6.52 points. These results further suggest that PVC-Judge's performance gains are attributable to visual consistency alignment rather than IF or VQ cues.

Gemini 3 ProGPT-5.1EditRewardEditScore Avg@4Qwen3-VL-8B-InstructPVC-Judge
87.83%76.23%64.73%47.12%72.96%82.75%

Thank you for the detailed responses and helpful clarifications. The additional information has largely addressed my concerns, and I am therefore raising my rating to Weak Accept. Good luck!

Reviewer_U3XkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GEditBench v2, an image editing benchmark containing 1,200 examples across 23 tasks, including an open-set category. It also proposes PVC-Judge, an 8B pairwise visual consistency evaluator trained on synthetic preferences constructed through object-centric, human-centric, and VLM-based pipelines. The authors introduce VCReward-Bench to assess agreement with human judgments. Experiments report higher average judgment accuracy than the base model on two meta-evaluation benchmarks and provide a leaderboard of 16 editing models across instruction following, visual quality, and visual consistency.

Critical strengths
  1. Evaluation of unintended changes across a range of editing tasks. The paper evaluates instruction following, visual quality, and visual consistency separately, allowing unintended changes to be examined alongside editing accuracy and image quality. The benchmark includes both predefined tasks and open-set instructions drawn from real-world requests.

  2. Improved average agreement with human judgments. PVC-Judge achieves higher average accuracy than its base model and GPT-5.1 on both the proposed and an external meta-evaluation benchmark. These comparisons provide evidence for the effectiveness of the overall training pipeline under the reported evaluation settings.

Critical weaknesses
  1. A focused ablation would help clarify the contribution of the preference construction design. Section 4.2 combines region decoupling, human-attribute decoupling, and preference filtering. The reported results support the combined pipeline, but the contribution of these design choices remains less clear. A controlled comparison against a simplified construction pipeline, with the key finding summarized in the main text, would strengthen the methodological evidence.

  2. The scope of the generalization claim could be stated more precisely. The external benchmark results provide evidence of transfer beyond the proposed test set. Section 3.2 additionally links pairwise training to mitigating scoring ceilings and generalization to new editing models, while Fig. 2 compares evaluation formats. The main text would benefit from distinguishing these claims and explaining the evidence for each, or qualifying them accordingly. Human-agreement results on open-set instructions would provide further support.

  3. The pair-selection criteria for VCReward-Bench should be summarized in the main text. Section 4.3 introduces the benchmark, but a brief explanation of its selection criteria would help readers interpret the human-agreement results. In particular, it would be useful to state how preferences in instruction following, visual quality, and overall assessment affect pair selection, and whether cases with conflicting preferences are included. This would clarify the scope of the meta-evaluation.

  4. The link between regional metrics and the judge's VC criterion could be clearer in the main text. The color-alteration example in Section 4.2 explains a task-specific metric. Connecting this example to the attributes that should remain unchanged within the edited object, and to the trained judge's decision criterion, would clarify the evaluation scope. A concise description of score aggregation and disagreement handling would also help readers follow the preference construction procedure.

  5. Additional backbone validation and a qualified task-level statement would strengthen the conclusions. The proposed training procedure is evaluated only with Qwen3-VL-8B. Validation with another open-source VLM family, such as InternVL, would help assess whether the gains extend beyond this backbone. Separately, Section 5.1 states that PVC-Judge outperforms its base model on every task, whereas Table 7 reports 75.69% versus 76.23% on portrait beautification. The main-text statement should be qualified, with a brief discussion of task-level exceptions and failure cases.

Questions for authors

None

3 comments
AuthorsOpenReview ↗

Response to Reviewer U3Xk (Part 1 / 2)

We sincerely appreciate the reviewer's detailed and constructive feedback. We are pleased to see that you found our evaluation comprehensive and PVC-Judge more human-aligned. Below, we provide clarifications and additional results addressing each comment.


W1: A focused ablation would help clarify the contribution of the preference construction design.

A1: Thanks for the question. We clarify that region decoupling, human-attribute decoupling, and preference filtering form a sequential pipeline in which each stage provides the prerequisites for the next. To address this concern, we conduct additional stage-wise controlled ablations to validate their contributions respectively.

For region decoupling, a natural simplified alternative is to ask a single VLM to ground the edited region directly. However, we find that it often captures visual differences between the input and output rather than the intended editing target, and tends to produce overly broad bounding boxes for multi-object edits. Parsing the instruction with an LLM before VLM grounding increases target-region grounding success from 72% to 90% on 50 test samples selected from GEditBench v2.

We then evaluate the task-specific metric computation and filtering stage, including human-attribute metrics, on a separate local editing subset of our human-annotated pool (~11k valid pairs, not included in VCReward-Bench). The primary metrics retain ~6k pairs with 91.2% human agreement, versus 70.4% for the unfiltered pool, showing that they effectively separate reliable preference pairs from ambiguous ones for training. Moreover, the auxiliary validations serve as double-checks, and removing them reduces human agreement by 5.3 points.

These results further support the contribution of instruction-guided grounding and metric-based filtering to preference label quality.


W2: The scope of the generalization claim could be stated more precisely.

A2: Thank you for your suggestions. We will precisely clarify the evidence supporting each claim. In Sec. 3.2, lines 212- 215, we provided a conceptual analysis of why the scoring ceiling arises in pointwise paradigm from the perspectives of model training and out-of-distribution adaptation. In addition, the positive results on EditReward-Bench (which already contains outputs of editing models unseen in training, e.g., MagicBrush, gpt-image-1, Gemini-2.5-image-preview) in Fig. 7 provides empirical evidence for PVC-Judge's generalization to new editing models.

To further validate the scoring ceilings, we additionally annotate 630 pairs among three top generators (GPT-Image-1.5, Nano Banana Pro, Seedream 4.5), which lie outside the training distributions of both PVC-Judge and the pointwise EditReward. These pairs are built from 210 image-instruction pairs of VCReward-Bench (10 per task), with those lacking a clear human preference or failing the Appendix D filtering discarded. On the retained 374 pairs, PVC-Judge achieves 78.61% human agreement, ahead of GPT-5.1 (73.26%) and EditReward (60.70%). Compared with their accuracies on VCReward-Bench, EditReward drops by 6.7 points, about twice PVC-Judge's drop (3.21points), consistent with our scoring ceiling analysis of pointwise evaluation on strong unseen generators.


W3: The pair-selection criteria for VCReward-Bench should be summarized in the main text.

A3: Thanks for pointing this out. As described in Appendix D, pairs are filtered by Pareto-dominance rule: a pair is valid only if the preferred image is not worse on Instruction Following (IF), Visual Quality (VQ), and Overall dimensions, and strictly better on Visual Consistency (VC). Therefore, VCReward-Bench contains no pairs with conflicting preferences, which isolates VC discrimination from IF/VQ trade-offs. From the resulting ~21k valid pairs, two experts with Master's degrees selected pairs with relatively small yet human-resolvable quality differences, while ensuring diversity in image-instruction queries. We will summarize these criteria in Sec. 4.3.

AuthorsOpenReview ↗

Response to Reviewer U3Xk (Part 2 / 2)

W4: The link between regional metrics and the judge's VC criterion could be clearer in the main text. The color-alteration example in Section 4.2 explains a task-specific metric. Connecting this example to the attributes that should remain unchanged within the edited object, and to the trained judge's decision criterion, would clarify the evaluation scope. A concise description of score aggregation and disagreement handling would also help readers follow the preference construction procedure.

A4: Thanks for your suggestions. We agree that the link to the VC criterion should be explicit. As defined in line 197, our VC criterion concerns preserving content and attributes that the instruction does not request changing. Regional metrics provide task-specific proxies for this criterion.

In the color alteration example (lines 316-318), LPIPS and CLIP-EMD assess perceptual and semantic preservation outside the edited region, while lightness-channel SSIM and DINO similarity assess structural and layout preservation within the edited object. These metric-derived preferences train PVC-Judge to prefer outputs that better preserve the attributes these metrics capture. The full task-to-metric mapping appears in Appendix C.1.

The score aggregation and conflict handling are described in Sec. 4.2 Score-based Robust Preference Synthesis: primary metrics are z-score normalized per (Iin,Inst)(I_{in}, Inst) group, top/bottom 30% candidates become Winners/Losers, pairs are kept under regional Pareto dominance and auxiliary majority voting, and pairs with primary–auxiliary conflicts are discarded. We will revise this paragraph into a step-by-step description for clearer presentation.


W5: Additional backbone validation and a qualified task-level statement would strengthen the conclusions.

A5: Thanks for pointing this out. We chose Qwen3-VL-8B because it is one of the most widely used open-source VLMs, and its 8B scale offers a sweet spot between multimodal capability, training cost, and deployment efficiency. Importantly, our preference construction pipelines and pairwise training objective are architecture-agnostic and do not rely on backbone-specific components. While we acknowledge that validation on another VLM family would further broaden the empirical evidence, reproducing the full training and evaluation pipeline under matched settings would exceed our current computational budget. We will release the pipeline and training code to facilitate evaluation on other backbones.

We thank the reviewer for noting this wording inconsistency in Sec. 5.1. We will revise the statement to: PVC-Judge outperforms its base model on 20 of the 21 tasks, with portrait beautification as the exception (−0.54%).

Thank you for the detailed and constructive response. The authors have provided helpful clarifications and additional experimental results that address most of my concerns, particularly regarding the preference construction pipeline, generalization evaluation, and benchmark design. Although the additional backbone validation remains limited, the authors have acknowledged this limitation and appropriately qualified their claims. Overall, I appreciate the authors' efforts, and I would like to maintain my original score.

Reviewer_iNsFOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces GEditBench v2, a benchmark with 1,200 examples covering 23 image editing tasks. It also proposes PVC-Judge, an 8B pairwise visual consistency evaluator trained with region-based metrics and VLM-generated preferences, and VCReward-Bench for evaluating agreement with human judgments. The paper benchmarks 16 editing models across instruction following, visual quality, and visual consistency.

Critical strengths
  • The benchmark covers a broad and useful range of tasks, including identity reference, chart editing, text translation, hybrid edits, and open-set requests. This makes the evaluation relevant to practical image editing.
  • The proposed open-source PVC-Judge is a useful contribution. It improves over its base model and GPT-5.1 on both reported VC evaluation sets. The human preference evaluation and the strong overall correlation with Arena rankings provide encouraging evidence for the approach.
  • The region-based preference construction is well motivated. Separating edited and non-edited regions, with task-specific checks for identity and structure, provides a practical way to distinguish intended edits from unwanted changes.
Critical weaknesses
  • The trade-off between pairwise and pointwise evaluation is not fully examined. Pairwise judgments show higher human agreement, but obtaining a model ranking requires selecting comparison partners and fitting scores. The paper does not quantify the cost of adding a new model or compare ranking reliability under a matched evaluation budget.
  • Validation on close comparisons between strong editing models is limited. The training candidates come from BAGEL, Kontext, Step1X-Edit, and Qwen-Image-Edit variants, without outputs from the leading proprietary editors evaluated in the main leaderboard. It is therefore unclear whether the learned judge can reliably distinguish the smaller quality differences between stronger models. Aggregate accuracy does not resolve this concern, and the human annotation protocol lacks details on annotator agreement and disagreement resolution.
  • Although Gemini 3 Pro and GPT-5.1 are compared for VC, the main leaderboard still uses GPT-4o for IF and VQ. Without a comparison using stronger judges for these dimensions, the stability of the rankings, especially among the top models, remains unclear.
Questions for authors
  • What comparison protocol and evaluation budget are needed to add a new model to the leaderboard? Does pairwise evaluation still improve ranking reliability over pointwise evaluation under a matched budget?
  • How many annotators evaluated each pair, how were disagreements resolved, and what was the inter-annotator agreement? How well does PVC-Judge agree with humans on close comparisons between the strongest generators?
  • How stable are the main rankings, especially among the top models, when GPT-4o is replaced by Gemini 3 Pro, GPT-5.1, or a newer MLLM for IF and VQ?
  • The preference construction pipeline uses multiple models and task-specific metrics. Which components are necessary, and how does this approach compare with using a single strong VLM to label all preference pairs, in terms of human agreement and annotation cost?
4 comments
AuthorsOpenReview ↗

Response to Reviewer iNsF (Part 1 / 3)

We sincerely thank the reviewer for the detailed and constructive feedback. We are pleased to see that you found our benchmark comprehensive and practically relevant, PVC-Judge useful and human-aligned, and our region-based preference construction pipeline well motivated. Below, we provide detailed responses to all comments.


W1/Q1: What comparison protocol and evaluation budget are needed to add a new model to the leaderboard? Does pairwise evaluation still improve ranking reliability over pointwise evaluation under a matched budget?

A1: Protocol and budget for adding a new model. Thanks for the question. Under the exhaustive protocol used for our leaderboard, a new model is compared with all NN existing models on Q=1,200Q=1,200 queries for each dimension, after which we refit the Elo scores while reusing all previous comparisons. The incremental cost is NQNQ judgments per dimension, i.e., 19,200 judgments when adding a model to the existing 16-model leaderboard. VC evaluation with PVC-Judge takes ~4 GPU-hours on a single A100-80G, with batch size=1. The score fitting with Bradley-Terry model is lightweight and usually takes ~10 minutes on a single CPU, so the evaluation cost is dominated by the pairwise judgments.

We acknowledge this increasing cost, which we have discussed as a limitation in Appendix A. To reduce it, we plan to explore anchor-based comparisons and hierarchical matchups in future work.

A2: Matched budget comparison. For a fair comparison, we use EditScore, a pointwise evaluator built on the same Qwen3-VL-8B backbone as PVC-Judge, and increase its self-ensemble budget KK from 4 to 16 per image, matching the incremental N=16N=16 pairwise judgments per query in A1. The results on VCReward-Bench and EditReward-Bench below show that EditScore's average human agreement accuracy improves from 53.52% to 57.15%, with diminishing gains (K=4→8K=4\to8: +2.28 points, K=8→16K=8\to16: +1.35 points), while PVC-Judge achieves 82.13%.

MethodEditReward-BenchVCReward-BenchAvg. Accuracy
EditScore Avg@457.8349.2053.52
EditScore Avg@860.1251.4755.80
EditScore Avg@1661.9352.3657.15
PVC-Judge82.4481.8282.13

For ranking reliability, we use EditScore's Semantic Consistency scores (averaged over KK forward passes) of the two candidates to obtain pairwise VC results, while keeping the IF and VQ results unchanged. We then fit the Bradley-Terry model and Elo ratings in the same way as PVC-Judge. Under the same budget, the PVC-Judge ranking achieves Spearman's ρ\rho=0.929 with the Arena ranking, compared with 0.865 for EditScore Avg@16. These results suggest that pairwise evaluation provides both stronger human alignment and more reliable rankings under a matched budget.

AuthorsOpenReview ↗

Response to Reviewer iNsF (Part 2 / 3)

W2/Q2: How many annotators evaluated each pair, how were disagreements resolved, and what was the inter-annotator agreement? How well does PVC-Judge agree with humans on close comparisons between the strongest generators?

A3: Annotation details. Thanks for pointing this out. Our annotation team comprises 18 trained annotators, and each pair is independently labeled by 2 annotators on all four dimensions (IF, VQ, VC, and Overall). Disagreements are first adjudicated by a senior expert, with a second PhD-level expert reviewing unresolved edge-cases. Pairs are discarded if consensus cannot be reached.

We measure inter-annotator agreement on daily spot checks. Batches with agreement below 90% are relabeled. The average daily spot-check agreement was 93.6%. The annotated pool (covers 4,388 input prompts, ~65k pairs) is then filtered by the Pareto-style rule described in Appendix D, yielding ~21k valid pairs.

Finally, two experts with Master's degrees curated VCReward-Bench from these valid pairs, selecting pairs whose quality differences are relatively small yet human-resolvable, while ensuring diversity in image-instruction prompts. We will add these details to the revised manuscript.

A4: Close comparison between the strongest generators. As detailed in A3, VCReward-Bench is explicitly curated toward pairs with small quality differences, so PVC-Judge's performance on it (81.82% vs. 76.89% for GPT-5.1) already reflects high agreement on close comparisons.

To directly address the concern, we additionally annotate 630 pairs among top-3 generators unseen in training (GPT-Image-1.5, Nano Banana Pro and Seedream 4.5), built from 210 randomly sampled image-instruction pairs from VCReward-Bench (10 per task). Pairs without a clear human preference or failing the Appendix D filtering are discarded. On the retained 374 valid pairs, PVC-Judge achieves 78.61% human agreement, ahead of EditReward (60.70%) and GPT-5.1 (73.26%). These results further validate that PVC-Judge remains reliable on close comparisons among the strongest generators.


W3/Q3: How stable are the main rankings, especially among the top models, when GPT-4o is replaced by Gemini 3 Pro, GPT-5.1, or a newer MLLM for IF and VQ?

A5: Thanks for the question. To address the concern, we further evaluate IF and VQ with GPT-5.1 for the top-6 models (Nano Banana Pro, Seedream 4.5, GPT-Image-1.5, FLUX.2 [klein] 9B, Qwen-Image-Edit 2511 and FLUX.2 [dev] Turbo) in GEditBench v2, covering all 15 model pairs, and refit Elo while retaining PVC-Judge for VC. GPT-4o and GPT-5.1 rankings are highly consistent: Spearman's ρ\rho are 0.943 for IF, 0.886 for VQ, and 0.943 for Overall, with a maximum shift of only one rank. The Overall top three and their order remain unchanged, validating the stability of the main rankings.

AuthorsOpenReview ↗

Response to Reviewer iNsF (Part 3 / 3)

Q4: The preference construction pipeline uses multiple models and task-specific metrics. Which components are necessary, and how does this approach compare with using a single strong VLM to label all preference pairs, in terms of human agreement and annotation cost?

A6: Necessity of components. Our preference construction pipeline mainly consists of two components: (i) Task-Adaptive Spatial Decoupling, which combines an LLM and a VLM to localize the target edit region, and (ii) Region-Specific Metric Ensemble, which scores the decoupled regions with primary metrics and auxiliary validations. We examine the necessity of each below.

(1) LLM-VLM combination is necessary for grounding accuracy: In our experiment, a single VLM asked to ground the edit directly tends to return regions that differ between input and output rather than the intended target, and its bounding boxes usually over-extend for multi-object edits. Parsing the instruction with an LLM first raises target-region success from 72% to 90% on 50 test samples selected from GEditBench v2.

(2) Multiple task-specific metrics are necessary for quality control: As discussed in Sec. 4.2, the primary metrics serve as the criterion of rejection sampling for constructing high-confidence preference pairs, which provide a stable training signal. To further validate this, we evaluate the effect of these metrics on the remaining local editing subset of our human-annotated pool (~11k valid pairs, not included in VCReward-Bench). The primary metrics retain ~6k pairs with 91.2% human agreement, versus 70.4% for the unfiltered pool, showing that they effectively separate reliable preference pairs from ambiguous ones for training. Moreover, the auxiliary validations serve as double-checks, and removing them reduces human agreement by 5.3 points.

A7: Comparison with a single VLM.

(1) Human Agreement: On the local editing subset of VCReward-Bench in Table 7, PVC-Judge improves over its base model by 9.19 points (71.67% -> 80.86%) and surpasses GPT-5.1 (74.21%), suggesting that our construction pipeline could yield a higher-quality preference signal than a single strong VLM, like GPT-5.1. For a direct comparison, we further evaluate GPT-5.1 on the same ~6k retained pairs in A6(2): GPT-5.1 reaches 80.1% human agreement, below our pipeline (91.2%).

(2) Annotation Cost: Our pipeline uses only small open-source models, including Qwen3-4B and Qwen3-VL-8B, together with lightweight algorithms, and runs on a single A100-80G GPU, whereas a strong VLM such as Qwen3-VL-235B-A22B needs 8xH100 GPUs, and GPT-5.1 costs approximately 0.003 USD per pair (~11k USD for our ~180k instructions × C72\times\ C_7^2 candidate pairs). Moreover, our metrics ensemble strategy enables rejection sampling for quality control at no extra cost, whereas a single VLM typically relies on a self-consistency strategy to estimate uncertainty, at 8~16x the inference cost.

Reviewer_iNsFOpenReview ↗

Official Comment by Reviewer iNsF

Thank you for the detailed response, which addresses my concerns. I hope GEditBench v2 will facilitate the evaluation of rapidly evolving leading image models, and be easy for the research community to use. I maintain my positive rating of the submission.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank You and Summary of Revisions

We sincerely thank all reviewers and the Area Chair for the time and care they put into reviewing our paper and rebuttal. We appreciate the reviewers' recognition of the importance of evaluating visual consistency as a dedicated dimension, the broad task coverage of GEditBench v2, the practical value of VCReward-Bench for meta-evaluation, and the effectiveness of PVC-Judge. Their constructive feedback helped us further improve the clarity and rigor of the paper.

We have addressed each reviewer's comments in the individual responses and incorporated the corresponding clarifications and discussions into the revised manuscript. Below, we briefly summarize our updates in the revised PDF, with all changes highlighted in blue.

  • Main Text: We clarified the link between the regional metrics and the VC criterion (lines 317-320; U3Xk), numbered the four steps of preference synthesis (lines 345-355; U3Xk), added details on VCReward-Bench's annotation and pair selection procedures (lines 397-406; iNsF, U3Xk, B4C8), and stated PVC-Judge's performance and generalization claims more precisely (lines 427-431 and lines 451-452; U3Xk).
  • Appendix B.3: We described the protocol and budget for adding a new model to the leaderboard (page 20, lines 1060-1070; iNsF).

We once again thank all reviewers for their time, insightful comments, and constructive suggestions, which have greatly helped us improve our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.