- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces GEditBench v2, a benchmark for general instruction-guided image editing comprising 1,200 samples across 22 predefined editing tasks and an open-set category containing 100 real-world instructions. To address the difficulty of automatically evaluating visual consistency—the preservation of non-target content, identity, and structure—the authors propose PVC-Judge, a pairwise evaluator fine-tuned from Qwen3-VL-8B. Its training preferences are constructed using object-centric and human-centric region-decoupled metrics, supplemented by VLM-based annotations for global editing tasks. The authors also introduce VCReward-Bench, an expert-annotated preference benchmark for measuring agreement between visual consistency evaluators and human judgments. Experiments report that PVC-Judge outperforms existing open-source evaluators and achieves higher average accuracy than GPT-5.1 on this benchmark.
Critical strengths
-
Important and timely evaluation problem. As image editing models become increasingly capable, distinguishing successful instruction execution from the preservation of content outside the intended editing scope becomes more important. Treating visual consistency as a separate evaluation dimension addresses a meaningful practical need.
-
Broad and practically relevant task coverage. Beyond common local, global, and reference-guided edits, the benchmark includes chart editing, in-image text translation, enhancement, and hybrid editing. The open-set category further extends evaluation beyond a closed taxonomy and introduces more challenging, unconstrained instructions.
-
Human-annotated meta-evaluation. In addition to developing an evaluator, the authors construct VCReward-Bench using expert preference annotations. Evaluating agreement with human judgments provides a more direct basis for assessing evaluator reliability than relying solely on automatically synthesized preferences.
Critical weaknesses
-
Insufficient detail on the reliability of human annotations. Although Appendix D describes the annotation interface, evaluation dimensions, and available preference options, it does not adequately specify the number and qualifications of annotators, the number of independent annotations per pair, the label aggregation or disagreement-resolution procedure, or inter-annotator agreement statistics. Since VCReward-Bench provides central evidence for the claimed human alignment of PVC-Judge, these omissions make it difficult to assess the reliability of the benchmark labels and the credibility of the reported performance improvements.
-
The filtering protocol does not sufficiently isolate visual consistency and may introduce cross-dimensional confounding. Appendix D requires the VC-preferred candidate to be no worse than the other candidate in IF, VQ, and Overall. However, this rule still permits pairs in which one candidate is preferred across all dimensions. An evaluator could therefore potentially predict the VC label using instruction following, visual quality, or overall preference alone. Consequently, the current results do not sufficiently substantiate the claim that PVC-Judge is specifically aligned with human judgments of visual consistency.
Questions for authors
See Weaknesses
2 comments
Thank you for the detailed responses and helpful clarifications. The additional information has largely addressed my concerns, and I am therefore raising my rating to Weak Accept. Good luck!
Response to Reviewer B4C8
We sincerely appreciate the reviewer for the detailed and constructive feedback. We are glad that you found the visual consistency evaluation problem important and timely, our task coverage broad, and our meta-evaluation practical.
We hope our responses address your concerns, and we would be grateful if you would reconsider your rating in light of them. Below, we provide detailed responses to all comments.
W1: Insufficient detail on the reliability of human annotations.
A1: Thanks for pointing this out. Our annotation team comprises 18 annotators, all trained using a detailed manual with annotation guidelines and representative examples. The manual is iteratively refined as disagreements arise. Each pair is independently labeled by 2 annotators on IF, VQ, VC, and Overall dimensions.
Labels agreed upon by both annotators are accepted directly. Disagreements are first adjudicated by a senior expert, with a second PhD-level expert reviewing unresolved edge-cases. Pairs are discarded if consensus cannot be reached.
For inter-annotator agreement, we spot-check agreement between the two annotators on a daily sample of pairs. Batches with agreement below 90% are relabeled. The average daily spot-check agreement was 93.6%. The annotated pool (covers 4,388 input prompts, ~65k pairs) is then filtered by the Pareto-style rule described in Appendix D, yielding ~21k valid pairs.
Finally, two experts with Master’s degrees curated VCReward-Bench from these valid pairs, selecting pairs whose quality differences are relatively small yet human-resolvable while ensuring diversity in image-instruction prompts. We will add these details to the revised manuscript.
W2: The filtering protocol for VCReward-Bench does not sufficiently isolate visual consistency.
A2: Thanks for the question. We agree that the Pareto-style rule in Appendix D alone does not rule out all-dimension dominance. Beyond it, the two experts conduct a second-stage curation (see A1) that prioritizes subtle yet human-resolvable differences, which further limits cross-dimensional confounding.
To address this concern further, we examine the distribution of preference labels in VCReward-Bench. Among its 3,506 pairs, 3,147 (89.8%) are tied in both IF and VQ, 323 (9.2%) have one preference in IF or VQ, and only 36 (1.0%) prefer the same candidate across all dimensions. All-dimension dominance thus accounts for a very small fraction of the benchmark. In the 3,147-pair subset, IF and VQ are both tied, so any Overall preference can only arise from VC.
We then evaluate the judges on the 3,147-pair subset. The table below shows that our PVC-Judge remains the strongest open-source evaluator, outperforming its base model, Qwen3-VL-8B-Instruct, by 9.79 points and GPT-5.1 by 6.52 points. These results further suggest that PVC-Judge's performance gains are attributable to visual consistency alignment rather than IF or VQ cues.