- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents VIEScore2, a unified evaluator for generated and edited images. The model predicts perceptual quality and semantic consistency scores, as well as spatial locations of artifacts and prompt–image misalignment. The authors convert heterogeneous localization annotations into a common 16 × 16 grid format and train the model with supervised fine-tuning followed by GRPO. They evaluate its agreement with human judgments and its defect localization performance across several benchmarks.
Critical strengths
- The joint treatment of overall scoring and defect localization is useful. The localized outputs also make the scores easier to inspect.
- The common grid format offers a practical way to train on datasets with different types of spatial annotations.
Critical weaknesses
- The paper presents VIEScore2 as a potential reward model, but does not directly test whether using it as a reward improves a generative model.
- The results do not yet make clear how well its scoring and localization generalize to the defects produced by recent generators. Although SDG-30K includes newer models, a generator-level breakdown would be helpful.
Questions for authors
-
Since reward modeling is an important potential use of VIEScore2, could the authors conduct a small-scale experiment using it as the reward in one training method, such as Flow-GRPO, DanceGRPO, or DiffusionNFT? The key question is whether optimizing with VIEScore2 improves generated images under an independent evaluation. If feasible, a comparison with an established reward-model baseline under the same generator and training budget would make the result easier to interpret, but even a limited experiment would be informative.
-
Many images in the training datasets come from earlier generators, and Table 14 largely focuses on older models. As generation and editing models improve, the types and visibility of artifacts may change. Could the authors report results separately for the newer generators already represented in SDG-30K and, if feasible, conduct a small additional evaluation on models such as FLUX.2 [klein] or Z-Image? I see this as a generalization question rather than a requirement for a large new benchmark.
-
I found two examples in Figure 3 difficult to interpret. In the SynthScars example, the annotated region appears to cover only part of the unclear text, whereas the GPT-5.6-sol prediction covers more of the text. In the SDG-30K example, the stated asymmetry in the porcelain figure’s facial gold inlays is hard to discern even when enlarged. Could the authors show the original annotations for these cases and check whether similar annotation ambiguities occur elsewhere in the training or test data?
5 comments
Response to Authors
Thank you for the detailed responses and additional experiments, which have addressed my main concerns regarding reward modeling and generalization to recent generators.
Regarding Q3, I would like to further discuss how defect identification aligns with human preferences. In my practical use of image generation models, the subtle asymmetry in the SDG-30K porcelain figure in Figure 3 seems acceptable: when using “symmetrical face,” I might intend broadly similar patterns on both sides rather than strict mirror symmetry. In contrast, the unclear text in the SynthScars example seems less acceptable, and I would prefer the localization to cover the entire defective text region. These examples suggest a possible distinction between matching dataset annotations and identifying defects that users actually care about. While these are only a few examples and reflect my personal preferences, they leave me uncertain about this aspect of VIEScore2. Could the authors clarify what evidence supports its alignment with human judgments of defect presence, extent, and importance, beyond overall score correlation and agreement with existing spatial annotations?
We thank the reviewer for the follow-up. We agree that agreement with benchmark annotations does not necessarily imply the detection of perceptually meaningful defects. To address this, we conducted a user study evaluating defect validity, perceived importance, and localization quality.
User Study. We sampled 30 images: 24 from four held-out benchmarks (SynthScars, SDG-30K, HAD, and AbHuman; 6 each) and 6 from in-domain test sets (RichHF and ImagenWorld; 3 each). 8 raters independently evaluated VIEScore2's predictions and benchmark annotations, presented in randomized, unlabeled order.
| VIEScore2 | Annotation | Krippendorff's α | |
|---|---|---|---|
| Q1: Defect presence | 63% | 60% | 0.37 |
| Q2: Perceived importance | 62% | 63% | 0.02 |
| Q3: Full coverage / Over-localization | 38% / 34% | 43% / 24% | 0.17 |
| Q4: Missed defects | 20% | 24% | 0.20 |
| Q5: Preference (decided pairs) | 53% | 47% | 0.14 |
At least seven of eight raters agree on defect presence for 30 of 60 markings (12 unanimously).
Takeaway. VIEScore2 matches benchmark annotations on defect presence and importance and misses fewer defects, at the cost of slightly broader regions; raters agree on whether a defect exists but not on how much it matters (α ≈ 0), so importance is inherently subjective. Consistently, on RichHF's graded human heatmaps, VIEScore2's recall rises from 0.24 to 0.66 with defect intensity (GPT-5.6-sol: 0.39 to 0.50).
Regarding the two examples:
- Porcelain figure: VIEScore2 identifies deviations from the prompt's explicit "symmetrical face", consistent with the SDG-30K annotation. Whether such asymmetry is acceptable is subjective, as the low importance agreement above shows.
- SynthScars: VIEScore2 covers the full text width but only its lower part, because the text spans two grid rows and the annotation converted to 16×16 keeps only the bottom row. This under-coverage is an artifact of the grid label rather than typical behavior: in the user study, VIEScore2 leaves defects partially covered or missed less often than the benchmark annotations (28% vs. 33%).
We appreciated the reviewer's concrete suggestions and ran each of the requested experiments.
New samples from recent generators (used in Q1 and Q2). We generated 8 images for each of the 200 DrawBench prompts with Z-Image-Turbo and FLUX.2-klein-4B, and scored every image with VIEScore2, PickScore, and HPSv2.1. For each prompt, we compare the image VIEScore2 scores highest, or lowest, with a random one of the 8:
On both generators, neither of which was seen in training, the image VIEScore2 scores highest has a higher PickScore than a random one, and the image it scores lowest has a lower one. On these recent generators, VIEScore2’s ordering therefore agrees with an independent preference model.
VIEScore2 as a reward signal (W1, Q1).
Using VIEScore2 to train generators lies beyond the scope of the submission, which focuses on evaluation. Following the reviewer’s suggestion, we nevertheless ran two small-scale experiments, and both results are encouraging.
(a) Best-of-N selection. On the samples above, choosing the image VIEScore2 scores highest improves PickScore over random selection on both generators. Within a prompt, VIEScore2 correlates only weakly with PickScore and HPSv2.1 (Spearman 0.09–0.16), so it adds a complementary signal.
(b) RL fine-tuning (new experiment). We fine-tuned FLUX.2-klein-base-4B with DiffusionNFT (LoRA, 160 steps), using VIEScore2’s overall score or PickScore as the reward under the same training budget, and evaluated on DrawBench with metrics that neither reward uses. As in DiffusionNFT, all models are sampled without CFG, which is why the base model has a negative ImageReward.
VIEScore2 improves every metric over the base model and matches or slightly exceeds PickScore on all four. This includes HPSv2.1 and ImageReward, which are preference models of the same kind as PickScore.
Recent generators (W2, Q2).
VIEScore2's grid IoU on SDG-30K by generator (same protocol as Table 1):
On every generator, VIEScore2 ranks second to fourth among the 12 methods of Table 1, above all closed models and all prior trained evaluators except SDG, and GRPO helps on all four. SDG, whose detector is trained on SDG-30K, is first. Compared with SANA-1.5, the earliest of the four, VIEScore2's grid IoU on the three later generators is 0.015 lower on average. The other methods change by −0.032 to +0.007 over the same split, so this difference is not specific to VIEScore2.
FLUX.2 [klein] and Z-Image. The samples at the top come from the two models the reviewer suggested. They have no defect labels, but VIEScore2's score still orders them sensibly (table at the top), and it localizes visible defects such as misspelled letters.
Unseen sources. A new leave-one-dataset-out experiment (LoRA SFT on a 15% subsample; full table in our response to Reviewer AAhw) shows that VIEScore2 transfers partially to a source it never sees: on ImagenWorld editing, grid IoU is 0.167 without any ImagenWorld training data, against 0.296 with it.
Figure 3 annotations (Q3).
We thank the reviewer for pointing out these ambiguities. Both examples use the datasets' original annotations, converted to our evaluation format. The original annotation covers the entire banner. Figure 3 displays its 16×16 grid conversion, where cells with less than 50% overlap are excluded. Against the original polygon, VIEScore2 still achieves higher pixel IoU than GPT-5.6-sol (0.54 vs. 0.40), whose prediction extends beyond the annotated region. SDG-30K. The annotation identifies a subtle prompt mismatch: the requested "symmetrical face" conflicts with the asymmetric gold inlays. While difficult to discern, VIEScore2 localizes the annotated region (grid IoU 0.61). We will include the original annotations alongside their grid conversions in the revised figure and clarify the ambiguity of these examples.
We hope these experiments and results have adequately addressed reviewer’s concerns, and we sincerely appreciate the reviewer’s consideration of these improvements in their final evaluation.