- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how annotation protocols should be matched to different text-to-image evaluation skills. It introduces an 18-skill taxonomy and conducts controlled comparisons for three protocol families: brush-based annotation versus global Likert ratings for visual artifacts; word-level annotation with gap tokens versus BQA and Likert ratings for text rendering; and reference-assisted annotation versus unassisted BQA for reference-dependent skills. Each controlled study uses 30 prompts, three image-generation models, and the same six annotators. The paper also presents a 43-prompt evaluation of four generation models and an automated implementation using existing VLMs and a pretrained artifact detector.
Critical strengths
-
The paper addresses a meaningful evaluation-design problem through controlled comparisons. Treating annotation design as an experimental variable is useful, and keeping the images and annotator panel fixed makes the observed differences informative. In particular, the increase in artifact annotation agreement from α = 0.39 to 0.82 provides evidence that the reporting mechanism substantially affects reproducibility.
-
The proposed protocols provide actionable information beyond a global judgment. Word-level labels and gap tokens identify correct, missing, and inserted text, while brush masks localize perceived defects. Reference images also address a concrete knowledge bottleneck, reducing “Unsure” responses from 63% to 2.5%. These results support the practical value of more explicit annotation tasks.
Critical weaknesses
-
The strongest result establishes improved consistency, but does not establish that the resulting score preserves the intended evaluation target. In Section 4.1, the Likert baseline measures perceived realism and the absence or severity of artifacts, whereas the brush-based score measures the fraction of image area marked as defective. These quantities need not reflect the same quality differences: a small but conspicuous facial or anatomical defect may have a substantial perceptual impact while receiving only a small area penalty. The higher α therefore supports more reproducible annotation of affected area, but does not by itself establish a better measure of artifact quality. This matters for the interpretation of Table 2, where artifact scores of 0.995–0.998 are taken as evidence that recent models have largely closed the artifact-suppression gap. The current evidence does not distinguish genuine quality saturation from limited sensitivity of the area-based score. This is the principal gap between the reported reliability improvement and the broader model-evaluation conclusions.
-
The independence of the human reference used to validate automatic evaluation is unclear. Appendix C, page 18, states that BQA answers in the full evaluation were pre-filled by a VLM and then confirmed or corrected by annotators. If Figure 6 uses these annotations as its human reference, the reported correlations may be affected by anchoring to the suggested answers. The manuscript does not identify the pre-filling model, quantify how often suggestions were corrected, or establish whether the correlations persist against an unassisted human reference. This concern is limited to the affected BQA results in the full evaluation; the controlled protocol comparisons in Section 4 explicitly excluded such suggestions and remain informative.
-
The limited scale and coverage leave practical generalizability and annotation efficiency insufficiently established. Each controlled study uses only 30 prompts, three generation models, and a fixed panel of six annotators. The full evaluation contains 43 prompts, with some skills represented by only three or four prompts (Table 6). These experiments support the observed agreement gains under the studied conditions, but provide limited evidence that the gains persist across the broader prompt distributions and annotator populations encountered in large annotation projects. Section 4.4 motivates the sample size by the stabilization of reliability estimates; this does not establish broader coverage or generalizability. Furthermore, the claim that finer-grained recording requires only a few additional interactions is not supported by measured annotation time. Stabilization with fewer annotators therefore does not, by itself, demonstrate lower total annotation cost. The unresolved issue is whether the protocols retain their benefits at a comparable human annotation budget in a broader setting.
Questions for authors
-
Artifact-score validity: clarification and additional experiment. What is the intended construct of the brush-derived scalar score: affected area, perceived artifact severity, or overall artifact quality? To support its use for the latter two, please evaluate both the Likert and brush-derived scores on a held-out set of matched images with localized defects introduced or repaired. An independent panel, blinded to the protocol outputs, should adjudicate the direction of improvement and relative artifact severity. The set should include small but perceptually salient defects. Reporting how well each protocol recovers these independently established differences, with uncertainty estimates, would directly test whether the consistency gain preserves meaningful quality information. If the intended target is affected area alone, the interpretation of Table 2 should be narrowed accordingly.
-
Human-reference independence: clarification and additional experiment. Does Figure 6 use the VLM-assisted BQA annotations described in Appendix C? Please identify the pre-filling model and report the proportion of suggestions that annotators corrected. Please also obtain unassisted annotations from independent annotators for a small subset of the same BQA items, then compare the Figure 6 correlations against assisted and unassisted references on those same items, with uncertainty estimates. This would directly establish whether the automatic-evaluation conclusions remain supported without suggested answers.
-
Generalizability and annotation cost: additional experiment. Please repeat the key comparisons with a new, independent group of annotators to assess whether the agreement gains generalize beyond the original six-person panel. Please measure annotation time per image and total human effort, and compare the protocols under matched total annotation-time budgets. Evaluating both agreement and performance against the independent quality criterion requested above would establish whether the protocols provide better evaluation signals per unit of human effort. This focused extension would be more informative than simply collecting additional ratings from the existing six-person panel.
4 comments
tRqa-2: independence of the Figure 6 human reference (W2, Q2)
"Does Figure 6 use the VLM-assisted BQA annotations described in Appendix C? Please identify the pre-filling model and report the proportion of suggestions that annotators corrected."
Yes. For the yes/no skills, Figure 6 uses the full-suite annotations, which were collected with the suggestions on screen. Suggestions were shown only for binary yes/no questions in Section 5, never to the skill-aligned protocols in Section 4 and never for brush, word-level, anchor or aesthetics annotations. The pre-fill model was gpt-5-mini, which is not one of the judges in Figure 6 (GPT-5, Molmo2-8B, Qwen3.5-9B), so no judge is compared with its own suggestions. Appendix C will name it.
Override rate. Of 6,090 answers given with a suggestion on screen, annotators confirmed 87.0%, reversed 7.9% and set 5.1% to Unsure. Reversals were concentrated on "No" suggestions: annotators reversed 35.8% of the suggestions that said "No", against 2.6% of those that said "Yes". The human majority differs from the suggestion on 6.7% of the 991 questions.
Why suggestions. Binary questions about objects, attributes, counts and relations are easy for current VLMs, so a suggestion mostly saves the annotator a click while the decision stays with the human. The override pattern agrees: even with a small pre-fill model (gpt-5-mini), most suggestions were confirmed, and correction rates were highest on the more challenging skills (camera 20%, view 18%, action 12%).
Evidence against anchoring.
- The human majority agrees with the suggestions on 93.3% of questions, less than with Molmo2-8B (95.1%) and Qwen3.5-9B (96.1%), which the annotators never saw. GPT-5, the judge closest to the pre-fill model, is at 93.4%, so it is not favoured either.
- Where humans reversed a suggestion, Molmo2 and Qwen side with the humans 80% and 78% of the time.
- On the pre-filled skills, gpt-5-mini's own correlation with the human reference (mean ρ 0.58) is no higher than GPT-5's (0.60) or Qwen's (0.62). A reference anchored to it would favour it.
Unassisted annotations. We could not collect unassisted annotations of these items within the response period. The three checks above address the same question with the existing data: a human reference anchored on the suggestions would favour the pre-fill model and the judges that resemble it, and it does not.
Pre-filled and mixed skills in Figure 6. Mean correlations for GPT-5 / Molmo2 / Qwen are 0.60 / 0.45 / 0.62 on pre-filled yes/no skills and 0.63 / 0.60 / 0.55 on skills that mix yes/no with anchor-Likert or word-level questions. The revised figure will give N and a bootstrap interval for every bar.
tRqa-3: generalization and annotation cost (W3, Q3)
"Please repeat the key comparisons with a new, independent group of annotators ... Please measure annotation time per image and total human effort, and compare the protocols under matched total annotation-time budgets. Evaluating both agreement and performance against the independent quality criterion requested above would establish whether the protocols provide better evaluation signals per unit of human effort."
New independent annotators. The comparisons of Sections 4.1 to 4.3 were repeated with annotators who had no part in the original study.
| Skill | Baseline α, original | Baseline α, new panel | Skill-aligned α, original | Skill-aligned α, new panel |
|---|---|---|---|---|
| Artifacts: Likert vs brush (ranks) | 0.39 | 0.51 | 0.61 | 0.65 |
| Artifacts: Likert vs brush (as in the paper) | 0.39 | 0.51 | 0.82 | 0.67 |
| Text: yes/no vs word-level, each word | 0.63 | 0.60 | 0.96 | 0.92 |
| Reference: from memory vs with anchors† | 0.47 | 0.50 | 0.57 | 0.56 |
| Reference: share of Unsure answers† | 63.0% | 52.3% | 2.5% | 2.3% |
† Only four of the five crowd annotators completed Study 3b.
The ordering of protocols is the same in every family, and the two panels' consensus scores correlate at 0.77 to 0.96.
Time. The application logged the time on screen for every image.
| Comparison | Median s per image, baseline | Median s per image, ours | Mean s per image, baseline | Mean s per image, ours |
|---|---|---|---|---|
| Artifacts: Likert vs brush | 7 | 15 | 19 | 46 |
| Text: Likert vs word-level | 7 | 6 | 11 | 20 |
| Reference: from memory vs with anchors† | 6 | 6 | 10 | 9 |
† Only four of the five crowd annotators completed Study 3b.
Anchor annotation costs the same as its baseline, so its gain is free. Word-level annotation costs the same on the typical image and more on average (20 against 11 seconds per image). Brush annotation costs about twice a rating.
Matched budgets for artifacts. Agreement is a property of the protocol and barely moves with panel size: Likert α is 0.49 with two annotators and 0.51 with five, brush 0.68 and 0.67. At a matched budget of about 95 annotator-seconds per image, that is five Likert annotators (α 0.51) or two brush annotators (α 0.68). Against the independent severity criterion the two scores perform alike (table under tRqa-1), and the brush record adds localization.
Scope. Section 4.4 states that 30 prompts were chosen because the intervals of baseline and skill-aligned protocols separate at that size for artifacts and text rendering, which supports the comparison and is not a claim of broad coverage. Coverage of other prompts and models comes from Section 5 (43 prompts, two further models), and coverage of other annotators from the replication above.
Thank you for your response; I now have a clearer understanding of the paper, and I will increase my score by one point.
We thank the reviewer for recognising the value of treating annotation design as an experimental variable, the informative fixed-panel comparison and the actionable information our protocols provide, and for three precise requests. We ran the two experiments that could be completed in the response period: a blinded independent panel that judges relative artifact severity (tRqa-1), and a replication of the key comparisons with new annotators and measured time (tRqa-3). Both used the same application and images as Section 4, with no AI suggestions.
tRqa-1: what the brush score measures (W1, Q1)
Construct. The scalar is affected area: where the artifacts are and how much of the image they cover. The claim of the protocol is that marking what one sees gives a more reproducible record than compressing it into one number, and that the record keeps the location of each defect.
Does brush area still reflect severity? To test this as the reviewer suggests, we asked six new annotators, blind to both protocols, to compare the 90 within-prompt image pairs of Section 4.1 and pick the image with less severe artifacts (76 pairs have a majority). We then checked how often the paper's Likert and brush scores pick the same image as this blinded majority:
Brush area and Likert agree with the blinded judges equally often (difference −1.3 points [−11.8, +8.8]), and both agree more often than a single blinded annotator agrees with the other five (66%). The close numbers are the finding: brushing gains consistency without losing the severity signal of a rating.
This answers the question the reviewer raises: the consistency gain does not come at the cost of the quality signal. We do not claim that area is a better severity measure than a rating, and the revised text will say so.
Benchmark artifacts (Table 2). We agree with the reviewer on this point, which Reviewer eDES also raised (eDES-1). One minus the mean area saturates (0.995 to 0.998 across models) because artifacts are small, about 1% of the image when present. In the revised Table 2 we will replace this column with two scores computed from the brush annotations: the share of artifact-free images (an image counts as having an artifact when at least three of six annotators mark it; higher is better) and the number of consensus regions per image (connected regions marked by at least three of six annotators; lower is better). Both spread the models over a wider range than the submitted score:
The sentence about models having closed the artifact gap will read: 72 to 81% of images are artifact-free, and artifacts cover about 1% of the image when present.
Introduced or repaired defects. We used naturally occurring pairs because they could be collected and adjudicated within the response period, and they contain the defects that current models actually produce, most of which are small (about 1% of the image). A set with defects introduced or repaired by inpainting is a natural extension, and the masks collected here identify where to edit.