acceptodds
Under review as a conference paper at ICLR 2027

Skill-Aligned Annotation for Reliable Text-to-Image Evaluation

Abstract

Text-to-image (T2I) generation has advanced rapidly, making reliable evaluation critical as performance differences between models narrow. Existing evaluation practices typically apply uniform annotation mechanisms, such as Likert-scale or binary question answering (BQA), across heterogeneous evaluation skills, despite fundamental differences in their nature. In this work, we revisit T2I evaluation through the lens of skill-aligned annotation, where annotation strategies reflect the underlying characteristics of each evaluation skill. We systematically compare skill-aligned annotation against uniform baselines and show that it produces more consistent evaluation signals, with higher inter-annotator agreement and improved stability across models. In controlled comparisons with the same annotators, skill-aligned protocols raise inter-annotator agreement (Krippendorff's ) from as low as to , cut annotator abstention from to , and stabilize with four to five annotators. Finally, we present an automated pipeline that instantiates the proposed evaluation protocol, enabling scalable and fine-grained evaluation with spatially grounded feedback. Our work highlights that improving the foundations of image evaluation can increase reliability and efficiency without simply scaling annotation effort. We hope this motivates further research on refining evaluation protocols as a central component of reliable model assessment.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Text-to-Image Generation Evaluation, Evaluation of Image Generation, Evaluation Alignment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tRqaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how annotation protocols should be matched to different text-to-image evaluation skills. It introduces an 18-skill taxonomy and conducts controlled comparisons for three protocol families: brush-based annotation versus global Likert ratings for visual artifacts; word-level annotation with gap tokens versus BQA and Likert ratings for text rendering; and reference-assisted annotation versus unassisted BQA for reference-dependent skills. Each controlled study uses 30 prompts, three image-generation models, and the same six annotators. The paper also presents a 43-prompt evaluation of four generation models and an automated implementation using existing VLMs and a pretrained artifact detector.

Critical strengths
  1. The paper addresses a meaningful evaluation-design problem through controlled comparisons. Treating annotation design as an experimental variable is useful, and keeping the images and annotator panel fixed makes the observed differences informative. In particular, the increase in artifact annotation agreement from α = 0.39 to 0.82 provides evidence that the reporting mechanism substantially affects reproducibility.

  2. The proposed protocols provide actionable information beyond a global judgment. Word-level labels and gap tokens identify correct, missing, and inserted text, while brush masks localize perceived defects. Reference images also address a concrete knowledge bottleneck, reducing “Unsure” responses from 63% to 2.5%. These results support the practical value of more explicit annotation tasks.

Critical weaknesses
  1. The strongest result establishes improved consistency, but does not establish that the resulting score preserves the intended evaluation target. In Section 4.1, the Likert baseline measures perceived realism and the absence or severity of artifacts, whereas the brush-based score measures the fraction of image area marked as defective. These quantities need not reflect the same quality differences: a small but conspicuous facial or anatomical defect may have a substantial perceptual impact while receiving only a small area penalty. The higher α therefore supports more reproducible annotation of affected area, but does not by itself establish a better measure of artifact quality. This matters for the interpretation of Table 2, where artifact scores of 0.995–0.998 are taken as evidence that recent models have largely closed the artifact-suppression gap. The current evidence does not distinguish genuine quality saturation from limited sensitivity of the area-based score. This is the principal gap between the reported reliability improvement and the broader model-evaluation conclusions.

  2. The independence of the human reference used to validate automatic evaluation is unclear. Appendix C, page 18, states that BQA answers in the full evaluation were pre-filled by a VLM and then confirmed or corrected by annotators. If Figure 6 uses these annotations as its human reference, the reported correlations may be affected by anchoring to the suggested answers. The manuscript does not identify the pre-filling model, quantify how often suggestions were corrected, or establish whether the correlations persist against an unassisted human reference. This concern is limited to the affected BQA results in the full evaluation; the controlled protocol comparisons in Section 4 explicitly excluded such suggestions and remain informative.

  3. The limited scale and coverage leave practical generalizability and annotation efficiency insufficiently established. Each controlled study uses only 30 prompts, three generation models, and a fixed panel of six annotators. The full evaluation contains 43 prompts, with some skills represented by only three or four prompts (Table 6). These experiments support the observed agreement gains under the studied conditions, but provide limited evidence that the gains persist across the broader prompt distributions and annotator populations encountered in large annotation projects. Section 4.4 motivates the sample size by the stabilization of reliability estimates; this does not establish broader coverage or generalizability. Furthermore, the claim that finer-grained recording requires only a few additional interactions is not supported by measured annotation time. Stabilization with fewer annotators therefore does not, by itself, demonstrate lower total annotation cost. The unresolved issue is whether the protocols retain their benefits at a comparable human annotation budget in a broader setting.

Questions for authors
  1. Artifact-score validity: clarification and additional experiment. What is the intended construct of the brush-derived scalar score: affected area, perceived artifact severity, or overall artifact quality? To support its use for the latter two, please evaluate both the Likert and brush-derived scores on a held-out set of matched images with localized defects introduced or repaired. An independent panel, blinded to the protocol outputs, should adjudicate the direction of improvement and relative artifact severity. The set should include small but perceptually salient defects. Reporting how well each protocol recovers these independently established differences, with uncertainty estimates, would directly test whether the consistency gain preserves meaningful quality information. If the intended target is affected area alone, the interpretation of Table 2 should be narrowed accordingly.

  2. Human-reference independence: clarification and additional experiment. Does Figure 6 use the VLM-assisted BQA annotations described in Appendix C? Please identify the pre-filling model and report the proportion of suggestions that annotators corrected. Please also obtain unassisted annotations from independent annotators for a small subset of the same BQA items, then compare the Figure 6 correlations against assisted and unassisted references on those same items, with uncertainty estimates. This would directly establish whether the automatic-evaluation conclusions remain supported without suggested answers.

  3. Generalizability and annotation cost: additional experiment. Please repeat the key comparisons with a new, independent group of annotators to assess whether the agreement gains generalize beyond the original six-person panel. Please measure annotation time per image and total human effort, and compare the protocols under matched total annotation-time budgets. Evaluating both agreement and performance against the independent quality criterion requested above would establish whether the protocols provide better evaluation signals per unit of human effort. This focused extension would be more informative than simply collecting additional ratings from the existing six-person panel.

4 comments

We thank the reviewer for recognising the value of treating annotation design as an experimental variable, the informative fixed-panel comparison and the actionable information our protocols provide, and for three precise requests. We ran the two experiments that could be completed in the response period: a blinded independent panel that judges relative artifact severity (tRqa-1), and a replication of the key comparisons with new annotators and measured time (tRqa-3). Both used the same application and images as Section 4, with no AI suggestions.

tRqa-1: what the brush score measures (W1, Q1)

"What is the intended construct of the brush-derived scalar score: affected area, perceived artifact severity, or overall artifact quality? ... An independent panel, blinded to the protocol outputs, should adjudicate the direction of improvement and relative artifact severity. ... If the intended target is affected area alone, the interpretation of Table 2 should be narrowed accordingly."

Construct. The scalar is affected area: where the artifacts are and how much of the image they cover. The claim of the protocol is that marking what one sees gives a more reproducible record than compressing it into one number, and that the record keeps the location of each defect.

Does brush area still reflect severity? To test this as the reviewer suggests, we asked six new annotators, blind to both protocols, to compare the 90 within-prompt image pairs of Section 4.1 and pick the image with less severe artifacts (76 pairs have a majority). We then checked how often the paper's Likert and brush scores pick the same image as this blinded majority:

ScoreAgrees with blinded majority (76 pairs)Pairs with a 3+ vote margin (40)
Likert73.7% [62.0, 84.7]81.2% [68.1, 93.4]
Brush area72.4% [60.5, 82.9]85.0% [71.4, 97.1]

Brush area and Likert agree with the blinded judges equally often (difference −1.3 points [−11.8, +8.8]), and both agree more often than a single blinded annotator agrees with the other five (66%). The close numbers are the finding: brushing gains consistency without losing the severity signal of a rating.

This answers the question the reviewer raises: the consistency gain does not come at the cost of the quality signal. We do not claim that area is a better severity measure than a rating, and the revised text will say so.

Benchmark artifacts (Table 2). We agree with the reviewer on this point, which Reviewer eDES also raised (eDES-1). One minus the mean area saturates (0.995 to 0.998 across models) because artifacts are small, about 1% of the image when present. In the revised Table 2 we will replace this column with two scores computed from the brush annotations: the share of artifact-free images (an image counts as having an artifact when at least three of six annotators mark it; higher is better) and the number of consensus regions per image (connected regions marked by at least three of six annotators; lower is better). Both spread the models over a wider range than the submitted score:

Model1 − mean area, submitted (↑)Artifact-free images, revised (↑)Consensus regions per image, revised (↓)
Nano-Banana0.99881.4%0.26
Wan-2.50.99779.1%0.53
FLUX.2-dev0.99672.1%0.63
Z-Image-Turbo0.99572.1%1.14

The sentence about models having closed the artifact gap will read: 72 to 81% of images are artifact-free, and artifacts cover about 1% of the image when present.

Introduced or repaired defects. We used naturally occurring pairs because they could be collected and adjudicated within the response period, and they contain the defects that current models actually produce, most of which are small (about 1% of the image). A set with defects introduced or repaired by inpainting is a natural extension, and the masks collected here identify where to edit.

tRqa-2: independence of the Figure 6 human reference (W2, Q2)

"Does Figure 6 use the VLM-assisted BQA annotations described in Appendix C? Please identify the pre-filling model and report the proportion of suggestions that annotators corrected."

Yes. For the yes/no skills, Figure 6 uses the full-suite annotations, which were collected with the suggestions on screen. Suggestions were shown only for binary yes/no questions in Section 5, never to the skill-aligned protocols in Section 4 and never for brush, word-level, anchor or aesthetics annotations. The pre-fill model was gpt-5-mini, which is not one of the judges in Figure 6 (GPT-5, Molmo2-8B, Qwen3.5-9B), so no judge is compared with its own suggestions. Appendix C will name it.

Override rate. Of 6,090 answers given with a suggestion on screen, annotators confirmed 87.0%, reversed 7.9% and set 5.1% to Unsure. Reversals were concentrated on "No" suggestions: annotators reversed 35.8% of the suggestions that said "No", against 2.6% of those that said "Yes". The human majority differs from the suggestion on 6.7% of the 991 questions.

Why suggestions. Binary questions about objects, attributes, counts and relations are easy for current VLMs, so a suggestion mostly saves the annotator a click while the decision stays with the human. The override pattern agrees: even with a small pre-fill model (gpt-5-mini), most suggestions were confirmed, and correction rates were highest on the more challenging skills (camera 20%, view 18%, action 12%).

Evidence against anchoring.

  • The human majority agrees with the suggestions on 93.3% of questions, less than with Molmo2-8B (95.1%) and Qwen3.5-9B (96.1%), which the annotators never saw. GPT-5, the judge closest to the pre-fill model, is at 93.4%, so it is not favoured either.
  • Where humans reversed a suggestion, Molmo2 and Qwen side with the humans 80% and 78% of the time.
  • On the pre-filled skills, gpt-5-mini's own correlation with the human reference (mean ρ 0.58) is no higher than GPT-5's (0.60) or Qwen's (0.62). A reference anchored to it would favour it.

Unassisted annotations. We could not collect unassisted annotations of these items within the response period. The three checks above address the same question with the existing data: a human reference anchored on the suggestions would favour the pre-fill model and the judges that resemble it, and it does not.

Pre-filled and mixed skills in Figure 6. Mean correlations for GPT-5 / Molmo2 / Qwen are 0.60 / 0.45 / 0.62 on pre-filled yes/no skills and 0.63 / 0.60 / 0.55 on skills that mix yes/no with anchor-Likert or word-level questions. The revised figure will give N and a bootstrap interval for every bar.

tRqa-3: generalization and annotation cost (W3, Q3)

"Please repeat the key comparisons with a new, independent group of annotators ... Please measure annotation time per image and total human effort, and compare the protocols under matched total annotation-time budgets. Evaluating both agreement and performance against the independent quality criterion requested above would establish whether the protocols provide better evaluation signals per unit of human effort."

New independent annotators. The comparisons of Sections 4.1 to 4.3 were repeated with annotators who had no part in the original study.

SkillBaseline α, originalBaseline α, new panelSkill-aligned α, originalSkill-aligned α, new panel
Artifacts: Likert vs brush (ranks)0.390.510.610.65
Artifacts: Likert vs brush (as in the paper)0.390.510.820.67
Text: yes/no vs word-level, each word0.630.600.960.92
Reference: from memory vs with anchors†0.470.500.570.56
Reference: share of Unsure answers†63.0%52.3%2.5%2.3%

† Only four of the five crowd annotators completed Study 3b.

The ordering of protocols is the same in every family, and the two panels' consensus scores correlate at 0.77 to 0.96.

Time. The application logged the time on screen for every image.

ComparisonMedian s per image, baselineMedian s per image, oursMean s per image, baselineMean s per image, ours
Artifacts: Likert vs brush7151946
Text: Likert vs word-level761120
Reference: from memory vs with anchors†66109

† Only four of the five crowd annotators completed Study 3b.

Anchor annotation costs the same as its baseline, so its gain is free. Word-level annotation costs the same on the typical image and more on average (20 against 11 seconds per image). Brush annotation costs about twice a rating.

Matched budgets for artifacts. Agreement is a property of the protocol and barely moves with panel size: Likert α is 0.49 with two annotators and 0.51 with five, brush 0.68 and 0.67. At a matched budget of about 95 annotator-seconds per image, that is five Likert annotators (α 0.51) or two brush annotators (α 0.68). Against the independent severity criterion the two scores perform alike (table under tRqa-1), and the brush record adds localization.

Scope. Section 4.4 states that 30 prompts were chosen because the intervals of baseline and skill-aligned protocols separate at that size for artifacts and text rendering, which supports the comparison and is not a claim of broad coverage. Coverage of other prompts and models comes from Section 5 (43 prompts, two further models), and coverage of other annotators from the replication above.

Thank you for your response; I now have a clearer understanding of the paper, and I will increase my score by one point.

Reviewer_showOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the reliability of human annotation protocols for text-to-image (T2I) evaluation. The authors argue that existing frameworks often apply uniform mechanisms, such as binary question answering (BQA) or Likert-scale ratings, to heterogeneous evaluation skills, potentially reducing consistency and reliability.

The proposed skill-aligned annotation framework selects mechanisms according to the characteristics of individual skills. The authors extend a taxonomy to 18 T2I skills and introduce brush-based annotation for visual artifacts, word-level annotation with gap tokens for text rendering, and anchor-based annotation for reference-dependent skills. Other skills retain conventional BQA or Likert ratings.

Controlled within-subject experiments compare annotation mechanisms on the same prompts and generated images with the same annotators. The proposed protocols improve inter-annotator agreement, reduce extreme disagreement and abstention, and often stabilize with four to five annotators. A full evaluation suite profiles several T2I models, while an agentic pipeline uses VLMs and an artifact detector to automate skill-specific evaluation. The paper highlights the importance of annotation design for consistent and fine-grained T2I assessment.

Critical strengths
  1. The paper identifies a meaningful methodological issue: heterogeneous evaluation skills may be poorly served by a uniform annotation interface. The skill-aligned design selects mechanisms appropriate to the structure of each judgment and yields more informative feedback, including spatial masks and word-level labels. This focus on improving the foundations of evaluation is relevant to benchmark reliability and model diagnostics.

  2. The same annotators evaluate the same images under alternative protocols, reducing potential confounding from annotator and item differences. For visual artifacts, Krippendorff’s alpha rises from 0.39 with Likert ratings to 0.82 with brush annotation. For text rendering, word-level annotation provides higher agreement than BQA and global Likert ratings. For reference-dependent skills, anchors reduce the Unsure rate from 63% to 2.5%. These results offer substantial evidence that protocol design affects annotation consistency.

Critical weaknesses
  1. The paper mainly focuses on improving annotation consistency and stability, and the experimental results demonstrate encouraging improvements in inter-annotator agreement. However, the accuracy of the resulting annotations could be further explored. In particular, it would be helpful to investigate whether the proposed skill-aligned protocols also lead to more accurate assessments of generated image quality and prompt adherence.

  2. The core experiments rely on a relatively small annotator panel consisting of six participants, with 90 generated images per evaluated skill. Although the within-subject design effectively controls for annotator differences, the limited number of annotators may not adequately capture variations in expertise, cultural backgrounds, and subjective evaluation standards.

  3. The authors use a fixed group of evaluators and a fixed evaluation dataset, raising concerns about the generalizability of their findings to independent groups of annotators, diverse prompts, and unseen images.

Questions for authors
  1. Annotation Accuracy: The paper primarily focuses on improving annotation consistency and stability. Have the authors considered evaluating whether the proposed skill-aligned protocols also improve annotation accuracy? Comparisons against expert-verified annotations or independently established ground truth could provide a more comprehensive assessment.

  2. Generalizability Across Annotators: The core experiments involve six annotators. Could the authors validate the protocols with an independent annotator group representing different expertise levels or cultural backgrounds, to test whether the observed improvements generalize beyond the current panel?

  3. Generalizability Across Evaluation Samples: Could the authors provide validation on unseen prompts, different T2I models, or an independent evaluation set to show that the improvements remain consistent in diverse scenarios?

  4. Comparability of Agreement Measurements: The protocols produce ordinal Likert scores, continuous artifact ratios, and word-level binary judgments. Could the authors clarify the distance functions and aggregation procedures used for Krippendorff’s alpha in each case, and examine robustness under alternative agreement measures?

  5. Reliability of Retrieved Visual Anchors: How do the authors ensure that automatically retrieved anchors are relevant and representative? Could inappropriate anchors introduce systematic bias into the resulting judgments?

2 comments

We thank the reviewer for recognising the relevance of the problem, the richer feedback of skill-aligned protocols, and the controlled same-panel comparison. The two generalization questions asked for an independent annotator group, and we ran one: new annotators repeated the Section 4 comparisons on the same images, and a further group gave blinded pairwise judgments.

show-1: consistency and accuracy (W1, Q1)

"Have the authors considered evaluating whether the proposed skill-aligned protocols also improve annotation accuracy? Comparisons against expert-verified annotations or independently established ground truth could provide a more comprehensive assessment."

For the skills in Section 4, human judgment is the gold standard: whether a region looks like an artifact, how close an image is to an artist's style, or whether a partly rendered word is readable is decided by people, and no superior signal exists to measure accuracy against. What a protocol can improve is how consistently different people reach the same judgment, so that is what we optimise and measure. For factual skills with a ground truth (object presence, counts, spatial relations), the paper keeps plain yes/no questions.

The closest available independent reference is a blinded pairwise panel: six further annotators, who saw neither protocol's output, picked the image with the less severe artifacts in each of the 90 within-prompt pairs of Section 4.1. The Likert and brush scores pick the image that panel prefers equally often (73.7% and 72.4% of pairs), so the gain in consistency does not come with a loss of agreement with independent judges.

show-2: six annotators (W2, Q2)

"Could the authors validate the protocols with an independent annotator group representing different expertise levels or cultural backgrounds, to test whether the observed improvements generalize beyond the current panel?"

To show that our protocols generalize beyond the original panel, five crowd annotators repeated the exact same experiments, on the same images and application as Section 4. Five annotators suffice for this comparison, since agreement barely changes with panel size: for artifacts, Likert α is 0.49 with two annotators and 0.51 with five, and brush α 0.68 and 0.67. The table below reports results for the original and the new panel.

How we measure brush agreement. The paper computes α on the raw brushed area. Following Reviewer eDES's suggestion to rescale this score, we also compute it on ranks (ordinal α), which asks only whether annotators order the images the same way. We report both because brushed area is very skewed: most marks cover about 1% of the image and a few cover up to 24%, so on raw area those few images dominate α. This is why the two panels differ on raw area (0.82 and 0.67) but agree on ranks (0.61 and 0.65). Under both measures, brush agreement stays well above Likert.

SkillBaseline α, originalBaseline α, new panelSkill-aligned α, originalSkill-aligned α, new panel
Artifacts: Likert vs brush (ranks)0.390.510.610.65
Artifacts: Likert vs brush (as in the paper)0.390.510.820.67
Text: yes/no vs word-level, each word0.630.600.960.92
Reference: from memory vs with anchors†0.470.500.570.56
Reference: share of Unsure answers†63.0%52.3%2.5%2.3%

† Only four of the five crowd annotators completed Study 3b.

The skill-aligned protocol has the higher agreement in every family for the new panel, as for the original one; for reference skills the gain in agreement is modest and the main effect of anchors is to remove Unsure answers. The two panels also reach a similar consensus: their scores correlate at 0.77 (Likert) and 0.80 (brush) for artifacts, 0.96 for word-level text (per word) and 0.81 for anchored answers.

show-3: fixed panel and fixed data (W3, Q3)

"Could the authors provide validation on unseen prompts, different T2I models, or an independent evaluation set to show that the improvements remain consistent in diverse scenarios?"

Holding the panel, prompts and images fixed is a requirement of the experiment, not a convenience. As Reviewer eDES notes, T2I evaluation papers often change annotation templates, prompt sets and rater pools at the same time, "making it hard to tell why agreement improved". Keeping the six annotators, 30 prompts and 90 images "strictly fixed across competing protocols" is what lets the paper show that low agreement "is largely an artifact of the reporting interface rather than rater negligence". Generalization must then be tested separately, and the new evidence confirms it:

  • Annotators. An additional panel of five crowd annotators reproduces the ordering of protocols in every skill family on the Section 4 images, and its agreement is stable across panel size. The results therefore generalize beyond the original raters.
  • Prompts and models. Section 5 is an independent evaluation set by design: the protocols developed on the Section 4 set are applied unchanged to 43 new prompts and two further models (Nano-Banana, Wan-2.5). They work at this scale, and the model rankings they produce are stable under annotator subsampling (Figure 5).

show-4: agreement measures (Q4)

"Could the authors clarify the distance functions and aggregation procedures used for Krippendorff's alpha in each case, and examine robustness under alternative agreement measures?"

The levels are ordinal for 1 to 5 ratings, interval for continuous scores (brushed area, word-level accuracy), and nominal for yes/no answers and word tiles. The unit is the image for image-level scores and the question or tile otherwise. Unsure answers are treated as missing. Intervals are bootstrapped over items. A new appendix table will state this per protocol.

The ordering of protocols in the artifact and text families holds under a second coefficient suited to each data type: ICC for the brushed area, which is continuous, and Fleiss κ for yes/no answers and word tiles, which are categorical (baseline → skill-aligned):

Familyα (paper)Second coefficient
Artifacts: Likert → brush0.39 → 0.82ICC(2,1): 0.40 → 0.82
Text: yes/no → word-level, each word0.63 → 0.96Fleiss κ: 0.63 → 0.96
Reference: from memory → with anchors0.47 → 0.57Fleiss κ: 0.57 → 0.56

For reference skills the main gain is coverage: anchors cut the Unsure rate from 63.0% to 2.5% in the original panel and from 52.3% to 2.3% in the new one, so nearly every item gets a usable answer. Agreement on the answers given rises modestly under α (0.47 to 0.57) and stays level under κ (0.57 and 0.56) because the from-memory κ covers only the 37% of items annotators already knew.

show-5: reliability of retrieved anchors (Q5)

"How do the authors ensure that automatically retrieved anchors are relevant and representative? Could inappropriate anchors introduce systematic bias into the resulting judgments?"

Retrieval has two steps, which the revised appendix will describe. A language model reads each validation question and decides whether it refers to an external entity or style; if so, it writes a search query (the artist's name, the landmark). An image search returns results for that query and the first three are shown. Before annotation, we verified every retrieved anchor by hand to confirm that it shows the intended entity or style.

A single off-target image could bias a judgment, which is why three anchors are shown. The data indicate the anchors were adequate for both panels: with anchors the Unsure rate falls from 63.0% to 2.5% (original) and from 52.3% to 2.3% (new), and agreement rises modestly in both. In the new panel, 97% of Unsure answers became Yes or No once the anchors were shown.

Reviewer_eDESOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how the choice of human annotation interface affects the reliability of text-to-image (T2I) model evaluation. The authors argue that applying uniform annotation mechanisms (such as binary question answering or 1-to-5 Likert scales) across fundamentally different evaluation skills introduces avoidable annotator disagreement and high abstention rates. Building on the Gecko benchmark, they expand the evaluation taxonomy to 18 skills (16 semantic categories plus visual artifacts and aesthetic quality) and use a single GPT-5 prompt to jointly tag prompts and generate dependency-structured validation questions. In a controlled within-subject study (6 annotators evaluating 90 images per skill across 30 prompts and 3 models), they compare uniform baselines against three skill-aligned mechanisms: pixel-level brush masks for visual artifacts, word-level tiles with virtual gap tokens for text rendering, and reference-anchored Likert scales for skills requiring external knowledge (landmarks, artistic styles, and named entities). They show substantial gains in inter-annotator agreement (Krippendorff's alpha) and faster convergence across annotator subsets. Finally, they run the full protocol on a 43-prompt benchmark across four T2I models and evaluate how well an automated VLM-agent pipeline correlates with the collected human judgments.

Critical strengths

1.Controlled within-subject isolation of annotation mechanisms (Section 4). A recurring issue in T2I evaluation papers is that annotation templates, prompt sets, and rater pools are changed simultaneously, making it hard to tell why agreement improved. By holding the 6-annotator panel, the 30 prompts, and the 90 generated images strictly fixed across competing protocols, the paper cleanly demonstrates that low inter-annotator agreement on artifacts and reference-heavy prompts is largely an artifact of the reporting interface rather than rater negligence. Showing that the exact same annotators move from alpha = 0.39 to 0.82 on artifacts simply by switching from a global Likert scale to brushing defective regions is a compelling and practical result for benchmark design.

2.Thoughtful interface designs that preserve diagnostic granularity without heavy overhead (Sections 4.1 to 4.3). The three proposed mechanisms directly address concrete failure modes in existing T2I benchmarks. In particular, introducing clickable gap tokens between expected words in the word-level text protocol is a clean solution for penalizing hallucinated or repeated words without requiring complex character-level alignment. Likewise, surfacing three retrieved visual anchors alongside a graded similarity scale (Anchor Likert) directly resolves the prior-knowledge bottleneck in evaluating specific artists, landmarks, and named entities, turning a 63.0% "Unsure" rate into usable ratings (2.5% abstention) that stabilize around 4 annotators.

Critical weaknesses
  1. Scale compression in the pixel-area artifact score distorts both the Extreme Disagreement Rate (EDR) comparison and benchmark discriminability (Section 4.1 and Table 2). To compute Krippendorff's alpha, EDR, and model scores for the brush protocol, each brushed mask is converted into a scalar ratio by dividing marked artifact pixels by the total number of pixels in the image. However, common T2I artifacts (such as deformed hands, extra limbs, or garbled facial features) typically occupy only a tiny fraction of the overall image canvas (often 0.2% to 3% of total pixels). Because all resulting scores are compressed near zero (or near 1.0 when inverted in Table 2), defining EDR as a 40% difference across the theoretical [0, 1] range is not comparable to a 2-point gap on a 1 to 5 Likert scale: for two annotators to trigger an extreme disagreement under the brush metric, one would have to brush at least 40% more of the entire image canvas than the other (and even the 20% threshold in Table 5 requires a 20 percentage point difference in whole-image area). This same compression causes the Visual Artifacts column in the full benchmark (Table 2) to saturate between 0.995 and 0.998 across all four models, leaving almost no discriminative spread and implicitly penalizing a small, severe facial distortion less than a larger region of mild background texture noise. 2.AI pre-population bias and very small per-skill sample sizes in the full evaluation suite (Sections 5 and 6). First, Appendix C (lines 965 to 968) notes that to accelerate annotation in Section 5, all BQA answers were initialized by a VLM and displayed to human annotators as AI suggestions to confirm or correct. Because 11 of the 18 skills rely on BQA, showing VLM predictions upfront introduces strong anchoring and confirmation bias, which confounds the human score convergence analysis in Figure 5 for those skills. Second, the full suite in Section 5 comprises only 43 prompts total (Appendix F, Table 6), resulting in several skills having only 3 to 6 prompts and 3 to 6 validation questions in total (for example, View has 3 questions, Camera has 4, Mood/Feeling has 4, and Language Complexity has 6). Benchmarking four models and computing per-skill VLM-human Spearman correlations (Figure 6, where multiple bars had to be omitted due to constant predictions) on 3 to 4 questions per skill is statistically fragile.
Questions for authors
  1. Impact of artifact score normalization (Section 4.1 and Table 2): If you compute Krippendorff's alpha and EDR for the brush protocol after normalizing the artifact pixel ratios by the empirical range observed across the 90 images (e.g., min-max or percentile scaling, or normalizing by foreground/union mask area rather than total image pixels), what are the resulting alpha and EDR values? Additionally, if you report spatial overlap (such as pairwise IoU or Dice coefficient across annotators' brush masks) directly, how well do annotators agree on exact localization.
  2. Effect of VLM pre-population in Section 5 (Appendix C, line 965): In the full suite evaluation, how frequently did the six human annotators actually override the VLM-initialized BQA suggestions? Can you report the rank stability curve (Figure 5) restricted strictly to the skills that were annotated from scratch without AI suggestions?
  3. Empirical annotation time (Section 4.4): In Section 4.4, you argue that skill-aligned protocols require minimal extra effort beyond the initial visual inspection phase. Did your web application log per-item completion times during the Section 4 experiments, and if so, what is the average wall-clock time per image for Brush vs. Likert, Word-Level vs. BQA/Likert, and Anchor Likert vs. standard BQA? 4.VLM evaluation granularity and missing Word-Level agent details (Section 6 and Appendix H): In Figure 6, is the Spearman correlation computed across the 4 aggregated model scores per skill (n = 4) or across individual image-level scores? Also, Section 6 (line 449) describes a word-level VLM agent that labels expected words and gaps individually, but Appendix H (line 1168) states that the system has four agents and omits the word-level agent. Could you clarify how the word-level VLM evaluation was prompted and executed?

Suggestion for future research: Rather than weighting all brushed pixels equally across the canvas, weighting artifact masks by semantic salience (for example, weighting regions overlapping with detected faces, hands, or primary subjects higher than background pixels) could prevent score saturation while preserving the high inter-annotator agreement of brush annotations.

4 comments

We thank the reviewer for the careful reading, and for recognising that our fixed-panel design isolates the effect of each annotation mechanism and makes the weaknesses of conventional protocols visible, as well as the effectiveness of the skill-aligned protocols. We address each question and concern below.

To test generalization beyond the original panel, as also requested by Reviewer show, five crowd annotators, none from the original panel, repeated the same experiments. They used the same application and images as Section 4, with no AI suggestions, and the application logged the time spent on every image. Results for both panels are reported below.

eDES-1: scale compression of the brush score (W1, Q1)

"Scale compression in the pixel-area artifact score distorts both the Extreme Disagreement Rate (EDR) comparison and benchmark discriminability ... If you compute Krippendorff's alpha and EDR for the brush protocol after normalizing the artifact pixel ratios by the empirical range ... what are the resulting alpha and EDR values? ... how well do annotators agree on exact localization."

We agree with both observations and we recompute α and EDR below under the scalings the reviewer suggests.

Alpha. Interval α is invariant to linear rescaling, so min-max normalisation does not affect alpha. We report the non-linear and rank-based versions below.

Scaling of the brush scoreα [95% CI]
Raw area (paper), and min-max0.82 [0.56, 0.89]
Clipped at the 2.5 to 97.5 percentiles0.72 [0.55, 0.82]
Percentile ranks (ordinal)0.61
Likert baseline0.39 [0.27, 0.50]

Under every scaling, the brush protocol remains well above the Likert baseline (α 0.61 to 0.82 against 0.39), so its advantage does not depend on how the area is scaled.

The revised Section 4.1 will report the ordinal α next to the interval one. The new independent panel gives the same picture: 0.65 for brush against 0.51 for Likert.

EDR. Following the reviewer's suggestion, we normalise by the empirical range. An annotator's brush score on an image is the share of the image they marked. The observed range of the brush score, from the lowest to the highest score over all annotators and images, is 0 to 24% of the image. Following the paper's definition of EDR, a gap of two Likert points, which corresponds to 40% of the scoring range, we set the brush threshold to 40% of this observed range (about 10 points of image area). Because a few large marks set this range, we also report a threshold at 40% of the 2.5 to 97.5 percentile range (about 2.6 points for the original panel). Under both ranges, the brush protocol has fewer extreme disagreements than Likert in both panels:

ThresholdOriginal panelIndependent panel
Brush, 40% of the observed range2.2%3.3%
Brush, 40% of the 2.5 to 97.5 percentile range18.9%14.4%
Likert, 40% of the range (2 points)62.2%48.9%

Localization. Because people draw differently, free-hand marks of the same defect vary in stroke and extent, so pixel-exact overlap is naturally low: mean pairwise IoU is 0.14 and Dice 0.19 over pairs where at least one annotator marked something. What matters for reliability is whether annotators mark the same regions. To measure this, we take the union of the six masks on each image, split it into connected regions (ignoring regions smaller than 0.05% of the image), and count how many annotators marked inside each region. Regions marked by at least three of six annotators cover 72% of the total marked area, and at least five of six annotators agree on whether an image has any artifact for 72% of images. The revision will state this explicitly: the protocol reliably identifies the defective regions, while exact boundaries vary with each annotator's stroke.

Benchmark discriminability (Table 2). The reviewer also notes that scale compression limits how well the brush score separates models. We agree: in Table 2, one minus the mean brushed area ranges only from 0.995 to 0.998 across the four models. However, rescaling does not resolve this. Min-max scaling is linear and leaves the relative gaps between models unchanged, and the saturation comes from artifacts being small (about 1% of the image when present), not from the scale. Moreover, normalising by the union of the masks instead of the image removes the size of the defect from the score, so it measures how much annotators overlap rather than how much of the image is affected. Instead, in the revised Table 2 we will replace this column with two scores computed from the brush annotations: the share of artifact-free images (an image counts as having an artifact when at least three of six annotators mark it), and the number of consensus regions per image (connected regions marked by at least three of six annotators). Higher is better for the first, as for the submitted score; lower is better for the second. These scores spread the models over a wider range than the submitted one:

Model1 − mean area, submitted (↑)Artifact-free images, revised (↑)Consensus regions per image, revised (↓)
Nano-Banana0.99881.4%0.26
Wan-2.50.99779.1%0.53
FLUX.2-dev0.99672.1%0.63
Z-Image-Turbo0.99572.1%1.14

eDES-2: AI pre-population and per-skill sample sizes (W2, Q2)

"In the full suite evaluation, how frequently did the six human annotators actually override the VLM-initialized BQA suggestions? Can you report the rank stability curve (Figure 5) restricted strictly to the skills that were annotated from scratch without AI suggestions?"

AI suggestions were shown only for binary yes/no questions in Section 5, never to the skill-aligned protocol in Section 4 and never for brush, word-level, anchor or aesthetics annotations. The pre-fill model was gpt-5-mini, which is not one of the judges in Figure 6.

Override rate. Of 6,090 answers given with a suggestion on screen, annotators confirmed 87.0%, reversed 7.9% and set 5.1% to Unsure. Reversals were concentrated on "No" suggestions: annotators reversed 35.8% of the suggestions that said "No", against 2.6% of those that said "Yes".

Why AI-suggestions. Binary questions about objects, attributes, counts and relations are easy for current VLMs, so a suggestion mostly saves the annotator a click while the decision stays with the human. The override pattern agrees: even with a small pre-fill model (gpt-5-mini), most suggestions were confirmed, and correction rates were highest on the more challenging skills (camera 20%, view 18%, action 12%).

Evidence against anchoring.

  • The human majority agrees with the suggestions on 93.3% of questions, less than with Molmo2-8B (95.1%) and Qwen3.5-9B (96.1%), which the annotators never saw. GPT-5, the judge closest to the pre-fill model, is at 93.4%, so it is not favoured either.
  • Where humans reversed a suggestion, Molmo2 and Qwen side with the humans 80% and 78% of the time.
  • On the pre-filled skills, gpt-5-mini's own correlation with the human reference (mean ρ 0.58) is no higher than GPT-5's (0.60) or Qwen's (0.62). A reference anchored to it would favour it.

Figure 5 without pre-filled skills. On the skills annotated entirely without suggestions (artifacts and aesthetics), the rank correlation with the full panel is 0.97 at three annotators and 0.99 at four. On the anchor-Likert questions of the mixed skills it is 0.91 at four and 0.97 at five. Model rankings therefore remain stable on skills that never showed AI suggestions, so the pre-filled suggestions do not drive the stability shown in Figure 5.

Per-skill sample size (Figure 6). Each correlation in Figure 6 is computed over images, not over questions or over the four model averages. Every prompt is generated by four models, so a skill with 3 to 6 prompts contributes 12 to 24 images (4 prompts give 16 images). The revised Figure 6 will state the number of images behind each bar and add 95% confidence intervals, so the skills estimated from few images are visible. The bars omitted from Figure 6 are skills where a judge gave every image the same score, which leaves a rank correlation undefined; this also happens for weather, which has 28 images. The skill distribution follows the natural frequency of skills in the Gecko prompt set from which we sample (Appendix F): entities and attributes appear in most prompts, while camera, view and mood are rare. The reliability comparisons of Section 4 use dedicated sets of 30 prompts each and do not depend on this distribution.

eDES-3: annotation time (Q3)

"Did your web application log per-item completion times during the Section 4 experiments, and if so, what is the average wall-clock time per image ..."

The original sessions were not timed, so we measured time in the new independent-panel replication, where the application logs the time on screen for every image.

ComparisonMedian s per image, baselineMedian s per image, oursMean s per image, baselineMean s per image, ours
Artifacts: Likert vs brush7151946
Text: Likert vs word-level761120
Reference: from memory vs with anchors†66109

† Only four of the five crowd annotators completed Study 3b.

Anchor annotation costs the same as its baseline. Word-level annotation takes the same time as a rating on the typical image, but a minority of images take much longer, which raises its mean to 20 seconds against 11. Brush annotation takes about twice as long as a rating, 8 seconds more per image. Section 4.4 will report these times.

Summary. Of the three skill-aligned protocols, anchors cost no extra time, word-level annotation costs more on average (20 against 11 seconds per image), and brush annotation costs about twice a rating. In return, brush annotation gives clearly higher agreement than Likert and records where each defect is, which a single rating cannot.

eDES-4: Figure 6 granularity and the word-level agent (Q4)

"In Figure 6, is the Spearman correlation computed across the 4 aggregated model scores per skill (n = 4) or across individual image-level scores? ... Appendix H (line 1168) states that the system has four agents and omits the word-level agent."

The correlation is over images. For each skill we pair the mean human score with the VLM score for every image whose prompt contains that skill (N = prompts with the skill × 4 models). The caption will say so and give N per bar.

The word-level agent was used for all text-rendering results. It receives the image and the expected sentence split into word tiles and gap tokens, and returns yes or no per tile. Accuracy is computed as for the human protocol. Appendix H will list five agents and include its prompt.

eDES-5: salience-weighted masks (suggestion)

We thank the reviewer for the suggestion to weight masks by the salience of each defect, and will add it to Future Work. The pairwise severity judgments we collected can be used to fit such weights.


Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their careful and constructive reviews. The reviewers recognised that:

  • The problem matters: a single annotation interface serves heterogeneous evaluation skills poorly, which limits benchmark reliability (show, tRqa).
  • The designs are thoughtful: brushed masks, word-level tiles with gap tokens and retrieved visual anchors give richer, more diagnostic feedback without heavy overhead (eDES, show, tRqa).
  • The evidence is clean: a controlled within-subject design attributes the gains in agreement to the interface itself, a compelling and practical result for benchmark design (eDES, show, tRqa).

We would like to bring the attention of the reviewers and the AC to the fixed-panel design. As noted by Reviewer eDES, holding annotators, prompts and images fixed is a requirement of the comparison, not a limitation: it is what isolates the annotation mechanism. Generalization is tested separately, by the new experiments below and by Section 5 (43 new prompts, two further models).

New experiments, both consistent with the main paper:

  • Independent crowd panel. Five paid crowd annotators repeated the Section 4 experiments: the skill-aligned protocol again has higher agreement in every skill family.
  • Blinded severity judgments. Six further annotators, blind to both protocols, picked the less severe image in 90 pairs: brush area agrees with them as often as a Likert rating, so the gain in consistency keeps the quality signal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.