- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
SQUARE-Bench evaluates LMMs as judges of AI-generated images. It organizes approximately 18K question-answer pairs into four aspects and 38 subcategories using generated and real images. Dual-answer annotations distinguish observed image content from the intended generation outcome. The paper evaluates 23 models, compares them with five independent human experts, and fine-tunes three models on the benchmark. An auxiliary experiment uses adapted models to guide iterative image editing. This produces mixed gains and lower scores on all six quality metrics for both editors relative to single-pass editing.
Critical strengths
- A useful evaluation objective and broad resource scope. Assessing the reliability of multimodal evaluators is important. Bringing semantic, quality, authenticity, and responsibility tasks into a shared evaluation framework, with diverse image sources and a broad model comparison, could support more informed evaluator selection.
- Explicit separation of depicted content and generation targets. The dual-answer annotations provide a useful way to distinguish whether the generator failed to realize the target or the evaluator misread the resulting image. The paper also reports substantial variation across evaluation aspects and acknowledges negative downstream editing results, providing a more informative picture than an aggregate leaderboard alone.
Critical weaknesses
- The construction process does not sufficiently establish what the benchmark measures. To support conclusions about evaluator reliability, the taxonomy and QA strategy need to capture consequential errors in actual image assessment. The paper defines 38 subcategories but provides limited validation of this coverage. Diagnosis within these categories also requires reliable labels. Expert cross-checking and adjudication are described, yet agreement rates and the nature of disputed judgments are unreported. This matters particularly when independent human experts score below several models: without examining those disagreements and human errors, it is unclear how much the gap reflects evaluator capability versus differences in interpretation or judgment standards.
- Incomplete test composition and numerical inconsistencies undermine score reliability. Test counts for most of the 38 subcategories and the aggregation rule are missing, preventing assessment of balance and reliability, but the disclosed 6-question and 12-question subsets already yield unstable category estimates. Several reported values also raise concerns about numerical consistency. For example, Table 5 reports 81.82% Human Bias accuracy for fine-tuned Ovis2.5-9B, while Appendix F (p.24, L1270–1278) specifies 12 test questions, with invalid responses counted as incorrect. No integer correct count on 12 questions can produce this percentage. Such discrepancies require clarification of the scoring procedure across models and categories.
Questions for authors
- What evidence shows that the taxonomy and QA strategy capture important errors in actual T2I evaluation? Please report annotation agreement, representative disputed label, and human-error cases to clarify the gap between human and LMM judges.
- Please provide category counts, per-model evaluation denominators, and the aggregation formula. How do the authors explain the numerical inconsistencies, exemplified by 81.82% under the stated 12-question protocol? Please check the tables for similar discrepancies and explain whether the same questions and scoring rules were used across models.
2 comments
Comment Part2
Response to W2 & Q2: Test composition and scoring
The test set contains 2,875 questions, with 412 in Semantics, 368 in Quality, 1,792 in Authenticity, and 303 in Responsibility. In the revised full-test comparison, accuracy within each aspect will be the number of correct answers divided by the number of scheduled questions in that aspect. Overall accuracy will be the total number of correct answers divided by 2,875. Thus, each question contributes equally to the overall score.
Thank you for catching the inconsistency in the fine-tuned Ovis2.5-9B Human Bias score. One of the 12 questions triggered a CUDA out-of-memory error before a prediction was saved. The original aggregation divided the nine correct answers by the 11 saved predictions, yielding 81.82%. We reran the missing question, and the model answered it incorrectly. We will correct Table 5 to 75.00% (9/12) and clarify that empty, unparseable, or missing predictions receive zero credit under the fixed-denominator protocol.
The submitted Responsibility scores for LLaVA-NeXT and CogAgent-18B excluded the two multi-image categories they could not process, Cultural Fairness and Human Bias. We agree that this coverage difference should have been explicit. In the revision, we will identify unsupported questions and report both a comparison on the same supported questions for every model and a full-test comparison with zero credit for unsupported items. We will state the denominators used and apply the same scoring rules when updating the affected tables and rankings.
Comment Part1
We thank the reviewer for the constructive questions about what SQUARE-Bench measures, how disagreements were resolved, and how the reported scores were computed. We agree that the submission should have made the evidence for these claims more explicit.
Response to W1a & Q1: Relevance of the taxonomy and QA design
We appreciate the reviewer’s question about how the taxonomy relates to actual T2I evaluation. Table 7 documents the sources used to curate the benchmark, including established benchmarks for image generation, synthetic-image detection, safety, and image editing. We use prompts and captions from these sources to bring established evaluation challenges into SQUARE-Bench, and we generate new outputs using 22 T2I models. Where source category metadata is available, it informs the initial categorization. The final question targets a property or failure visible in the resulting image.
To check whether these questions capture recurring failures that matter to the evaluation, we examined 796 questions from the test set with both ground-truth labels, traceable to the original generation batch, after screening for prompt cues. Visual GT and Intended GT differed on 516 questions across all 22 generators. In each of the 35 subcategories represented in this analysis, such discrepancies occurred in outputs from at least two generators. Attribute binding illustrates why this matters. One prompt requests a red giraffe, but the generated image depicts a naturally brown-and-orange giraffe. The image can look plausible and still fail an explicit requirement. The QA item asks about the giraffe’s visible color, so the benchmark can record the generation failure and separately assess whether the evaluator recognized it. Among the attribute-binding questions, Visual GT and Intended GT differed on 19 items, whose images came from 12 different T2I generators. We will add this analysis to show how the taxonomy groups recurring failures and how the questions test the specific visual evidence needed to judge them.
Response to W1b & Q1: Reviewer agreement, disputed labels, and the human–LMM gap
We appreciate the reviewer’s question about how much of the human–LMM gap reflects evaluator capability and how much may reflect differences in interpretation. We first quantified agreement during QA review. Three reviewers independently assessed each proposed question and answer and submitted their decisions before discussion or adjudication. This analysis covers all question, excluding the automatically generated binary real/fake questions. Of all review decisions, 83.09% approved the item, 12.24% flagged an incorrect answer, and 4.67% flagged a problem with the question. All three reviewers gave the same decision on 78.76% of items. Pairwise agreement was 85.37%, compared with 70.76% expected by chance under the observed vote distribution, yielding a Fleiss’ κ of approximately 0.50. To examine the score gap, we compared Qwen3-VL-32B with the Human (Best) reference on the same 2,875 test questions. We grouped the non-binary items by the three pre-adjudication review decisions that could be matched to the test version.
The provenance-based real/fake questions account for 74 of the 161 additional correct answers. On the 1,200 non-binary questions unanimously approved by all three reviewers, the gap is six answers, or 0.50 percentage points. Larger differences appear in groups where reviewers disagreed or flagged problems. These groups may also differ in difficulty and task composition, so we will report the breakdown alongside the aggregate score and qualify our interpretation of the overall gap. The individual cases illustrate why this distinction matters. In one binary item, the human reference labeled a generated image as a real photograph, while Qwen correctly identified its synthetic provenance. In a bicycle-wheel item, the initial assessment and check disagreed about whether the hub and axle were visible. Human adjudication added an option stating that the components could not be distinguished. The human reference chose that option, while Qwen chose the option stating that the components were missing or merged.