acceptodds
Withdrawn from ICLR 2027

WeGenBench: A Multidimensional Diagnostic Benchmark towards Text-to-Image Model Optimization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent text-to-image models achieve impressive visual fidelity, yet existing benchmarks often reduce their behavior to coarse aggregate scores and provide limited guidance for targeted improvement. We introduce WeGenBench, a bilingual, multidimensional benchmark for diagnosing text-to-image generation capabilities. It contains 4,000 prompts across general image generation and visual text rendering, evenly balanced between Chinese and English. In addition to scenario labels, each prompt is annotated with fine-grained capability tags that capture challenges such as spatial relations, attribute binding, complex actions, negation, typography, and bilingual rendering. Aggregating results over the intersection of scenarios and tags reveals localized failure patterns and suggests hypotheses for targeted data curation and post-training. We further develop interpretable vision-language-model (VLM)-based evaluation protocols for semantic alignment and aesthetic quality, together with OCR- and VLM-based metrics for visual text rendering. These protocols provide structured rationales and dimension-level diagnostics and show stronger or comparable agreement with human judgments than the evaluated baselines on our validation set. Finally, we benchmark a broad collection of state-of-the-art systems, revealing persistent compositional, linguistic, and typographic bottlenecks that can be obscured by aggregate evaluation.

Primary area:
datasets and benchmarks
Keywords:
Text-to-Image Generation, Benchmark, Multidimensional Evaluation, Model Diagnosis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bGvfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes WeGenBench, a bilingual and multidimensional diagnostic benchmark for text-to-image generation. The benchmark contains 4,000 prompts evenly divided between Chinese and English and between general image generation and visual text rendering, with each prompt annotated with a scenario label and multiple fine-grained capability tags. It introduces interpretable evaluation protocols: checklist-based QA and CoT-based deduction for semantic alignment, adaptive level-wise anchor matching with structural constraints for aesthetic quality, and OCR- and VLM-based metrics for visual text rendering that capture spelling, completeness, and readability. Experiments show that no single system leads uniformly across dimensions, semantic alignment and exact text accuracy can diverge substantially, long and compositional prompts remain difficult, and performance differs across languages and typographic settings.

Critical strengths
  1. The paper contributes a 4,000-prompt Chinese-English benchmark with scenario labels and fine-grained capability tags, enabling both application-level evaluation and failure localization that can point to candidate optimization directions.

  2. This paper provides tailored protocols for semantic alignment, aesthetic quality, and visual text rendering, with the semantic-alignment and aesthetic-quality metrics explicitly validated against human judgments.

Critical weaknesses
  1. The contribution is largely incremental and the conclusions are unsurprising. Many T2I benchmarks already exist, and the paper's main findings--long compositional prompts are hard, text rendering diverges from semantic alignment, and multi-entity binding/negation remain bottlenecks--are already well known. Apart from fine-grained tags, the benchmark offers no clearly significant new capability, and checklist-based QA is a common evaluation method. The novelty is not enough for a qualified work.

  2. The tag-level diagnostic utility is asserted but not empirically validated. The paper claims tags can guide optimization, but its suggestions are generic, such as ''attribute-binding failures motivate multi-entity supervision.'' It does not show that tag-level scores predict intervention success or reveal insights beyond aggregate scores, which weakens the main claimed advantage.

Questions for authors
  1. Can the authors state concretely what verifiable evaluation capability WeGenBench adds beyond existing T2I benchmarks, other than scale and tag granularity?

  2. Can the authors provide a case showing that tag-level scores actually predict or guide an effective intervention, rather than merely describing known bottleneck?

Reviewer_Ai4cOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces WeGenBench, a bilingual, multidimensional diagnostic benchmark for text-to-image generation. It contains 4,000 prompts across general image generation and visual text rendering, evenly balanced between Chinese and English. Each prompt has a scenario label and fine-grained capability tags. The authors also develop evaluation protocols for semantic alignment and aesthetic quality, together with metrics for visual text rendering. They validate the semantic-alignment and aesthetic-quality protocols against human judgments and benchmark a collection of text-to-image models to identify capability-specific bottlenecks.

Critical strengths
  1. The benchmark has practical value. By incorporating general-purpose images, visual text, Chinese and English, as well as scene and capability labels into a unified evaluation framework, it provides a convenient starting point for investigating specific failure modes.
  2. The authors attempt to validate the automatic metrics. Rather than directly using the proposed metrics to rank models, the paper compares semantic and aesthetic metrics against human pairwise judgments and model-level rankings.
Critical weaknesses
  1. Unclear distinction from existing benchmarks. The paper does not clearly compare WeGenBench with the closest benchmarks, making its incremental coverage and evaluation capabilities difficult to assess.
  2. Insufficiently established unique contribution. Although WeGenBench combines bilingual prompts, capability tags, and multiple metrics, the paper does not clearly show what this combination enables beyond existing evaluation approaches.
Questions for authors
  1. How does WeGenBench differ from the closest existing benchmarks in its prompt coverage, capability tags, bilingual design, and evaluation metrics? Which of these differences constitutes its main contribution?
  2. What analyses become possible by evaluating general image generation and visual text rendering within WeGenBench that would be difficult to obtain from existing benchmarks? Which findings in Section 4.2 demonstrate this added value?
  3. How are checklist questions generated and weighted, and how are their answers aggregated? What determines the penalties in CoT-based Deduction, and how are the aesthetic anchors constructed and selected? A worked scoring example would clarify these procedures.
  4. The CoT evaluator is fine-tuned on expert reviews and its checkpoint is selected using human judgments. Are the prompts, images, and judgments used for those steps separate from the final evaluation in Table 1?
  5. How many prompts did each model complete for Table 2, and how are missing prompts distributed across languages, scenarios, and difficulty levels? Do the main conclusions hold when models are compared on the same set of prompts?
Reviewer_dLQCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces WeGenBench, a bilingual text-to-image generation benchmark consisting of 4,000 Chinese and English prompts. Rather than simply ranking models, the benchmark aims to identify specific capability weaknesses and provide guidance for data construction and post-training optimization.

The main contributions include:

  1. A multidimensional benchmark with fine-grained capability tags covering attribute binding, spatial relations, complex actions, negation, and text rendering.
  2. Automated evaluation protocols for semantic alignment, aesthetic quality, and visual text rendering, including VLM-based assessment and OCR-based metrics.
  3. Experimental evaluations of multiple image generation models, demonstrating improved agreement with human judgments on the validation set and identifying capability-specific weaknesses.
Critical strengths
  1. Interesting and practically relevant benchmark. The paper goes beyond aggregate model scores and attempts to identify specific weaknesses in image generation capabilities. The fine-grained capability taxonomy could be useful for model evaluation and targeted optimization.

  2. Relatively comprehensive evaluation. The benchmark covers both Chinese and English prompts, with evaluations across semantic alignment, aesthetic quality, and text rendering. The proposed evaluation methods also show improved agreement with human judgments compared with the evaluated baselines.

Critical weaknesses
  1. Lack of end-to-end validation for model optimization.

    • The paper identifies weaknesses in capabilities such as attribute binding, complex actions, and negation.
    • However, it does not demonstrate whether these diagnostic results can effectively guide data construction or post-training to improve model performance.
    • Given the paper's motivation of supporting model optimization, an end-to-end experiment connecting capability diagnosis to actual model improvements would make the contribution more convincing.
  2. The reliability of automatic evaluation remains insufficiently established.

    • Although the proposed metrics show improved agreement with human judgments, their pairwise agreement is still relatively limited, particularly for aesthetic quality.
    • VLM-based evaluation may introduce evaluator bias, especially for subjective dimensions such as aesthetics.
    • More evidence is needed to establish the robustness and generalizability of these evaluation methods across different models and visual styles.
Questions for authors
  1. Can the authors provide an end-to-end experiment demonstrating that the identified capability weaknesses can guide targeted data construction or post-training and lead to measurable improvements? A comparison with general or randomly selected training data would be particularly useful.

  2. How robust are the VLM-based evaluators across different image generation models and visual styles? In particular, how do the authors address potential evaluator bias and the relatively limited pairwise agreement with human aesthetic judgments?

  3. Have the authors evaluated whether the fine-grained capability-level scores consistently correlate with human judgments, beyond the reported aggregate and model-level correlations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.