- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes WeGenBench, a bilingual and multidimensional diagnostic benchmark for text-to-image generation. The benchmark contains 4,000 prompts evenly divided between Chinese and English and between general image generation and visual text rendering, with each prompt annotated with a scenario label and multiple fine-grained capability tags. It introduces interpretable evaluation protocols: checklist-based QA and CoT-based deduction for semantic alignment, adaptive level-wise anchor matching with structural constraints for aesthetic quality, and OCR- and VLM-based metrics for visual text rendering that capture spelling, completeness, and readability. Experiments show that no single system leads uniformly across dimensions, semantic alignment and exact text accuracy can diverge substantially, long and compositional prompts remain difficult, and performance differs across languages and typographic settings.
Critical strengths
-
The paper contributes a 4,000-prompt Chinese-English benchmark with scenario labels and fine-grained capability tags, enabling both application-level evaluation and failure localization that can point to candidate optimization directions.
-
This paper provides tailored protocols for semantic alignment, aesthetic quality, and visual text rendering, with the semantic-alignment and aesthetic-quality metrics explicitly validated against human judgments.
Critical weaknesses
-
The contribution is largely incremental and the conclusions are unsurprising. Many T2I benchmarks already exist, and the paper's main findings--long compositional prompts are hard, text rendering diverges from semantic alignment, and multi-entity binding/negation remain bottlenecks--are already well known. Apart from fine-grained tags, the benchmark offers no clearly significant new capability, and checklist-based QA is a common evaluation method. The novelty is not enough for a qualified work.
-
The tag-level diagnostic utility is asserted but not empirically validated. The paper claims tags can guide optimization, but its suggestions are generic, such as ''attribute-binding failures motivate multi-entity supervision.'' It does not show that tag-level scores predict intervention success or reveal insights beyond aggregate scores, which weakens the main claimed advantage.
Questions for authors
-
Can the authors state concretely what verifiable evaluation capability WeGenBench adds beyond existing T2I benchmarks, other than scale and tag granularity?
-
Can the authors provide a case showing that tag-level scores actually predict or guide an effective intervention, rather than merely describing known bottleneck?