- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
UltraText Bench evaluates dense English and Chinese text rendering in image generation. It contains 432 prompts across 24 scene categories and three workload levels, with exact target strings and structured references for individual text regions. A VLM judge assigns six scores, aggregated into four dimensions and a composite. The paper compares 24 model configurations and presents qualitative examples of rendering errors and disagreements with the judge.
Critical strengths
The collection of bilingual prompts and structured region references is a potentially useful resource. Explicitly associating target strings with locations/ regions could support systematic evaluation of dense text across varied scenes.
Critical weaknesses
-
The paper does not establish that its evaluator reliably measures the intended capabilities. Section 3.2 mentions ten participants but does not adequately describe the human-evaluation protocol or report quantitative human–judge agreement. We read from the paper that GPT Image 2 outputs receive maximum scores on all six dimensions (above 99%), but Figure A9 shows degraded code text receiving maximum accuracy and readability scores. Figure A7 demonstrates the opposite problem, with recognizable target content receiving zero accuracy and completeness. For a benchmark paper, this undermines the central contribution.
-
Appendix G documents incomplete judge checkpoint? And that GPT Image 2 does not have full coverage? Are all models compared on equivalent prompt subsets then?
Questions for authors
-
How many images did the ten participants assess? What exactly did the ten-person human evaluation assess? How were the images sampled across models, and score ranges?
-
Appendix H.2 identifies substantial overlap with existing benchmarks, particularly InfoTextBench. What specific evaluation capability does UltraText provide beyond these resources, and which result demonstrates that contribution?