acceptodds
Withdrawn from ICLR 2027

UltraText Bench: A Comprehensive Bilingual Benchmark for Evaluating Visual Text Rendering in Image Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

14 views

Abstract

Dense visual text requires image generators to reproduce long strings across mul- tiple regions with correct placement and legibility. As short-string rendering improves, evaluation must test sustained performance across more demanding scenes. We introduce UltraText Bench, a bilingual benchmark for prompt-only generation of dense visual text. It contains 432 prompts spanning 24 real-world scene categories and three difficulty levels, split equally between English and Chinese. Each human-reviewed prompt supplies exact strings for four to twelve text regions, paired with structured references for their content, placement, and visual attributes. We use the Q-Judger vision-language model to assess each image against the complete reference, reporting text fidelity, text clarity, spatial quality, and scene quality. Across 24 model configurations, these dimensions reveal differ- ent strengths: Z-Image-Turbo gains 3.81 clarity points over Z-Image-Base while losing 14.76 fidelity points under the reported settings. Performance also varies with workload; Qwen-Image-2512’s English composite falls from 86.50 at L1 to 42.86 at L3. Ten participants took part in human evaluation of the automatic scores

Primary area:
datasets and benchmarks
Keywords:
benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_gbDROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

UltraText Bench evaluates dense English and Chinese text rendering in image generation. It contains 432 prompts across 24 scene categories and three workload levels, with exact target strings and structured references for individual text regions. A VLM judge assigns six scores, aggregated into four dimensions and a composite. The paper compares 24 model configurations and presents qualitative examples of rendering errors and disagreements with the judge.

Critical strengths

The collection of bilingual prompts and structured region references is a potentially useful resource. Explicitly associating target strings with locations/ regions could support systematic evaluation of dense text across varied scenes.

Critical weaknesses
  • The paper does not establish that its evaluator reliably measures the intended capabilities. Section 3.2 mentions ten participants but does not adequately describe the human-evaluation protocol or report quantitative human–judge agreement. We read from the paper that GPT Image 2 outputs receive maximum scores on all six dimensions (above 99%), but Figure A9 shows degraded code text receiving maximum accuracy and readability scores. Figure A7 demonstrates the opposite problem, with recognizable target content receiving zero accuracy and completeness. For a benchmark paper, this undermines the central contribution.

  • Appendix G documents incomplete judge checkpoint? And that GPT Image 2 does not have full coverage? Are all models compared on equivalent prompt subsets then?

Questions for authors
  • How many images did the ten participants assess? What exactly did the ten-person human evaluation assess? How were the images sampled across models, and score ranges?

  • Appendix H.2 identifies substantial overlap with existing benchmarks, particularly InfoTextBench. What specific evaluation capability does UltraText provide beyond these resources, and which result demonstrates that contribution?

Reviewer_cKyAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

UltraText Bench evaluates dense visual text rendering in prompt-only image generation. It contains 432 English and Chinese prompts across 24 scene categories and three difficulty levels, paired with structured references for individual text regions. The evaluation uses Q-Judger to produce six image-level ratings, aggregated into text fidelity, clarity, spatial quality, scene quality, and a composite score. The authors compare 24 model configurations and report qualitative inspection involving ten human participants.

The dataset design has practical value. However, the paper provides insufficient evidence that the automatic evaluator reliably measures the fine-grained text errors central to the benchmark.

Critical strengths
  • Useful coverage and workload design. The benchmark covers 24 scene categories, two languages, and three difficulty levels. Each prompt specifies 4–12 text regions, with workloads reaching 5,230 target characters. This provides a useful setting for evaluating sustained text reproduction across diverse layouts and carriers.
  • Structured region references. Each region includes its target text, position, size, type, carrier, and importance. The generator receives only the natural-language prompt containing the target strings, while the structured reference is reserved for evaluation. These annotations provide a useful foundation for diagnosing omissions, placement errors, and failures involving supporting text.
Critical weaknesses
  • The human evaluation does not quantitatively establish evaluator reliability. Section 3.2 mentions ten participants but does not sufficiently describe the sampling procedure, annotation protocol, or assignment of images to participants. No quantitative human–judge agreement, inter-rater agreement, or human verification rate for maximum-rated images is reported. This is particularly concerning because 83.6% of valid GPT Image 2 images receive maximum scores on all six dimensions, while Figure A9 illustrates visibly degraded text despite maximum ratings. The example does not establish the frequency of evaluator errors, but it makes systematic validation essential.

  • The evaluation does not demonstrate that the region annotations are used reliably. Despite the detailed references, Q-Judger returns only six holistic image-level ratings, without region-level transcriptions, matches, or error evidence. The scoring prompt also lacks intermediate score anchors and an explicit rule for aggregating errors across regions. Providing the complete reference therefore does not establish that every region was checked. This limits both the interpretability and the evidential basis of the claimed whole-scene evaluation.

  • The experimental comparisons contain substantial confounding factors. English and Chinese prompts are independently authored, and their L3 workloads average approximately 2,387 and 744 characters, respectively. Difficulty levels change text load and region count together. Model configurations also differ in resolution and sampling settings. Although the paper acknowledges these limitations, the comparisons support descriptive observations about the evaluated configurations rather than isolated conclusions about language, text length, or acceleration.

Questions for authors
  1. Can you provide the human evaluation protocol and quantitative results, including sample selection, annotation instructions, inter-rater agreement, and human–judge agreement? In particular, what fraction of randomly sampled maximum-rated images contains character errors, omissions, or unreadable text?
  2. Is there quantitative evidence that Q-Judger detects errors across all requested regions, especially small or low-importance regions? A region-level audit or controlled perturbation study measuring sensitivity to substitutions, omissions, and misplaced text would materially strengthen my assessment.
  3. Do the reported language, difficulty, and Base/Turbo trends persist in comparisons that match workloads or control generation settings? Which conclusions remain supported under those controls?
Reviewer_jEutOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a text-to-image benchmark, UltraText Bench, targeting to generating text-rich images. This benchmark contains 432 prompts covering 24 types of domains, and three different levels according to the text length, and two languages including English and Chinese. Each input text prompt illustrates 4-12 text regions, with structured references describing the positions and sizes for each text prompt, which is used for evaluation instead of input. Authors borrow Q-Judger to evaluate the model. Four dimensions are considered for evaluation, including text fidelity, text clarity, spatial quality, and scene quality. These dimensions contribute to a composite score with weights of 60%, 30%, 5%, and 5%, respectively. Evaluations on 24 models show that current text-to-image models still face challenges in text-rich image generation.

Critical strengths
  1. The problem of text-rich image generation is important and practical in real application scenarios.
  2. The structured reference is valuable and more concrete compared to text-only image quality evaluation.
Critical weaknesses
  1. The paper writing is very bad. The main pages do not contain enough interesting and valuable information. I suspect that the paper is written by AI without any manually check or the authors wrote by themselves but did not polish the writing enough before submission. There are many details without clear previous explanation or very confusing. I will only give two examples: (a). In the abstract, the sentence " Qwen-Image-2512’s English composite falls from 86.50 at L1 to 42.86 at L3" is very unclear. There is no explanation of "L1/3" but they are mentioned in the abstract. (b). In line "EN design bands are 250–600, 600–1200, and 1200+ GT characters". What is GT?
  2. The discussion of evaluation score and method is not strong and reliable enough. Although in Sec. 3.2, authors mention that there are 10 human participants, they fail to mention details include the number of evaluated images, the sampling procedure, the number of annotators per image, whether automatic scores were hidden from annotators, and agreement both among annotators and between humans and the judge. I notice that in the Appendix I, they acknowledge that they are missing inter-rater agreement and high-score alignment statistics, this is a severe weakness, which can not be accepted even mentioned in the limitation section.
  3. The contribution is not strong enough. There is no technical contribution. Besides, even for a benchmark work, the difference is not clear compared to previous work such as STRICT [1] and other related work mentioned in the paper. Even the paper claim that one contribution is considering more scenes (categories(, this is not strong enough. Authors need to conduct a comprehensive investigation of the exact scenes (categories) that previous benchmarks contain. Besides, the overall dataset only contains 423 prompts which is very small and not valuable at this time. Using ICLR standard, this is definitely not enough.

[1] STRICT: Stress-Test of Rendering Image Containing Text

Questions for authors
  1. How was the human evaluation conducted?
  2. How reliable are maximum scores? In a randomly sampled subset of maximum-score images, how often do human annotators identify text or layout errors?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.