- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces TextCanvasBench, a benchmark for evaluating text-rich image generation. The core methodological contribution is a two-level evaluation framework with four diagnostic dimensions: D1 (glyph structure quality), D2 (complete-text accuracy), D3 (layout and style fidelity), and D4 (semantic alignment). The benchmark is constructed via a three-stage pipeline: data collection, paired Chinese/English prompt generation, and frozen D3/D4 specification construction. The Experiments reveals differences in long-text rendering capabilities that existing benchmark scores can obscure. Its detailed scores distinguish between models that receive similar scores on other benchmarks, while its rankings align more closely with human judgments than those of most benchmarks.
Critical strengths
- The decomposition of text-rich generation evaluation into four dimensions with frozen specifications is a principled advance over aggregate scoring.
- The paper is well written and easy to follow, and the authors conducted comprehensive experiments with a benchmark covering a wide range of domains.
Critical weaknesses
- Insufficient evidence for the claimed alignment with human judgment. The paper reports Spearman rank correlations between TextCanvasBench rankings and human judgments, substantially higher than those reported for the compared benchmarks. However, the paper does not sufficiently describe the protocol used to construct the human rankings, including the number of annotators, sample size, evaluation tasks, blinding procedures, inter-annotator agreement, and statistical uncertainty.
- Limited novel insights beyond the established importance of long-text rendering. I did not gain many heuristic insights from the experiments conducted on the benchmark. The challenge of rendering long text in generated images has already been emphasized in prior work, including the studies cited by the paper. Although TextCanvasBench provides a more comprehensive evaluation framework, the paper does not sufficiently demonstrate what new understanding of model capabilities, failure mechanisms, or practical strategies its benchmark enables beyond what is already known.
Questions for authors
Can you provide the detailed prompt in Figure 1? As far as I know, Qwen-Image-2.0 should not behave so poorly on long-text image generation. I want to check this on my own device.
Thank you. References: revised manuscript; additions in blue.
Q1: What supports the claimed alignment with human judgments?
Response: Thank you for highlighting the need to substantiate alignment with human judgments. We provide the protocol and reliability statistics below.
(1) Samples/raters. From the initial five-model pool (complete generations/matching prompts), we sampled 200 by domain/subcategory proportions, then selected 100 covering 8 domains/38 subcategories, 50/language. Selection favored longer text within coverage constraints, not uniform sampling. Nine models: 900 images, each independently rated by 6 bilingual graduate students: 5,400 records/21,600 dimension scores.
(2) Criteria/blinding. Raters were briefed and viewed full prompts/text/images. D1: glyph integrity/legibility; D2: text accuracy/completeness; D3: explicit layout/style; D4: meaning/text-object associations. Scores: 1-5, severe failure to full/near-full fulfillment. Model identities, automatic scores and other ratings were hidden; outputs randomized.
(3) Aggregation. Average six raters/four dimensions per image, then 50 images/model/language to rank nine models.
(4) Agreement. Ordinal Krippendorff's alpha (Table 9, p. 21):
(5) Ranking/CI (Table 10, p. 21):
20,000 prompt-cluster bootstrap resamples retain all models/ratings per prompt, comparing resampled human with fixed automatic rankings. See Appendix D.1, pp. 20-21; interface: Figure 12, p. 21.
Q2: What new insights go beyond the established difficulty of long-text rendering?
Response: Thank you. Our task representation enables requirement-level diagnosis and matched analysis of unchanged information, beyond text-length comparisons.
(1) Task-design advantage: jointly specifying text, presentation and relations. Prompts provide complete required lettering, scenes and explicit organization/style (Figure 8, p. 15). Specifications retain text-block identities, roles and subject-value-unit/object relations. D1-D4 separately assess glyphs, textual accuracy, layout/style and semantics. This locates failed requirements even at similar total scores and enables fixed-content comparisons under changing surrounding requirements.
(2) New insight: readable strings do not ensure correct information. In Figure 7b, p. 10, Seedream keeps core menu requirements and requested container count fixed, but expands surrounding text from 256 to 1024 target characters. It changes the required RMB 12 per skewer to RMB 12 per portion: readable text changes pricing meaning. Boogu must assign Brown Rice 37%, yet outputs both 43% and 37%: required strings coexist with conflicting values. Of its 21 images with available D1>=90, 15 contain readable substitutions or definite binding errors (Figure 7a, p. 10). Original association swaps preserve strings, changing D2 by -0.2 points but targeted D4 associations by -100.0; color edits and valid spacing redesigns produce different responses (Figure 5b, p. 9). These are targeted checks, not full D3/D4 totals. The finding exposes field/relation errors that surface text quality can obscure, motivating separate diagnosis.
(3) New insight: added surrounding text can damage unchanged information. We analyze 240 existing outputs: 12 scenes, five models, N=256/1024 crossed with requested containers K=4/16. Four core facts/scene stay fixed. At matched scene/model/K/actual resolution, increasing N changes the resolved-fact fraction with correct content and associations (Figure 13a, p. 24):
GPT-Image-2/Nano-Banana-Pro change by -3.1/+1.0 pp, retaining 189/192 and 191/192 audited facts. Tracking the same facts separates damage to existing information from errors in added text, rather than merely comparing unrelated length groups. This supplies paired training/testing conditions with fixed facts and expanded surrounding text, checking omissions, unit substitutions and binding conflicts. Model-design implications: Table 13, p. 25. The analysis is added in Section 4.4, pp. 9-10; full settings/results: Appendix D.3, pp. 23-25, marked in blue.
Q3: Can you provide Figure 1 prompts and clarify the model?
Response: Thank you for flagging the label; we apologize for the error. Figure 1 uses Qwen-Image, not Qwen-Image-2.0, now corrected. The supplement "Figure 1 Reproduction Material" provides complete short/long prompts and an additional English dataset example with its full visual description and required text for independent testing.