acceptodds
Under review as a conference paper at ICLR 2027

TextCanvasBench: A Comprehensive Benchmark for Complex and Long-Text Image Generation

Abstract

Recent open- and closed-source text-to-image (T2I) models can generate text-rich images, including scientific figures and presentation slides, yet benchmark evaluation has not kept pace with this capability. Existing text-rendering benchmarks typically focus on short text, a small number of text regions, and image-level or OCR-based aggregate metrics, limiting their ability to assess long-form content, multi-region layouts, and block-level stylistic and semantic relations in practical T2I scenarios. To address these gaps, we introduce TextCanvasBench, a benchmark of 1,600 specification-guided samples spanning 8 domains and 38 subcategories of real-world text-rich visuals, from commercial posters and academic slides to digital interface panels. The required text ranges from short labels to more than 1,000 characters, and the annotations identify text blocks and their roles. We evaluate generation through four complementary dimensions, moving from the whole image to individual text blocks and regions. At the macro image level, we assess glyph quality, the completeness and accuracy of the required text, and whether the image conveys the prompt's overall meaning. At the micro block and region level, we assess position, orientation, arrangement and reading order, font, size, and color, as well as the facts and associations tied to individual blocks. In our experiments, we evaluate nine state-of-the-art open- and closed-source T2I models (e.g., Boogu-Image and GPT-Image-2). The results show that TextCanvasBench covers a wider range of practical text-rich scenarios and reveals differences in long-text rendering capabilities that existing benchmark scores can obscure. Its detailed scores distinguish models with similar scores on other benchmarks, while its rankings agree more closely with human judgments than those of most benchmarks in our comparison.

Primary area:
datasets and benchmarks
Keywords:
Text-to-Image Generation, Text-Rich Image Generation, Long-Text Rendering, Benchmark, Fine-Grained Evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_G4hkOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces TextCanvasBench, a benchmark for evaluating text-rich image generation. The core methodological contribution is a two-level evaluation framework with four diagnostic dimensions: D1 (glyph structure quality), D2 (complete-text accuracy), D3 (layout and style fidelity), and D4 (semantic alignment). The benchmark is constructed via a three-stage pipeline: data collection, paired Chinese/English prompt generation, and frozen D3/D4 specification construction. The Experiments reveals differences in long-text rendering capabilities that existing benchmark scores can obscure. Its detailed scores distinguish between models that receive similar scores on other benchmarks, while its rankings align more closely with human judgments than those of most benchmarks.

Critical strengths
  1. The decomposition of text-rich generation evaluation into four dimensions with frozen specifications is a principled advance over aggregate scoring.
  2. The paper is well written and easy to follow, and the authors conducted comprehensive experiments with a benchmark covering a wide range of domains.
Critical weaknesses
  1. Insufficient evidence for the claimed alignment with human judgment. The paper reports Spearman rank correlations between TextCanvasBench rankings and human judgments, substantially higher than those reported for the compared benchmarks. However, the paper does not sufficiently describe the protocol used to construct the human rankings, including the number of annotators, sample size, evaluation tasks, blinding procedures, inter-annotator agreement, and statistical uncertainty.
  2. Limited novel insights beyond the established importance of long-text rendering. I did not gain many heuristic insights from the experiments conducted on the benchmark. The challenge of rendering long text in generated images has already been emphasized in prior work, including the studies cited by the paper. Although TextCanvasBench provides a more comprehensive evaluation framework, the paper does not sufficiently demonstrate what new understanding of model capabilities, failure mechanisms, or practical strategies its benchmark enables beyond what is already known.
Questions for authors

Can you provide the detailed prompt in Figure 1? As far as I know, Qwen-Image-2.0 should not behave so poorly on long-text image generation. I want to check this on my own device.

1 comment

Thank you. References: revised manuscript; additions in blue.

Q1: What supports the claimed alignment with human judgments?

Response: Thank you for highlighting the need to substantiate alignment with human judgments. We provide the protocol and reliability statistics below.

(1) Samples/raters. From the initial five-model pool (complete generations/matching prompts), we sampled 200 by domain/subcategory proportions, then selected 100 covering 8 domains/38 subcategories, 50/language. Selection favored longer text within coverage constraints, not uniform sampling. Nine models: 900 images, each independently rated by 6 bilingual graduate students: 5,400 records/21,600 dimension scores.

(2) Criteria/blinding. Raters were briefed and viewed full prompts/text/images. D1: glyph integrity/legibility; D2: text accuracy/completeness; D3: explicit layout/style; D4: meaning/text-object associations. Scores: 1-5, severe failure to full/near-full fulfillment. Model identities, automatic scores and other ratings were hidden; outputs randomized.

(3) Aggregation. Average six raters/four dimensions per image, then 50 images/model/language to rank nine models.

(4) Agreement. Ordinal Krippendorff's alpha (Table 9, p. 21):

LanguageD1D2D3D4
English.783.761.622.722
Chinese.736.701.683.720

(5) Ranking/CI (Table 10, p. 21):

LanguageSpearman rho95% CI
English.950[.917,.950]
Chinese.917[.883,.917]

20,000 prompt-cluster bootstrap resamples retain all models/ratings per prompt, comparing resampled human with fixed automatic rankings. See Appendix D.1, pp. 20-21; interface: Figure 12, p. 21.

Q2: What new insights go beyond the established difficulty of long-text rendering?

Response: Thank you. Our task representation enables requirement-level diagnosis and matched analysis of unchanged information, beyond text-length comparisons.

(1) Task-design advantage: jointly specifying text, presentation and relations. Prompts provide complete required lettering, scenes and explicit organization/style (Figure 8, p. 15). Specifications retain text-block identities, roles and subject-value-unit/object relations. D1-D4 separately assess glyphs, textual accuracy, layout/style and semantics. This locates failed requirements even at similar total scores and enables fixed-content comparisons under changing surrounding requirements.

(2) New insight: readable strings do not ensure correct information. In Figure 7b, p. 10, Seedream keeps core menu requirements and requested container count fixed, but expands surrounding text from 256 to 1024 target characters. It changes the required RMB 12 per skewer to RMB 12 per portion: readable text changes pricing meaning. Boogu must assign Brown Rice 37%, yet outputs both 43% and 37%: required strings coexist with conflicting values. Of its 21 images with available D1>=90, 15 contain readable substitutions or definite binding errors (Figure 7a, p. 10). Original association swaps preserve strings, changing D2 by -0.2 points but targeted D4 associations by -100.0; color edits and valid spacing redesigns produce different responses (Figure 5b, p. 9). These are targeted checks, not full D3/D4 totals. The finding exposes field/relation errors that surface text quality can obscure, motivating separate diagnosis.

(3) New insight: added surrounding text can damage unchanged information. We analyze 240 existing outputs: 12 scenes, five models, N=256/1024 crossed with requested containers K=4/16. Four core facts/scene stay fixed. At matched scene/model/K/actual resolution, increasing N changes the resolved-fact fraction with correct content and associations (Figure 13a, p. 24):

ModelFidelity change (pp)Exploratory 95% scene-bootstrap CI
Seedream 5 Pro-12.6[-24.0,-3.4]
Boogu Turbo-22.9[-40.8,-4.4]

GPT-Image-2/Nano-Banana-Pro change by -3.1/+1.0 pp, retaining 189/192 and 191/192 audited facts. Tracking the same facts separates damage to existing information from errors in added text, rather than merely comparing unrelated length groups. This supplies paired training/testing conditions with fixed facts and expanded surrounding text, checking omissions, unit substitutions and binding conflicts. Model-design implications: Table 13, p. 25. The analysis is added in Section 4.4, pp. 9-10; full settings/results: Appendix D.3, pp. 23-25, marked in blue.

Q3: Can you provide Figure 1 prompts and clarify the model?

Response: Thank you for flagging the label; we apologize for the error. Figure 1 uses Qwen-Image, not Qwen-Image-2.0, now corrected. The supplement "Figure 1 Reproduction Material" provides complete short/long prompts and an additional English dataset example with its full visual description and required text for independent testing.

Reviewer_4hAbOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces TextCanvasBench, a benchmark designed to evaluate the generation capabilities of text-to-image models in scenarios involving long texts and complex typography. The benchmark comprises 1,600 samples spanning 8 domains and 38 subcategories, and evaluates models across four dimensions: glyph quality, text accuracy, layout style, and semantic consistency. Experiments show that it is more effective than existing benchmarks at distinguishing the text generation capabilities of different models and is more consistent with human judgments.

Critical strengths
  1. Comprehensive scene coverage. A benchmark test set consisting of 8 domains and 38 subcategories, totaling 1,600 samples, was constructed to test more realistic and diverse text-to-image generation scenarios.

  2. Detailed evaluation dimensions. From the overall image to the local text blocks, this benchmark test assesses character quality, text accuracy, layout style, and semantic consistency, making it easier to identify specific types of model failure issues.

  3. Stronger model discrimination ability. Experiments on 9 generation models show that this benchmark test can further distinguish models that have similar scores in existing benchmark tests, and its evaluation results are more consistent with human judgments.

Critical weaknesses
  1. There are inconsistencies in some of the statistical data. For example, the main text states that the dataset contains 42,853 labeled text blocks across 1,600 samples; however, Table 3 in the appendix reports an average of 26.76 text blocks per image. Based on this, the total should be 26.76 × 1,600 = 42,816, which differs from the figure in the main text by 37. Conversely, if the total of 42,853 is correct, the average should be approximately 26.78.

  2. The analysis of the reasons for the model's failure is still not thorough enough. The current analysis mainly focuses on the phenomena and failure modes, and has not further explored the relationship between these issues and the model architecture, training data, text representation, or generation mechanism. Therefore, its guidance significance for future model design and optimization is still limited.

Questions for authors
  1. There are inconsistencies in some of the statistical data. For example, the main text states that the dataset contains 42,853 labeled text blocks across 1,600 samples; however, Table 3 in the appendix reports an average of 26.76 text blocks per image. Based on this, the total should be 26.76 × 1,600 = 42,816, which differs from the figure in the main text by 37. Conversely, if the total of 42,853 is correct, the average should be approximately 26.78.

  2. The analysis of the reasons for the model's failure is still not thorough enough. The current analysis mainly focuses on the phenomena and failure modes, and has not further explored the relationship between these issues and the model architecture, training data, text representation, or generation mechanism. Therefore, its guidance significance for future model design and optimization is still limited.

2 comments

Thank you for the constructive feedback. References are to the revised manuscript; additions are blue.

Q1: How are inconsistent dataset statistics corrected?

Response: Thank you for catching these reporting errors. We apologize for the oversight and have corrected the affected statistics. The 1,600-sample annotation file contains 42,853 nonempty text blocks: mean 26.783125, rounded to 26.78, not 26.76. The 1-10 and 11-30 block groups are 450/704, not 452/702; >30 remains 446. Affected character-count cross-tabulation cells were corrected consistently. Median, P90 and range match the original table. Dataset/per-sample annotations were not changed. See Appendix A, Table 3, p. 13.

Q2: How do failures relate to architecture, training data, text representation and generation?

Response: Thank you for requesting a deeper connection to model design. We added an audit and matched comparisons of 240 existing outputs, yielding two findings and corresponding design implications (Section 4.4, pp. 9-10; Appendix D.3, pp. 23-24).

(1) Beyond glyph quality, models need to preserve field identities and semantic associations. English D1/D2 is 92.88/53.10 for GLM-Image and 93.28/49.80 for Z-Image-Turbo (Table 2). Figure 7b, p. 10, locates errors behind this separation. Seedream holds the core menu item, required price/unit and requested container count fixed, then expands surrounding required text from 256 to 1024 target characters. The higher-load output replaces the required RMB 12 per skewer with RMB 12 per portion. Boogu must associate Brown Rice with 37%, but assigns it both 43% and 37%. Thus readable fields can be substituted, and required strings can appear with conflicting subject-value bindings.

The audit covers four core facts/scene and 192 fact instances/model. Boogu has 81 missing/substituted instances and 12 definite binding errors/conflicts; 15 of its 21 images with available D1>=90 still contain readable substitutions or definite binding errors (Figure 7a). Original association swaps preserve strings, changing mean D2 by only -0.2 points but targeted D4 associations by -100.0 (Figure 5b, p. 9); these are targeted checks, not full D4 scores.

Architecture/representation implications. Public designs in Table 12, p. 24, address complementary aspects:

ModelPublic designAdditional objective
GLM-ImageGlyph Encoder; AR + diffusionGlobal content/relation constraints
Z-Image-TurboJoint text-visual S3-DiTFull-text identity/coverage
Qwen-ImageQwen2.5-VL + MMDiT; paragraph/template trainingRequired-instance coverage
Boogu TurboInstruction encoder + DiT; character coverageEntity-value-unit consistency

The errors motivate coupling glyph/text features with explicit field and instance identities, not only independent strings. Supervision should check substitutions, duplicates and contradictory bindings; rendering should preserve the intended correspondences, not merely the presence of target text.

(2) Expanding surrounding text can interfere with unchanged core information. We use 12 scenes (6/language), five models (GPT-Image-2, Nano-Banana-Pro, Seedream, Qwen-Image, Boogu), and target characters N=256/1024 crossed with requested containers K=4/16: 240 outputs, one per condition. Four predefined core facts remain fixed per scene. At matched scene/model/K/actual resolution, we increase N. Fidelity is the resolved-core-fact fraction with correct content and associations. Figure 13a, p. 24, reports:

ModelFidelity change (pp)Exploratory 95% scene-bootstrap CI
Seedream 5 Pro-12.6[-24.0,-3.4]
Boogu Turbo-22.9[-40.8,-4.4]

GPT-Image-2/Nano-Banana-Pro change by -3.1/+1.0 pp and retain 189/192 and 191/192 audited facts. Tracking the SAME facts distinguishes damage to existing information from errors confined to added text; Seedream's unit substitution illustrates this interference.

Training/generation implications. Paired training examples can keep core facts fixed while expanding surrounding text, supervising both new-text accuracy and unchanged-field retention. This extends character/paragraph coverage with an explicit preservation objective. Planning/layout assignment can retain field identities, while rendering and later corrections recheck values, units and relations for omissions, substitutions or conflicts. We have not inspected actual training corpora or controlled internal modules; these are design directions grounded in public designs and output evidence, not claims that particular data are missing or a module caused the errors. Additions are marked in blue.

Thank you for your response.

I have no further questions and maintain my recommendation to accept.

Reviewer_vbrtOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TextCanvasBench, a benchmark for evaluating text-to-image generation involving long text and complex layouts. It contains 1,600 samples spanning eight domains and 38 subcategories, with paired Chinese and English prompts. The evaluation separates four dimensions: glyph quality (D1), textual correctness (D2), layout and style (D3), and semantic alignment (D4). Experiments on nine generators, cross-benchmark comparisons, and controlled image edits aim to demonstrate improved model discrimination and agreement with human judgments.

The benchmark addresses a practically valuable problem, and the four-dimensional evaluation is useful for diagnosing different failure modes. However, the current validation does not sufficiently establish the reliability of the evaluation itself.

Critical strengths
  • Practical relevance. Long text, multiple text regions, and complex layouts are important requirements for applications such as infographics, slides, posters, and interfaces. Evaluating textual content together with presentation and semantic associations is a worthwhile direction.

  • Useful diagnostic decomposition. Separating glyph integrity, textual accuracy, layout/style fidelity, and semantic alignment helps expose failures that aggregate text scores can conceal. In particular, distinguishing correct strings from correct text-image associations is valuable. Preparing fixed, prompt-derived specifications before evaluating model outputs is also a sensible design choice.

Critical weaknesses
  1. The human evaluation lacks essential methodological details. Agreement with human rankings is a central validation claim, but the paper does not report the number of evaluated images, the number of annotators, the assessment criteria, or how individual judgments were aggregated. Inter-annotator agreement and statistical uncertainty are also missing. Consequently, the reported Spearman correlations are difficult to assess and provide insufficient evidence for the reliability of the proposed evaluation dimensions.

  2. Some scoring rules and experimental results require clarification. The main text allows unresolved or unknown observations, whereas Appendix C.1 states that all requirements are resolved before aggregation. The operational treatment of unresolved cases, evaluation failure rates, and effective sample coverage are therefore unclear. Additionally, Figure 6 reports a TextCanvasBench D1 score of 0.458 for Qwen-Image, substantially below the 0.8549/0.8675 reported in Table 2. Different subsets or configurations could explain this discrepancy, but these differences need to be documented to make the results interpretable and reproducible.

  3. D4 provides limited coverage of long-text semantics. The core textual-semantics component evaluates at most five selected facts per sample, even when the required text contains thousands of characters. A high D4 score therefore offers limited evidence about semantic preservation across the complete text. For example, GPT-Image-2 achieves D4 scores around 98 while its D2 scores are around 69. These dimensions measure different properties, but the paper should investigate whether selective fact coverage contributes to this gap and whether D4 detects meaningful errors outside the selected facts.

  4. Model coverage and ranking robustness are insufficiently established. The main evaluation includes nine generators, but only seven are shared across all columns in the cross-benchmark comparison, and the shared-evaluator ablation includes five. The open-weight group also includes FLUX.1 and SD3.5, which perform substantially worse on this task. Broad performance differences make aggregate dispersion less informative about the benchmark's ability to distinguish similarly capable models. The conclusions would benefit from comparisons among competitive models, repeated generations, and sensitivity analyses across model subsets.

Questions for authors
  1. Could you provide the complete human-evaluation protocol, including sample and annotator counts, assessment criteria, aggregation procedures, inter-annotator agreement, and confidence intervals for the reported correlations?

  2. How are unresolved observations handled in the reported scores? What are the evaluation failure rates and valid-sample coverage for each model? What subset or configuration differences explain the Qwen-Image D1 discrepancy between Figure 6 and Table 2?

  3. How are the five core semantic facts selected, and what fraction of the required content do they cover? Does D4 reliably penalize meaningful omissions or semantic errors outside these selected facts?

  4. Do the model-discrimination and human-agreement conclusions remain stable when restricting comparisons to similarly capable models, excluding the weakest baselines, and accounting for variation across repeated generations?

3 comments
AuthorsOpenReview ↗

Comment1 & 2

Thank you for the careful review.

Q1: Human-evaluation protocol and reliability?

Response: Thank you for this question.

(1) Samples/raters. From the initial five-model pool (complete generations/matching prompts), we sampled 200 by domain/subcategory proportions, then selected 100 covering 8 domains/38 subcategories, 50/language. Selection favored longer text within coverage constraints, not uniform sampling. Nine models: 900 images, each independently rated by 6 bilingual graduate students: 5,400 records/21,600 dimension scores.

(2) Criteria/blinding. Raters were briefed and viewed full prompts/text/images. D1: glyph integrity/legibility; D2: text accuracy/completeness; D3: explicit layout/style; D4: meaning/text-object associations. Scores: 1-5, severe failure to full/near-full fulfillment. Model identities, automatic scores and other ratings were hidden; outputs randomized.

(3) Aggregation. Average six raters/four dimensions per image, then 50 images/model/language to rank nine models.

(4) Agreement. Ordinal Krippendorff's alpha (Table 9, p. 21):

LanguageD1D2D3D4
English.783.761.622.722
Chinese.736.701.683.720

(5) Ranking/CI (Table 10, p. 21):

LanguageSpearman rho95% CI
English.950[.917,.950]
Chinese.917[.883,.917]

20,000 prompt-cluster bootstrap resamples retain all models/ratings per prompt, comparing resampled human with fixed automatic rankings. See Appendix D.1, pp. 20-21; interface: Figure 12.

Q2: How are unknowns scored, what are failure/coverage rates, and why does Qwen-Image's D1 differ between Figure 6 and Table 2?

Response: Thank you for flagging this; we apologize for the unclear descriptions. Appendix C.1 is corrected; new analysis appears in Appendix D.2, pp. 22-23, Tables 11-12, marked blue.

(1) Unknowns, confirmed errors and technical failures. Confirmed omissions, substitutions and requirement violations receive the prescribed penalties, rather than being classified as unknown. Bounded reinspection uses text identity, semantic regions/neighbors, full-resolution views and supported local enlargements with image context; positions stay in original coordinates. Request or output-format failures receive bounded retries. Genuinely indeterminate checks remain unknown and are excluded from score aggregation, not automatically treated as passes or violations. Appendix C.1 no longer asserts that all observations are resolved before aggregation.

(2) Supplementary evaluation and coverage. Targeted reinspection and bounded technical retries supplement the existing process. D3 coverage rises from 77.06%-88.13% to 98.45%-99.38%. Technical failure rates below are residual image-level failures after retries; coverage/failure rates combine both languages. Tables 11-12 report the nine-model results:

ModelOriginal coverage %Supplementary coverage %Technical failure %Original D3 meanSupplementary D3 meanOverall bounds (separate set)
GPT Image 286.4499.150.8591.4797.2196.32-97.17
Seedream 5 Pro84.1599.090.5890.6195.6894.76-95.67
Nano-Banana-Pro86.7999.030.2289.3595.1094.02-94.63
Boogu Turbo86.9499.340.2588.8194.6293.85-94.51
Z-Image-Turbo87.1999.380.1687.9091.4290.68-91.30
Qwen Image86.1699.240.1686.2890.3489.22-89.97
GLM77.0698.450.2881.1579.7977.43-78.98
FLUX88.1399.120.1355.6931.8632.07-32.95
SD3.585.3198.520.2541.4221.6621.66-23.14

(3) Scores, ranking and residual unknowns. Supplementary evaluation changes some absolute scores substantially. We report both versions, making the original scoring evidence explicit. Both versions weight language means equally in the same nine-model set; means use nonempty totals. All nine language-averaged D3 rank positions remain unchanged.

The overall bounds hold resolved judgments fixed and assign residual unknowns their minimum/maximum scores. Their widths are 0.61-1.55 points on a 0-100 scale, limiting the influence of remaining unknowns within the overall record set despite coverage below 100%. Bounds concern the separately aggregated overall records in the table.

(4) Figure 6 versus Table 2. Figure 6 uses 10 prompts emphasizing longer text, one image per prompt from each of five generators (50 images). Table 2 instead uses scored records from the full evaluation pool. Thus Qwen-Image's 0.458 and 0.8549/0.8675 (normalized to 0-1) are not repeated measurements on the same samples. The small subset is exploratory and selection-sensitive. The completed 200-prompt comparison, three generation runs and model-subset analyses are reported in Appendix D.5, pp. 27-28, Tables 17-19, and our response to Comment 4.

Q3: How are the five facts selected, how much content do they cover, and does D4 penalize meaningful errors outside them? Does selective coverage contribute to GPT-Image-2's D2-D4 gap?

Response: Thank you for highlighting the fixed fact budget's limitation for long-text semantics. We clarify selection and add coverage measurements, controlled error tests, and expanded D4 results. References below are to the revised manuscript.

(1) Fact selection and counting units. A model extracts semantic requirements from the complete required text, followed by human review. Selection prioritizes the main message, key values, entity relations and table subject-attribute-value bindings, retaining units, conditions, negations and referents. Pure position/font/arrangement belongs to D3; semantic label-value and text-object associations remain in D4.

The five-fact limit concerns checklist entries per sample. Coverage counts independently identified necessary facts: semantic mapping identifies fully covered facts and deduplicates them within each input. Entries and covered facts therefore use different counting units.

(2) Content coverage. On the same finalized inputs, we construct a five-entry control and expanded checklist, mapping both to an independently extracted and reviewed necessary-fact inventory. A fact counts as covered only when both mappings confirm full preservation of its content and bindings. All 1,600 inputs are audited, yielding 45,086 necessary facts (Table 14, p. 25):

SettingCovered factsCoverage
Five-entry control13,68530.35%
Expanded checklist36,82281.67%

Coverage is pooled covered facts divided by pooled necessary facts, not a macro-average of per-input percentages. Expansion adds 51.32 percentage points, directly checking more values, conditions and relations. Coverage remains below 100%; high D4 therefore means fulfillment of checked requirements, not complete preservation of all textual semantics.

(3) Errors outside the selected facts. An error covered by neither selected entries nor other D4 requirements receives no direct penalty in the original setting. We test expansion with correct/error image pairs introducing omissions, value/unit errors, object-association errors and condition/negation errors. Both settings evaluate identical pairs. Detection requires evidence identifying the inserted error, not just a lower score (Table 15, p. 26):

ScopeTestedEligibleFive-entry detectionExpanded detection
Shared facts121111/1111/11
Added facts12120/1212/12

Eligibility requires both settings to pass the correct control. One shared pair fails this criterion and is retained but excluded from the rate. In these controlled tests, expansion preserves detection on eligible shared facts and detects the inserted errors in previously unchecked facts.

(4) Expanded D4 and the D2-D4 gap. Scoring formulas and weights are unchanged (Table 16, p. 26):

ModelZH D4EN D4Check coverage %Technical failure %Old -> expanded rank
GPT Image 299.0398.9999.930.221 -> 1
Nano-Banana-Pro96.4396.0299.420.252 -> 2
Seedream 5 Pro92.0491.3499.920.393 -> 3
Z-Image-Turbo80.8980.4299.870.946 -> 4
Boogu Turbo74.4373.7699.851.344 -> 5
Qwen Image61.7561.3499.531.095 -> 6
GLM61.1560.1999.870.197 -> 7
FLUX4.3617.1099.870.258 -> 8
SD3.51.017.7699.740.169 -> 9

Language scores average complete-score images (0-100). Check coverage and residual API/format failure rates combine languages. Original ranks average languages equally; expanded ranks use pooled complete-score images in the same nine-model set. The order changes from Boogu-Qwen-Z-Image-Turbo to Z-Image-Turbo-Boogu-Qwen: superiority on a small fact selection need not extend to broader semantic checking.

GPT-Image-2's pooled expanded D4 remains 99.01, versus approximately 98 originally: high D4 is not confined to the five-entry setting. D2 measures required-text accuracy/completeness; D4 measures specified semantic requirements. Omissions/substitutions may incur D2 penalties, while expanded D4 directly checks meanings and associations. The composite combines D1-D4 to assess glyphs, text, layout/style and semantics. We report D4 fact coverage and additional error detection alongside its scores to characterize semantic evaluation scope. Selection details: B.3, p. 16; new experiments: D.4, pp. 25-27. Additions are blue.

Q4: Do model-discrimination and human-agreement conclusions remain stable for similarly capable models, after excluding weak baselines and accounting for repeated generations?

Response: Thank you for the suggestions. We expanded the shared-evaluator experiment and analyzed model subsets, within-family comparisons, repeated generations, and human-ranking agreement.

(1) Model set and prompt sampling. The original five-model ablation excluded FLUX.1/SD3.5. Our nine-model comparison excludes them and adds Z-Image and Boogu Base, retaining both Base/Turbo pairs. Each benchmark has its own 200 prompts; models share prompts within each benchmark. All use D1 (0–100). Prompts are selected as follows:

  • CVTG-2K (2,000): 25 randomly sampled from each of eight strata: CVTG/CVTG-Style × 2/3/4/5 text regions.
  • LongText-Bench (320): 50 randomly sampled per stratum: Chinese/English × short/long text (four strata).
  • TextAtlasEval (4,000): 50 randomly sampled from each of four subsets.
  • TextCanvasBench (1,600): retain the previous 100 and add 100, covering eight domains and prioritizing longer text, tables and multi-region samples; 100 Chinese-description and 100 English-description prompts.

For TextCanvasBench, we perform three independent generation runs on the same 200 prompts, holding model versions, generation parameters and evaluation settings fixed.

ModelCVTG-2KLongText-BenchTextAtlasEvalOurs R1Ours R2Ours R3MeanRange
GPT99.2297.9599.4398.2997.9797.5097.920.79
Nano95.2894.3297.5998.3397.6896.1897.402.15
Seedream94.7193.5497.2492.6392.7894.6293.341.99
GLM93.4194.3692.4692.4692.5192.9792.650.51
Z-Image94.0393.2791.4790.7890.4690.2290.490.56
Z-Image-Turbo86.9484.6392.1189.6989.4589.3489.490.35
Boogu Base93.2687.8791.3482.8581.7578.4881.034.37
Boogu Turbo93.2887.6390.9681.6680.8177.8980.123.77
Qwen95.0083.7091.2779.1480.2778.0779.162.20
Cross-model spread12.2814.258.4719.1917.7019.6118.76—

Mean/Range summarize the three TextCanvasBench run means; cross-model spread is highest minus lowest model mean. Z-Image denotes the additional base model, distinct from Z-Image-Turbo.

(2) Closely related models. Z-Image exceeds Z-Image-Turbo by 1.09, 1.01, and 0.88 points across runs; Boogu Base exceeds Turbo by 1.19, 0.94, and 0.59. Both within-family comparisons retain their direction. Boogu Base-minus-Turbo differences on CVTG-2K, LongText-Bench, and TextAtlasEval are −0.02, +0.24, and +0.38, versus +0.91 for the TextCanvasBench three-run mean. These comparisons show the same direction across all three runs.

(3) Separation and repeated-generation stability. Without FLUX.1/SD3.5, TextCanvasBench retains cross-model spreads of 19.19, 17.70, and 19.61, exceeding 12.28, 14.25, and 8.47 on the comparison benchmarks. Population variance over the nine model means (denominator 9) is 43.16, 42.20, and 58.07 across our runs, versus 9.06, 21.88, and 9.83 for CVTG-2K, LongText-Bench, and TextAtlasEval. Corresponding standard deviations are 6.57/6.50/7.62 versus 3.01/4.68/3.14. Both range and variance show larger separation on this model set without the weakest baselines.

Three-run averaging yields GPT > Nano > Seedream > GLM > Z-Image > Z-Image-Turbo > Boogu Base > Boogu Turbo > Qwen, matching run 2. Each model's across-run range is 0.35–4.37 points (GLM: 0.51). Rank changes across runs are confined to GPT/Nano and Boogu Turbo/Qwen; the other five models retain their positions. Thus, broad separation persists, with some close pairs exchanging order.

(4) Human-ranking agreement. Keeping existing human model means fixed, we recomputed Spearman correlations for the original nine models and the seven-model subset excluding FLUX.1/SD3.5. Automatic composites weight D1–D4 equally; the supplementary version retains D1/D2 and uses supplementary D3/D4.

Automatic evaluationNine-model EN ρNine-model ZH ρSeven-model EN ρSeven-model ZH ρ
Original0.95000.91670.89290.8214
Supplementary D3/D40.98330.98330.96430.9643

Agreement remains high after excluding weak baselines in both versions. Supplementary experiments appear in revised Appendix D.5, pp. 27–28: Table 17 (complete D1 comparison), Table 18 (dispersion), and Table 19 (human-ranking correlations). All additions are marked in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.