acceptodds
Under review as a conference paper at ICLR 2027

ExtremeTextBench: Evaluating Long-Form Text Rendering and Typeface Fidelity in Text-Rich Image Generation

Abstract

Recent image-generation models have made rapid progress in text rendering. However, most existing benchmarks evaluate rendering accuracy under conventional settings, focusing primarily on short words, phrases, or moderately long text. As model capabilities continue to advance, these settings offer increasingly limited discriminative power and provide little insight into whether models can satisfy demanding design requirements, including long-form text rendering, small-scale text rendering, and precise typeface control. We introduce ExtremeTextBench, a bilingual benchmark that systematically evaluates two underexplored capabilities: rendering text under extreme length and size conditions, and adhering to explicit typeface constraints. ExtremeTextBench comprises 600 English and Chinese prompts, 3,695 annotated text segments, 78 typeface families, and 24 design categories drawn from realistic text-rich applications, including posters, presentation slides, e-commerce graphics, menus, and invitations. By analyzing performance across text-length and text-size ranges (with target lengths of up to 428 words in English and 1,584 characters in Chinese, and an observed small-text range of pixels), ExtremeTextBench separately examines how rendering performance varies along these two dimensions, providing a more discriminative assessment of current state-of-the-art models. We also introduce a typeface-control evaluation method to assess whether generated text faithfully follows explicitly specified typefaces. Experiments on 16 state-of-the-art image-generation models reveal persistent limitations in rendering ultra-long text and text at small scales, while precise typeface control remains an open challenge. ExtremeTextBench fills an important gap in existing evaluation by providing a realistic and diagnostic testbed for the next generation of text-capable image-generation models.

Primary area:
datasets and benchmarks
Keywords:
benchmark, quality assessment, Text-to-image generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NAisOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ExtremeTextBench, a benchmark for evaluating image generation models on long-text rendering, small-text rendering, and adherence to specified typefaces. It contains 600 English and Chinese prompts, 3,695 text segments, 78 typeface families, and 24 design categories. Experiments on 16 models show that long and small text remain challenging, and accurate text rendering does not necessarily imply adherence to the requested typeface.

Critical strengths
  1. The paper is well written and easy to follow, with a clear presentation of the benchmark construction and evaluation protocol.
  2. ExtremeTextBench addresses a meaningful evaluation problem by assessing long-text rendering, small-text rendering, and typeface fidelity in practical text-rich design scenarios.
Critical weaknesses
  1. The paper does not clearly show what makes the long-text setting challenging. The authors report length statistics and results across length ranges, but lacks examples showing how long the text is and where models fail. Longer text also tends to involve smaller fonts and more complex layouts, so it is difficult to tell which factor causes the performance drop.

  2. The reason for combining font evaluation with long-text evaluation is not clear enough. Font control has practical uses, but the paper mostly evaluates text accuracy, without showing what new insights come from combining them. Moreover, font evaluation only includes sufficiently matched text regions of at least 50 pixels in height, leaving out the small-text setting that the paper emphasizes.

  3. The added value over existing benchmarks remains unclear. The evaluation mainly uses OCR-based text scores and multimodal-model-based font scores, and the results mostly describe score differences between models. Since existing benchmarks use different metrics and evaluation procedures, a wider score spread does not by itself establish a better benchmark.

Questions for authors

see weakness above

Reviewer_fJc8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets a real and timely gap: existing text-rendering benchmarks (LongText-Bench, 66 words; CVTG-2K, 30 words) are saturated at 96–99, and explicit typeface control is a hard requirement in real design workflows that prior work only touches coarsely. The construction effort is solid and the writing is clear. However, the paper does not establish the validity bounds of its own measurements, and one of its headline claims is not supported by the experimental design.

Critical strengths

1)Well-chosen problem, honestly scoped. The saturation argument is correct and the related-work discussion names concrete limitations of each predecessor rather than gesturing at them. 2)Annotation density enables the intended slices. Binding segment text, segment role, family, and a reference rendering per segment is what makes multi-axis analysis possible, and the 2,000→600 curation with ≥3 reviewers per item on five criteria is documented more candidly than most benchmark papers. 3)The per-segment reference design for typeface is the right call. Comparing against a rendering of the same segment in the same family removes confounds of size, weight, and background that a single global family sample would introduce, and the rubric explicitly rejects credit for mere serif/sans-serif category match.

Critical weaknesses

1)OCR error is an unbounded confound in the headline metric. Text F1 depends entirely on concatenated OCR output. At 428 words and ≤20px, the OCR pipeline's own error rate becomes the measurement floor, so "models degrade on long/small text" cannot be distinguished from "OCR degrades on long/small text." No upper-bound experiment is reported. The authors already have the ideal control: compose the reference renderings into full layouts, run the identical OCR pipeline, and report the resulting F1 and detector recall — this separates "not rendered" from "rendered but not detected." This is standard practice for text-rendering benchmarks. Relatedly, §3.5.3's local F1 is conditioned on detection; per-model detection coverage should be reported, otherwise the denominators differ across models. There is also a direct confound: Seedream 4.5 and HiDream-O1-Image generate at 2048² and are downsampled to 1024², which specifically damages the small-text regime the paper targets — please re-evaluate at native resolution. 2)Typeface track has a selection bias and an unvalidated judge. Font is computed only over crops with LCS ≥ 0.5 and size ≥ 50px, i.e. exactly where the model already rendered correct, legible content. Weaker models are thus scored on a smaller and easier subset. The per-model eligibility rate (coverage) is never reported — if a strong model covers 80% and a weak one 20%, the Font column in Table 2 is not comparable, and the "correctness does not imply typeface fidelity" comparison (Qwen-3.0 Pro 57.67 vs Nano Banana 2.0 77.31) may be a coverage artifact. Report coverage, and recompute Font on a matched (minimum common) coverage. Second, the judge has no negative control: pairing crops with the wrong family reference (especially near-duplicates: Helvetica/Arial, SimHei/Heiti SC — and note Source Han Sans and Noto Sans SC are literally the same family under two names, yet counted as two) would reveal whether the 0–3 score measures family identity at all, versus "looks like plausible text." Third, Fig. 6(c)(d) spans only 1.0–2.5 on a 3-point scale, so the judge essentially never uses 0 or 3 and the scale has collapsed; this undermines claims such as the 25.78-point Helvetica–Montserrat gap. Fourth, using GPT-5.5 both to generate the target text and to judge typeface invites correlated model-family preference — add a second judge from another vendor. 3) The "wider separation" argument does not yet exclude noise or saturation. Table 3 compares dispersion (range/SD/pairwise gap) across different metrics (Word Accuracy vs Text F1). Dispersion depends on metric scale, ceiling location, and difficulty, so cross-metric SD comparison is not a valid measure of discriminative power — being harder is not the same as being more discriminative. Moreover, only one sample is generated per prompt and no seed variance is reported, while several CIs in Table 2 are already ±3 points; if seed-to-seed variation is of the same order as inter-model gaps, larger spread may simply mean more noise. Please report multi-seed variance (≥3 seeds, ≥6 models) and express separation as an effect size. Also, the bottom of Table 2 (18.24 / 32.38 / 46.40) is failure-saturated — differences there carry no capability ordering and should be discussed as a floor effect. 4) "Joint" evaluation is overstated. Task II requires size ≥ 50px, which excludes the ≤20px regime entirely. Typeface fidelity is therefore never measured under the paper's other headline condition, so the three capabilities are not evaluated jointly. Either revise the claim or add a small-scale typeface evaluation in the sub-50px regime.

Questions for authors
  1. What Text F1 and detector recall does the OCR pipeline achieve on composed reference renderings, and how does that upper bound decay in the small-text and long-text bins?
  2. What is the Font eligibility rate per model, and does the Table 2 ordering change under matched coverage?
  3. What does the judge score for crops paired with wrong-family references, and what is its accuracy on near-duplicate pairs (Helvetica/Arial, SimHei/Heiti SC)?
  4. How large is seed-to-seed variance in Text F1 relative to the reported inter-model gaps?
  5. Table 1 compares maximum lengths (428 vs 66), but the distribution here is 46–428. Could you report median/P90 and length-conditioned comparisons instead?
Reviewer_9vavOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ExtremeTextBench, a bilingual benchmark for evaluating long-form text rendering, small text, and typeface fidelity in image generation. It contains 600 prompts and evaluates 16 image generation models using OCR-based metrics and a multimodal font similarity judge.

Critical strengths
  • Evaluating long text and precise typography is relevant to practical image generation applications.
  • The benchmark covers multiple design scenarios and provides useful comparisons across recent image generation models.
Critical weaknesses
  • The proposed Text F1 metric is based on multiset matching and ignores text order. A model could produce the correct words in the wrong order and still receive a high score. This is a significant limitation for evaluating long-form text rendering.
  • Typeface fidelity is evaluated only on regions with sufficient text matching and a minimum size of 50 pixels. This excludes difficult small-text regions and may introduce selection bias when comparing models with different text rendering capabilities.
Questions for authors
  • Can the authors report sequence-sensitive metrics, such as edit distance, and examine whether model rankings change?
  • How does typeface fidelity change when accounting for unmatched or excluded text regions? Could the authors report the evaluation coverage for each model?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.