acceptodds
Withdrawn from ICLR 2027

AttriTextBench: Fine-Grained Attribute Evaluation for Unified Visual Text Generation and Editing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent state-of-the-art models have advanced rapidly in both text-to-image generation and instruction-based editing. Despite this progress, improving visual text synthesis remains a challenging and evolving frontier for image generation models. However, existing benchmarks primarily focus on content accuracy or rely on holistic whole-image evaluation. The former overlooks attribute control, while the latter cannot localize failures at individual text targets. We therefore introduce , a fine-grained benchmark for attribute-level evaluation of visual text generation and editing. , AttriTextBench covers generation tasks and editing tasks such as text addition and modification across multiple rendering scenes. , we explicitly evaluate multiple attributes, such as font, size and weight. Specifically, we curate editing instructions from real user traffic and derive generation prompts from real images. Each target records its text content and applicable attributes. , we decompose each user prompt into atomic operations. For each operation, we resolve a target bounding box in which OCR measures content and a visual judge agent assesses each applicable attribute with a binary verdict. Meanwhile, masked background comparison evaluates background preservation for editing tasks. AttriTextBench contains 721 generation prompts with 10,803 text segments and 2,854 editing instructions with 3,449 operations. Experiments on 14 models show that AttriTextBench exposes fine-grained weaknesses in visual text control and provides actionable guidance for future model development.

Primary area:
datasets and benchmarks
Keywords:
Visual Text, Generation and Editing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rYDTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AttriTextBench, a large scale comprehensive benchmark combining both generation, editing and convering different text attributes and scene text generation tasks. To support this, the authors created a comprehensive data curation pipeline and conducted evaluation on a large set of frontier models, and observe that despite strong generation accuracy, there remains a substantial gap between content accuracy and attribute control accuracy.

Critical strengths
  1. Clear motivation and contribution. The curated large scale dataset serves as a solid contribution for comprehensively evaluating generative models' ability for generating visual text.

  2. The observation of current models lack good attribute control is a helpful insight.

Critical weaknesses
  1. The highest pearson correlation of the agent judgement for the automated pipeline and human judgement described in Section 4 is 0.581. Though indicating positive correlation, this still raises concerns about the robustness of the proposed automated evaluation pipeline, especially for highly complex attribute judgement.
Questions for authors
  1. What are the failure cases where automated evaluation and human judgement differ? A detailed analysis of what patterns are potentially judged incorrectly would be helpful for justifying the soundness and robustness of the proposed benchmark.
Reviewer_xQZxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AttriTextBench, a fine-grained benchmark for visual text generation and editing. It performs region-level evaluations of text content and five visual attributes across eight scenarios and four tasks. The benchmark combines atomic operations, OCR-based content scoring, agent-based attribute evaluation, and background preservation metrics. The results indicate a significant gap between content accuracy and attribute control capabilities, particularly in text-dense and multi-operation scenarios.

Critical strengths
  1. This paper introduces AttriTextBench. This framework uses real images and user instructions to evaluate text generation and editing capabilities across various scenarios, operations, and five visual attributes.

  2. Its region-level metrics separately assess text content, visual attributes, and background preservation, thereby facilitating the analysis of model failures.

  3. Test results for 14 models reveal significant weaknesses in attribute control, dense text generation, and multi-operation editing, highlighting the value of this benchmark for model diagnosis.

Critical weaknesses
  1. The use of the term “Agent” may not be fully appropriate. Mainstream agents are generally expected to have capabilities such as tool use, task planning, or autonomous decision-making. In contrast, the “Agent” in this paper is closer to a multi-model data processing workflow that mainly relies on MLLMs for visual evaluation. Therefore, it does not fully reflect the typical capabilities of an agent, and the term may be somewhat misleading.

  2. The analysis of failure cases is not sufficiently in-depth. Although the paper identifies several major performance bottlenecks, the discussion mainly stays at the level of observed results. It does not clearly distinguish whether the errors are caused by region localization, text rendering, attribute control, or background preservation. We recommend adding controlled experiments or visualizations to better analyze the effects of these factors.

  3. The writing of the paper could be further improved. Some expressions are not sufficiently precise, and the use of language and tense is inconsistent. For example, the “Contributions” section states “We introduce AttriTextBench,” while the “Conclusions” section states “We presented AttriTextBench.” We recommend using a more consistent writing style and carefully reviewing the paper for grammatical and language issues.

Questions for authors
  1. The use of the term “Agent” may not be fully appropriate. Mainstream agents are generally expected to have capabilities such as tool use, task planning, or autonomous decision-making. In contrast, the “Agent” in this paper is closer to a multi-model data processing workflow that mainly relies on MLLMs for visual evaluation. Therefore, it does not fully reflect the typical capabilities of an agent, and the term may be somewhat misleading.

  2. The analysis of failure cases is not sufficiently in-depth. Although the paper identifies several major performance bottlenecks, the discussion mainly stays at the level of observed results. It does not clearly distinguish whether the errors are caused by region localization, text rendering, attribute control, or background preservation. We recommend adding controlled experiments or visualizations to better analyze the effects of these factors.

  3. The writing of the paper could be further improved. Some expressions are not sufficiently precise, and the use of language and tense is inconsistent. For example, the “Contributions” section states “We introduce AttriTextBench,” while the “Conclusions” section states “We presented AttriTextBench.” We recommend using a more consistent writing style and carefully reviewing the paper for grammatical and language issues.

Reviewer_uA74OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces AttriTextBench to address limitations in task coverage, attribute assessment, and localized error diagnosis in existing visual text benchmarks. Its main contributions are a unified data representation for generation and editing, annotations for five text attributes, and an atomic evaluation protocol combining region localization, OCR, and a visual judge.

Critical strengths
  1. Actionable, fine-grained diagnosis. The protocol separates text content, attribute compliance, and background preservation, identifying failures that whole-image scores or OCR accuracy alone may miss. This makes the benchmark useful for targeted model improvement.
  2. Unified coverage of generation and editing. A shared operation representation spans eight scenes and five text attributes, providing a structured framework for comparing visual text capabilities across both generation and editing tasks.
Critical weaknesses
  1. Limited validation of the attribute judge. The reported correlation with human labels is moderate (r=0.581) and is evaluated only on 196 editing examples. Without per-attribute results, human inter-annotator agreement, or ranking-stability analysis, it remains unclear whether judge errors materially affect the final scores and conclusions. Do the main model rankings remain stable under human evaluation?
  2. Insufficient detail on data construction. What are the specific sources of the editing image–instruction pairs and generation source images? How are text segments, attribute types, and attribute values selected? Which vision-language models, prompt templates, and filtering criteria are used? The current description is insufficient to assess data representativeness or reproduce the construction process.
Questions for authors
  1. Unclear workflow in Figure 3. Figure 3 does not clearly connect “Select IDs,” “Union,” and the context required for each attribute judgment, making it difficult to understand how target regions are selected, merged, and used for subsequent scoring.
  2. Insufficient validation of output localization. The paper states that source-image boxes are manually verified, but does not clarify whether verification also covers output-region relocation, OCR region selection, and merging. Localization accuracy is not reported. How is missing target text distinguished from text that is present but not successfully localized?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.