- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces AttriTextBench, a large scale comprehensive benchmark combining both generation, editing and convering different text attributes and scene text generation tasks. To support this, the authors created a comprehensive data curation pipeline and conducted evaluation on a large set of frontier models, and observe that despite strong generation accuracy, there remains a substantial gap between content accuracy and attribute control accuracy.
Critical strengths
-
Clear motivation and contribution. The curated large scale dataset serves as a solid contribution for comprehensively evaluating generative models' ability for generating visual text.
-
The observation of current models lack good attribute control is a helpful insight.
Critical weaknesses
- The highest pearson correlation of the agent judgement for the automated pipeline and human judgement described in Section 4 is 0.581. Though indicating positive correlation, this still raises concerns about the robustness of the proposed automated evaluation pipeline, especially for highly complex attribute judgement.
Questions for authors
- What are the failure cases where automated evaluation and human judgement differ? A detailed analysis of what patterns are potentially judged incorrectly would be helpful for justifying the soundness and robustness of the proposed benchmark.