- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces TechImage-Bench for evaluating whether text-to-image models can produce specification-faithful technical illustrations in biology, engineering, and general domains. The benchmark comprises 654 tasks, 6,076 hierarchical criteria, and 44,131 binary checks derived from source text and reference figures through an LMM-assisted pipeline with expert refinement.
Critical strengths
Technical image generation is a meaningful limitation of existing text-to-image benchmarks.
TechImage-Bench provides 654 tasks, 6,076 criteria, and 44,131 binary checks across three domains. The hierarchical rubrics enable more detailed error diagnosis than a single holistic score.
Critical weaknesses
Insufficient validation of the proposed evaluation methodology. The central contribution is a rubric-based evaluation framework, yet the paper does not systematically compare it against existing approaches or direct LMM-based evaluation. Although human agreement results are reported, the lack of comparative evidence makes it difficult to establish whether rubric decomposition actually improves evaluation accuracy or reliability.
Potential evaluator bias in the main results. All models are evaluated exclusively using OpenAI o4-mini, while several evaluated generators also come from OpenAI. Prior research has raised concerns about LLM evaluators favoring models from the same provider. Without cross-evaluator validation or independent human judgments, potential same-provider bias cannot be ruled out, casting doubt on the fairness of the reported model rankings.
Questions for authors
How does rubric-based evaluation compare with existing evaluation methods in terms of agreement with human experts?
Have the authors tested alternative evaluators from independent model providers to rule out same-provider bias? Do the model rankings remain consistent when evaluated by non-OpenAI models or domain experts?
How many samples were used for human validation, how were they selected, and what was the inter-annotator agreement?
What are the individual contributions of task-specific rubrics, domain-specific rubrics, and hierarchical scoring?
Thank you for your valuable feedback!
Comparison with Direct LMM-Based Evaluation
We conducted additional experiments comparing our method with direct LMM-based evaluation. As shown in the table, our rubric-based evaluation exhibits a significant positive Spearman correlation with human judgments, whereas direct LMM-based evaluation does not correlate with human judgments.
Fairness of Model Rankings
o4-mini is a language model developed by OpenAI, which we use as an evaluator to assess multiple image generation models. Although some of the evaluated models, such as GPT-Image and GPT-Image-2, are also developed by OpenAI, they differ from o4-mini in their model architectures and training data.
As described in line 822, we conducted experiments to verify the consistency between LMM evaluation results and human judgments.
Furthermore, we performed additional experiments using Google’s Gemini-3.1-Pro as the evaluator. As shown in the table below, its results also closely align with human judgments, indicating that our fine-grained rubrics yield consistent agreement with human evaluations across evaluators from different providers, supporting the stability of image generation model rankings.
Human Validation
Refer to line 804 onward for details on the experts and evaluation criteria. We recruited 20 domain experts and randomly sampled 100 tasks across the three domains, covering 3,773 rubric items. To improve the consistency and reliability of the human evaluations, each item was first assessed by one expert and then randomly assigned to another expert for review.
Refer to line 235 for details on dataset construction. All 654 tasks and their reference images were screened and reviewed by domain experts to ensure the scientific validity, evaluation value, and feasibility of the tasks.
Individual Contributions of TechImage-Bench Components
As described in line 246, we explain the rationale for designing task-specific and domain-specific rubrics.
In lines 288 and 379, we explain that hierarchical scoring is designed to measure the degree of overall compliance with the constraints and provide richer and more diagnostic feedback, allowing for a more precise characterization of the limitations of current models in technical image generation.
These three components play different roles. To assess their individual contributions, we constructed three sub-benchmarks and conducted additional experiments. As shown in the table, the full benchmark achieves substantially higher correlation with human judgments than any of the three sub-benchmarks, indicating that combining these components improves alignment between automated evaluation and human judgments.
Our task-specific rubrics focus on the technical correctness of task-specific content, while domain-specific rubrics address quality requirements shared across tasks within a domain. Hierarchical scoring aggregates fine-grained checks to characterize the extent to which the technical requirements of each criterion are satisfied. Together, these components enable our benchmark to diagnose fine-grained errors from multiple technical perspectives and provide an overall assessment that better aligns with human judgments.