acceptodds
Under review as a conference paper at ICLR 2027

TechImage-Bench: Rubric‑Based Evaluation for Technical Image Generation

Abstract

We study technical image generation, where a model must synthesize information‑dense, scientifically precise illustrations from detailed descriptions rather than merely produce visually plausible pictures. To quantify the progress, we introduce TechImage-Bench, a rubric-based benchmark that targets biology schematics, engineering/patent drawings, and general technical illustrations. For 654 figures collected from real textbooks and technical reports, we construct detailed image instructions and a hierarchy of rubrics that decompose correctness into 6,076 criteria and 44,131 binary checks. Rubrics are derived from surrounding text and reference figures using large multimodal models, and are evaluated by an automated LMM-based judge with a principled penalty scheme that aggregates sub-question outcomes into interpretable criterion scores. We benchmark several representative text-to-image models on TechImage-Bench and find that, despite strong open-domain performance, the best base model reaches only 0.813 rubric accuracy and 0.596 criterion score overall, revealing substantial gaps in fine-grained scientific fidelity. Finally, we show that the same rubrics provide actionable supervision: feeding failed checks back into an editing model for iterative refinement boosts a strong generator from 0.660 to 0.865 in rubric accuracy and from 0.382 to 0.697 in criterion score. TechImage-Bench thus offers both a rigorous diagnostic for technical image generation and a scalable signal for improving specification-faithful scientific illustrations.

Primary area:
datasets and benchmarks
Keywords:
technical image generation, visual synthesis, large multimodal model, generative model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5e5TOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TechImage-Bench for evaluating whether text-to-image models can produce specification-faithful technical illustrations in biology, engineering, and general domains. The benchmark comprises 654 tasks, 6,076 hierarchical criteria, and 44,131 binary checks derived from source text and reference figures through an LMM-assisted pipeline with expert refinement.

Critical strengths

Technical image generation is a meaningful limitation of existing text-to-image benchmarks.

TechImage-Bench provides 654 tasks, 6,076 criteria, and 44,131 binary checks across three domains. The hierarchical rubrics enable more detailed error diagnosis than a single holistic score.

Critical weaknesses

Insufficient validation of the proposed evaluation methodology. The central contribution is a rubric-based evaluation framework, yet the paper does not systematically compare it against existing approaches or direct LMM-based evaluation. Although human agreement results are reported, the lack of comparative evidence makes it difficult to establish whether rubric decomposition actually improves evaluation accuracy or reliability.

Potential evaluator bias in the main results. All models are evaluated exclusively using OpenAI o4-mini, while several evaluated generators also come from OpenAI. Prior research has raised concerns about LLM evaluators favoring models from the same provider. Without cross-evaluator validation or independent human judgments, potential same-provider bias cannot be ruled out, casting doubt on the fairness of the reported model rankings.

Questions for authors

How does rubric-based evaluation compare with existing evaluation methods in terms of agreement with human experts?

Have the authors tested alternative evaluators from independent model providers to rule out same-provider bias? Do the model rankings remain consistent when evaluated by non-OpenAI models or domain experts?

How many samples were used for human validation, how were they selected, and what was the inter-annotator agreement?

What are the individual contributions of task-specific rubrics, domain-specific rubrics, and hierarchical scoring?

1 comment

Thank you for your valuable feedback!

Comparison with Direct LMM-Based Evaluation

We conducted additional experiments comparing our method with direct LMM-based evaluation. As shown in the table, our rubric-based evaluation exhibits a significant positive Spearman correlation with human judgments, whereas direct LMM-based evaluation does not correlate with human judgments.

MethodAvg. Spearman ρ
Direct LMM-based evaluation-0.0897
Rubric-based evaluation0.8344

Fairness of Model Rankings

o4-mini is a language model developed by OpenAI, which we use as an evaluator to assess multiple image generation models. Although some of the evaluated models, such as GPT-Image and GPT-Image-2, are also developed by OpenAI, they differ from o4-mini in their model architectures and training data.

As described in line 822, we conducted experiments to verify the consistency between LMM evaluation results and human judgments.

Furthermore, we performed additional experiments using Google’s Gemini-3.1-Pro as the evaluator. As shown in the table below, its results also closely align with human judgments, indicating that our fine-grained rubrics yield consistent agreement with human evaluations across evaluators from different providers, supporting the stability of image generation model rankings.

PercentileAccScore
50%0.0140.050
90%0.0200.067
100%0.0220.071

Human Validation

Refer to line 804 onward for details on the experts and evaluation criteria. We recruited 20 domain experts and randomly sampled 100 tasks across the three domains, covering 3,773 rubric items. To improve the consistency and reliability of the human evaluations, each item was first assessed by one expert and then randomly assigned to another expert for review.

Refer to line 235 for details on dataset construction. All 654 tasks and their reference images were screened and reviewed by domain experts to ensure the scientific validity, evaluation value, and feasibility of the tasks.

Individual Contributions of TechImage-Bench Components

As described in line 246, we explain the rationale for designing task-specific and domain-specific rubrics.

In lines 288 and 379, we explain that hierarchical scoring is designed to measure the degree of overall compliance with the constraints and provide richer and more diagnostic feedback, allowing for a more precise characterization of the limitations of current models in technical image generation.

These three components play different roles. To assess their individual contributions, we constructed three sub-benchmarks and conducted additional experiments. As shown in the table, the full benchmark achieves substantially higher correlation with human judgments than any of the three sub-benchmarks, indicating that combining these components improves alignment between automated evaluation and human judgments.

Our task-specific rubrics focus on the technical correctness of task-specific content, while domain-specific rubrics address quality requirements shared across tasks within a domain. Hierarchical scoring aggregates fine-grained checks to characterize the extent to which the technical requirements of each criterion are satisfied. Together, these components enable our benchmark to diagnose fine-grained errors from multiple technical perspectives and provide an overall assessment that better aligns with human judgments.

BenchmarkAvg. Spearman ρ
Full benchmark0.83438
Task-specific0.54582
Domain-specific0.72131
Criteria-only0.69485
Reviewer_TpazOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TechImage-Bench, a benchmark for evaluating the fine-grained correctness of technical images generated from detailed textual descriptions. It contains 654 tasks across biology, engineering, and general technical illustration, organized into 6,076 criteria and 44,131 binary scoring points. The dataset is constructed using source images and surrounding text, multimodal-model-assisted description and rubric generation, and human refinement. An o4-mini evaluator answers each scoring question separately, and the results are aggregated into point-level accuracy and an exponentially penalized criterion score.

The authors evaluate ten image-generation models and report substantial remaining errors, particularly in engineering illustrations. They also use failed checks to guide iterative image editing, reporting an increase in overall accuracy from 0.660 to 0.865 after ten iterations. Human rubric-quality assessments and comparisons between automated and human scores provide initial evidence of benchmark quality.

Critical strengths

The paper addresses a meaningful evaluation gap with a substantial, interpretable benchmark. Technical illustrations can appear convincing while containing consequential errors in component counts, connections, labels, or process directions. The hierarchical rubrics make these failures explicit and localizable. The collection of 654 tasks and 44,131 checks across three domains could therefore be a useful resource for evaluating specification adherence beyond overall visual quality.

The evaluation produces actionable feedback. Failed scoring points identify concrete targets for image editing, and the refinement experiments provide a useful demonstration of this application. Connecting evaluation with correction increases the practical value of the benchmark, even though the magnitude and attribution of the reported gains require stronger validation.

Critical weaknesses

Human–model agreement is insufficiently characterized at the individual-check level. Table 5 provides encouraging evidence that automated and human evaluations yield similar aggregate Accuracy and Score values. However, similar aggregate scores do not necessarily imply agreement on which specific checks pass or fail. This distinction is particularly important because the benchmark is intended to diagnose fine-grained errors and provide targeted editing feedback. The manuscript does not report check-level agreement or sufficiently specify the human-evaluation protocol, including sample size, sampling strategy, annotation redundancy, and disagreement resolution. Reporting these details, together with check-level agreement and an analysis of common discrepancies, would strengthen confidence in the evaluator’s diagnostic reliability.

The rationale and robustness of the exponential criterion score are not sufficiently established. The paper explains that 0.5ei0.5^{e_i} assigns a higher score when a fixed number of failed checks is concentrated within fewer criteria rather than distributed across multiple criteria. However, it does not adequately justify why this preference, or the specific halving penalty, reflects expert judgments of technical-image quality. Because the score depends on the number rather than the proportion of failed checks, criteria containing more checks receive lower scores at the same failure rate. Related checks may also penalize the same underlying error multiple times. For example, in Figure 4, P8.6 combines the requirements of P8.4 and P8.5, so a single floating or disconnected arrow may trigger multiple failed checks. It is unclear how the exponential scoring rule accounts for this dependence, or whether such overlapping checks affect the robustness of scores and model rankings.

Questions for authors

Section 5.1 states that the authors record “the best results produced during the editing process.” Does Table 3 report the output at iteration N, or the best output observed up to that iteration? The number of candidates and the selection criterion are also unspecified. These details matter because selecting the best result across iterations can contribute to the observed gains. While the experiment demonstrates that the overall refinement procedure improves benchmark scores, it does not isolate the benefit of rubric-specific feedback from additional editing attempts and possible score-based selection. Clarifying the protocol and comparing against generic editing feedback under the same budget and selection procedure would make the interpretation more convincing.

Other questions see Critical Weaknesses.

4 comments

We appreciate your invaluable advice!

Consistency Between LMM and Human Judgments

We further analyzed the checks on which human and automated evaluations disagreed. The discrepancies mainly occurred when an image broadly met the intended requirements but contained local errors or ambiguities. Examples include blurry text and layouts that were generally correct but had elements in incorrect positions or relationships. Human evaluators tended to use information from the surrounding image to infer what unclear or flawed content was intended to convey, sometimes marking checks as passed despite these defects. In contrast, the automated evaluator made judgments based more strictly on the rubrics and the visible content of the image.

We recruited 20 domain experts and randomly sampled 100 tasks across the three domains, covering 3,773 rubric items. To improve the consistency and reliability of the human evaluations, each item was first assessed by one expert and then randomly assigned to another expert for review.

Score Design and Robustness of Model Rankings

Our criteria are manually designed to assess image quality from different technical perspectives, with each criterion containing a set of fine-grained checks. Under the exponential scoring scheme, a criterion’s score decreases to 0.0625 when four checks fail, after which additional failed checks have a negligible impact on the criterion score and the overall score. Our statistics show that when all checks under a criterion fail, its score does not exceed 0.031250, indicating that its effect on the overall score is nearly the same as that of a zero score. To further assess the robustness of the scoring scheme, we compared scoring based on the number of failed checks with scoring based on the proportion of failed checks. The maximum difference between the two methods was 0.009921, suggesting that scoring based on the number of failed checks does not materially affect the overall results.

MetricMaximum
Criterion score when all checks fail0.031250
Score difference between methods0.009921

Furthermore, element presence and the correctness of relationships between elements represent distinct technical requirements. When a required element is missing, both its presence requirement and the relationship requirements involving that element may be unmet, so the corresponding checks should all be marked as failed. These checks assess the image against distinct technical requirements rather than repeatedly assess it against the same requirement and do not affect the robustness of scores or model rankings.

Comparison with Generic Editing Feedback

Thank you for highlighting this important point. Table 3 in the original submission reports the cumulative scores obtained up to iteration N. At each iteration, we generate three candidate images and use an evaluator model to select the one that best follows the editing instructions. The selected image serves as both the output of the current iteration and the input to the next.

To further demonstrate the effectiveness of error feedback from fine-grained rubrics, we conducted additional comparisons with generic editing feedback and ordinary self-refinement. As shown in the table, generic editing feedback and ordinary self-refinement yield limited improvements after multiple iterations. These results indicate that targeted error feedback from fine-grained rubrics guides image editing more effectively, enabling larger accuracy gains with fewer iterations.

Rubric-based refinementGeneric editing feedbackOrdinary self-refinement
Original0.6600.6600.660
10.729 (+0.069)0.663 (+0.003)0.684 (+0.024)
20.769 (+0.109)0.689 (+0.029)0.694 (+0.034)
30.800 (+0.140)0.704 (+0.044)0.724 (+0.064)
40.820 (+0.160)0.715 (+0.055)0.745 (+0.085)
50.831 (+0.171)0.730 (+0.070)0.755 (+0.095)

Thank you again for your thoughtful feedback and the time you have devoted to reviewing our work. We would greatly appreciate your consideration of our responses. If any questions or concerns remain, we would be happy to provide further clarification.

We sincerely appreciate your time and careful review.

Thank you for the response and additional analyses, which address some of my concerns. I understand that distinct technical requirements may fail because of the same underlying error. However, my concern is more specifically about overlap between composite checks and their constituent checks: in Figure 4, P8.6 appears to combine the requirements of P8.4 and P8.5. Could the authors clarify why the combined check is scored additionally when its constituent requirements have already been scored separately?

Thank you again for your reviewing.

To assess what P8.6 adds beyond P8.4 and P8.5, we recomputed the model rankings after excluding P8.6 while retaining the other checks. The Spearman correlation with the human-derived rankings dropped from 0.8344 to 0.7783. The example below illustrates what P8.6 captures that the two individual checks can miss.

Evaluation settingSpearman ρ with human rankings
Full benchmark0.8344
Without P8.60.7783

This result reflects our observation during rubric design that arrows must not only connect elements correctly at the graphical level but also convey the intended relationships accurately at the semantic level. A typical example involves two visual elements, each with a text label. One arrow connects the graphical elements themselves, while a second connects their labels. Since each rubric item is evaluated independently, P8.5 would be marked true because the arrows themselves do not visually overlap. P8.4 would also be marked true because each arrow connects a source to a target. However, P8.6 would be marked false because the two connections overlap semantically by expressing the same relationship.

Our rubrics were specifically designed based on LLM performance, allowing LLMs to interpret them correctly and assign scores during evaluation. Nevertheless, we greatly appreciate this suggestion, which helps us make the rubrics clearer to human readers. We have clarified the wording of P8.5 (“Does each arrow graphically avoid being floating, overlapping, or unclear?”) and P8.6 (“Does each arrow clearly connect the source and target elements semantically without floating, overlapping, or unclear situations?”). This modification will be included in the dataset, and we reran the evaluation with the revised rubrics and found that the results remained essentially unchanged within the margin of error (e.g., for GPT-Image-1.5, GPT-Image-2, and Nano Banana 2, the scores before and after the rubric revision were, in that order, 0.765 to 0.764, 0.813 to 0.813, and 0.801 to 0.799 with o4-mini, and 0.770 to 0.768, 0.811 to 0.809, and 0.798 to 0.795 with Gemini-3.1-Pro).

If any questions or concerns remain, please let us know. We would be glad to provide further clarification.

Reviewer_YsQQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TechImage-Bench, a benchmark of 654 technical image generation tasks across biology, engineering, and general illustrations. It evaluates fine-grained correctness through hierarchical rubrics comprising 6,076 criteria and 44,131 binary checks. Experiments assess ten image generation models and show that rubric-based feedback can guide iterative image refinement.

Critical strengths
  1. The benchmark is useful for evaluating fine-grained correctness in technical images, including structures, relationships, and labels.
  2. The experiments cover ten models across three domains, with detailed examples that help explain the results.
  3. The paper is well presented and easy to read.
Critical weaknesses
  1. Using the same evaluator to provide refinement feedback and select improved outputs risks favoring its own preferences. Without independent human or cross-evaluator validation, it remains unclear whether higher scores reflect genuine improvements in technical correctness. Comparisons with ordinary self-refinement or alternative feedback under matched budgets are also missing.
  2. The claim that “rubrics provide actionable supervision” is not accompanied by a cost analysis. Individually evaluating scoring points, generating feedback, and repeatedly editing images may incur substantial overhead. The practical efficiency of this refinement process remains unclear.
Questions for authors
  1. How many samples were included in the human validation studies, how were they selected, and what was the agreement among annotators?
  2. Could overlapping checks penalize the same underlying error multiple times? For example, Figure 4 repeatedly checks the “alveolus” label across criteria, while element-presence, relationship, and error-avoidance checks may overlap. How was this redundancy addressed, particularly given that each failed check halves the corresponding criterion score?
2 comments

Thanks for your invaluable advice!

Independent Validation and Comparisons

We use expert-validated, fine-grained rubrics to provide refinement feedback, translating specific requirements that generated images fail to satisfy into editing instructions. This keeps the refinement process focused on explicit requirements for technical correctness.

Refer to line 822 onward for our analysis of the agreement between LMM-based evaluations and human judgments. The results support the reliability of our fine-grained rubric-based evaluation and suggest that higher scores reflect greater technical correctness. We also used Gemini-3.1-Pro as an evaluator and found that its results were consistent with human judgments, further supporting the reliability of our rubric-based evaluation.

PercentileAccScore
50%0.0140.050
90%0.0200.067
100%0.0220.071

To further demonstrate the effectiveness of error feedback from fine-grained rubrics, we conducted additional comparisons with alternative feedback and ordinary self-refinement. As shown in the table, alternative feedback and ordinary self-refinement yield limited improvements after multiple iterations. These results indicate that targeted error feedback from fine-grained rubrics guides image editing more effectively, enabling larger accuracy gains with fewer iterations.

Rubric-based refinementAlternative feedbackOrdinary self-refinement
Original0.6600.6600.660
10.729 (+0.069)0.663 (+0.003)0.684 (+0.024)
20.769 (+0.109)0.689 (+0.029)0.694 (+0.034)
30.800 (+0.140)0.704 (+0.044)0.724 (+0.064)
40.820 (+0.160)0.715 (+0.055)0.745 (+0.085)
50.831 (+0.171)0.730 (+0.070)0.755 (+0.095)

Cost Analysis of Rubric-Based Evaluation

Our main contribution is TechImage-Bench and its fine-grained rubrics. The refinement with rubrics experiment aims to show that, beyond evaluation, these rubrics can provide specific error feedback to guide models in improving the technical correctness of generated images. Evaluating a single check costs an average of USD 0.003, while evaluating one task costs an average of USD 0.202. These results highlight the potential of rubric-based evaluation for large-scale benchmarking.

Human Validation

Refer to line 804 onward for details on the experts and evaluation criteria. We recruited 20 domain experts and randomly sampled 100 tasks across the three domains, covering 3,773 rubric items. To improve the consistency and reliability of the human evaluations, each item was first assessed by one expert and then randomly assigned to another expert for review.

Refer to line 235 for details on dataset construction. All 654 tasks and their reference images were screened and reviewed by domain experts to ensure the scientific validity, evaluation value, and feasibility of the tasks.

Rubric Design

Thank you for highlighting this important aspect of our benchmark’s rubric design!

Each criterion represents a semantically complete evaluation unit, and different criteria assess distinct technical requirements, collectively measuring the technical correctness of an image from multiple perspectives. Our rubrics do not repeatedly check for the same issue. Instead, they assess the technical correctness of an image from different perspectives. In particular, the presence of an element, the correctness of a relationship, and the avoidance of an error are distinct technical requirements. When a required element is missing, both the requirement that it be present and the requirement that it have a specific relationship with another element are unmet. The corresponding checks are therefore marked as failed, and the failures are reflected in the scores of their respective criteria.

Thank you again for your insightful feedback. It has been invaluable in refining our work. We would be grateful if you could review our rebuttal. Should any questions remain, we would be happy to clarify them.

We sincerely appreciate your time and effort.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers, AC, SAC and PC for their efforts in improving our work.

We appreciate all positive feedback on our work: meaningful evaluation gap (Tpaz: “addresses a meaningful evaluation gap”; 5e5T: “... is a meaningful limitation of existing text-to-image benchmarks”); useful and substantial benchmark (YsQQ: “useful for evaluating ... in technical images”; Tpaz: “a substantial, interpretable benchmark ... a useful resource ...”); more detailed error diagnosis (Tpaz: “The hierarchical rubrics make these failures explicit and localizable”; 5e5T: “The hierarchical rubrics enable more detailed error diagnosis ...”); actionable feedback (Tpaz: “produces actionable feedback”); experiments with detailed examples (YsQQ: “... with detailed examples that help explain the results”); well presented and easy to read (YsQQ: “well presented and easy to read”).

We addressed all concerns as follows:

ReviewerConcernOur Response
YsQQ(i) Evaluator bias and refinement comparisons; (ii) Cost analysis; (iii) Human validation; (iv) Rubric design.(i) Added Gemini-3.1-Pro validation and refinement baselines; (ii) Reported evaluation costs; (iii) Clarified human validation; (iv) Explained the distinct technical requirements assessed by rubric items.
Tpaz(i) Human–LMM agreement; (ii) Scoring and checks; (iii) Refinement comparisons.(i) (iii) addressed after discussion. (i) Analyzed check-level human–LMM disagreements; (ii) Tested scoring robustness; removing P8.6 reduced the Spearman correlation. Illustrated graphical versus semantic checks and clarified P8.5/P8.6 wording; reevaluation showed similar scores and unchanged rankings; (iii) Clarified the selection protocol and added refinement baselines.
5e5T(i) Agreement with humans; (ii) Evaluator bias; (iii) Human validation; (iv) Component contributions.All addressed after discussion. (i) Compared against direct LMM evaluation and clarified its protocol and near-zero correlation; (ii) Added Gemini-3.1-Pro validation and reported per-model scores and unchanged cross-evaluator rankings; (iii) Clarified human validation; (iv) Added experiments to assess component contributions.

Finally, we once again express our sincere gratitude to everyone for your important contributions to improving our work.

Sincerely,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.