acceptodds
Withdrawn from ICLR 2027

Faithful, Enriched, and Precise: Benchmarking Natural-Science Illustration Generation by T2I models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Scientific illustrations are essential tools for communicating research findings, especially in natural science, where they visualize complex concepts and processes. As Text-to-Image (T2I) models become increasingly capable, researchers have started to use them for scientific illustration generation. However, existing benchmarks often assess outputs at a holistic level, overlooking fine-grained elements, while scientific reasoning ability and output conciseness remain under-quantified. We introduce FEPBench, a benchmark built from carefully selected high-quality scientific illustrations across multiple disciplines and layout types. With the assistance of multimodal large language models (MLLMs) and human experts, we provide fine-grained atom set annotations and systematically evaluate T2I models along three dimensions: instruction faithfulness, reasoning enrichment, and semantic precision. Our evaluation further decomposes model performance across visual, textual, relation, and layout elements. Results show that even state-of-the-art (SOTA) closed-source models, such as GPT Image 2 and Nano Banana Pro, still suffer from text-rendering bottlenecks, limited reasoning enrichment, and difficulty balancing generation richness with precision. These findings provide practical guidance for improving and deploying T2I models in scientific illustration generation across diverse scientific research scenarios. Our benchmark dataset, atom set annotations, and code are all available at https://anonymous.4open.science/r/FEDBench-1DDD.

Primary area:
datasets and benchmarks
Keywords:
benchmark,text-to-image model,illustration generation,natural science

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_yw6NOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

FEPBench evaluates scientific illustration generation using 1,300 reference figures from three scientific domains. It uses semantic atoms to measure Instruction Faithfulness (IF), Reasoning Enrichment (RE), and Semantic Precision (SP), and compares ten generation systems.

Critical strengths

Separating instruction following, additional scientific content, and unsupported content is useful. The repository provides 1,300 reference images, corresponding annotations, and evaluation code.

Critical weaknesses
  1. The evaluation criteria do not fully match the stated task. The task is to generate scientific illustrations from text, but evaluation depends on a particular reference figure and a predefined set of atoms. A scientific concept can have several correct visual representations. The paper does not adequately explain why recovering reference details omitted from the prompt should receive more credit than another equally correct representation. RE rewards coverage of annotated content beyond the prompt; this alone does not establish scientific reasoning. SP may also penalize valid content absent from the support set. The metrics therefore do not clearly distinguish recovery of a particular reference representation from the ability to generate scientifically sound illustrations.

  2. Essential annotation-quality evidence is missing, and the released implementation conflicts with the metric definitions. Appendix B.3 leaves the independent annotation audit’s sampling fraction and random seed as “[to be filled].” The described precision, recall, F1, and provenance-agreement results are also missing. The code assigns required atoms to IF and optional atoms to RE. However, 3,871 atoms across 931 samples are marked both optional and explicitly mentioned, causing content already mentioned in the prompt to enter RE. The released annotations also do not match the loader’s expected schema, lack the information expected by the layout evaluator, and yield some statistics inconsistent with Table 8. It remains unclear which evaluation version produced the reported results.

Questions for authors
  1. How does the benchmark fairly evaluate different but equally correct scientific representations? What evidence supports interpreting RE as scientific reasoning?
  2. Was the independent annotation audit completed? Please provide its sampling details, quality statistics, and expert-review records for open-world atoms.
  3. Please explain the IF/RE assignments and discrepancies between the data, code, and paper. Would correcting them change the model rankings?
Reviewer_zWwNOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FEPBench, a benchmark for natural-science illustration generation containing 1,300 reference illustrations across three scientific domains, single- and multi-panel layouts, and two prompt formats. The evaluation decomposes illustrations into visual, textual, relational, and layout atoms, and reports Instruction Faithfulness (IF), Reasoning Enrichment (RE), and Semantic Precision (SP). Supported content includes prompt-required atoms, reference-derived supplementary atoms, and expert-approved open-world additions. The authors evaluate nine text-to-image models and one code-based system, analyze performance across domains and generation conditions, and validate automated evaluation using two multimodal judges and three experts on 1,200 generated figures.

Critical strengths
  1. The benchmark addresses a useful evaluation gap with a structured diagnostic framework. Scientific illustrations can appear plausible while containing incorrect labels, missing relations, or misplaced components. Evaluating these elements separately, while distinguishing instruction satisfaction from supplementary content and unsupported additions, provides actionable information that a holistic score can obscure. The coverage of 1,300 illustrations across multiple disciplines and layouts makes this a potentially useful community resource.

  2. The evaluation includes meaningful reliability checks and a broad system comparison. The paper evaluates ten systems under both prompt formats and examines domain, layout, and complexity effects. It also reports validation on 1,200 generated figures independently scored by three experts, with human–human ICC values of 0.88–0.94 and GPT–human rank correlations of 0.84–0.89, alongside a second evaluator and a three-seed experiment for four open-source models. These checks provide substantial support for the consistency of scoring under the proposed rubric.

Critical weaknesses
  1. The incremental contribution is not sufficiently established. The paper does not introduce a new generation model or agentic method; its contribution centers on the benchmark and evaluation protocol. A benchmark can constitute a substantial contribution on its own, but this makes it especially important to demonstrate the added value over existing scientific-illustration evaluation. The paper does not sufficiently show, on the same generated figures, which consequential errors or model differences existing holistic or checklist-based methods miss and FEPBench captures. Findings such as difficulties with text rendering and multi-panel layouts are useful, but do not by themselves establish a substantial advance. The ImgCoder comparison could provide additional insight into code-based versus direct image generation, yet its implementation and inference budget are insufficiently specified to support that interpretation.

  2. The core evaluation framework requires further validation. Appendix B.3 leaves the independent annotation audit incomplete, and Appendix B.4 does not clearly document the implemented open-world expansion procedure or its statistics. RE measures coverage of annotated supplementary content, which does not necessarily establish scientific reasoning, while SP may penalize valid additions missing from the support set. The human-agreement study supports scoring consistency but does not resolve this coverage concern. Separate results for reference-derived and open-world RE, together with an expert audit of unsupported content, would help establish what the metrics capture and whether annotation omissions affect model rankings.

Questions for authors
  1. What important errors or model differences does FEPBench identify that existing holistic or checklist-based evaluation fails to capture? Can the authors compare these approaches on the same generated figures, using expert judgments to establish which disagreements reflect meaningful improvements in evaluation?

  2. It is unclear whether the supported atom set was frozen before inspecting evaluated model outputs and, if output-driven expansion was used, whether all systems were rescored against the same updated set. Please clarify these procedures and report separate counts of reference-derived and open-world reasoning atoms, candidate acceptance and disagreement statistics, and the completed independent-annotation audit described in Appendix B.3 (now is "[to be filled]").

  3. Although its backbone is specified, ImgCoder’s output format (e.g., SVG, plotting code, or another representation), rendering pipeline, and inference budget remain unclear. Please describe how it generates and renders figures, including text, and whether it uses multiple calls or execution-based refinement. These details are needed to interpret its IF advantage over the image generators.

  4. Scientifically valid and relevant additions may be classified as unsupported simply because they are absent from the annotated support set. Please provide an expert audit of such cases to distinguish genuine errors from annotation omissions, and assess whether accounting for valid additions changes SP scores or model rankings. Reporting RE separately for reference-derived and open-world atoms would also clarify whether higher scores reflect recovery of omitted reference details or other expert-approved enrichment.

  5. At what level were the correlations in Table 4 computed: individual figures, model averages, or another aggregation level? Please specify the ICC variant. Please also reconcile the ImgCoder IF values of 0.678 in the main text versus 0.676 in Table 2, and 0.670 versus 0.667 for structured prompts in the text and Table 3.

Reviewer_z6X5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces FEPBench to evaluate text-to-image models’ ability to generate natural-science illustrations. The authors collect 1,300 illustrations of scientific mechanisms and structures, provide both free-form and structured prompts for each illustration, and annotate four types of atoms: text, visual entities, relations, and layout. They further distinguish between prompt-required content, permitted additions, and unsupported additions, measuring instruction faithfulness, enrichment, and semantic precision, respectively. The paper evaluates nine T2I models and ImgCoder and analyzes the effects of scientific domain, layout, prompt format, and content complexity.

Critical strengths

The problem addressed by this paper has practical value. Scientific illustrations must accurately represent text, entities, and relations; overall visual appeal or image–text similarity alone is insufficient. Analyzing omissions and incorrect additions separately can help us understand specific model failure modes.

Critical weaknesses

The experiments lack specialized scientific illustration systems such as AutoFigure and PaperBanana. I am unclear why the authors did not evaluate these systems. They already use content planning, iterative checking, and correction to improve generated illustrations, whereas the experiments in this paper still focus primarily on base T2I models.

There is insufficient evidence that the new metrics are more accurate or useful than existing evaluation methods. TIFA and DSG have already studied item-by-item content verification, and existing scientific illustration benchmarks also consider accuracy, completeness, logic, and conciseness. Dividing evaluation into more categories does not, by itself, demonstrate greater effectiveness. The reported human–judge agreement is valuable, but it primarily shows that automatic scores approximate expert scores under the FEP rubric. I would like the authors to compare FEP with existing evaluation methods on the same generated images, using independent expert judgments as a reference, and provide concrete examples where their assessments differ. This would establish the paper’s contribution more directly than further expanding the model leaderboard.


The authors state:

The authors take full responsibility for all text, claims, code, figures, annotations, and experimental results presented in this work.

see in line 786, "The fraction of this subset is [to be filled], and the sampling seedis [to be filled]. "?

why has this information not been completed?

Questions for authors

I would also like more information about the qualifications of the experts involved:

  1. How many experts participated in the project overall?
  2. What are their disciplinary backgrounds, and how do they collectively cover physics, materials science, ecology, biology, and medicine?
  3. Have they obtained doctoral degrees, and what relevant research experience do they have?
  4. Are they among the paper’s authors?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.