- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
FEPBench evaluates scientific illustration generation using 1,300 reference figures from three scientific domains. It uses semantic atoms to measure Instruction Faithfulness (IF), Reasoning Enrichment (RE), and Semantic Precision (SP), and compares ten generation systems.
Critical strengths
Separating instruction following, additional scientific content, and unsupported content is useful. The repository provides 1,300 reference images, corresponding annotations, and evaluation code.
Critical weaknesses
-
The evaluation criteria do not fully match the stated task. The task is to generate scientific illustrations from text, but evaluation depends on a particular reference figure and a predefined set of atoms. A scientific concept can have several correct visual representations. The paper does not adequately explain why recovering reference details omitted from the prompt should receive more credit than another equally correct representation. RE rewards coverage of annotated content beyond the prompt; this alone does not establish scientific reasoning. SP may also penalize valid content absent from the support set. The metrics therefore do not clearly distinguish recovery of a particular reference representation from the ability to generate scientifically sound illustrations.
-
Essential annotation-quality evidence is missing, and the released implementation conflicts with the metric definitions. Appendix B.3 leaves the independent annotation audit’s sampling fraction and random seed as “[to be filled].” The described precision, recall, F1, and provenance-agreement results are also missing. The code assigns required atoms to IF and optional atoms to RE. However, 3,871 atoms across 931 samples are marked both optional and explicitly mentioned, causing content already mentioned in the prompt to enter RE. The released annotations also do not match the loader’s expected schema, lack the information expected by the layout evaluator, and yield some statistics inconsistent with Table 8. It remains unclear which evaluation version produced the reported results.
Questions for authors
- How does the benchmark fairly evaluate different but equally correct scientific representations? What evidence supports interpreting RE as scientific reasoning?
- Was the independent annotation audit completed? Please provide its sampling details, quality statistics, and expert-review records for open-world atoms.
- Please explain the IF/RE assignments and discrepancies between the data, code, and paper. Would correcting them change the model rankings?