- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
The paper proposes a benchmark for converting molecule/reaction diagrams within scientific figures to structured SMILES strings. The best setting tested (Gemini 3.8 Flash, generate end-to-end 3 times + selection) achieved 50.0%.
Critical strengths
- The proposed benchmark claims to provide a contribution over previous benchmarks in that the figures more realistically represent scientific figures, as they are not individual cropped-out molecular diagrams but instead entire scientific figures.
Critical weaknesses
- My primary concern is that the benchmarked methods are not representative of the best available methods for the task. The models used were small 8-billion open-weight models and Gemini 3.8 Flash, yet AI systems scientists use on a daily basis involve frontier models (GPT Sol/Terra, Claude Opus/Sonnet, etc.). The proposed benchmark would be much more relevant if the authors benchmarked these frontier models to provide evidence that it is able to distinguish them, and that much room for improvement remains for frontier models. Update: This is partially addressed with the inclusion of e2e molecule extraction results of two frontier models. Evaluation of other settings (tool-cascade/agentic, reaction extraction) and some analyses (sensitivity to cropping/standardization) still rely on the weaker models.
- Novelty compared to prior benchmarks appears somewhat incremental. The paper claims existing benchmarks “mainly evaluate images of individual molecules”, but there are benchmarks that evaluate diagrams of reactions such as RxnScribe’s evaluation, RxnBench (Li et al., 2026), and works that the authors included in their related work section. Also see Q1 below. Update: With the comparison table the authors provide, I now see that the proposed benchmark is likely the first to simultaneously possess multiple features (target specified by query, output full structure, molecule/reaction). What one learns from having the combination over having the individual features is not entirely clear to me yet.
Update: I've raised my informativeness and soundness scores from 1 to 2, and overall score to 2 (weak reject).
Questions for authors
- The Related Work section currently focuses more on prior methods and only contains brief descriptions of differences between PaperMolBench and prior benchmarks. To facilitate evaluating the exact contributions of PaperMolBench over prior benchmarks, it would be helpful to see a table with a systematic comparison in terms of various features and other basic information (e.g. benchmark size). Update: The authors have answered this question.
7 comments
Response to Reviewer LBLz (Part 2 of 2): Comparison with Prior Benchmarks
W2: "Novelty compared to prior benchmarks appears somewhat incremental. [...] there are benchmarks that evaluate diagrams of reactions such as RxnScribe's evaluation, RxnBench (Li et al., 2026) [...]"
Q1: "[...] it would be helpful to see a table with a systematic comparison in terms of various features and other basic information (e.g. benchmark size)."
Thank you for this suggestion; we address the two points together. We added the following comparison table to Related Work (Table 1), which now also covers RxnBench:
| Benchmark | Input | Query-specified target | Structures generated and scored | Task | Size |
|---|---|---|---|---|---|
| Classical OCSR test sets (USPTO, CLEF, JPO, UOB, Staker, ACS) | single-molecule image | no | yes | molecule | 331 to 50,000 images each |
| WildMol (Fang et al., 2025) | molecule image cropped from PDF documents | no | yes | molecule | 20,000 images |
| MolRecBench-Wild (Yang et al., 2026) | molecule image cropped from papers | no | yes | molecule | 5,029 images from 820 papers |
| RxnScribe dataset (Qian et al., 2023) | reaction diagram | no; all reactions are parsed | no; locations and roles of entities only | reaction | 1,378 diagrams, 3,776 reactions |
| RxnBench (Li et al., 2026) | reaction scheme; full document | yes, as a question | no; multiple choice | reaction understanding | 1,525 questions on 305 schemes; 540 questions on 108 articles |
| PaperMolBench | full scientific figure | yes, natural-language query | yes | molecule and reaction | 355 cases on 149 figures from 129 papers |
PaperMolBench differs from prior benchmarks in combining three properties:
- the input is a full scientific figure rather than a cropped single-molecule image;
- the target is specified by a natural-language query, so the model must first understand the figure and locate the queried molecule or reaction in it;
- the model must generate the complete structure itself, and is scored on whether the structure matches exactly.
Regarding the two works you mention: the RxnScribe evaluation targets the parsing of reaction diagrams and scores the locations and roles of the entities, not whether the molecular structures are recognized correctly; RxnBench evaluates reaction understanding through multiple-choice questions and does not require generating structures. Both evaluate capabilities different from, and complementary to, those evaluated by PaperMolBench. The difference is also visible in the results: RxnBench reports that the best model already reaches 96.2% on its single-figure QA, whereas on PaperMolBench, where the structure must be generated, the strongest frontier model reaches only 67.6% (see W1).
We agree that the sentence in the introduction, "existing benchmarks mainly evaluate images of individual molecules", did not cover the work on reaction diagrams; the revised introduction now covers these benchmarks as well.
We hope these additions address your concerns, and we would be grateful if you would consider them in your final assessment. We are happy to provide any further clarification.
Upon skimming the authors' response, it seems likely that my primary concern is addressed -- I'll likely raise my score but I'll have to go through the details first. Thanks for your patience!
Response to Reviewer LBLz (Part 1 of 2): Frontier-Model Results
Thank you for the constructive comments, and for recognizing that evaluating full scientific figures rather than cropped molecular diagrams is the more realistic setting. Following your suggestions, we evaluated two frontier models and added a comparison with prior benchmarks. The revised paper is uploaded, with changes highlighted in blue.
Thank you for this suggestion. We evaluated Claude Opus 5.5 and GPT-6-Sol on all 204 Molecule Extraction cases, with the same prompts and scoring as for Gemini 3.8 Flash. The results are added to the main table (Table 3):
(EM, %. Gemini's numbers differ slightly from the 44.1 and 50.0 in the submission because the revision scores all systems against one unified version of the references.)
These results provide both kinds of evidence you ask for.
The benchmark distinguishes frontier models. With End-to-End Generation, Opus 5.5 is 12.7 points above GPT-6-Sol and 20.1 points above Gemini 3.8 Flash; both gaps are significant under paired tests (Table 10, Appendix B.1).
Much room for improvement remains for frontier models. Even the best configuration (Opus 5.5 with selection) fails on about one third of the cases, and accuracy drops sharply as the target molecule grows:
(EM with End-to-End Generation, %.)
In addition, about half of Opus 5.5's errors (34 of 70) recover the correct atom connectivity and differ from the reference only in stereochemistry. Large molecules and stereochemistry are thus concrete challenges in real scientific figures that remain open even for the strongest model. These analyses are added to Appendix B.3 (Table 12).