acceptodds
Under review as a conference paper at ICLR 2027

PaperMolBench: Benchmarking and Understanding Molecular Structure Recognition in Scientific Figures

Abstract

Scientific figures in chemistry and biology contain complex molecular structures and figure logic. Existing benchmarks mainly focus on single-molecule images and therefore do not measure whether visual language models (VLMs) can understand real scientific figures for research in these domains. We introduce PaperMolBench, a human-annotated benchmark for extracting target molecular structures and reactions from full scientific figures, comprising 355 test cases. We evaluate representative frameworks, including Tool-Cascaded Method, Agentic Pipeline, and End-to-End Generation, together with several widely used large models. On Molecule Extraction, the best configuration among evaluated methods reaches only 50.0% exact-match (EM) accuracy. To understand the gap, we build a capability diagnostic framework for the simple yet competitive end-to-end generation approach, covering localization, identification, and transcription. Using InternVL3.5-8B, we observe low accuracy even on synthetic molecular images, with further drops on crops from scientific figures, revealing a clear distribution gap. Further analysis indicates that effective molecular structure extraction requires addressing both limited data scale and distribution mismatch, and that linear text representations such as SMILES do not naturally capture molecular topology. These findings highlight key directions for future VLMs in chemistry.

Primary area:
datasets and benchmarks
Keywords:
Optical Chemical Structure Recognition, Vision-Language Models, Scientific Figure Understanding, Multimodal Benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_LBLzOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2(+1)
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper proposes a benchmark for converting molecule/reaction diagrams within scientific figures to structured SMILES strings. The best setting tested (Gemini 3.8 Flash, generate end-to-end 3 times + selection) achieved 50.0%.

Critical strengths
  1. The proposed benchmark claims to provide a contribution over previous benchmarks in that the figures more realistically represent scientific figures, as they are not individual cropped-out molecular diagrams but instead entire scientific figures.
Critical weaknesses
  1. My primary concern is that the benchmarked methods are not representative of the best available methods for the task. The models used were small 8-billion open-weight models and Gemini 3.8 Flash, yet AI systems scientists use on a daily basis involve frontier models (GPT Sol/Terra, Claude Opus/Sonnet, etc.). The proposed benchmark would be much more relevant if the authors benchmarked these frontier models to provide evidence that it is able to distinguish them, and that much room for improvement remains for frontier models. Update: This is partially addressed with the inclusion of e2e molecule extraction results of two frontier models. Evaluation of other settings (tool-cascade/agentic, reaction extraction) and some analyses (sensitivity to cropping/standardization) still rely on the weaker models.
  2. Novelty compared to prior benchmarks appears somewhat incremental. The paper claims existing benchmarks “mainly evaluate images of individual molecules”, but there are benchmarks that evaluate diagrams of reactions such as RxnScribe’s evaluation, RxnBench (Li et al., 2026), and works that the authors included in their related work section. Also see Q1 below. Update: With the comparison table the authors provide, I now see that the proposed benchmark is likely the first to simultaneously possess multiple features (target specified by query, output full structure, molecule/reaction). What one learns from having the combination over having the individual features is not entirely clear to me yet.

Update: I've raised my informativeness and soundness scores from 1 to 2, and overall score to 2 (weak reject).

Questions for authors
  1. The Related Work section currently focuses more on prior methods and only contains brief descriptions of differences between PaperMolBench and prior benchmarks. To facilitate evaluating the exact contributions of PaperMolBench over prior benchmarks, it would be helpful to see a table with a systematic comparison in terms of various features and other basic information (e.g. benchmark size). Update: The authors have answered this question.
7 comments
AuthorsOpenReview ↗

Response to Reviewer LBLz (Part 1 of 2): Frontier-Model Results

Thank you for the constructive comments, and for recognizing that evaluating full scientific figures rather than cropped molecular diagrams is the more realistic setting. Following your suggestions, we evaluated two frontier models and added a comparison with prior benchmarks. The revised paper is uploaded, with changes highlighted in blue.

W1: "The proposed benchmark would be much more relevant if the authors benchmarked these frontier models to provide evidence that it is able to distinguish them, and that much room for improvement remains for frontier models."

Thank you for this suggestion. We evaluated Claude Opus 5.5 and GPT-6-Sol on all 204 Molecule Extraction cases, with the same prompts and scoring as for Gemini 3.8 Flash. The results are added to the main table (Table 3):

ModelEnd-to-End GenerationThree calls + selection
Gemini 3.8 Flash45.651.5
GPT-6-Sol52.956.9
Claude Opus 5.565.767.6

(EM, %. Gemini's numbers differ slightly from the 44.1 and 50.0 in the submission because the revision scores all systems against one unified version of the references.)

These results provide both kinds of evidence you ask for.

The benchmark distinguishes frontier models. With End-to-End Generation, Opus 5.5 is 12.7 points above GPT-6-Sol and 20.1 points above Gemini 3.8 Flash; both gaps are significant under paired tests (Table 10, Appendix B.1).

Much room for improvement remains for frontier models. Even the best configuration (Opus 5.5 with selection) fails on about one third of the cases, and accuracy drops sharply as the target molecule grows:

Heavy atoms in the target moleculeCasesClaude Opus 5.5GPT-6-Sol
at most 158881.872.7
16 to 308058.845.0
more than 303641.722.2

(EM with End-to-End Generation, %.)

In addition, about half of Opus 5.5's errors (34 of 70) recover the correct atom connectivity and differ from the reference only in stereochemistry. Large molecules and stereochemistry are thus concrete challenges in real scientific figures that remain open even for the strongest model. These analyses are added to Appendix B.3 (Table 12).

AuthorsOpenReview ↗

Response to Reviewer LBLz (Part 2 of 2): Comparison with Prior Benchmarks

W2: "Novelty compared to prior benchmarks appears somewhat incremental. [...] there are benchmarks that evaluate diagrams of reactions such as RxnScribe's evaluation, RxnBench (Li et al., 2026) [...]"

Q1: "[...] it would be helpful to see a table with a systematic comparison in terms of various features and other basic information (e.g. benchmark size)."

Thank you for this suggestion; we address the two points together. We added the following comparison table to Related Work (Table 1), which now also covers RxnBench:

BenchmarkInputQuery-specified targetStructures generated and scoredTaskSize
Classical OCSR test sets (USPTO, CLEF, JPO, UOB, Staker, ACS)single-molecule imagenoyesmolecule331 to 50,000 images each
WildMol (Fang et al., 2025)molecule image cropped from PDF documentsnoyesmolecule20,000 images
MolRecBench-Wild (Yang et al., 2026)molecule image cropped from papersnoyesmolecule5,029 images from 820 papers
RxnScribe dataset (Qian et al., 2023)reaction diagramno; all reactions are parsedno; locations and roles of entities onlyreaction1,378 diagrams, 3,776 reactions
RxnBench (Li et al., 2026)reaction scheme; full documentyes, as a questionno; multiple choicereaction understanding1,525 questions on 305 schemes; 540 questions on 108 articles
PaperMolBenchfull scientific figureyes, natural-language queryyesmolecule and reaction355 cases on 149 figures from 129 papers

PaperMolBench differs from prior benchmarks in combining three properties:

  • the input is a full scientific figure rather than a cropped single-molecule image;
  • the target is specified by a natural-language query, so the model must first understand the figure and locate the queried molecule or reaction in it;
  • the model must generate the complete structure itself, and is scored on whether the structure matches exactly.

Regarding the two works you mention: the RxnScribe evaluation targets the parsing of reaction diagrams and scores the locations and roles of the entities, not whether the molecular structures are recognized correctly; RxnBench evaluates reaction understanding through multiple-choice questions and does not require generating structures. Both evaluate capabilities different from, and complementary to, those evaluated by PaperMolBench. The difference is also visible in the results: RxnBench reports that the best model already reaches 96.2% on its single-figure QA, whereas on PaperMolBench, where the structure must be generated, the strongest frontier model reaches only 67.6% (see W1).

We agree that the sentence in the introduction, "existing benchmarks mainly evaluate images of individual molecules", did not cover the work on reaction diagrams; the revised introduction now covers these benchmarks as well.

We hope these additions address your concerns, and we would be grateful if you would consider them in your final assessment. We are happy to provide any further clarification.

Upon skimming the authors' response, it seems likely that my primary concern is addressed -- I'll likely raise my score but I'll have to go through the details first. Thanks for your patience!

Reviewer_ovPrOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents PaperMolBench, a benchmark designed to evaluate molecular structure and reaction extraction directly from full scientific figures. The dataset consists of 355 test cases across 149 real published figures requiring models to parse spatial cues, multi-panel layouts, and figure logic. The authors benchmark tool-cascaded pipelines, agentic workflows, and end-to-end VLMs, supplemented by continued pretraining, fine-tuning, and representation probing on open-source models.

Critical strengths
  1. Realistic task formulation: The work addresses an authentic research challenge by evaluating full, multi-panel scientific figures rather than relying solely on isolated single-molecule crops.
  2. Detailed diagnostic breakdowns: Comparing full figures against ground-truth crops and standardized RDKit renderings clearly decouples visual localization errors from molecular recognition errors.
Critical weaknesses
  1. Narrow benchmark sample size: With only 355 evaluation cases across 149 figures and just 72 cases in the diagnostic subset, the benchmark offers limited statistical power and inadequate coverage across chemical subdisciplines.

  2. Missing comparison on established OCSR benchmarks: The paper lacks bidirectional evaluation; the proposed models and baselines are not tested on classical molecular benchmarks (such as USPTO, CLEF, or standard ChemDraw benchmarks), leaving it unclear how they perform against established small model standards.

  3. Missing competitive non-LLM baselines: The tool cascaded pipeline relies solely on DECIMER while omitting stronger, modern graph prediction systems like MolScribe and MolGrapher, leaving out essential reference comparisons.

  4. Absence of an oracle crop upper bound: The paper does not evaluate top performing dedicated OCSR small models on ground truth crops, making it difficult to judge whether generalist VLMs are actually competitive against mature specialist tools.

  5. Under-trained model adaptation: Continued pretraining is capped below 1,900 steps and supervised fine-tuning runs for only a single epoch, indicating severely under-converged training rather than a rigorous assessment of model capacity.

  6. Inconclusive fine-tuning gains on the primary task: While fine-tuning improves accuracy on cropped single molecule images, performance gains on full figures remain marginal and unstable, demonstrating that the adaptation recipe fails on the central benchmark task.

  7. Narrow open-source model coverage: Empirical training experiments are confined to two 8B models (InternVL3.5 and Qwen3-VL), lacking comparisons across different parameter scales or vision encoder backbones.

Questions for authors

Please address all the issues raised in the Weaknesses section, particularly regarding the benchmark scale, the missing comparisons with classical dedicated small models, and the training convergence.

6 comments
AuthorsOpenReview ↗

Response to Reviewer ovPr (Part 1 of 3): Benchmark Size, Coverage, and Classical OCSR Benchmarks

Thank you for the detailed comments, and for recognizing the full-figure task setting and the diagnostic design. We ran additional experiments for every point raised; the results are included in the revised paper, with changes highlighted in blue.

W1: "Narrow benchmark sample size: [...] the benchmark offers limited statistical power and inadequate coverage across chemical subdisciplines."

Statistical power. The conclusions of the paper rest on large gaps, all of which are significant under paired tests (p < 0.001; Table 10, Appendix B.1):

Comparison (Molecule Extraction, 204 cases)EM gap
Claude Opus 5.5 vs. Gemini 3.8 Flash20.1 points
Claude Opus 5.5 vs. GPT-6-Sol12.7 points
End-to-End Generation vs. Tool-Cascaded Method (MolScribe)17.6 points

(Opus 5.5 and GPT-6-Sol are new results; see Table 3.)

The current scale is therefore sufficient to support the paper's conclusions. On the 72-case diagnostic subset, all results of the training analysis average five generations per case, which reduces sampling variance. In addition, every case was annotated independently by two annotators and adjudicated by a third; we prioritized annotation quality.

Coverage. We added coverage statistics for the Molecule Extraction set (Table 7, Appendix A.2):

AspectStatistics
Sources83 figures from 79 papers
Domainsplant natural products; human and animal metabolism and medicinal chemistry; microbial biosynthesis; enzymology; chemistry and materials
Molecules192 unique molecules; 3 to 72 heavy atoms; molecular weight 56 to 1036
Structural featuresstereocenters (85 cases); E/Z double bonds (47); formal charges (33); R-groups (32); elements such as P, S and F (30)
Query typesby name (113 cases); by position (86)
Figure layoutsingle-panel (64 figures); multi-panel (19)
Figure typesabout half are pathway and reaction-mechanism diagrams; the rest are structure overviews and experimental-data figures that contain structures

The benchmark is thus diverse in molecular structure, query type, and figure type.

W2: "Missing comparison on established OCSR benchmarks: [...] the proposed models and baselines are not tested on classical molecular benchmarks (such as USPTO, CLEF, or standard ChemDraw benchmarks) [...]"

Thank you for this suggestion. We evaluated all models on eight classical OCSR benchmarks (CLEF, JPO, UOB, USPTO, Staker, ACS, Indigo, ChemDraw), with 200 randomly sampled images per benchmark:

ModelMean EM over the 8 benchmarks (%)
Qwen3-VL-8B, official2.8
Qwen3-VL-8B, RDKit-rendered SFT31.9
Qwen3-VL-8B, Real-depiction SFT61.9
InternVL3.5-8B, official65.3
InternVL3.5-8B, RDKit-rendered SFT58.6
InternVL3.5-8B, Real-depiction SFT67.6
MolScribe (as reported in its paper)87.0
  • Real-depiction SFT outperforms RDKit-rendered SFT on all eight benchmarks (by 30.0 points for Qwen and 9.0 points for InternVL). These benchmarks are independent of our training data, so this externally validates the paper's core finding that the depiction distribution of the training images matters.
  • The specialist MolScribe remains ahead on these single-molecule benchmarks. On real scientific figures, however, the ordering reverses: even when given the annotated target crop, MolScribe reaches only 34.7% EM, below Gemini's 55.0% (see W4). Performance on classical benchmarks therefore does not carry over to real scientific figures, which is the motivation for PaperMolBench.

Per-benchmark results are in Appendix D (Table 19).

AuthorsOpenReview ↗

Response to Reviewer ovPr (Part 2 of 3): MolScribe Baselines and Training Convergence

W3: "Missing competitive non-LLM baselines: The tool cascaded pipeline relies solely on DECIMER while omitting stronger, modern graph prediction systems like MolScribe and MolGrapher [...]"

Thank you for this suggestion. We replaced DECIMER with MolScribe in the Tool-Cascaded Method and re-evaluated all 204 cases:

MethodEM (%)
Tool-Cascaded Method with DECIMER (original)18.6
Tool-Cascaded Method with MolScribe (new)27.9
End-to-End Generation (Gemini 3.8 Flash)45.6
Three calls + selection (Gemini 3.8 Flash)51.5

MolScribe is indeed stronger than DECIMER (by 9.3 points). Yet even with MolScribe, the cascade remains 17.6 points below End-to-End Generation. The gap comes from the cascade itself: the detector fails to find the target molecule in about 15% of the cases, and specialist recognizers have limited accuracy on real scientific depictions (see W4). The MolScribe results are added to the main table (Table 3).

W4: "Absence of an oracle crop upper bound: The paper does not evaluate top performing dedicated OCSR small models on ground truth crops, making it difficult to judge whether generalist VLMs are actually competitive against mature specialist tools."

We ran this experiment. On the 72 diagnostic cases, MolScribe and Gemini receive exactly the same three inputs:

ModelFull figureAnnotated target cropRDKit image
MolScribe0.034.783.3
Gemini 3.8 Flash56.755.056.1

(EM, %; Gemini is the mean of five generations.)

  • On ground-truth crops, Gemini outperforms MolScribe by 20 points. Even with perfect localization, a generalist VLM already surpasses the specialist tool on real scientific depictions.
  • MolScribe reaches 83.3% on RDKit images but drops to 34.7% on the real depictions of the same molecules. Its recognition ability is not in question; real depictions simply differ from its training distribution. This agrees with the paper's conclusion on the distribution gap.
  • MolScribe cannot process full figures directly (0.0%), and full figures are the task this paper targets.

These results are added to Table 5 (Section 5.1).

W5: "Under-trained model adaptation: Continued pretraining is capped below 1,900 steps and supervised fine-tuning runs for only a single epoch, indicating severely under-converged training [...]"

We examined convergence from both the training loss and the downstream metrics; both indicate that training has converged.

The training loss has plateaued. CPT makes one full pass over the entire corpus (about 0.9B tokens), with a constant learning rate after warmup. By the end of training the loss has essentially stopped decreasing: in all three models, the mean loss over the last 10% of steps is only 0.4% to 0.5% lower than over the preceding 10%.

Downstream metrics saturate after about 800 CPT steps. We fine-tuned and evaluated a checkpoint every 200 CPT steps. FPS on full figures:

CPT step040080012001600
InternVL3.5-8B37.438.642.639.441.1
Qwen3-VL-8B29.731.038.034.737.3
Qwen3-VL-4B (new)27.331.534.432.535.2

All three models improve clearly over the first 800 steps and stay essentially flat afterwards. Training longer would therefore not change the paper's conclusions.

The loss curves and the curves corresponding to the table above are shown in the new Figure 5 (Appendix C.2); results for all checkpoints are in Tables 16 to 18.

AuthorsOpenReview ↗

Response to Reviewer ovPr (Part 3 of 3): Full-Figure Gains, Model Coverage, and Presentation

W6: "Inconclusive fine-tuning gains on the primary task: While fine-tuning improves accuracy on cropped single molecule images, performance gains on full figures remain marginal and unstable [...]"

The training experiments are a controlled analysis of two factors, data scale and depiction distribution, designed to identify what limits full-figure extraction and to point to directions for improving it. The pattern you point out is one of the findings of this analysis, and the paper states it as such in Section 5.4 and in the conclusion.

Training does improve full-figure extraction. Results on full figures (EM / FPS):

ModelOfficialAfter CPT and SFT
InternVL3.5-8B12.8 / 34.017.2 / 42.1
Qwen3-VL-8B3.6 / 19.88.9 / 33.7
Qwen3-VL-4B (new)4.4 / 20.07.2 / 35.1

Larger gains on crops and smaller gains on full figures: this contrast is itself informative. The real depictions added in SFT are 5,205 single-molecule images. They help the model recognize how molecules are drawn in the literature, which is why Real-depiction SFT outperforms RDKit-rendered SFT on crops for all three models (Table 15, Appendix C.4). But they are not full figures: they cannot teach the model to locate the target molecule in a complex figure, and their number is limited. The advantage therefore does not fully carry over to full figures.

This points to the next step. As stated in the conclusion, improving full-figure extraction calls for combining scalable synthetic data with supervision from real scientific figures.

W7: "Narrow open-source model coverage: Empirical training experiments are confined to two 8B models (InternVL3.5 and Qwen3-VL), lacking comparisons across different parameter scales or vision encoder backbones."

Thank you for this suggestion.

Parameter scale. We added Qwen3-VL-4B, with exactly the same training and evaluation pipeline as the 8B model. The 4B results closely match the 8B results (left of the arrow: SFT only; right: CPT followed by SFT):

MetricQwen3-VL-4B (new)Qwen3-VL-8B
RDKit image, EM16.9 → 34.417.5 → 34.4
Target crop, FPS33.2 → 43.734.0 → 43.5
Full figure, FPS27.3 → 34.829.7 → 35.8
Advantage of Real-depiction over RDKit-rendered SFT on crops (FPS)8.0 points4.9 points

(Values after CPT are averaged over the checkpoints beyond 800 CPT steps.)

Vision encoder. InternVL3.5 and Qwen3-VL use different vision encoders, and the paper's conclusions are consistent across the two. We added this clarification to Section 5.2.

In summary, the effect of CPT and the advantage of real-depiction training hold across parameter scales and across vision encoders. Full results for the 4B model are in Table 18 (Appendix C.5).

Presentation.

Finally, thank you for the feedback on presentation. In the revision, all diagnostic and training results follow one evaluation protocol and all tables use one reference version; we removed an auxiliary table that used a different protocol and made the table rows explicit (for example, the official base and the SFT-only model are now listed separately). The new statistical analyses and additional experiments are placed in the appendix, so that the main text keeps a single line of argument.

We hope these additions address your concerns, and we would be grateful if you would consider them in your final assessment. We are happy to provide any further clarification.

Reviewer_uaJGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PaperMolBench evaluates query-specified molecule and reaction extraction from complete scientific figures. It contains 355 cases and compares several extraction pipelines, followed by paired image diagnostics and training studies on two open VLMs.

Critical strengths
  • Keeping the complete figure and asking for a particular target makes this more relevant to literature extraction than isolated-molecule recognition. The annotation procedure includes independent labels and third-person adjudication.
  • The paired real-depiction versus RDKit SFT study is useful. It shows a crop-recognition benefit that does not consistently carry over to full figures, which helps separate recognition improvements from the full extraction problem.
Critical weaknesses
  • Some system differences are small relative to the test set. The 44.1% versus 46.1% molecule EM difference amounts to four cases out of 204, and many queries share a source figure. Without paired uncertainty estimates that account for this grouping, it is hard to tell which system differences are reliable.
  • Appendix A.1 says seven references shared with the diagnostic set were revised before the paired analysis, and one unparseable reference remains in the denominator. I couldn't tell whether all system results use the same final labels. For a benchmark paper, the reference version and treatment of invalid labels need to be consistent and explicit.

Overall: Weak acceptance. The full-figure task and paired analyses are useful contributions. I am less convinced by the ranking of closely performing systems until the label-version and uncertainty questions are resolved.

Questions for authors
  • Can you report paired uncertainty estimates for the main system comparisons, grouping queries by source figure?
  • Were Tables 2 and 4 scored against the same final reference version? Please clarify the seven revisions and report the effect of correcting or excluding the unparseable reference.
2 comments
AuthorsOpenReview ↗

Response to Reviewer uaJG: Paired Uncertainty Estimates and Reference Version

Thank you for the positive assessment of the full-figure task and the paired analyses, and for two precise questions. Both are addressed in the revised paper (changes highlighted in blue).

W1 / Q1: "Can you report paired uncertainty estimates for the main system comparisons, grouping queries by source figure?"

Thank you for this suggestion. We ran paired analyses for the main system comparisons: 95% confidence intervals from a bootstrap that resamples source figures (so queries from the same figure stay together), and exact McNemar tests. The results are added to Appendix B.1 (Table 10):

Comparison (204 Molecule Extraction cases, 83 figures)EM difference95% CIMcNemar p
End-to-End Generation vs. Tool-Cascaded Method (DECIMER)+27.018.2 to 35.8below 0.001
End-to-End Generation vs. Tool-Cascaded Method (MolScribe, new)+17.68.0 to 27.1below 0.001
Agentic Pipeline vs. End-to-End Generation+2.0−2.4 to 6.20.48
Three calls + selection vs. End-to-End Generation+5.91.0 to 11.30.017
Three calls + selection vs. Agentic Pipeline+3.9−2.0 to 10.00.18
Claude Opus 5.5 vs. Gemini 3.8 Flash (End-to-End, new)+20.113.8 to 26.9below 0.001

The main conclusions of the paper are reliable. End-to-End Generation clearly outperforms the Tool-Cascaded Method, also after the recognizer is replaced with the stronger MolScribe; the gaps between the newly added frontier models are significant as well.

The pair you point to is indeed within the uncertainty. The Agentic Pipeline and End-to-End Generation differ by 2.0 points, with an interval that contains zero. Three calls + selection is 5.9 points above End-to-End Generation (p = 0.017), but this is not significant after correction for multiple comparisons. The revision therefore states explicitly that it does not rank End-to-End Generation, the Agentic Pipeline and three calls + selection against each other (Appendix B.1), and the main table (Table 3) now reports a confidence interval for every system.

W2 / Q2: "Were Tables 2 and 4 scored against the same final reference version? Please clarify the seven revisions and report the effect of correcting or excluding the unparseable reference."

In the revision, all tables are scored against one final version of the references. In the submission, Table 4 (the 72 diagnostic cases) already used the final version, whereas for Table 2 (204 cases) the seven references shared with the diagnostic set had not yet been synchronized. We have unified the references and rescored all systems:

SystemSubmissionUnified references
Tool-Cascaded Method (DECIMER)18.118.6
End-to-End Generation44.145.6
Agentic Pipeline46.147.5
Three calls + selection50.051.5

Each system changes by at most three cases (0.5 to 1.5 points), and the ranking of the systems is unchanged.

An itemized list of the seven revisions is added to Appendix A.1: four complete missing stereochemistry annotations on sugar units; two correct substituents (hydroxyl groups that had been recorded as methyl groups); and one rewrites a string that could not be parsed into standard SMILES, without changing the molecule. The last one is the unparseable reference you refer to. After this correction, the benchmark no longer contains any unparseable reference; the correction changes the score of only one system on one case (the Tool-Cascaded Method gains one correct case).

We hope these additions address your concerns, and we are happy to provide any further clarification.

AuthorsOpenReview ↗

A Gentle Reminder

Dear Reviewer uaJG,

Thank you again for your time and effort in reviewing our paper. As the discussion period ends on November 18 (AoE), we would like to kindly remind you of our response. We have addressed both of your questions: the paired uncertainty estimates grouped by source figure are in Appendix B.1 (Table 10), and all results in the revised paper are now scored against a single reference version, with the seven revisions itemized in Appendix A.1.

If our response resolves your concerns, we would be grateful if you could consider updating your assessment. We are also happy to answer any further questions before the discussion period closes.

Thank you very much for your attention.

Best regards,

Authors

Reviewer_PTWQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PaperMolBench is a human-annotated benchmark for extracting molecular structures from complete scientific figures rather than from cropped single-molecule images. Two tasks are defined: Molecule Extraction, where a natural-language query selects one target and the system returns its SMILES, and Reaction Extraction, where the system returns reactant and product SMILES for a specified reaction. Scoring uses exact match of canonical isomeric SMILES and an averaged chirality-aware Morgan fingerprint similarity, with a joint exact match for reactions. The benchmark contains 355 cases from 149 figures in 129 papers, of which 204 are molecule cases and 151 reaction cases, plus a 72-case diagnostic subset in which every molecule query is paired with its full figure, a verified target crop, and an RDKit rendering of the reference. Three system families are compared on molecule extraction with one proprietary model: a tool cascade with detection plus a dedicated recognizer, direct end-to-end generation, and an agentic pipeline that edits and re-renders the predicted structure. A test-time variant makes three generation calls and has the model select among deduplicated candidates. Reaction extraction is evaluated with end-to-end generation only. The diagnostic study compares the proprietary model with two open eight-billion-parameter vision-language models across the three input conditions. This motivates a training study in which both open families undergo continued pretraining on a mixture that includes synthetic multi-molecule recognition data, with checkpoints along each run followed by independent supervised fine-tuning in two paired variants differing only in whether added examples use their original depictions or RDKit renderings of the same molecules. Finally, probes test whether visual features and language-backbone image-token states separate alternative renderings of the same molecule from single-edit perturbations.

Critical strengths

1.The benchmark targets a real gap: the figure is retained in full and a natural-language query must select the target before recognition, rather than evaluating pre-cropped single molecules (L035-049). 2.Annotation is two independent annotators plus a third adjudicator, with adjudication sheets retaining initial labels and written reasons for corrections (L652-686). 3.Reference-level honesty is unusual: one unparseable reference is kept in the denominator and scores zero for every system, seven references updated before the paired analysis are disclosed, and one ambiguous product label is explicitly mapped to the wildcard form (L688-698). 4.The diagnostic subset is properly paired, with the same molecule presented as full figure, verified crop and RDKit rendering, which isolates localization from recognition rather than confounding them (L143-150). 5.The training study is a matched paired design, with target labels and all other data shared between the two fine-tuning variants and only the depiction source differing, and five generations per evaluation point (L349-359).

Critical weaknesses

1.Every headline system number depends on a proprietary endpoint whose identity the authors could not confirm: the API reported a channel identifier supplied by an intermediary (L812-820), so the molecule and reaction system tables are not reproducible and are not pinned to a verifiable model version. 2.That model is evaluated with a single greedy generation while the open models use five (L757-768), so the claim that its exact match does not improve with crops or renderings (L304-310) rests on one deterministic pass over 72 items, where the reported spread of 59.7, 51.4 and 54.2 is about four to six cases. 3.No confidence intervals or repeated runs appear for any system result; on 204 cases the 50.0 against 46.1 gap between the sampling configuration and the agentic pipeline is eight cases (L218-241). 4.The tool cascade is handicapped by its components rather than by its design: 30 of 204 cases fail at the detection stage and score zero (L812-816), and the recognizer is one specific tool while stronger recognition models are discussed in related work but not substituted. 5.The paired fine-tuning conclusion counts checkpoints along a single continued-pretraining run as if they were independent observations, 10 of 11 and 9 of 10 (L369-373), when the per-checkpoint margins are often one to four exact-match points on 72 cases and the trajectories are visibly non-monotone (L918-960). 6.The probing section has no control task despite citing the control-task literature (L104-106): positives are renderings of the identical molecule and negatives differ by one edit, so high separability is close to what any image-encoding representation would give, and the macro-average weights a 29-pair category equally with a 2,611-pair one (L406-418, L1008-1022).

Questions for authors

1.Can you rerun the molecule and reaction system tables against a directly verifiable endpoint, with the model version pinned, given that the current numbers come from a channel identifier reported by an intermediary (L812-820)? 2.Please report exact-match differences with confidence intervals, for instance bootstrap intervals over cases or exact McNemar tests on the paired 72-case diagnostic. The claim that crops and renderings do not help the proprietary model (L304-310) is the pivot of the paper and currently has no uncertainty attached. 3.Would you evaluate that model with five generations at the same temperature used for the open models (L757-768)? A single greedy pass is not comparable to a five-generation mean, and the conclusion may be an artifact of the asymmetry. 4.What does the tool cascade reach with a stronger recognizer and with the 30 detection failures handled by a fallback (L812-816)? As reported, 18.1 percent conflates detector recall with recognition ability. 5.Rather than counting checkpoints along one run (L369-373), can you report at least two independent continued-pretraining runs per family, or a paired test over cases at matched checkpoints? The limitations note that run-to-run variation is unmeasured (L445-452), which is precisely the variation the claim needs. 6.For the probes, what do control probes on randomized or shuffled representations give, and what are the per-category AUCs weighted by pair count rather than macro-averaged (L1008-1022)? 7.What is the release plan for the figures, crops, renderings and references, given that the ethics statement raises attribution and redistribution for figure assets (L454-462)? For a benchmark paper this determines whether the contribution is usable. 8.The same official checkpoints appear at 12.8, 12.5 and 16.7 exact match in different tables (L263-269, L918-940, L962-995). Could you state one canonical decoding protocol and report all baselines under it?

4 comments
AuthorsOpenReview ↗

Response to Reviewer PTWQ (Part 1 of 3): Endpoint, Confidence Intervals, and Five Generations

Thank you for the careful reading, and for recognizing the task setting, the annotation procedure and the paired designs. We ran additional experiments or analyses for all eight questions. Weaknesses 1 to 6 correspond to Questions 1, 3, 2, 4, 5 and 6 and are answered together with them. The revised paper is uploaded, with changes highlighted in blue.

Q1 (W1): "Can you rerun the molecule and reaction system tables against a directly verifiable endpoint, with the model version pinned [...]?"

We have verified that the model used is indeed gemini-3.8-flash, and the results of our verification runs agree with the submission. The corresponding description in Appendix B is updated accordingly.

Q2 (W3): "Please report exact-match differences with confidence intervals [...]. The claim that crops and renderings do not help the proprietary model is the pivot of the paper and currently has no uncertainty attached."

Thank you for this suggestion. The revision reports confidence intervals (bootstrap resampling source figures) for all system and diagnostic comparisons, and exact McNemar tests for the system comparisons (Table 3, Appendix A.4, and Table 10 in Appendix B.1).

For the claim you identify as the pivot of the paper (Gemini 3.8 Flash, 72 diagnostic cases, mean of five generations):

ComparisonEM difference95% CI
Target crop vs. full figure−1.7−6.5 to 3.3
RDKit image vs. full figure−0.6−6.6 to 5.4
RDKit image vs. target crop+1.1−4.1 to 6.8

All three differences are close to zero and all intervals contain zero. The claim therefore holds: providing Gemini with the target crop or the RDKit image does not improve its EM.

For the system comparisons on the 204 cases:

ComparisonEM difference95% CIMcNemar p
End-to-End Generation vs. Tool-Cascaded Method (DECIMER)+27.018.2 to 35.8below 0.001
Three calls + selection vs. End-to-End Generation+5.91.0 to 11.30.017
Three calls + selection vs. Agentic Pipeline+3.9−2.0 to 10.00.18

The advantage of End-to-End Generation over the Tool-Cascaded Method is robust. For the gap you mention, three calls + selection against the Agentic Pipeline (51.5 against 47.5 after unifying the references), the interval contains zero, and the revision states explicitly that it does not rank these two.

Q3 (W2): "Would you evaluate that model with five generations at the same temperature used for the open models?"

Yes. We re-evaluated Gemini 3.8 Flash under exactly the same setting as the open models (temperature 0.7, five generations):

InputSubmission (single generation)Mean of five generations
Full figure59.756.7
Target crop51.455.0
RDKit image54.256.1

The conclusion is unchanged and clearer: Gemini's EM is nearly identical across the three inputs (55.0 to 56.7), and the intervals of all paired differences contain zero (see Q2). Your assessment is correct: the spread among the three numbers in the submission reflects single-generation variation and disappears under the five-generation mean. The diagnostic table (Table 5 in the revision) now reports the five-generation means throughout, so that all models are evaluated under one setting.

AuthorsOpenReview ↗

Response to Reviewer PTWQ (Part 2 of 3): Tool Cascade and Paired Fine-Tuning Comparison

Q4 (W4): "What does the tool cascade reach with a stronger recognizer and with the 30 detection failures handled by a fallback? As reported, 18.1 percent conflates detector recall with recognition ability."

We replaced the recognizer with MolScribe and added two kinds of fallback for the detection failures (Table 11, Appendix B.2):

Tool-Cascaded configurationEM
DECIMER (configuration in the submission)18.6
MolScribe27.9
MolScribe; on detection failure, pass the full figure to the recognizer28.4
MolScribe; on detection failure, fall back to End-to-End Generation36.3
End-to-End Generation (reference)45.6
Three calls + selection (reference)51.5

With the stronger recognizer and the fallback, the Tool-Cascaded Method improves from 18.6 to 36.3, but remains 9.3 points below End-to-End Generation (95% CI 1.4 to 17.4) and 15.2 points below three calls + selection (5.7 to 24.9). The gain from the last fallback itself comes from End-to-End Generation.

To separate detection from recognition, we analyzed the stages on the 72 diagnostic cases: the detector returns no box in 13 cases and boxes that miss the target in 7; the target is detected in 52 cases, and Gemini selects the correct box in 50 of them. Selection is thus rarely the problem. The loss comes from detector recall and from recognition on real depictions: even when MolScribe is given the annotated target crop directly, its EM is only 34.7 (Table 5). This is an inherent limitation of the cascaded design: an error at the detection or recognition stage cannot be recovered downstream.

Q5 (W5): "Rather than counting checkpoints along one run, can you report at least two independent continued-pretraining runs per family, or a paired test over cases at matched checkpoints?"

We adopted the second option you suggest, and additionally added one independent run.

Paired test over cases (Table 15, Appendix C.4). The revision no longer counts checkpoints. For each model, we compare the two SFT variants case by case at matched checkpoints, combine the per-checkpoint differences into a single statistic, and obtain the confidence interval by resampling source figures; checkpoints are not treated as independent observations.

ModelEM difference on target cropsFPS difference on target cropsEM difference on full figures
InternVL3.5-8B+2.0 (−1.9 to 7.1)+1.1 (−2.1 to 4.9)−0.9 (−2.5 to 0.8)
Qwen3-VL-8B+2.7 (0.7 to 5.1)+4.9 (3.3 to 6.7)+0.4 (−1.0 to 2.2)
Qwen3-VL-4B (new)+5.6 (1.3 to 11.1)+8.0 (4.7 to 11.9)+0.9 (−1.6 to 4.1)

(Real-depiction SFT minus RDKit-rendered SFT; 95% CIs in parentheses.)

An independent run (Table 18). We ran the same pipeline on Qwen3-VL-4B, starting from its official weights; its continued pretraining is a fully independent run. Its results agree with those of the 8B model, with a larger advantage.

On target crops, the advantage of Real-depiction SFT passes the paired test for both Qwen models, and has the same direction for InternVL with an interval that contains zero; on full figures none of the three models shows a clear difference. This matches the statement in the paper: real depictions help recognition when the target is provided as a crop, but give no consistent full-figure advantage.

Independent external validation (Table 19, Appendix D). On eight classical OCSR benchmarks, Real-depiction SFT outperforms RDKit-rendered SFT on all eight for both model families: mean EM is 30.0 points higher for Qwen (27.4 to 32.6) and 9.0 points higher for InternVL (6.8 to 11.2). These benchmarks are unrelated to the diagnostic subset and much larger (1,600 images per model).

AuthorsOpenReview ↗

Response to Reviewer PTWQ (Part 3 of 3): Control Probes, Release Plan, and Decoding Protocol

Q6 (W6): "The probing section has no control task [...] so high separability is close to what any image-encoding representation would give, and the macro-average weights a 29-pair category equally with a 2,611-pair one."

We added the controls you ask for, and we also rebuilt the probe data: every image of every positive and negative now receives its own independent random layout (random coordinates, rotation, mirroring), which rules out separation based on layout similarity. The results (AUC, weighted by pair count; Table 6 and Appendix E in the revision):

SettingImage-token states of the language backboneVisual features
Trained probe0.911 (0.891 to 0.931)0.844 (0.821 to 0.869)
No probe; cosine similarity of the frozen features0.7610.605
Trained probe with shuffled labels0.7230.560
Randomly initialized vision encoder, probe retrainednot applicable0.516
Raw pixels, probe retrainednot applicable0.537
  • Not every image encoding achieves this. A probe retrained on the features of an encoder with the same architecture but random weights reaches only 0.516, and a probe retrained on raw pixels only 0.537; both are close to chance.
  • The separability is not produced by the capacity of the probe. With shuffled labels, the probe does not exceed the level obtained without any training (0.723 against 0.761, and 0.560 against 0.605).
  • The conclusion holds under pair-count weighting. The table above is weighted by pair count rather than macro-averaged.

Q7: "What is the release plan for the figures, crops, renderings and references [...]?"

We will publicly release the following, so that the benchmark can be used in full:

  • all queries and references, together with the annotation and adjudication records;
  • the bounding boxes of the target crops and the RDKit renderings for the diagnostic subset;
  • the evaluation code and the raw outputs of all systems in the paper.

For the figures themselves: those whose source license permits redistribution will be released directly with the dataset, with attribution; for the remaining figures, we provide the DOI of the source paper, the figure number and a retrieval script, so that users can reconstruct identical inputs locally. This release plan is added to the Reproducibility and Ethics Statements.

Q8: "The same official checkpoints appear at 12.8, 12.5 and 16.7 exact match in different tables. Could you state one canonical decoding protocol and report all baselines under it?"

Thank you for pointing this out. The three numbers correspond to three different settings, and one of them is not the official model:

NumberWhere (submission)Model and setting
12.8Table 4official InternVL3.5-8B; temperature 0.7, mean of five generations
12.5step 0 of the all-checkpoint tablethe model fine-tuned without continued pretraining (SFT-only); temperature 0.7, mean of five generations
16.7auxiliary fixed-mixture SFT table in the appendixofficial InternVL3.5-8B; temperature 0, single generation

The revision uses one protocol throughout: all models in the diagnostic and training experiments, including Gemini, are reported as the mean of five generations at temperature 0.7. The auxiliary single-generation table is removed; in the all-checkpoint tables (Tables 16 to 18), step 0 is now explicitly labeled SFT-only and the official model is listed in a separate row.

We hope these additions address your concerns, and we are happy to provide any further clarification.

AuthorsOpenReview ↗

A Gentle Reminder

Dear Reviewer PTWQ,

Thank you again for your time and effort in reviewing our paper. As the discussion period ends on November 18 (AoE), we would like to kindly remind you of our response. We have addressed all eight of your questions in three parts, including the five-generation evaluation of Gemini (Table 5), the confidence intervals and paired tests (Appendix B.1), the paired comparison of the two SFT variants (Appendix C.4), and the control probes (Table 6, Appendix E).

If our response resolves your concerns, we would be grateful if you could consider updating your assessment. We are also happy to answer any further questions before the discussion period closes.

Thank you very much for your attention.

Best regards,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of the Revision

We thank all reviewers for their time and constructive comments. We have uploaded a revised paper in which all changes are highlighted in blue. The main revisions are:

  • Frontier models (Table 3, Appendix B.3). Following Reviewer LBLz, we evaluated Claude Opus 5.5 and GPT-6-Sol on all 204 Molecule Extraction cases. End-to-End Generation reaches 65.7% and 52.9% EM; the best configuration reaches 67.6%, so about one third of the cases remain unsolved.
  • Stronger specialist baseline (Tables 3 and 5, Appendix B.2). Following Reviewers ovPr and PTWQ, the Tool-Cascaded Method is also run with MolScribe as the recognizer, with fallbacks for detection failures, and MolScribe is evaluated directly on the annotated target crops.
  • Statistical analysis (Table 3, Appendices A.4, B.1 and C.4). Following Reviewers uaJG, PTWQ and ovPr, we report confidence intervals from a bootstrap over source figures and paired comparisons with McNemar tests; the two SFT variants are now compared with a paired test over cases instead of checkpoint counts.
  • One protocol and one reference version (Table 5, Appendix A.1). Following Reviewers uaJG and PTWQ, all results are scored against a single version of the references, and the Gemini diagnostics now average five generations, as for the open models.
  • Classical OCSR benchmarks (Appendix D). Following Reviewer ovPr, we evaluated the official and fine-tuned models on eight classical benchmarks.
  • A third open model and a convergence analysis (Table 18, Appendix C.2). Following Reviewer ovPr, we added Qwen3-VL-4B and report the convergence of the training loss and of the downstream metrics.
  • Comparison with prior benchmarks (Table 1). Following Reviewer LBLz, Related Work now contains a systematic comparison table.
  • Control probes (Table 6, Appendix E). Following Reviewer PTWQ, we rebuilt the probe data and added control probes.
  • Coverage statistics and release plan (Appendix A.2; Reproducibility and Ethics Statements). Added following Reviewers ovPr and PTWQ.

The conclusions of the paper are unchanged. The best result on Molecule Extraction is updated from 50.0% to 67.6%, obtained with a newly added frontier model. We respond to each reviewer below and are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.