- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents RxnMiner, a multimodal agent that integrates a vision-language model, the Pi Agent framework, MolParser, R-group reconstruction tools, and RDKit to extract structured reaction data from chemical figures. The authors also introduce ChemRxn-Bench and report an improvement in reaction extraction F1 from 21% to 71% using Qwen as the backbone. However, the paper has notable limitations in methodological novelty and experimental design.
Critical strengths
-
Extracting complete reaction records from complex chemical literature is a valuable task, particularly when dealing with shared templates and implicitly specified reactants.
-
ChemRxn-Bench contains 603 figures and 8,547 manually annotated reaction records, providing a more rigorous evaluation benchmark for complete reaction extraction.
Critical weaknesses
W1. Existing systems like OpenChemIE and ChemEAGLE already support R-group substitution, reactant reconstruction, and chemistry-specific tool integration. The paper does not sufficiently clarify its contributions beyond these prior approaches.
W2. Table 2 reports F1 scores of 71.0, 69.0, and 64.0 for Qwen, Kimi, and DeepSeek, respectively, whereas Appendix D.2 reports 78.4, 76.3, and 71.2 for the corresponding models, without adequately explaining these discrepancies. Additionally, the abstract reports ChemEAGLE's F1 as 25.1%, while Table 1 shows that 25.1% corresponds to Precision and the actual F1 is 25.0%.
Questions for authors
Can the authors report average inference time, token consumption, the number of tool call to assess its practical efficiency?
6 comments
[Response 2/3] - Reconstruction Examples and Component Tests
RxnMiner generalizes better across different types of reaction figures. RxnMiner handles the following cases, which both baselines get wrong (App. D.6):
| Case | OpenChemIE | ChemEAGLE | RxnMiner |
|---|---|---|---|
| Ring size given as a repeat count (12-membered ring) | 6-membered | 6-membered | 12-membered |
| The same change needed at two sites | One site | One site | Both sites |
| Color-coded fragment origins | Fragments swapped | Fragments swapped | Correct assignment |
| Aromatic precursors (Ar¹ and Ar²) | Rings read as saturated | Rings read as saturated | Aromatic rings recovered |
| Nested aryl substituent | Scaffold misread | Template could not be parsed | Correct |
To isolate the substitution step itself, we also gave our engine and the OpenChemIE / ChemEAGLE functions exactly the same inputs (App. D.7):
| Test | RxnMiner | OpenChemIE / ChemEAGLE |
|---|---|---|
| Stereocentres and double-bond geometry kept after substitution (6 probes) | 6 / 6 | 2 / 6 |
| Same result for 24 equivalent atom orderings of one stereoisomer | 24 / 24 | 12 / 24 |
| R group "somewhere on a ring" (ChemEAGLE's own Fig. 6d), correct connectivity | 5 / 6 | cannot be expressed |
Two design choices explain these results. RxnMiner uses MolParser's Extended SMILES representation to encode ring-position-variable groups, whereas the tested baseline interfaces require fixed attachment atoms (App. C.1). And our engine keeps each atom's neighbour order when it substitutes, so wedge bonds survive. When reconstruction does fail, the reason goes back to the model, which can rebuild the structure from the figure or mark it as unresolved instead of returning the generic template. To be fair about our own limits: on Fig. 6d only 2 of the 6 products pass a strict round trip, because of a stereochemistry issue elsewhere in our pipeline.
In short, RxnMiner goes beyond existing systems by supporting more complex R-group substitutions, preserving stereochemistry, and reconstructing complete reactions more accurately through an integrated chemistry toolchain. We have made this clearer in the revised paper.
[Response 3/3] - Runtime, Tokens, and Tool Calls
Q1: Can the authors report average inference time, token consumption, the number of tool call to assess its practical efficiency?
Thank you for this practical question. Full statistics are in App. D.8 (Tables 18 and 20).
| System | RF1 | Time / fig, s | Tool calls / fig | Tokens in / out per fig | Cost / fig ($) | Cost / correct reaction ($) |
|---|---|---|---|---|---|---|
| RxnMiner (Qwen) | 70.8 | 1,344 | 64.8 | 4.74 M / 64.7 k | 0.59 | 0.06 |
| RxnMiner (Kimi K3) | 69.0 | 1,623 | 68.3 | 3.00 M / 45.9 k | 1.93 | 0.21 |
| RxnMiner (DeepSeek) | 63.6 | 1,915 | 78.7 | 6.30 M / 120.9 k | 0.67 | 0.08 |
| Bare Pi (Qwen) | 59.6 | 1,134 | 45.0 | 2.73 M / 60.7 k | 0.33 | 0.04 |
| Vanilla Qwen | 20.8 | 247 | – | 6.5 k / 16.0 k | 0.007 | 0.002 |
| ChemEAGLE (GPT) | 25.0 | 547 | – | 74.4 k / 16.3 k | 0.55 | ≈0.19 |
| OpenChemIE | 5.6 | 5 | – | local models | – | – |
RxnMiner is slower than ChemEAGLE (about 22 vs. 9 minutes per figure), but costs about the same per figure and about one third as much per correctly extracted reaction. Note that ChemEAGLE runs on GPT, so the backbones differ, and our Qwen runs were served locally and priced at the hosted Qwen rate.
We thank the reviewer again for the valuable comments. We believe these clarifications and new experiments strengthen the paper, and they are included in the revised version.
[Response 1/3] - Score Corrections and Method Contributions
Dear Reviewer,
We sincerely thank you for the careful reading and for recognizing the value of complete reaction extraction and of a rigorous benchmark for it. Below we respond to each point; all changes are in the revised PDF and marked in red.
W2: Table 2 reports F1 scores of 71.0, 69.0, and 64.0 for Qwen, Kimi, and DeepSeek, whereas Appendix D.2 reports 78.4, 76.3, and 71.2, without adequately explaining these discrepancies. Additionally, the abstract reports ChemEAGLE's F1 as 25.1%, while Table 1 shows that 25.1% corresponds to Precision and the actual F1 is 25.0%.
Thank you for carefully identifying these inconsistencies. We apologize for the errors in reporting these results. After re-examining the original evaluation records, we identified two issues and have corrected them in the revised manuscript.
The first is that the paper mixed two ways of computing reaction F1. Table 2 computes F1 for each figure and then averages over figures. App. D.2 instead averaged precision and recall over figures first and computed F1 from those averages, which gives higher numbers. App. D.2 now uses the same per-figure definition as Table 2.
The second is rounding: some values were rounded before they went into the tables, which is why many of them ended in ".0". The per-figure results themselves were correct, and we have now recomputed every number from the stored outputs. The ChemEAGLE value in the abstract was its precision by mistake and is now its F1.
For these three backbones, the Table 2 scores changed by at most 0.4 points. The larger changes in App. D.2 reflect the change in aggregation. The ranking is unchanged. We apologize for the confusion.
W1: Existing systems like OpenChemIE and ChemEAGLE already support R-group substitution, reactant reconstruction, and chemistry-specific tool integration. The paper does not sufficiently clarify its contributions beyond these prior approaches.
Thank you for this important concern. We agree that OpenChemIE and ChemEAGLE introduced R-group substitution, reactant reconstruction, and chemistry-specific tool integration;RxnMiner builds on these ideas and extends them in several ways: (1) more expressive R-group substitution, supporting complex Markush structures and preserving stereochemistry; (2) more accurate reconstruction of complete reactions, including reactants, products, and their corresponding labels; and (3) a richer chemistry toolchain, integrating tools such as MolParser and special-designed tools for structure recognition, reconstruction, and validation. Together, these improvements lead to substantially more accurate reaction extraction from real figures, and we added App. D.6 and D.7 to show this directly.
Firstly,RxnMiner is ahead on every source, including the figures each baseline was originally developed on:
Much of the gap comes from substrate-scope figures, where every product has to be paired with its own label and reactants. OpenChemIE struggles with more complex substrate-scope figures. The three ChemEAGLE examples below illustrate two assembly errors: assigning products to the wrong reaction role and merging distinct experiments into one record (App. D.6).