- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents AutoChem, a multimodal multi-agent framework for reconstructing experiment-level chemical reaction records from scientific papers and supporting information. It integrates textual extraction, molecular structure recognition, procedure inheritance, and cross-modal entity alignment. AutoChem achieves 79.56% micro-F1 on 22 expert-annotated papers, substantially outperforming direct LLM extraction. The reconstructed knowledge is further organized into a reaction-centric knowledge graph and a 344-question reaction-optimization benchmark.
Critical strengths
- The paper addresses a meaningful challenge in chemical literature understanding. The 79.56% micro-F1 substantially exceeds the same-backbone GPT-5-mini baseline (47.30%), demonstrating the effectiveness of structured reconstruction.
- Explicit procedure inheritance and entity resolution are valuable for recovering distributed experimental information. The ablations in Table 3 provide convincing evidence for the importance of procedure inheritance.
Critical weaknesses
- Limited methodological novelty and insufficient competitive comparisons. Most components rely on established LLM-agent, entity-linking, and multimodal extraction techniques. Comparisons with recent systems such as ChemMiner, MERMaid, and ReactionSeek are missing.
- Incomplete validation of key contributions and downstream utility. The ablations omit ChemVision and cross-modal alignment, making the contribution of multimodal integration unclear.
Questions for authors
- How were the input formats and supported extraction fields aligned between AutoChem and OpenChemIE? Are comparisons with newer multimodal extraction systems available?
- Are there existing text-only, vision-only, or cross-modal alignment ablations that isolate the contribution of multimodal integration?
- Were the automatically generated optimization questions independently validated by chemistry experts? How were potentially incorrect reaction groupings identified?
- Are the KG-RAG improvements statistically significant under paired evaluation? How does KG retrieval compare with retrieving unstructured source passages?
8 comments
Response to Weaknesses #1
We thank the reviewer for this point. It is true that the individual building blocks of AutoChem (e.g., LLM agents, entity linking, multimodal extraction) are techniques established in prior works. However, it is important to note that we did not claim novelty for any single component - instead, AutoChem is a system-level contribution. AutoChem provides an end-to-end solution, from raw PDFs to structured records and a knowledge graph (KG), for a real problem that remains unsolved: reconstructing complete experiment-level reaction records from full papers and SI.
The difficulty in addressing the real-world challenge is that the evidence for one experiment could distribute across the whole document, specifically,
- Reaction conditions are often stated once in a general procedure (GP) and inherited by many entries;
- Reaction compounds are referred to by local labels (e.g., "3a") that are defined only in schemes, tables, or the SI;
- Reaction outcomes are reported separately in substrate-scope tables.
Contributions of our paper include
- Problem formulation: We formulate the task as the resolution of this distributed evidence.
- Two targeted components in one end-to-end pipeline: Existing systems do not treat these resolution steps as first-class. The Procedure Miner resolves GP inheritance, and the Cross-modal Chem Registry resolves local labels.
- Benchmark: We provide an expert-annotated, 14-field full-document benchmark on 22 papers that evaluates complete records rather than isolated fragments.
Comparison with existing systems: Table R3-W1-1 compares AutoChem with the systems suggested by the reviewer. These systems differ in target unit and input, and none treats GP inheritance as an explicit step. Their outputs therefore need to be adapted to our schema for a fair comparison.
Table R3-W1-1. Positioning comparison between AutoChem and closely related systems.
| Dimension | ChemMiner | MERMaid | ReactionSeek | AutoChem (ours) |
|---|---|---|---|---|
| Primary modality | Text + image + table | Vision-centric VLM | Text + graphics | Text + vision, fused |
| Target unit | Fragment-level reaction info | Visual elements → KG | Self-contained OS procedures | Experiment-level record |
| GP inheritance | Not a first-class step | Not a design target | Less critical (OS self-contained) | Explicit Procedure Miner |
| Cross-modal resolution | Coreference within text | Visual coreference | Within standardized entries | Cross-modal Chem Registry |
| Corpus regime | Heterogeneous literature | Graphics-heavy PDFs | Organic Syntheses | Raw papers + SI |
Head-to-head comparison. Among these systems, ChemMiner's output is closest to our schema. Therefore, for benchmarking, we adapted ChemMiner to our 14-field full-document protocol and evaluated it on the same 22-paper corpus against the same expert annotations. We also introduce OpenChemIE, a pipeline of specialized chemistry extraction models, as a non-agent baseline. Results are shown in Table R3-W1-2. The large gap (14.47 vs. 79.56 F1), with both precision and recall low for both baselines, shows that existing systems do not solve end-to-end, experiment-level record extraction from full papers and SI. This is consistent with the design differences highlighted in Table R3-W1-1: these systems target different units and inputs, and none treats GP inheritance or cross-modal label resolution as an explicit step.
Table R3-W1-2. Adapted comparisons under the same 14-field full-document protocol.
| Model | Correct Predictions | Total Predictions | Precision (%) | Recall (%) | F1 (%) |
|---|---|---|---|---|---|
| OpenChemIE | 355 | 1,427 | 24.88 | 2.15 | 3.95 |
| ChemMiner | 1,573 | 5,199 | 30.26 | 9.51 | 14.47 |
| AutoChem (ours) | 12,894 | 15,870 | 81.29 | 78.11 | 79.56 |
MERMaid and ReactionSeek. We did not include these two systems in the head-to-head comparison. The main reason is that, their inputs exclude part of the evidence that our protocol measures - so that their scores would mainly reflect input coverage rather than extraction quality. Specifically,
-
MERMaid extracts reaction data from figures and tables. Main-text and SI procedures, where GP conditions are typically stated, fall outside its scope, which makes it complementary to AutoChem.
-
ReactionSeek was validated on Organic Syntheses, where each article is a single, fully specified procedure. Applying it to our corpus would first require assembling each entry's procedure, including inherited GP conditions. That assembly is precisely the step our evaluation measures.
Paper Revisions. We have revised the manuscript to clarify AutoChem’s positioning, report the adapted ChemMiner comparison, and explain why direct comparisons with MERMaid and ReactionSeek are not well-defined given their different target units and corpus conventions.
Response to Weaknesses #2
We thank the reviewer for this point. We agree that the contributions of ChemVision and Cross-modal Entity Alignment should be examined more directly. Within the rebuttal period, we address this through an expert audit of their outputs, together with existing evidence on how image-derived records enter the downstream resources.
Scope of the analysis. Two considerations shaped this design. First, the 14-field extraction metric scores reaction-level fields; it does not score cross-modal identity links (SAME_AS / NAME_RESOLVED_BY), which are the primary output of alignment. A field-level F1 ablation would therefore capture alignment's main function only indirectly. Second, verifying image-side records at the structure level requires manual inspection against the original figures and tool-based SMILES conversion, which was not feasible at corpus scale within the rebuttal period. We acknowledge that a corpus-level quantitative ablation of ChemVision remains a limitation of the current version.
Audit design. We audited 10 reactions (Q01–Q10) covering five reaction families: Buchwald–Hartwig, Sonogashira, Ni/photoredox, radical decarboxylative C–C, and photocatalysis. For each case, a chemistry expert examined the following:
- the ChemVision-only output, verified against the original figure and SI over six fields (substrate, product, stereochemistry, catalytic system, conditions, outcomes), and
- the links that Cross-modal Entity Alignment adds between the independently produced ChemText and ChemVision outputs: SAME_AS links between textual and visual mentions of the same compound, and NAME_RESOLVED_BY links that identify unresolved image-side compounds by their textual names. The expert judged whether each link was identity-correct.
ChemVision-only results. Reported outcomes were correctly attributed in 9/10 cases. This included multi-product attribution in Q06, where the figure reports results for both product 3a (19%, 96% ee) and the TEMPO-trapping adduct 6 (69%); the adduct is not part of the paired text-side record. Condition recovery was substantially weaker. Only Q06 and Q09 matched all condition fields shown in the figure, and even in Q06 the figure refers to "standard conditions" whose details are given only in the SI. Typical failures included missing atmosphere or reagent amounts; missed entry-specific footnote conditions (e.g., Q02 used 4 mol% Pd/10 mol% L8 instead of the footnote-specific 2 mol% Pd/5 mol% L8 for 36 h); missing catalyst or light information (Q07–Q08); failed condition-string extraction (Q03/Q04); and target extraction failure (Q10). Structure-level errors also occurred: the image-side product structure was missing in Q06 and Q09, and substrate structures were partially incorrect in Q03 and Q09 (Appendix A.5). From these experimental results, we see that Image-side extraction is informative for structures and outcomes, but it is insufficient on its own.
Alignment results. All links proposed by alignment in the audited cases were judged identity-correct. In Q05, the two modalities assigned different roles (text: substrate; image: reagent), while the identity link was correct. Without alignment, these mentions would remain separate entity nodes; with alignment, each compound is connected to both its textual and visual evidence.
Summary. These audited cases suggest a division of duty as follows:
- ChemVision can contribute structures and outcomes that are absent from the paired text records (e.g., Q06),
- ChemText supplies most experimental conditions.
- Cross-modal Entity Alignment does not extract additional fields; instead, it links the two pathways at the level of chemical identity and provenance.
Results on Cross-Modal Alignment. cross-modal alignment was highly reliable on the audited links: all 11 SAME_AS/NAME_RESOLVED_BY links were identity-correct; 7/10 involved label mismatches (e.g., 2 vs. 2a, 1a vs. 1b, 1o vs. 1), and 2/10 image products had no SMILES (Q06 3a, Q09 6) but were resolved by text names. One case (Q05) had a role mismatch (text: substrate; image: reagent) while the identity was correct. Without alignment, these cases would produce duplicate entity nodes and disconnected provenance; with alignment, they form a single reaction record.
Analysis of the above experiments. With the above experiments, we can confidently clarify that Cross-modal Entity Alignment is not an additional extractor: its main benefit is chemical-identity resolution, provenance linking, and knowledge-graph compactness, while ChemVision contributes complementary structures, reagents, and outcomes that are often missing or ambiguous in text.
Paper Revision. In the revised manuscript, we have added a comprehensive analysis in Appendix A.5. We hope these modifications can address the reviewer’s concern.
General response
Thank you for your valuable and constructive feedback. We sincerely appreciate your thoughtful comments on our methodological novelty, competitive comparisons, multimodal integration, benchmark reliability, and downstream utility. To address your concerns, we have carefully responded to each point one by one and revised the manuscript accordingly. Most importantly, we have clarified the system-level novelty of AutoChem and conducted additional comparisons with ChemMiner and OpenChemIE under a unified evaluation protocol. We have also conducted expert-audited case studies to clarify the complementary contributions of ChemVision and Cross-modal Entity Alignment, and performed a comprehensive source-grounded audit of all 344 optimization questions to assess their reliability. Furthermore, we have conducted paired statistical significance analyses for KG-RAG and introduced a matched unstructured text-based RAG baseline to demonstrate the benefits of reaction-centric knowledge organization. These additional experiments, analyses, and discussions of current limitations have been incorporated into the revised manuscript and appendices to strengthen the rigor, completeness, and clarity of our work. If there are any remaining concerns or aspects requiring further clarification, we would be happy to address them. Thank you again for your valuable feedback!