acceptodds
Under review as a conference paper at ICLR 2027

AutoChem: Reconstructing Chemical Reactions From Distributed Multimodal Evidence

Abstract

Chemical AI systems require detailed experiment-level reaction records, yet such data remain difficult to recover from scientific literature. Information about a single experiment is often distributed across reaction figures, Supporting Information (SI), and shared procedures, connected by local compound labels and procedural references. We formulate this challenge as reaction reconstruction and introduce AutoChem, a fully automated, multimodal multi-agent framework that converts raw papers and SI into structured reaction records. AutoChem explicitly resolves compound identities and general-procedure inheritance, aligns textual and visual evidence, and preserves experiment-specific conditions, outcomes, and provenance. On 22 expert-annotated papers covering 14 reaction-information fields, AutoChem achieves 79.56% micro-F1, outperforming direct extraction with the same GPT-5-mini backbone by 32.26 points. Ablations highlight procedure inheritance as a key contributor to extraction accuracy. The reconstructed records yield a reaction-centric knowledge graph with 11,138 nodes and 39,075 edges, together with 344 controlled-variable optimization questions: 179 for condition optimization and 165 for substrate optimization. Without retrieval, none of the evaluated LLM configurations significantly exceeds its random baseline after multiple-testing correction. Cross-paper KG retrieval improves Top-1 accuracy across three models, with the largest numerical gain observed for GPT-5-mini (from 45.9% to 50.3%), suggesting downstream utility of the automatically reconstructed knowledge. Code, benchmark, and demo are available at https://anonymous.4open.science/r/AutoChem.

Primary area:
foundation or frontier models, including LLMs
Keywords:
AI agent, multi-agent system, AI in chemical applications

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2uz8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents AutoChem, a multimodal multi-agent framework for reconstructing experiment-level chemical reaction records from scientific papers and supporting information. It integrates textual extraction, molecular structure recognition, procedure inheritance, and cross-modal entity alignment. AutoChem achieves 79.56% micro-F1 on 22 expert-annotated papers, substantially outperforming direct LLM extraction. The reconstructed knowledge is further organized into a reaction-centric knowledge graph and a 344-question reaction-optimization benchmark.

Critical strengths
  1. The paper addresses a meaningful challenge in chemical literature understanding. The 79.56% micro-F1 substantially exceeds the same-backbone GPT-5-mini baseline (47.30%), demonstrating the effectiveness of structured reconstruction.
  2. Explicit procedure inheritance and entity resolution are valuable for recovering distributed experimental information. The ablations in Table 3 provide convincing evidence for the importance of procedure inheritance.
Critical weaknesses
  1. Limited methodological novelty and insufficient competitive comparisons. Most components rely on established LLM-agent, entity-linking, and multimodal extraction techniques. Comparisons with recent systems such as ChemMiner, MERMaid, and ReactionSeek are missing.
  2. Incomplete validation of key contributions and downstream utility. The ablations omit ChemVision and cross-modal alignment, making the contribution of multimodal integration unclear.
Questions for authors
  1. How were the input formats and supported extraction fields aligned between AutoChem and OpenChemIE? Are comparisons with newer multimodal extraction systems available?
  2. Are there existing text-only, vision-only, or cross-modal alignment ablations that isolate the contribution of multimodal integration?
  3. Were the automatically generated optimization questions independently validated by chemistry experts? How were potentially incorrect reaction groupings identified?
  4. Are the KG-RAG improvements statistically significant under paired evaluation? How does KG retrieval compare with retrieving unstructured source passages?
8 comments
AuthorsOpenReview ↗

General response

Thank you for your valuable and constructive feedback. We sincerely appreciate your thoughtful comments on our methodological novelty, competitive comparisons, multimodal integration, benchmark reliability, and downstream utility. To address your concerns, we have carefully responded to each point one by one and revised the manuscript accordingly. Most importantly, we have clarified the system-level novelty of AutoChem and conducted additional comparisons with ChemMiner and OpenChemIE under a unified evaluation protocol. We have also conducted expert-audited case studies to clarify the complementary contributions of ChemVision and Cross-modal Entity Alignment, and performed a comprehensive source-grounded audit of all 344 optimization questions to assess their reliability. Furthermore, we have conducted paired statistical significance analyses for KG-RAG and introduced a matched unstructured text-based RAG baseline to demonstrate the benefits of reaction-centric knowledge organization. These additional experiments, analyses, and discussions of current limitations have been incorporated into the revised manuscript and appendices to strengthen the rigor, completeness, and clarity of our work. If there are any remaining concerns or aspects requiring further clarification, we would be happy to address them. Thank you again for your valuable feedback!

AuthorsOpenReview ↗

Response to Weaknesses #1

We thank the reviewer for this point. It is true that the individual building blocks of AutoChem (e.g., LLM agents, entity linking, multimodal extraction) are techniques established in prior works. However, it is important to note that we did not claim novelty for any single component - instead, AutoChem is a system-level contribution. AutoChem provides an end-to-end solution, from raw PDFs to structured records and a knowledge graph (KG), for a real problem that remains unsolved: reconstructing complete experiment-level reaction records from full papers and SI.

The difficulty in addressing the real-world challenge is that the evidence for one experiment could distribute across the whole document, specifically,

  • Reaction conditions are often stated once in a general procedure (GP) and inherited by many entries;
  • Reaction compounds are referred to by local labels (e.g., "3a") that are defined only in schemes, tables, or the SI;
  • Reaction outcomes are reported separately in substrate-scope tables.

Contributions of our paper include

  • Problem formulation: We formulate the task as the resolution of this distributed evidence.
  • Two targeted components in one end-to-end pipeline: Existing systems do not treat these resolution steps as first-class. The Procedure Miner resolves GP inheritance, and the Cross-modal Chem Registry resolves local labels.
  • Benchmark: We provide an expert-annotated, 14-field full-document benchmark on 22 papers that evaluates complete records rather than isolated fragments.

Comparison with existing systems: Table R3-W1-1 compares AutoChem with the systems suggested by the reviewer. These systems differ in target unit and input, and none treats GP inheritance as an explicit step. Their outputs therefore need to be adapted to our schema for a fair comparison.

Table R3-W1-1. Positioning comparison between AutoChem and closely related systems.

DimensionChemMinerMERMaidReactionSeekAutoChem (ours)
Primary modalityText + image + tableVision-centric VLMText + graphicsText + vision, fused
Target unitFragment-level reaction infoVisual elements → KGSelf-contained OS proceduresExperiment-level record
GP inheritanceNot a first-class stepNot a design targetLess critical (OS self-contained)Explicit Procedure Miner
Cross-modal resolutionCoreference within textVisual coreferenceWithin standardized entriesCross-modal Chem Registry
Corpus regimeHeterogeneous literatureGraphics-heavy PDFsOrganic SynthesesRaw papers + SI

Head-to-head comparison. Among these systems, ChemMiner's output is closest to our schema. Therefore, for benchmarking, we adapted ChemMiner to our 14-field full-document protocol and evaluated it on the same 22-paper corpus against the same expert annotations. We also introduce OpenChemIE, a pipeline of specialized chemistry extraction models, as a non-agent baseline. Results are shown in Table R3-W1-2. The large gap (14.47 vs. 79.56 F1), with both precision and recall low for both baselines, shows that existing systems do not solve end-to-end, experiment-level record extraction from full papers and SI. This is consistent with the design differences highlighted in Table R3-W1-1: these systems target different units and inputs, and none treats GP inheritance or cross-modal label resolution as an explicit step.

Table R3-W1-2. Adapted comparisons under the same 14-field full-document protocol.

ModelCorrect PredictionsTotal PredictionsPrecision (%)Recall (%)F1 (%)
OpenChemIE3551,42724.882.153.95
ChemMiner1,5735,19930.269.5114.47
AutoChem (ours)12,89415,87081.2978.1179.56

MERMaid and ReactionSeek. We did not include these two systems in the head-to-head comparison. The main reason is that, their inputs exclude part of the evidence that our protocol measures - so that their scores would mainly reflect input coverage rather than extraction quality. Specifically,

  • MERMaid extracts reaction data from figures and tables. Main-text and SI procedures, where GP conditions are typically stated, fall outside its scope, which makes it complementary to AutoChem.

  • ReactionSeek was validated on Organic Syntheses, where each article is a single, fully specified procedure. Applying it to our corpus would first require assembling each entry's procedure, including inherited GP conditions. That assembly is precisely the step our evaluation measures.

Paper Revisions. We have revised the manuscript to clarify AutoChem’s positioning, report the adapted ChemMiner comparison, and explain why direct comparisons with MERMaid and ReactionSeek are not well-defined given their different target units and corpus conventions.

AuthorsOpenReview ↗

Response to Weaknesses #2

We thank the reviewer for this point. We agree that the contributions of ChemVision and Cross-modal Entity Alignment should be examined more directly. Within the rebuttal period, we address this through an expert audit of their outputs, together with existing evidence on how image-derived records enter the downstream resources.

Scope of the analysis. Two considerations shaped this design. First, the 14-field extraction metric scores reaction-level fields; it does not score cross-modal identity links (SAME_AS / NAME_RESOLVED_BY), which are the primary output of alignment. A field-level F1 ablation would therefore capture alignment's main function only indirectly. Second, verifying image-side records at the structure level requires manual inspection against the original figures and tool-based SMILES conversion, which was not feasible at corpus scale within the rebuttal period. We acknowledge that a corpus-level quantitative ablation of ChemVision remains a limitation of the current version.

Audit design. We audited 10 reactions (Q01–Q10) covering five reaction families: Buchwald–Hartwig, Sonogashira, Ni/photoredox, radical decarboxylative C–C, and photocatalysis. For each case, a chemistry expert examined the following:

  • the ChemVision-only output, verified against the original figure and SI over six fields (substrate, product, stereochemistry, catalytic system, conditions, outcomes), and
  • the links that Cross-modal Entity Alignment adds between the independently produced ChemText and ChemVision outputs: SAME_AS links between textual and visual mentions of the same compound, and NAME_RESOLVED_BY links that identify unresolved image-side compounds by their textual names. The expert judged whether each link was identity-correct.

ChemVision-only results. Reported outcomes were correctly attributed in 9/10 cases. This included multi-product attribution in Q06, where the figure reports results for both product 3a (19%, 96% ee) and the TEMPO-trapping adduct 6 (69%); the adduct is not part of the paired text-side record. Condition recovery was substantially weaker. Only Q06 and Q09 matched all condition fields shown in the figure, and even in Q06 the figure refers to "standard conditions" whose details are given only in the SI. Typical failures included missing atmosphere or reagent amounts; missed entry-specific footnote conditions (e.g., Q02 used 4 mol% Pd/10 mol% L8 instead of the footnote-specific 2 mol% Pd/5 mol% L8 for 36 h); missing catalyst or light information (Q07–Q08); failed condition-string extraction (Q03/Q04); and target extraction failure (Q10). Structure-level errors also occurred: the image-side product structure was missing in Q06 and Q09, and substrate structures were partially incorrect in Q03 and Q09 (Appendix A.5). From these experimental results, we see that Image-side extraction is informative for structures and outcomes, but it is insufficient on its own.

Alignment results. All links proposed by alignment in the audited cases were judged identity-correct. In Q05, the two modalities assigned different roles (text: substrate; image: reagent), while the identity link was correct. Without alignment, these mentions would remain separate entity nodes; with alignment, each compound is connected to both its textual and visual evidence.

Summary. These audited cases suggest a division of duty as follows:

  • ChemVision can contribute structures and outcomes that are absent from the paired text records (e.g., Q06),
  • ChemText supplies most experimental conditions.
  • Cross-modal Entity Alignment does not extract additional fields; instead, it links the two pathways at the level of chemical identity and provenance.

Results on Cross-Modal Alignment. cross-modal alignment was highly reliable on the audited links: all 11 SAME_AS/NAME_RESOLVED_BY links were identity-correct; 7/10 involved label mismatches (e.g., 2 vs. 2a, 1a vs. 1b, 1o vs. 1), and 2/10 image products had no SMILES (Q06 3a, Q09 6) but were resolved by text names. One case (Q05) had a role mismatch (text: substrate; image: reagent) while the identity was correct. Without alignment, these cases would produce duplicate entity nodes and disconnected provenance; with alignment, they form a single reaction record.

Analysis of the above experiments. With the above experiments, we can confidently clarify that Cross-modal Entity Alignment is not an additional extractor: its main benefit is chemical-identity resolution, provenance linking, and knowledge-graph compactness, while ChemVision contributes complementary structures, reagents, and outcomes that are often missing or ambiguous in text.

Paper Revision. In the revised manuscript, we have added a comprehensive analysis in Appendix A.5. We hope these modifications can address the reviewer’s concern.

Reviewer_wPLbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces AutoChem, a multimodal multi-agent framework for reconstructing experiment-level chemical reaction records from scientific papers and Supporting Information. The framework integrates textual and visual extraction, compound identity resolution, general-procedure inheritance, and cross-modal entity alignment. The reconstructed records are further organized into a reaction-centric knowledge graph and used to construct reaction optimization benchmarks.

Critical strengths
  1. The paper addresses an important practical problem: reconstructing complete experimental records from information distributed across scientific documents and modalities.

  2. The explicit treatment of compound identifiers and general-procedure inheritance is well motivated and addresses a meaningful challenge in chemical literature extraction.

  3. The extraction results demonstrate substantial numerical improvements over the evaluated baselines, and the ablation study provides evidence supporting the importance of procedure inheritance.

Critical weaknesses
  1. Appendix A.3.1 states that the initial expert-review records were derived from the unified knowledge-graph edge table. Could the authors clarify whether annotators independently examined the original papers and Supporting Information to identify all reactions and missing fields, or primarily reviewed and corrected system-generated records?
  2. The paper emphasizes cross-modal entity alignment as a central component of AutoChem. However, Table 3 only reports ablations of SI Scout, Procedure Miner, and Entity Mapper. Could the authors provide comparisons among text-only extraction, vision-only extraction, text–vision combination without explicit alignment, and the full system?
  3. The paper presents downstream knowledge-graph utility as an important contribution. However, Figure 3 shows relatively modest Top-1 improvements of 1.5–4.4 percentage points. Comparisons against conventional text-based RAG and retrieval over non-graph structured reaction records would help determine whether the improvements arise specifically from the knowledge-graph representation rather than additional retrieved context.
  4. Although AutoChem achieves strong overall extraction performance, the reaction-condition field reaches only 45.34% F1. Could the authors provide a detailed analysis of the major error categories and quantify how incorrectly reconstructed conditions affect downstream knowledge-graph applications and optimization benchmarks?
  5. Approximately 71.5% of the benchmark questions contain only two options, resulting in a random Top-1 baseline of 46.4%. Could the authors report performance stratified by candidate count and discuss whether the predominance of binary-choice questions limits the benchmark's discriminative power?
Questions for authors
  1. Are the expert annotations constructed independently from the original papers, or are AutoChem-generated records used as the primary starting point?

  2. Can the authors provide comparisons against specified agents such as ChemEAGLE, ChemMiner, ReactionSeek, etc., and tool-augmented LLM baselines under comparable computational budgets?

  3. What are the independent contributions of ChemVision and Cross-modal Entity Alignment to the final extraction performance?

  4. Are the KG-RAG improvements statistically significant under paired testing, and how do they compare with standard text-based retrieval?

  5. How sensitive is the optimization benchmark to the equal weighting of yield and enantiomeric excess?

  6. What are the main sources of errors in reaction-condition extraction, and how do these errors affect downstream applications?

14 comments
AuthorsOpenReview ↗

General response

Thank you for your valuable and constructive feedback. We sincerely appreciate your thoughtful comments on annotation reliability, cross-modal entity alignment, downstream knowledge-graph utility, and benchmark evaluation. To address your concerns, we have carefully responded to each point and revised the manuscript accordingly. Most importantly, we have conducted an independent blind annotation study, evaluated AutoChem and the baselines against two gold standards, and added expert-audited case studies to validate the contributions of ChemVision and cross-modal entity alignment. We have also introduced a matched Text-RAG baseline and statistical significance analyses to further verify the effectiveness of KG-RAG. Furthermore, we have conducted detailed analyses of reaction-condition extraction errors and their downstream impact, reported performance stratified by candidate count, and examined the sensitivity of optimization results to different yield–ee weightings. We have additionally expanded comparisons with related chemistry extraction systems and clarified the limitations of our approach. These additional experiments, analyses, and discussions have been incorporated into the revised manuscript and appendices to strengthen the rigor, reliability, and clarity of our work. If there are any remaining concerns or aspects requiring further clarification, we would be happy to address them. Thank you again for your valuable feedback!

AuthorsOpenReview ↗

Response to Weaknesses #1

We thank the reviewer for this important question. The annotators primarily reviewed and corrected system-generated records: as described in Appendix A.3.1, reaction rows were initialized from AutoChem's unified KG edge table. The chemistry expert then re-read the original article and SI for each paper, verified every row against the source, corrected incomplete or incorrect entries, and added reactions missing from the initial export. We agree that this post-editing workflow can still introduce anchoring bias, particularly for recall and for plausible-looking systematic errors. To quantify this bias directly, we conducted a blind from-scratch annotation on a stratified subset.

To quantify this bias directly, we conducted a blind from-scratch annotation of a stratified subset. A chemistry-domain expert manually annotated 5 of the 22 papers (22.73%) and 219 reactions (18.83%) without access to AutoChem’s predictions or pre-filled cells.

Table R2-W1-1 reports field-level agreement. Agreement is high for identity-sensitive fields (i.e., substrate 97.42%, product 99.07%, intermediate 100.00%, catalyst 84.90%, solvent 87.40%) and lower for conditions (67.91%). The condition disagreements arise mainly because the blind annotator recorded additional procedural steps (e.g., "pre-stirred for 5 minutes"), while the reported temperature, time, and atmosphere were consistent. From the observation, we highlight that post-editing did not anchor the gold set on the molecular identities that the KG is built on - its effect is limited to how much procedural detail is recorded in the condition fields.

Table R2-W1-1. Field-level agreement between the primary gold set and the blind from-scratch annotation.

FieldAgreeDisagreeAgreement (%)
Substrate4151197.42
Product213299.07
Intermediate120100.00
Catalyst2534584.90
Ligand853769.67
Solvent2153187.40
Other components1364077.27
Conditions1466967.91

Table R2-W1-2 evaluates AutoChem and the baselines on the same 5 papers / 219 reactions against two gold standards: (i) the primary post-edited gold set and (ii) the blind from-scratch gold set. We highlight two important observations. First, the blind gold set is not uniformly harder: Gemini (+0.50) and Llama 4 (+3.54) slightly improve on it, indicating that the two gold sets differ mainly in field granularity (e.g., the level of procedural detail in conditions) rather than in overall difficulty. Second, AutoChem's drop is modest (−4.19), and it remains the strongest system, leading the next-best model (GPT-5-mini, 66.22 F1) by 10.68 points. The paper's main conclusion therefore holds under blind annotation.

Table R2-W1-2. Performance on the primary gold set vs. the blind-annotated subset. The blind subset is 5 papers / 219 reactions; the primary set is the full 22-paper corpus.

MethodPrecisionRecallF1ΔF1
AutoChem(Ours)85.38 → 84.0077.28 → 70.9781.10 →76.90−4.19
DeepSeek49.02 → 42.8913.57 → 11.7321.26 →18.42−2.84
Gemini69.77 → 72.7557.77 → 56.6563.20 →63.70+0.50
GPT-5-mini66.39 → 67.9366.95 → 64.5966.67 →66.22−0.45
Llama 449.30 → 55.6534.23 → 36.3140.41 →43.95+3.54
OpenChemIE23.31 → 24.542.56 → 2.504.61 →4.53−0.07
ChemMiner32.42 → 30.7419.85 → 17.5424.62 →22.34−2.28

In the revised manuscript, we have to state the initialization-and-verification workflow explicitly. We have also added the blind-subset protocol, agreement analysis, and dual-gold-set evaluation to the Appendix 3.2 of the revised manuscript. We believe this directly addresses the reviewer’s concern about post-editing bias while being transparent about the original annotation design.

AuthorsOpenReview ↗

Response to Weaknesses #2

We thank the reviewer for this constructive suggestion. A four-way comparison among 1) text-only, 2) vision-only, 3) text–vision without explicit alignment, and 4) the full system would indeed better justify our claim. Following the reviewer’s advice, we have conducted an expert-audited case study and revised the manuscript accordingly. As reaction-image extraction requires manual structure verification and tool-based SMILES conversion, comprehensive ablation experiments of the image-only method (i.e., the performance of the image-derived subset) is difficult. As an alternative, we report expert-audited case studies instead, which also showcase the importance of having the vision-part design in our system. Details of our analysis are as follows.

Case-study Design. We randomly sampled 10 reactions from papers in five of the seven reaction families: 1) Buchwald–Hartwig, 2) Sonogashira, 3) Ni/photoredox, 4) radical decarboxylative C–C, and 5) photocatalysis. For each case, one chemistry expert examined two outputs: (a) ChemVision-only and (b) the full ChemText + ChemVision + Cross-modal Entity Alignment pipeline. Output (a) was verified against the original reaction figure and SI across six field groups: substrate, product, stereochemistry, catalytic system, conditions, and outcomes. For (b), we show the triples produced by each module before alignment, together with the entity links that alignment adds: SAME_AS links between textual and visual mentions of the same compound, and NAME_RESOLVED_BY links that identify unresolved image-side compounds by their textual names. The expert judged whether each link was correct. Because ChemText and ChemVision run independently and alignment operates only on their outputs, the text-side triples are the standalone ChemText output, and the pre-alignment triples from both modules are exactly what a no-alignment variant would produce; no separate runs were therefore needed.

Results. Vision-only extraction reliably recovered products and outcomes in 9/10 cases and even recovered an additional TEMPO adduct (6, 69% yield) in Q06 that was absent from the text-side record. However, vision-only performed substantially worse in extracting complete reaction condition sets: only Q06 and Q09 fully matched all condition fields, while the remaining cases exhibited varying degrees of missing information or extraction errors. Typical failures included missing reaction atmosphere or reagent amounts, failure to capture entry-specific footnote conditions (e.g., Q02 used 4 mol% Pd/10 mol% L8 instead of the footnote-specific 2 mol% Pd/5 mol% L8 for 36 h), missing catalyst or light information (Q07–Q08), failed condition string extraction (Q03/Q04), and target extraction failure (Q10). Cross-modal alignment was highly reliable on the audited links: all 11 SAME_AS/NAME_RESOLVED_BY links were identity-correct; 7/10 involved label mismatches (e.g., 2 vs. 2a, 1a vs. 1b, 1o vs. 1), and 2/10 image products had no SMILES (Q06 3a, Q09 6) but were resolved by text names. One case (Q05) had a role mismatch (text: substrate; image: reagent) while the identity was correct. Without alignment, these cases would produce duplicate entity nodes and disconnected provenance; with alignment, they form a single reaction record. We therefore now clarify that Cross-modal Entity Alignment is not an additional extractor: its main benefit is chemical-identity resolution, provenance linking, and knowledge-graph compactness, while ChemVision contributes complementary structures, reagents, and outcomes that are often missing or ambiguous in text.

In the revised manuscript, we have added a comprehensive analysis in Appendix A.5 and summarize our analysis in the main text. We hope these modifications can address the reviewer’s concern.

Reviewer_HbzzOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents AutoChem, a training-free multimodal multi-agent framework for reconstructing experiment-level reaction records from chemical papers and Supporting Information. AutoChem combines textual extraction, visual extraction, cross-modal entity alignment, procedure inheritance, and reaction-centric knowledge-graph construction. It reports 79.56% micro-F1 over 14 fields on 22 expert-annotated papers, a 32.26-point gain over the same GPT-5-mini backbone used directly. It further constructs a 344-question reaction-optimization benchmark and evaluates cross-paper KG retrieval. The main empirical finding is that procedure inheritance is the largest contributor to extraction performance, while downstream KG retrieval yields numerical but not statistically significant gains.

Critical strengths
  1. Rather than assuming 1/k accuracy, the authors account for per-question candidate counts and tied gold answers, yielding 46.4% Top-1, and state that this reflects 246 of 344 questions having only two options. This is exactly the diagnostic that reveals how little headroom the benchmark has, and the authors supply it themselves.

  2. Procedure Miner degrades 10 of 14 fields with the largest losses in ligand amount (−42.04), catalyst amount (−37.67), and catalyst (−36.76). The mechanistic reading — these details are stated once in a general procedure and omitted from individual entries — is well matched to the pattern, and the losses concentrate precisely where the hypothesis predicts rather than uniformly. Where the signal is weak the authors say so: the SI Scout’s field-level effects are called mixed and nonsignificant, and the Entity Mapper’s runtime effect is explicitly described as not a demonstrated reduction.

Critical weaknesses
  1. Post-editing a system’s output is a documented source of bias, and the bias is not symmetric across the comparison. AutoChem’s errors arrive pre-filled in the cells the annotator checks; a systematic error — a consistently misassigned role, a consistently truncated condition bundle — is the kind least likely to be flagged, because it looks internally coherent across rows. Baseline errors enjoy no such protection. Recall is affected most severely: a reaction AutoChem never extracted must be noticed de novo from the source, while every reaction it did extract is already on the page.

  2. The authors state that the protocol “relied on expert source verification rather than independent duplicate annotation; accordingly, inter-annotator agreement is not reported.” For 1,163 reaction rows across 14 fields, a single annotator with no reliability estimate is below the standard for a dataset contribution.

Questions for authors
  1. Was any portion of the gold standard annotated from scratch, blind to AutoChem’s output? If not, can you annotate a stratified 20–30% subset this way and report agreement with the primary gold set, plus AutoChem and baseline scores on both?

  2. Are there existing chemical-extraction resources (e.g. ChemX,ChemTable) you considered as an external reference point?

  3. Can you report significance tests (McNemar or binomial) for the three KG-RAG comparisons, and align the abstract’s wording with whatever they show?

  4. What is extraction performance on the image-derived subset alone?

8 comments
AuthorsOpenReview ↗

General response

Thank you for your valuable and constructive feedback. We sincerely appreciate your thoughtful comments on our annotation reliability, evaluation methodology, and experimental validation. To address your concerns, we have carefully responded to each point one by one and revised the manuscript accordingly. Most importantly, we have conducted two independent blind annotations, reported inter-annotator agreement, and re-evaluated AutoChem and the baselines to validate the reliability of our results. We have also added statistical significance analyses for KG-RAG, expert-audited case studies for image-derived extraction, and further discussions on external benchmarks and future evaluations. These additional experiments and clarifications have been incorporated into the revised manuscript and appendices to strengthen the rigor and clarity of our work. If there are any remaining concerns or aspects requiring further clarification, we would be happy to address them. Thank you again for your valuable feedback!

AuthorsOpenReview ↗

Response to Weaknesses #1

To address the reviewer’s concern, we conducted a blind from-scratch annotation of a stratified subset: 5 of 22 papers (22.73%) and 219 reactions (18.83%), annotated by a chemistry expert with no access to AutoChem’s predictions or pre-filled cells. Because the annotation started from the original papers and SI, recall was not anchored to existing rows, and the blind set recovered 4 reactions absent from the primary gold. In these papers, AutoChem's F1 decreases from 81.10 (primary gold) to 76.90 (blind gold). This indicates a measurable but limited post-editing effect. AutoChem remains the strongest system, 10.68 points ahead of the best baseline (GPT-5-mini, 66.22), and the overall ranking is unchanged. Identity fields agree closely with the primary gold (substrate 97.42%, product 99.07%). Field-level agreement is high for identity-sensitive fields (substrate 97.42%, product 99.07%, intermediate 100.00%), with the main disagreement in ligand and condition granularity (mainly additional procedural steps such as pre-stirring, rather than in molecular identity or outcomes). These results indicate that the performance gain of our system comes from AutoChem’s technical advantage rather than post-editing bias.

A second independent blind annotation leads to the same conclusion (see our response to Weakness #2 below). Further details about the above experiments are provided in our response to your Question 1 below. We have also revised the manuscript by supplementing further details in Appendix 3.2.

AuthorsOpenReview ↗

Response to Weaknesses #2

We thank the reviewer for this important point. We agree that a single annotator without a reliability estimate is insufficient for a dataset contribution. We therefore added two independent from-scratch annotations of a stratified subset.

Annotation design. Two chemistry experts (Annotator A and Annotator B) independently annotated the same 5 of 22 papers from scratch (22.73% of the full set), each yielding 219 reactions (18.83% of the full set). Both worked only from the original papers and SI, without access to AutoChem's predictions (as in the primary gold set) or each other's annotations. Annotator A's set is the blind annotation also used in our response to Q1. Together with the primary gold set, this gives three gold sets for the same five papers, and we report three comparisons: Table R1-W2-1: Annotator A vs. Annotator B (inter-annotator agreement, IAA); Table R1-W2-2: Annotator B vs. the primary gold; Table R1-W2-3: all systems scored against the primary gold vs. Annotator B's gold

Inter-annotator agreement (IAA). Table R1-W2-1 below reports agreement between the two independent annotators. For the seven entity fields, weighted agreement is 97.96% (1,492 agree / 31 disagree). Substrate, product, intermediate, catalyst, ligand, and other components are at 100%, and solvent is 88.12%. The solvent differences come from 30 Minisci entries where Annotator B left 1,4-dioxane blank and one Ni–photoredox entry with i-PrOAc vs. 2-MeTHF. Condition agreement is lower in raw form (43.38%) because the annotators differ in granularity: 122 of 124 disagreements are information supplements (e.g., one annotator records "pre-stirred for 5 min" or "degassed for 5 min"), and only 2 are true conflicts (0.91% of reactions). The low raw condition agreement therefore reflects reporting detail rather than factual disagreement.

Table R1-W2-1. IAA between two independent annotators (5 papers, 219 reactions).

FieldAgreeDisagreeAgreement (%)
Substrate4320100.00
Product2190100.00
Intermediate120100.00
Catalyst3050100.00
Ligand1170100.00
Solvent2303188.12
Other components1770100.00
Entity fields (weighted)1,4923197.96
Conditions (raw)9512443.38
— information supplements—122—
— true conflicts—20.91

Agreement between Annotator B and the primary gold. Table R1-W2-2 compares the second annotator with the primary gold set on the 215 common reactions. On the 215 reactions shared by both sets, entity-level weighted agreement is 90.71%: substrate 97.42%, product 99.07%, intermediate 100.00%, catalyst 85.23%, solvent 98.67%, other components 77.27%, ligand 69.67%. The ligand gap mainly reflects naming/representation variants. Condition disagreements separate into 85 information supplements, 2 true conflicts, and 30 one-sided missing entries.

Table R1-W2-2. Agreement between the second annotator and the primary gold set.

FieldAgreeDisagreeAgreement (%)
Substrate4151197.42
Product213299.07
Intermediate120100.00
Catalyst2544485.23
Ligand853769.67
Solvent223398.67
Other components1364077.27
Entity fields (weighted)1,33813790.71
Conditions (raw)9811745.58
— information supplements—85—
— true conflicts—20.93
— one-sided missing—30—

System evaluation on Annotator B's gold. Table R1-W2-3 re-scores all systems on the second annotator’s gold set. On the same five papers, AutoChem drops from 81.10 (primary gold) to 79.05 F1 (Annotator B) but remains the strongest system, 11.35 points ahead of the next-best model, GPT-5-mini (67.70). Gemini (+1.58), GPT-5-mini (+1.04), and Llama 4 (+4.28) improve, while DeepSeek, ChemMiner, and OpenChemIE decrease slightly. The ranking is unchanged, consistent with the results on Annotator A's gold (see our response to your Question #1).

Table R1-W2-3. Performance on the primary gold set vs. the second annotator’s gold set.

MethodPrecision (historical → second)Recall (historical → second)F1 (historical → second)ΔF1
AutoChem (ours)85.38 → 85.7377.28 → 73.4181.10 → 79.05−2.04
DeepSeek-V4-Flash49.02 → 44.8413.57 → 12.4221.26 → 19.46−1.80
Gemini-3.8-Flash69.77 → 73.4457.77 → 57.9563.20 → 64.78+1.58
GPT-5-mini66.39 → 68.9966.95 → 66.4766.67 → 67.70+1.04
Llama 4 Maverick49.30 → 56.1334.23 → 37.1140.41 → 44.68+4.28
OpenChemIE23.31 → 24.542.56 → 2.534.61 → 4.59−0.02
ChemMiner (adapted)32.42 → 32.2819.85 → 18.6824.62 → 23.67−0.95

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Author Responses

Dear Area Chair,

Thank you very much for your time and efforts in evaluating our submission. We sincerely appreciate the constructive feedback provided by all three reviewers. We provide the following summary to facilitate your review of our responses and the corresponding revisions.

In response to the reviewers' requests, we have conducted extensive additional experiments, analyses, and methodological clarifications during the rebuttal period, addressing the substantive concerns raised by Reviewers Hbzz, wPLb, and 2uz8:

  • Annotation Reliability and Evaluation Validity (Reviewers Hbzz and wPLb): We conducted two independent blind annotations, reported inter-annotator agreement, and re-evaluated AutoChem and the baselines against two independently annotated gold standards to further validate the reliability and robustness of our evaluation results.

  • Multimodal Integration and Cross-modal Entity Alignment (Reviewers Hbzz, wPLb, and 2uz8): We conducted expert-audited case studies to validate image-derived chemical information extraction and demonstrate the complementary contributions of ChemVision and Cross-modal Entity Alignment to reaction-centric knowledge graph construction.

  • Methodological Novelty (Reviewers 2uz8) and Competitive Comparisons (Reviewers wPLb): We clarified the system-level novelty of AutoChem and its distinctions from existing chemistry information extraction systems. We also conducted additional comparisons with ChemMiner and OpenChemIE under a unified evaluation protocol to better characterize the advantages and limitations of our approach.

  • KG-RAG Effectiveness and Statistical Significance (Reviewers Hbzz, wPLb, and 2uz8): We introduced a matched unstructured Text-RAG baseline and conducted paired statistical significance analyses to further validate the effectiveness of KG-RAG and demonstrate the benefits of structured, reaction-centric knowledge organization for downstream chemical reasoning and optimization.

  • Benchmark Reliability and Downstream Evaluation (Reviewers wPLb and 2uz8): We performed a comprehensive source-grounded audit of all 344 optimization questions to assess benchmark reliability. Furthermore, we analyzed reaction-condition extraction errors and their downstream impact, reported performance stratified by candidate count, and examined the sensitivity of optimization results to different yield–ee weightings.

Additionally, we have carefully revised the manuscript and appendices to incorporate these new experimental results, analyses, and methodological clarifications, further strengthening the rigor, reliability, and completeness of our work.

We greatly appreciate the reviewers' constructive feedback and the time you are dedicating to evaluating our submission. We hope that these substantial additional experiments and revisions adequately address the reviewers' concerns.

Sincerely,

The Authors of Submission 32833

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.