acceptodds
Under review as a conference paper at ICLR 2027

UREA: Read Like a Chemist, Commit Like a Compiler

Abstract

The chemical literature is written to be read by chemists: reactions are distributed across schemes, tables, captions, and procedures, and faithful extraction has resisted automation. UREA (UREA Reaction Extracting Agent) is an agentic framework for such multimodal reading: a general-purpose vision-language model cross-references figures with procedures, zooms into regions at will, and maintains a compound ledger, while a deterministic chemistry compiler resolves shared definitions into structured records. The workflow combines structural validation, one-label-one-structure authoring, count reconciliation, and evidence pointers. Reading stays permissive; commitment makes its decisions explicit and checkable. Count reconciliation addresses silent truncation, where whole-document reading returns well-formed output covering only a fraction of a paper's reactions: the agent inventories experiments up front, reconciles its records against that inventory, and revisits shortfalls. We also audit and correct the benchmark's reference annotations; UREA achieves state-of-the-art 89.73 hard-match F1 on the corrected whole-document chemical reaction benchmark. The design transfers across model families and agent harnesses, and the largest observed performance swing in the ablations follows a change in reasoning effort. Adapted through task-specific prompts to standalone molecular recognition, the reader outperforms MinerU.Chem under author review on 500 development images. Together, these results motivate a design pattern that combines flexible reading with explicit, checkable records for domains where structured knowledge must stay faithful to source documents.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Scientific document understanding, Chemical reaction extraction, Multimodal Agents, Domain-specific languages, Neurosymbolic AI

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3cbSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents UREA, an agentic framework for extracting chemical reactions from scientific papers. It combines a vision-language model with a shared chemical workspace, a compiler, and several verification tools. The system allows the agent to revise extracted records based on structural checks, visual comparisons, and reaction count reconciliation. Experiments show strong results on reaction extraction and molecular recognition benchmarks.

Critical strengths
  1. The paper addresses a practical and challenging problem in chemical literature extraction.
  2. The shared workspace and verification workflow are well designed and help improve the consistency and completeness of extracted reactions.
  3. The experimental results are strong, and the ablation studies provide useful insights into the contributions of different components.
Critical weaknesses
  1. The methodological novelty is limited. The framework mainly integrates existing models and tools into an agentic workflow, without introducing a new learning method.
  2. The distinction from existing agent-based chemical extraction systems, such as ChemMiner and ChemEAGLE, is not sufficiently clear.
  3. The performance is highly sensitive to the reasoning setting. Reducing reasoning effort from max to medium leads to a large performance drop, raising concerns about robustness and efficiency.
  4. Some baseline comparisons are difficult to interpret because of changes in model configurations and benchmark annotations.
Questions for authors
  1. What is the main methodological contribution of UREA beyond integrating existing tools into a workflow?
  2. How does UREA fundamentally differ from previous agent-based chemical extraction systems?
  3. Why does performance drop so sharply when reasoning effort is reduced, and can the workflow be made more robust?
  4. Can the authors provide more controlled comparisons using the original baseline configurations and evaluation settings?
1 comment
AuthorsOpenReview ↗

Methodological contribution, agent comparison, reasoning robustness, and original-backbone ChemEAGLE rerun

Thank you for recognizing the practical significance of document-level extraction and the value of our workspace design and ablations. All revisions in the updated manuscript are highlighted in blue.

1. Methodological contribution beyond tool chaining (W1, Q1). UREA introduces an inference-time compilable representation and revision protocol rather than a static prompt-and-tool pipeline. Instead of emitting isolated JSON records one by one, the agent authors shared compound definitions (a ledger), reusable scaffolds, and source-linked reaction blocks in a line-oriented DSL. A deterministic compiler resolves cross-references so editing one shared definition automatically propagates across all dependent records. Deterministic chemical checks, cross-record audits, and renderback visual comparisons guide iterative in-place edits, while count-before-extract reconciliation checks emitted records against an upfront experiment inventory to prevent silent truncation.

2. Fundamental differences from ChemMiner and ChemEAGLE (W2, Q2).

  • ChemMiner uses a coreference dictionary and conflict-driven revisiting before a reaction agent assembles flat JSON records.
  • ChemEAGLE orchestrates specialist extraction agents and parsers via plan/action observers on individual scheme images.
  • UREA centers on a single-session, document-level compilable workspace with explicit cross-record dependency graphs, deterministic edit propagation, and upfront inventory reconciliation—mechanisms absent from both prior systems.

We will sharpen this contrast in Related Work and illustrate it with a logged source-to-edit-to-record trace. Empirically, under identical model, harness, reasoning effort, and matcher settings, UREA outperforms direct reading (89.73 vs. 62.53 F1), and across three additional full-corpus passes (114 runs), achieves 85.67 ± 1.44 F1 (retaining a +11.01 mean advantage even when crediting all misplaced direct-reading outputs post-hoc).

3. Reasoning effort, robustness, and cost–quality tradeoff (W3, Q3). To diagnose effort sensitivity, we ran a three-arm control (57 executions) across all 19 papers using gpt-5.6-luna and frozen prompts/tools, isolating environment readiness from reasoning effort:

Effort / Renderer SetupF1Mean Min/PaperPapers w/ Returned Renderbacks
Medium / original setup20.994.340/19
Medium / dependencies pre-provisioned22.064.494/19
Max / dependencies pre-provisioned81.9334.5019/19

Even with rendering dependencies pre-provisioned, medium-effort runs rarely engage verification (requesting rendering on 5/19 papers and returning images on 4/19), whereas max-effort runs complete visual verification on 19/19 papers and gain +59.87 F1. For instance, in one max-effort trace, visual inspection caught a misplaced hydroxyl group in a shared definition; fixing it once updated 34 final reaction records upon recompilation. Based on these findings, we identify three concrete robustness improvements in the revision: startup tool-readiness checks, programmatic verification-completion gates, and adaptive verification budgets.

4. Original-backbone ChemEAGLE rerun and protocol alignment (W4, Q4). To provide an aligned baseline comparison, we executed the released ChemEAGLE codebase (commit 506492ab, 84 verified files) using its original gpt-5-mini backbone (gpt-5-mini-2025-08-07 via OpenAI-compatible gateway) across all 19 papers. Following the released README pipeline, automatic PDF figure/table extraction yielded 155 images; running ChemEAGLE with its released prompts, specialist checkpoints, plan/action observers, and logged OPSIN/PubChem/CIR lookup relays produced valid JSON on 127 images (28 ended in native pipeline errors), totaling 691 predictions:

Reference InventoryEntriesPrecision (%)Recall (%)F1 (%)
Original annotations2,14919.836.389.65
Corrected executed-experiment inventory1,19841.2423.7930.17

Scope & disclosure: Minimal compatibility patches addressed cached BioBERT IDs, API-key parameters, bbox indexing, and updated PubChem SMILES fields (CPU execution, 90-min/paper timeout, 16,384-token request cap, omitting temperature/effort as in the released code rather than full-text Markdown integration). In Table 1/8, we will explicitly separate: (i) published scores (retaining ChemEAGLE's published 62.39 F1 on original references), (ii) this released-code gpt-5-mini rerun, (iii) same-backbone gpt-5.6-luna reruns, and (iv) OpenChemIE local models, specifying input paths, backbones, reference versions, and failure-handling policies for each.

Reviewer_np84OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

UREA is an agentic framework for whole-document chemical reaction extraction from organic-synthesis papers. A general-purpose vision-language model (GPT-5.6-Luna via Codex in headline runs) reads schemes, tables, captions, and procedures; it records its reading as shared compound definitions (a ledger), reusable scaffold templates, and experiment-specific reaction blocks in a line-oriented DSL. A deterministic chemistry compiler resolves shared references into structured records, with RDKit structural validation, an audit tool, render-back visual comparison, and count-before-extract reconciliation (the agent inventories expected experiments per source group up front, then reconciles emitted records against that inventory to catch silent truncation). The authors also audit and correct the reference annotations of a 19-paper benchmark (2149 original entries convert into 1198 "executed-experiment" references, verified by an independent blind review). On the corrected benchmark UREA reaches 89.73 F1 vs 62.53 for direct reading with the same model / harness / reasoning setting (paired whole-paper bootstrap 95% CI [11.62, 49.87]). It also leads on figure-level reaction extraction (D2: 75.24 hard F1) and R-group resolution (D3: 73.98), with ablations over instructions, tool availability, and reasoning effort, cross-model deployments at 75-84 F1, and a task-adapted molecular-recognition reader scoring 89% vs MinerU.Chem's 75.6% on 500 development images.

Critical strengths
  1. A clean same-model controlled comparison with open protocol disclosure. The headline 89.73 vs 62.53 holds model, harness, reasoning setting, reference denominator, and matcher fixed; the paper separates published vs rerun scores along four explicit protocol dimensions (Table 8), keeps failures in denominators, reports observed tool requests/returns when instruction-omission and runtime-blocking give different outcomes (Table 10), and quantifies reference-version agreement (37.94 F1) rather than silently aligning versions. This is what makes the core claim credible, and it seems above-average rigor for this literature.
  2. A genuinely novel interface plus a corrected, audited benchmark. Making shared chemical definitions and reaction blocks directly compilable in one session — so the agent edits reading decisions in place and reference resolution propagates across records — is not present in the cited agentic systems (ChemMiner, ChemEAGLE, MERMaid, AgentCAT), and count-before-extract reconciliation is a new named mechanism for silent truncation in whole-document extraction. The audited reconstruction of the benchmark references (682/1,198 entries changed, independent blind audit) is itself a reusable contribution.
Critical weaknesses
  1. No code, data, prompts, or corrected-reference inventory is accessible. The paper contains zero links to any artifact: no repository, no "code will be released" statement, no supplementary material. Appendix B ("Prompts and Reproducible Execution") contains a table of what each configuration changes but no actual prompt text; the toolchain (tools/rs.py) is described, not provided; and the corrected 1198-entry reference inventory — a stated contribution that defines what is being scored — cannot be inspected or reused. As submitted, none of the quantitative claims can be independently verified. This is fixable by release, but it is a hard gap at review time and the main reason I cannot rate higher.
  2. No execution-level variance for a stochastic agentic pipeline. Every UREA number is a single selected run (with retries and output selection) of a 4-hour stochastic process; the paired bootstrap resamples papers, not runs. For an agent whose behavior is shown to swing 73 points with a reasoning-effort setting, the run-to-run variance of the headline 89.73 is a first-class question that the paper does not answer. Related: the D4 molecular-recognition claim (+13.40 pp) rests on a development subset where prompt tuning used part of the evaluated images, so it is not a fair held-out comparison.
Questions for authors
  1. Will the corrected 1198-entry reference inventory, the toolchain (compile / check / audit / renderback / instantiate), and the full task prompts be released? If so, when — and can you point to an accessible artifact now (even anonymous)?
  2. What is the run-to-run variance of the headline D1 configuration (89.73)? If you have multiple runs, what is the mean / std of F1 across them? A single selected run of a stochastic 4-hour pipeline is not enough to pin down the core claim.
  3. For D4: can you report accuracy on a held-out split that was not used in prompt development, with independent (non-author) review? As presented, the +13.40 pp over MinerU.Chem is measured on a development set where prompt tuning used part of the evaluated images.
  4. Can you add a limitations section and a short outlook (e.g., scaling beyond 19 papers, non-organic chemistry, endpoint checkpoint drift)?
  5. Can you reword the abstract's "89.73 hard-match F1"? By your own definitions, D1 matching is structure-only (your "soft" equivalent), whereas "hard" applies to D2.
2 comments
AuthorsOpenReview ↗

Reproducibility artifacts, multi-pass variance (114 runs), D4 scope, and metric terminology

Thank you for your thorough review and for recognizing the rigor of our controlled comparisons, protocol disclosures, and benchmark audit. We are especially grateful that you noted the main blockers were artifact accessibility and run-to-run variance—both of which we address in full below. All corresponding revisions in the updated manuscript are highlighted in blue. 1. Reproducibility artifacts in Supplementary Material (W1, Q1). All requested artifacts were uploaded with our initial submission under “Supplementary Material → zip” on the OpenReview forum page (referenced in the Reproducibility Statement, p. 10 and Appendix B, p. 17). Inside urea-supplement/, the archive provides:

  • references/: the audited 1,198-entry executed-experiment reference inventory;
  • extraction/: the full deterministic toolchain (compile, check, audit, renderback, and scaffold instantiation);
  • prompts/: the complete task prompts for all configurations;
  • predictions/, eval/, and reproduce.py: evaluated outputs, evaluation scripts, and the one-command score-reproduction entrypoint.

In the revision, we will surface this archive and its directory structure prominently in the Introduction and Reproducibility Statement.

2. Execution policy and multi-pass run-to-run variability (W2, Q2). First, we clarify two details regarding the headline 89.73 F1 result: (a) it comes from a single dedicated batch of one run per paper across all 19 papers with zero cherry-picking across retries, and (b) 4 hours is the hard per-paper timeout, whereas actual execution averaged ~37 minutes per paper. We will make this explicit in Appendix B. Second, to quantify stochastic variability directly, we executed three full-corpus passes per method (114 total document runs across the 19 papers and 1,198 references) using gpt-5.6-luna at maximum reasoning effort under a frozen runtime and evaluation protocol. Every execution is included in the denominator:

Method / Evaluation ProtocolRun 1Run 2Run 3Mean ± Sample SD
UREA (predefined protocol)86.6384.0286.3785.67 ± 1.44
Direct reading (predefined protocol)64.8667.7468.6567.08 ± 1.98
Direct reading (post-hoc path-sensitivity)72.4274.5777.0274.67 ± 2.30
  • Low variance and consistent margin: UREA exhibits low run-to-run variance (SD=1.44\text{SD} = 1.44) and outperforms direct reading by +18.59 F1 on average under the predefined protocol.
  • Generous path-sensitivity check: Across the three direct-reading passes, 11 outputs were written to the run root instead of the prescribed workspace path and thus scored as empty under the predefined protocol. Even when we credit all 11 unmodified files post-hoc using the exact same matcher and full denominator, UREA still maintains a +11.01-point mean advantage (computed from unrounded scores).
  • Runtime transparency: These three-pass runs were executed on Nov 11–12 using Codex CLI 0.159.3 (whereas the initial submission used 0.154.0). We report them in a dedicated table alongside the historical headline run, noting the model alias and potential server-side checkpoint updates.

3. D4 development-set framing and independent review (W2, Q3). On the shared 500-image cohort, our direct-SMILES reader achieves 89.00% versus 75.60% for MinerU.Chem (+13.40 pp). Two chemists outside the author team independently reviewed these outputs; we will detail their review protocol in the appendix. Because part of this 500-image cohort informed prompt refinement, we do not claim held-out generalization for D4—we will explicitly frame and scope this result throughout the paper as a development-set comparison, and prioritize blind held-out benchmarking in future work.

4. Dedicated Limitations and Outlook section (Q4). We will add a dedicated Limitations and Outlook section covering: (i) the 19-paper organic-synthesis main-text scope, (ii) reasoning-effort sensitivity and compute cost, (iii) the scope of our three-pass variance study, (iv) D4's development-set status, and (v) hosted endpoint checkpoint drift. Our outlook highlights scaling to larger multi-domain corpora (including Supporting Information PDFs), blind held-out molecular recognition benchmarks, and pinned/open-weight endpoint evaluations.

5. Abstract metric terminology (Q5). Thank you for catching this inconsistency. We will revise the abstract from "89.73 hard-match F1" to "89.73 structure-matching F1." In D1, evaluation matches canonicalized reactant and product graphs after stereochemistry removal (making the scorer's soft and hard branches numerically identical on D1 references), whereas "hard" in D2 additionally enforces exact matching on drawn condition molecules.

Thank you for the detailed response. The supplementary materials and new experiments address my main concerns, so I am increasing my rating to weak accept.

I withdraw my criticism about missing artifacts. The archive includes the code, prompts, references, and predictions, and the reported scores reproduce. It would still be useful to include the complete source-linked correction and audit trail for the reference inventory.

The repeated runs provide stronger evidence for the main result. I would give more weight to the comparison with the path-corrected baseline: 85.67 ± 1.44 vs. 74.67 ± 2.30 F1. This remains a substantial improvement. Please report it prominently and distinguish these runs from the historical headline, given the runtime changes and possible checkpoint drift.

The development-set framing for D4 is appropriate, though it does not replace held-out evaluation. Please also describe the independent review and how disagreements were resolved. The planned limitations section and terminology correction address my remaining questions.

Reviewer_t5kJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

UREA extracts experimental reaction records from chemistry papers through a shared, compilable workspace, combining chemical construction, validation, visual inspection, and coverage reconciliation. A vision-language agent interprets the source evidence and revises shared definitions and reaction blocks, while deterministic tools compile and check the resulting records.

Critical strengths

The paper addresses a practically important problem beyond isolated molecule recognition. The full workflow achieves strong document-level extraction results, and the comparisons with same-backbone JSON-based variants provide useful evidence of its effectiveness.

Critical weaknesses
  1. The presentation substantially obscures the contribution. The paper introduces many system-specific terms before establishing a concrete end-to-end workflow. It is difficult to distinguish the proposed mechanisms from capabilities provided by the underlying VLM and standard chemical tools. A worked example tracing the source evidence, workspace edits, tool feedback, and final records would greatly improve clarity.

  2. The Shared-JSON control is not explained in sufficient operational detail. This comparison is important for understanding the central workspace contribution, but Table 4 gives limited detail on reference resolution and feedback integration in this variant. I appreciate that Section 4.4 explicitly evaluates representation together with its authoring instructions. A clearer account of what is preserved and what changes relative to the row/ledger workflow would help interpret the observed advantage and the role of the compilable representation.

Questions for authors
  1. Add a case study. Could you use an existing execution trace to illustrate how feedback leads to a source-grounded workspace correction?

  2. How does the Shared-JSON variant resolve shared references and integrate chemical checks and feedback, and which parts of this process differ from the full workflow?

  3. Revising the paper to reduce abstract terminology and explain the workflow more concretely would improve readability.

1 comment
AuthorsOpenReview ↗

Worked workflow example, Shared-JSON operational details, and presentation revisions

Thank you for recognizing the practical value of document-level reaction extraction and the strength of our same-backbone controls. In the revision, we will restructure the early sections to lead with a concrete end-to-end workflow before introducing system terminology: (1) inventorying experiments from the source, (2) authoring shared compound definitions and reaction blocks, (3) compiling and running deterministic checks, and (4) inspecting visual/diagnostic feedback against the source to revise the workspace. All changes in the revised PDF are highlighted in blue.

1. Worked example from an existing execution trace (W1, Q1). We will include the following trace from Scheme 1 of Lindman et al. (DOI: 10.1021/acs.joc.3c00972, p. 3) as a worked example in Section 3 (with full before-and-after renderings in Appendix X). In this trace, the initial workspace omitted the O-tert-butyl protecting groups on substrate 2i and product 3i, representing both with free hydroxyl side chains:

StageExecution Trace & System Behavior
Initial recordsCompilation succeeds; chemical check reports 0 errors and audit reports 0 conflicts because free-hydroxyl structures are chemically valid.
Visual feedbackVia renderback, the agent inspects the rendered structures side-by-side against the source scheme and detects the missing protecting groups.
Workspace editThe agent updates the shared definitions for 2i and 3i, revising the side-chain fragment from [C@H](O)C to [C@H](OC(C)(C)C)C.
RecompilationThe compiler automatically resolves references in reaction record 0027 to the updated structures while preserving the 88% yield, followed by a final visual check.

We extracted the before-and-after comparison images directly from the logged execution trace, and the corrected record appears in our frozen evaluation output. This example makes the division of labor concrete: the VLM identifies source-level semantic discrepancies that pass rule-based chemical checks, the workspace centralizes the fix in shared definitions, and the deterministic compiler propagates the update across referencing records.

2. Preserved mechanisms vs. differences in Shared-JSON (W2, Q2). To isolate the effect of the compilable workspace representation, Shared-JSON preserves the entire backend resolution and feedback loop while changing only the authoring format and its corresponding instructions (Section 4.4):

  • What is preserved: Both variants share the same label-based * reference resolver (compile-json invokes the exact same resolver as the row-based compiler, throwing compilation errors on undefined labels and propagating definition edits upon recompilation). Both also expose the identical tool suite (check, audit, renderback, and scaffold instantiation), allowing the agent to execute the same compile–inspect–revise loop.
  • What changes: Shared-JSON requires the agent to author and edit nested JSON objects (storing the compound ledger, template memory, and reaction records in JSONL files), whereas the full workflow uses flat, line-oriented compound definitions and reaction blocks paired with row-oriented authoring instructions.

We will expand Section 4.4 (and add a side-by-side comparison in the Appendix) to make this boundary explicit.

3. Reducing abstract terminology and improving readability (Q3). We will streamline system-specific terminology in the Introduction and Method overview, grounding every component in the concrete roles demonstrated above: source interpretation and visual verification by the VLM agent, reference resolution by the compiler, chemical validation by deterministic tools, and completeness verification against the initial experiment inventory.

Comments on the paper

1 comment
Area_Chair_qkBnOpenReview ↗

Reminder to Reviewers: Please Engage in the Discussion

Dear Reviewers,

As the authors have now submitted their rebuttal, I would kindly ask you to read their responses and engage in the discussion if you have not already done so.

Thank you!

Best regards,
AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.