- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents UREA, an agentic framework for extracting chemical reactions from scientific papers. It combines a vision-language model with a shared chemical workspace, a compiler, and several verification tools. The system allows the agent to revise extracted records based on structural checks, visual comparisons, and reaction count reconciliation. Experiments show strong results on reaction extraction and molecular recognition benchmarks.
Critical strengths
- The paper addresses a practical and challenging problem in chemical literature extraction.
- The shared workspace and verification workflow are well designed and help improve the consistency and completeness of extracted reactions.
- The experimental results are strong, and the ablation studies provide useful insights into the contributions of different components.
Critical weaknesses
- The methodological novelty is limited. The framework mainly integrates existing models and tools into an agentic workflow, without introducing a new learning method.
- The distinction from existing agent-based chemical extraction systems, such as ChemMiner and ChemEAGLE, is not sufficiently clear.
- The performance is highly sensitive to the reasoning setting. Reducing reasoning effort from max to medium leads to a large performance drop, raising concerns about robustness and efficiency.
- Some baseline comparisons are difficult to interpret because of changes in model configurations and benchmark annotations.
Questions for authors
- What is the main methodological contribution of UREA beyond integrating existing tools into a workflow?
- How does UREA fundamentally differ from previous agent-based chemical extraction systems?
- Why does performance drop so sharply when reasoning effort is reduced, and can the workflow be made more robust?
- Can the authors provide more controlled comparisons using the original baseline configurations and evaluation settings?
Methodological contribution, agent comparison, reasoning robustness, and original-backbone ChemEAGLE rerun
Thank you for recognizing the practical significance of document-level extraction and the value of our workspace design and ablations. All revisions in the updated manuscript are highlighted in blue.
1. Methodological contribution beyond tool chaining (W1, Q1). UREA introduces an inference-time compilable representation and revision protocol rather than a static prompt-and-tool pipeline. Instead of emitting isolated JSON records one by one, the agent authors shared compound definitions (a ledger), reusable scaffolds, and source-linked reaction blocks in a line-oriented DSL. A deterministic compiler resolves cross-references so editing one shared definition automatically propagates across all dependent records. Deterministic chemical checks, cross-record audits, and
renderbackvisual comparisons guide iterative in-place edits, while count-before-extract reconciliation checks emitted records against an upfront experiment inventory to prevent silent truncation.2. Fundamental differences from ChemMiner and ChemEAGLE (W2, Q2).
We will sharpen this contrast in Related Work and illustrate it with a logged source-to-edit-to-record trace. Empirically, under identical model, harness, reasoning effort, and matcher settings, UREA outperforms direct reading (89.73 vs. 62.53 F1), and across three additional full-corpus passes (114 runs), achieves 85.67 ± 1.44 F1 (retaining a +11.01 mean advantage even when crediting all misplaced direct-reading outputs post-hoc).
3. Reasoning effort, robustness, and cost–quality tradeoff (W3, Q3). To diagnose effort sensitivity, we ran a three-arm control (57 executions) across all 19 papers using
gpt-5.6-lunaand frozen prompts/tools, isolating environment readiness from reasoning effort:Even with rendering dependencies pre-provisioned, medium-effort runs rarely engage verification (requesting rendering on 5/19 papers and returning images on 4/19), whereas max-effort runs complete visual verification on 19/19 papers and gain +59.87 F1. For instance, in one max-effort trace, visual inspection caught a misplaced hydroxyl group in a shared definition; fixing it once updated 34 final reaction records upon recompilation. Based on these findings, we identify three concrete robustness improvements in the revision: startup tool-readiness checks, programmatic verification-completion gates, and adaptive verification budgets.
4. Original-backbone ChemEAGLE rerun and protocol alignment (W4, Q4). To provide an aligned baseline comparison, we executed the released ChemEAGLE codebase (
commit 506492ab, 84 verified files) using its originalgpt-5-minibackbone (gpt-5-mini-2025-08-07via OpenAI-compatible gateway) across all 19 papers. Following the released README pipeline, automatic PDF figure/table extraction yielded 155 images; running ChemEAGLE with its released prompts, specialist checkpoints, plan/action observers, and logged OPSIN/PubChem/CIR lookup relays produced valid JSON on 127 images (28 ended in native pipeline errors), totaling 691 predictions:Scope & disclosure: Minimal compatibility patches addressed cached BioBERT IDs, API-key parameters, bbox indexing, and updated PubChem SMILES fields (CPU execution, 90-min/paper timeout, 16,384-token request cap, omitting temperature/effort as in the released code rather than full-text Markdown integration). In Table 1/8, we will explicitly separate: (i) published scores (retaining ChemEAGLE's published 62.39 F1 on original references), (ii) this released-code
gpt-5-minirerun, (iii) same-backbonegpt-5.6-lunareruns, and (iv) OpenChemIE local models, specifying input paths, backbones, reference versions, and failure-handling policies for each.