acceptodds
Under review as a conference paper at ICLR 2027

DEBATE, NOT VOTING: AGENTIC ARBITRATION OVER WEAK OPTICAL CHEMICAL STRUCTURE RECOGNITION MODELS

Abstract

Optical chemical structure recognition (OCSR) turns a picture of a molecule into a machine-readable string, and it is the bottleneck between decades of printed chemistry and any computational use of it. Individual OCSR models remain unreliable on real documents: on a 1000-molecule sample of MolParser, three recent readers score 46.0%, 43.4% and 42.4% exact match, yet at least one of them is right on 60.8% of molecules. That gap is an ensembling problem, but it is not one that voting can solve, because the readers rarely agree and are rarely all wrong in the same way. We instead place the readers inside a multi-agent scaffold in which four language-model agents with cheminformatics tools argue about a candidate structure, write their findings to an append-only knowledge base, and are constrained by render-and-compare gates that admit a correction only when it is demonstrably better. The debate reaches 61.9%, beating the best single reader by 15.9 points and plurality voting by 9.2 points ( for both, robust to correction for all 27 tests we report), and is statistically indistinguishable from the any-reader-right ceiling. That aggregate hides two opposing effects: on the 661 molecules carrying no stereochemistry the debate is 3.9 points above the ceiling (), constructing structures no reader proposed, while on the 339 stereochemical ones it falls below. We then swap the arbitrating model. A stronger arbitrator adds 2.2 points (nominal ; neither this nor the +3.9 survives Bonferroni correction over all 27 tests, so we treat both as suggestive), and its mechanism is visible: it opens a correction on 423 of 1000 molecules against 120, and an interaction test indicates that it depends less on the individual readers (nominal , not correction-robust). We argue that the useful unit of progress here is the arbitration policy, not the reader, and that reported OCSR accuracy should be read against the union oracle rather than against the best single model.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
LLM-as-a-judge, AI Agents, OCSR, SMILES, Agent Debate, Specialised Agents, Loop

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.80 over 5 reviews, above 27% of the others.
Reviewer_ietUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies optical chemical structure recognition (OCSR) in real documents and proposes a multi-agent arbitration framework over frozen readers. Three specialist models produce candidate SMILES, and four vision-language-model agents select or revise these candidates using an append-only shared knowledge base, RDKit tools, and render-and-compare gates. Additional procedural modules support local inspection, stereochemical analysis, and scaffold-based regeneration.

On 1,000 randomly sampled MolParser molecules, the two arbitration configurations achieve exact-match accuracies of 59.7% and 61.9%, compared with 46.0% for the best individual reader and 52.7% for plurality voting. The better configuration is not significantly different from the 60.8% union-oracle reference. The paper further examines structural corrections, stereochemical errors, and reader dependence through subgroup analysis, per-molecule activity logs, and leave-one-reader-out experiments on 300 molecules.

Critical strengths
  1. Clear end-to-end gains on a real-document sample. On the same 1,000 molecules and identical reader outputs, the Opus configuration improves over the best individual reader and plurality voting by 15.9 and 9.2 percentage points, respectively, with the main comparisons remaining significant after multiplicity correction. This supports the practical value of a tool-using reasoning layer for improving existing OCSR outputs, even though the sources of the gains are not yet adequately isolated.
  2. Per-molecule analysis exposes gains and losses hidden by aggregate accuracy. The paper distinguishes cases corrected despite all readers being wrong from cases lost despite an available correct candidate, and analyzes stereochemical, Markush, and other residual errors. This goes beyond reporting a single accuracy number and reveals both the risk of chemically valid but incorrect outputs and the trade-off between correcting molecular constitution and preserving stereochemistry.
Critical weaknesses
  1. Missing baselines and component ablations prevent attribution to debate. The voting baselines can only select existing candidates, whereas the proposed system additionally receives VLM-based image interpretation, tool access, structural correction and regeneration, and more inference computation. Appendix A.2 explicitly acknowledges the absence of a zero-reader configuration, a tool-using single-agent baseline, and ablations of the debate loop, RDKit tools, or rendering gates. Consequently, the experiments establish that the complete system outperforms simple selection rules, but do not establish the advantage of debate emphasized by the title and central framing. They also leave the necessity of the specialist readers unresolved. This is my main concern about the methodological contribution.
  2. Claims about arbitrator capability and mechanism exceed what the experiment isolates. The configurations differ in both model and sampling temperature, and each is evaluated only once. The reported intervals capture molecule-sampling uncertainty, not run-to-run system variability. The overall 2.2-point gain, the pooled reader-dependence interaction, and the 3.9-point non-stereochemical gain over the oracle do not survive Bonferroni correction across all 27 tests. Moreover, increased correction-proposal frequency is a behavioral association, not an established cause of improved performance. The authors acknowledge these limitations, but statements that the effect is “real” or that a stronger arbitrator substitutes its own reading for a missing reader should be moderated or supported by repeated, sampling-matched experiments and intervention-based ablations.
Questions for authors
  1. Assessment-critical experiment: How does a single agent performing iterative correction compare when given the same underlying model, original image, candidates, and tools, with an approximately matched inference budget? How does a zero-reader configuration using the VLM and tools perform? These controls would directly affect my assessment of the contribution of debate and the necessity of the specialist readers.
  2. Assessment-critical experiment: What happens when multi-round debate or the render-and-compare gates are removed while retaining the remaining components? In particular, can component ablations distinguish the contributions of debate, tool-based verification, and structural correction, rather than only removing input readers?
  3. Experimental validation and scope of interpretation: I recognize that the Opus 4.8 API restrictions may prevent strict matching of sampling temperature. Could the authors repeat both configurations on the same subset under their existing settings to assess the stability of the 2.2-point accuracy difference and the reader-dependence interaction? Since repeated runs would still not eliminate the sampling confound, these findings should be interpreted as differences between arbitration configurations rather than effects attributable to model capability alone.
Reviewer_7VJ5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates multi-agent arbitration for optical chemical structure recognition (OCSR). Three frozen readers—MolScribe, DECIMER, and MolSight—produce candidate structures. Four agent roles then select or revise these candidates using RDKit tools, an append-only shared knowledge base, and render-and-compare gates. On a sample of 1,000 molecules from the MolParser corpus, the two arbitration configurations achieve exact-match accuracies of 59.7% and 61.9%, compared with 46.0% for the best individual reader and 52.7% for plurality voting. The paper further analyzes recoveries and losses relative to the union oracle, stereochemical and Markush failures, and changes in behavior when replacing the arbitrating model or withholding individual readers.

Critical strengths
  1. The experimental motivation is well justified and quantitatively grounded. The authors observe that the best individual OCSR model achieves only 46.0% accuracy, while at least one reader is correct on 60.8% of the examples, revealing substantial untapped complementarity among the readers. This gap motivates a concrete research question: can arbitration grounded in image evidence and cheminformatics tools more effectively select and correct candidate structures?
  2. The complete system achieves meaningful improvements on a practical recognition task.The error decomposition and statistical disclosures help characterize the system’s limitations. The paper goes beyond aggregate accuracy by distinguishing recovery of previously unavailable answers from destruction of available correct answers: the better configuration solves 76 cases where all readers fail but loses 65 cases where at least one reader succeeds. The stereochemical, Markush, and constitutional error analyses are informative. Explicit disclosure of post-hoc subgroup analysis, sampling differences, and multiplicity concerns also improves the interpretability of the results.
Critical weaknesses
  1. The experiments do not isolate the contribution of multi-agent debate or its verification mechanisms. The main comparisons are between the complete system and three readers or two voting rules. However, the complete system additionally benefits from a strong vision-language model, tool access, and multiple reasoning steps. As acknowledged in Appendix A.2, essential controls are missing: the same vision-language model reading the image directly, and a single agent using the same candidates and tools.Consequently, the gains cannot yet be attributed specifically to multi-agent interaction rather than the model’s own visual recognition ability, tool use, or additional inference computation.
  2. The methodological novelty is relatively limited. The proposed approach primarily integrates established ideas—multi-agent debate, a shared knowledge base, cheminformatics tools, and rendering-based verification—into optical chemical structure recognition. Its contribution lies mainly in application-specific system design and engineering integration, while substantive innovations in agent coordination or arbitration mechanisms beyond existing approaches are not clearly established. Although the combination has practical value, the methodological novelty appears limited.
Questions for authors
  1. Is the multi-agent architecture necessary? Under comparable token, call, or monetary budgets, how much does the complete system improve over the same vision-language model reading images directly and over a single agent using the same candidates and tools? Please also report latency and actual inference costs.
  2. Which components contribute to the gains? Can you separately ablate the debate loop, render-and-compare gates, and key tools, reporting both accuracy and beneficial versus harmful corrections? Who performs the visual comparison, and what operational rule determines acceptance? Please also reconcile the statements about component ablations in Section 3.3 and Appendix A.2.
  3. Are the arbitrator differences robust across repeated runs? Under sampling configurations that are as comparable as possible and explicitly documented, do the 2.2-point improvement and reader-dependence interaction persist across independent runs? Is intervention frequency consistently associated with improved net accuracy?
Reviewer_T5N4OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies optical chemical structure recognition using an arbitration system over three frozen recognition models. Four language-model agents use RDKit tools and shared records to inspect candidates, propose corrections, and select a final SMILES output. Corrections are checked by rendering them and comparing them with the input image. On 1,000 MolParser images, the authors report 61.9% exact-match accuracy, compared with 46.0% for the best reader and 52.7% for plurality voting. The paper also analyzes errors and compares two arbitrator models.

Critical strengths
  1. The paper addresses a practical task and reports substantial accuracy gains over the evaluated single-reader and voting baselines.

  2. The evaluation uses paired comparisons, confidence intervals, and exact McNemar tests. The main gains over the baselines remain significant after multiple-testing correction.

  3. The analysis examines both successful corrections and cases where the system loses a correct reader answer. Separate results for stereochemistry and Markush structures help explain the system’s limitations.

Critical weaknesses
  1. The methodological novelty is unclear. The paper does not explain what the proposed combination adds beyond existing agent workflows.
  2. The evaluation lacks matched single-agent baselines and component ablations, so the gains cannot be attributed specifically to debate.
  3. The related work provides little direct comparison with prior debate, verification, and iterative correction approaches.
  4. The arbitrator comparison uses different sampling settings and only one run per configuration, limiting conclusions about model capacity.
  5. The abstract contains too many numerical details.
Questions for authors
  1. How does the full system compare with a single agent using the same model, candidate structures, tools, and inference budget?
  2. What is the exact acceptance rule for the render-and-compare gate, and how does accuracy change when this gate or the debate loop is removed?
  3. Does the arbitrator improvement hold across repeated runs with matched sampling settings?
Reviewer_XcHYOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the Optical Chemical Structure Recognition (OCSR) problem, which involves recognizing the structure of molecules from line drawings and converting them into SMILES, the textual representation of molecules (e.g., CC(=O)Nc1ccc(O)cc1 for paracetamol). To address this research problem, four language-model agents with cheminformatics tools argue about a candidate structure. The experiment uses a 1,000-molecule sample from MolParser.

Critical strengths
  1. It addresses an overlooked domain in chemical and molecular study, which is Optical Chemical Structure Recognition (OCSR). It could have good applications in improving RDKit functionality.
  2. It uses Agentic AI effectively to address the research problem using a debate method.
Critical weaknesses
  1. The method is very simple, lacks novelty, and relies heavily on the prior knowledge of LLMs. It uses existing applications in research rather than proposing a new method. No zero-shot or fine-tuning was used, as the authors indicate in line 105: “Figure 1 gives the whole scaffold. Three OCSR models run once per image and are never fine-tuned or otherwise adapted.”
  2. The paper is focused more on feature engineering and evaluation, which is reflected in the four points of contribution listed by the authors: the first is a debate-based arbitration scaffold over frozen OCSR readers, while points 2, 3, and 4 are basically the evaluation and the results derived from it.
  3. The baseline in this study lacks a fine-tuning approach, which could be the "Occam's razor" approach for this research problem. MolParser, which the authors cite but use only the corpus part of, already used this approach by training their model on 7.7M samples, achieving a state-of-the-art accuracy of 76.9% on the WildMol-10k benchmark. I understand that the intention of the authors may be to build a paper explaining how LLMs interpret this research problem, but if this is the case, it should be reflected in the title, abstract, and method.
  4. The entire study seems to be centered around LLMs (Opus 4.8, Sonnet 4.6), but this was not reflected in the motivation for the work, abstract, and conclusion. The main focus on LLMs is evident from the results and discussions in Tables 2, 3, and 4. Yet, LLMs are never mentioned in the abstract, contribution points, related work, conclusion, or even in the title.
Questions for authors
  1. Why did the authors not use the MolParser model as a baseline model, which I believe addresses the same issue? This can be seen from the Figure 1 caption.ion in the MolParser paper, which stated, "Figure 1. MolParser uses an end-to-end transformer to extract the chemical structures to string expression from the real patent or literature." I understand that the authors used 1,000 samples from the MolParser corpus, but I am referring to the model itself, which was trained on a 7.7M paired dataset to address almost the same problem.

  2. It is not clear why the authors decided to construct their own 1,000 samples from the MolParser corpus, while the MolParser model has already been evaluated on USPTO, UoB, CLEF, JPO, ColoredBG, USPTO-10K (10,000), and WildMol-10K. Moreover, the Uni-Parser team has already published both a validation set (403) and a test sample consisting of 10K. If the authors used any of the USPTO, UoB, or MolParser benchmarks, the reader would be able to see how this agentic-based approach performs against reported models in the MolParser papers, which include the following models: OSRA 2.1, MolVec 0.9.7, Imago 2.0, Img2Mol, and MolGrapher. Indeed, the MolParser-Base score on most of these benchmarks was higher than 61.9, which is the highest score reported in this submission. This suggests that if MolParser-Base were included in this paper, it would likely achieve the best result. Why did the authors choose to construct their own benchmark when several benchmarks already exist and are used by other models to address the same problem?

  3. Can the author clarify exactly the motivation for this work? I found two conflicting statements in the manuscript. The first sentence in the abstract states: “Optical chemical structure recognition (OCSR) turns a picture of a molecule into a machine-readable string, and it stands between decades of printed chemistry and any computational use of it.” This indicates that the main motivation is to address the OCSR issue and obtain state-of-the-art results. However, in the introduction, the authors state: “We ask how much of that reserve a language-model agent system can recover, and which part of the system does the recovering.” This statement shifts the motivation to an explainability study! If this is the motivation of the work, it should be stated clearly in the abstract. Is the motivation simply to address OCSR and achieve state-of-the-art results, or is it to explain how LLMs approach this research problem (explainability)?

  4. What are the parameter sizes of MolScribe, DECIMER, MolSight, Sonnet 4.6, and Opus 4.8? We need the parameter counts in Table 1 to see how these baselines compare to each other.

Reviewer_fupWOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an agent-based arbitration framework for optical chemical structure recognition (OCSR). Three existing OCSR models generate candidate SMILES, which are then examined by multiple LLM agents with access to cheminformatics tools. The framework also includes scripted refinement stages and render-and-compare validation. On 1,000 molecules from MolParser, the Opus 4.8 configuration achieves 61.9% exact-match accuracy, outperforming the best individual reader (46.0%) and plurality voting (52.7%). The paper provides paired statistical comparisons and a detailed error analysis. However, the current experiments do not fully establish which components are responsible for the observed improvement.

Critical strengths
  1. The paper addresses a practical OCSR problem and evaluates molecules containing challenging features, including stereochemistry and Markush groups. The strict canonical-SMILES exact-match metric is appropriate for the stated database-use setting. Combining complementary readers, chemical validation tools, and an auditable record of agent decisions is a reasonable design.

  2. The evaluation is carefully paired at the molecule level. Both arbitrator configurations use the same 1,000 molecules and identical initial reader outputs. The paper reports Wilson intervals, exact McNemar tests, paired bootstrap intervals, multiple-testing correction, subgroup analyses, and a leave-one-reader-out experiment. The authors also acknowledge that approaching the union-oracle ceiling does not establish statistical equivalence to it.

Critical weaknesses
  1. The contribution of agentic arbitration is not sufficiently isolated. The paper does not include a zero-reader baseline in which the arbitrating VLM reads the molecular image directly, nor a single-agent baseline with access to the same chemical tools. The debate process, RDKit validation, render-and-compare gates, and scripted refinement stages are also not separately evaluated through ablations. Therefore, the improvement over voting cannot be attributed specifically to multi-agent arbitration; it may largely come from the arbitrating model’s own visual recognition and chemical reasoning. The render-and-compare decision is described qualitatively, without a sufficiently reproducible comparison rule or acceptance threshold.

  2. Each main arbitrator configuration is evaluated in a single run on the same 1,000 molecules. The paired bootstrap and McNemar analyses quantify uncertainty across molecules, not variation across independent executions of the complete pipeline. The sampling settings are also unmatched: Sonnet uses temperature 0, whereas Opus runs at the provider default because the API does not accept the same parameter. Thus, the reported 2.2-point difference cannot be cleanly attributed to the arbitrator model. The evaluation further relies on one MolParser sample, without an independent audit of ground-truth structures or validation on another OCSR corpus. The reader pool has very limited Markush wildcard coverage, which makes the results on that subset difficult to interpret as a test of arbitration.

Questions for authors
  1. Can the authors provide a direct comparison with a single VLM agent using the same image input and cheminformatics tools, together with ablations of the debate process, RDKit validation, and render-and-compare gates? Please specify exactly how the render-and-compare decision is made, including what evidence the evaluator sees and what criterion is used to accept or reject a correction.

  2. Can the authors repeat the complete pipeline several times on the same fixed molecule set and report per-run accuracy, mean ± standard deviation, and run-level uncertainty separately from molecule-level confidence intervals? The arbitrator comparison should use matched sampling settings where possible; if the APIs do not permit this, please provide repeated runs and a sensitivity analysis. Evaluation on an independent OCSR corpus or an expert-audited subset would also help establish the reliability and generalisability of the results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.