acceptodds
Under review as a conference paper at ICLR 2027

OrganicBench: Can Large Language Models Reason Like Chemists?

Abstract

Large language models have surpassed human experts in coding and mathematical theorem proving. However, they still struggle to tackle natural science problems in the real world, thereby limiting their potential for autonomous scientific discovery. We find that organic chemistry is a rich and challenging testbed grounded in real-world chemistry for evaluating AI chemists. To this end, we introduce OrganicBench, a multimodal benchmark comprising over 1,200 challenging organic chemistry problems grounded in published research literature. Given problem descriptions in images, SMILES strings, or molecular graphs, models are tasked with predicting reaction products, inferring key intermediates, and reasoning about reaction mechanisms. These tasks require models to understand realistic reaction conditions, apply correct knowledge, and perform complex reasoning in multistep synthetic routes, going beyond factual question answering based on memorization. Our evaluation demonstrates that frontier proprietary and open-source models fail to achieve competitive performance on these problems, and they exhibit consistent failure modes. Process-level analyses further reveal failures across multiple stages of problem-solving, exposing the gap between broad chemical knowledge and reliable complex reasoning. Overall, OrganicBench serves as an expert-level benchmark for measuring progress toward capable AI chemists.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Evaluation, Benchmark, AI Scientist, AI Chemist

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_V5s6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OrganicBench, a benchmark for evaluating the organic chemistry capabilities of LLMs. OrganicBench consists of three main tasks (Basic Principles, Reaction Prediction, and Route Completion) and supports three input modalities (Images, SMILES, and Molecular Graphs).

Critical strengths

I share the authors' concern that previous benchmarks have primarily focused on evaluating relatively simple perception-level capabilities, and I agree with the motivation behind this work.

  1. The paper is clearly and well-written, and it includes most of the essential elements expected of a benchmark paper. The Related Work section clearly distinguishes this work from prior studies, and the benchmark covers major tasks in organic chemistry.

  2. The paper provides comprehensive analyses of the benchmark results from multiple perspectives.

Critical weaknesses
  1. The number and diversity of the evaluated open-source and chemistry-specialized models seem somewhat limited. The evaluation mainly focuses on the Qwen3 series, and it would be beneficial to include results from a wider variety of models (e.g., GLM, Kimi, etc.). I would also like to see more diverse chemistry-specialized models included, even if they do not support all input modalities. If feasible, adding recent agentic frameworks (e.g., ChemCrow) would further strengthen the evaluation.

  2. It is unclear how the Reaction Prediction task substantially differs from conventional forward reaction prediction tasks. At first glance, it does not appear significantly different from widely used USPTO-based benchmarks. Furthermore, tasks (a) and (c) seem to have definitions similar to those in existing organic chemistry benchmarks (ChemCoTBench, RxnBench, and FukuyamaBench), even though these prior benchmarks may have limitations in their quality and depth of analysis.

Questions for authors
  1. Could the authors include evaluation results from a wider range of LLMs with different architectures, sizes, and purposes?

  2. Could the authors provide more detailed comparisons between OrganicBench and existing benchmarks, particularly regarding their task definitions?

  3. In organic chemistry, when reaction conditions are not specified in sufficient detail, multiple chemically plausible products may exist rather than a single correct answer. Could the benchmark account for such cases where multiple answers may be valid?

Since I share the authors' concerns about the limitations of existing benchmarks, I would be willing to revise my score upward if the issues raised above are adequately addressed.

12 comments

Dear Reviewer V5s6,

Thank you very much for taking the time to review OrganicBench and for your valuable feedback. In your comments, you noted the limited diversity of evaluated models(W#1, Q#1), the need for clearer comparisons of task definitions with existing benchmarks (W#2, Q#2), and the handling of reactions with multiple valid products (Q#3). During the rebuttal period, we conducted additional experiments across all 1209 problems and revised our manuscript to address each point. We hope these additional experiments and clarifications fully address your concerns regarding OrganicBench. Specifically, we have made the following updates:

  • Evaluated four additional general models (Kimi-K3.1-Pro, DeepSeek-V4.1-Flash, DeepSeek-V4-Pro, and GLM-5.3) across all supported input modalities (Section 4.1 and Table 1).
  • Evaluated two additional chemistry models (ChemSpark and ether0) under the SMILES modality (Section 4.1 and Table 1).
  • Reproduced the ChemCrow agent framework under the SMILES modality using three backbones (Gemini 3.1 Pro Preview, DeepSeek-V4.1-Flash, and Qwen3.8-27B) with five standard tools (Section 5.1 and Appendix D.5).
  • Clarified the differences between our Reaction Prediction task and conventional USPTO forward prediction benchmarks (Section 3.1 and Appendix A.2).
  • Added a comparison of task definitions between OrganicBench and existing benchmarks, including USPTO-based benchmarks, ChemCoTBench, ChemBench, ChemEval, RxnBench, SUPERChem, and FukuyamaBench (Section 2 and Appendix I).
  • Clarified how our curation pipeline and expert review incorporate multiple valid answers during dataset construction (Section 3.2.3).
  • Explained our equivalence scoring protocol, covering answer set matching, tautomer normalization, and stereochemical verification (Section 4.2 and Appendix D.2).
  • Established a human chemist baseline on a stratified 50-problem subset evaluated independently by two experts from the Chinese Chemistry Olympiad National Training Squad (Section 5.1 and Appendix D.5).
  • Evaluated the same human experts on 41 intermediate transformation steps (from 9 problems) alongside models under fully specified instructions to validate our single-step process analysis (Section 5.3.3 and Appendix F.4).

Here are our detailed responses to each of your questions and suggestions.↓

AuthorsOpenReview ↗

The author's detailed response [1/5]

W#1: The number and diversity of the evaluated open-source and chemistry-specialized models seem somewhat limited. ... Q#1: Could the authors include evaluation results from a wider range of LLMs with different architectures, sizes, and purposes?

Re: Thank you very much for this constructive suggestion. We fully agree that expanding the diversity of model families, chemistry-specialized models, and agentic frameworks broadens the empirical scope of OrganicBench and substantially strengthens our evaluation. Following your recommendation, we evaluated additional general-purpose models across the Kimi, DeepSeek, and GLM families, two additional chemistry-specialized models (ChemSpark and ether0), and a reproduction of the ChemCrow agentic framework across three diverse backbones on all 1209 problems.

First, to ensure fair comparison, every newly added model was evaluated across all input modalities it natively supports (image, SMILES, and molecular graph for Kimi-K3.1-Pro; SMILES and molecular graph for DeepSeek-V4-Pro and DeepSeek-V4.1-Flash; and SMILES for GLM-5.3, ChemSpark, and ether0). Due to rebuttal time constraints, GLM-5.3 has currently completed one full evaluation run, primarily because it frequently produces excessively lengthy reasoning chains that reach the maximum token limit and require extended inference time. We will complete the remaining runs for GLM-5.3 in the revised version. The overall and task-level accuracy are summarized below:

Table 1. Evaluation results (%) of additional general-purpose and chemistry-specialized models on OrganicBench across supported input modalities.

ModelInput ModalityBasic PrinciplesReaction PredictionRoute CompletionOverall Acc.
Kimi-K3.1-ProImage71.48%30.81%26.61%37.14%
Kimi-K3.1-ProSMILES70.98%32.13%31.99%38.57%
Kimi-K3.1-ProGraph71.64%29.83%27.69%36.56%
DeepSeek-V4.1-FlashSMILES70.98%28.81%33.87%36.34%
DeepSeek-V4.1-FlashGraph61.69%16.63%20.70%24.54%
DeepSeek-V4-ProSMILES64.18%26.21%25.27%32.42%
DeepSeek-V4-ProGraph64.51%24.96%18.28%30.85%
GLM-5.3SMILES68.16%22.51%20.16%29.86%
ChemSparkSMILES22.72%0.26%0.00%3.97%
ether0SMILES0.66%1.24%0.00%1.02%

Two key findings emerge from these results. First, general-purpose models across diverse architectures show the same pattern observed in Section 5.1, performing well on Basic Principles while dropping sharply on Reaction Prediction and Route Completion. Second, the newly evaluated chemistry-specialized models still perform poorly. ChemSpark achieves only 3.97% overall accuracy, and ether0 frequently fails to follow output formatting instructions and reaches only 1.02% even after format correction. This confirms that current chemistry-specialized models still lack expert-level chemical reasoning capability.

Second, following the official ChemCrow codebase, we reproduced the ChemCrow pipeline across all 1209 problems under the SMILES modality using three backbone models (Gemini 3.1 Pro Preview, DeepSeek-V4.1-Flash, and Qwen3.8-27B), equipping them with the five standard ChemCrow tools (MolSimilarity, SMILES2Weight, FunctionalGroups, ReactionPredict, and ReactionRetrosynthesis). The comparison against the original SMILES baselines is shown below. Table 2. Performance comparison (%) of three backbone models with and without the ChemCrow agentic framework under the SMILES modality.

BackboneBasic PrinciplesReaction PredictionRoute CompletionOverall Acc.
Gemini 3.1 Pro Preview73.80%38.70%31.18%43.77%
+ChemCrow70.15% (↓3.65%)33.94% (↓4.75%)26.61% (↓4.57%)39.21% (↓4.55%)
DeepSeek-V4.1-Flash70.98%28.81%33.87%36.34%
+ChemCrow65.17% (↓5.81%)25.00% (↓3.81%)30.65% (↓3.22%)32.26% (↓4.08%)
Qwen3.8-27B62.19%16.70%15.59%24.15%
+ChemCrow44.78% (↓17.41%)10.29% (↓6.41%)4.03% (↓11.56%)15.38% (↓8.77%)
AuthorsOpenReview ↗

The author's detailed response [2/5]

Across the 1209 problems, all three backbones actively invoke the provided tools, reaching tool usage rates of 67.99% (1301 total calls) for Gemini 3.1 Pro Preview, 86.60% (9560 calls) for DeepSeek-V4.1-Flash, and 49.63% (2317 calls) for Qwen3.8-27B. Nevertheless, overall accuracy drops across every backbone by 4.08% to 8.77% relative to the SMILES baseline. A primary cause is that the underlying reaction tools (ReactionPredict and ReactionRetrosynthesis) struggle on complex literature reactions and frequently return inaccurate predictions that mislead the reasoning process. This shows that building capable chemistry agents requires stronger chemical reasoning alongside much more reliable domain tools. In the revised manuscript, we have added the new general-purpose and chemistry-specialized models to Section 4.1 and Table 1 (Section 5.1), and included the ChemCrow agent evaluation and analysis in Section 5.1 and Appendix D.5.

Finally, to provide a direct human reference alongside these expanded model results, two experts from the Chinese Chemistry Olympiad National Training Squad independently completed a stratified 50-problem subset (13 textbook, 20 competition, and 17 research literature), as reported in Tables 3-4, as well as 41 intermediate transformation steps from 9 annotated problems in this subset to examine single-step structural execution, as shown in Table 5. Human experts outperform all frontier models on Research Literature and Route Completion, and achieve 98.78% accuracy on the single-step prompts, highlighting models' persistent gap in complex reaction reasoning and description-grounded structural transformation (Section 5.1, Section 5.3.3, Appendix D.5, and Appendix F.4).

Table 3. Accuracy comparison between frontier models and human chemistry experts on the representative subset of 50 problems grouped by question source.

SubjectTextbookCompetitionResearch LiteratureOverall
GPT-5.6 Sol89.74%66.67%1.96%50.67%
Gemini 3.1 Pro Preview79.49%66.67%11.76%51.33%
Claude Opus 574.36%70.00%9.80%50.67%
Human Expert80.77%75.00%29.41%61.00%

Table 4. Accuracy comparison (%) between frontier models and human chemistry experts on the representative subset of 50 problems grouped by task category.

SubjectBasic PrinciplesReaction PredictionRoute Completion
GPT-5.6 Sol75.76%48.28%30.00%
Gemini 3.1 Pro Preview81.82%49.43%23.33%
Claude Opus 572.73%47.13%36.67%
Human Expert68.18%56.90%65.00%

Table 5. Intermediate prediction accuracy on 41 transformation steps from 9 problems.

SubjectOriginal Single-Step PromptFully Specified Instruction
GPT-5.6 Sol68.29%100.00%
Qwen3.8-Flash-Next60.98%90.24%
Qwen3.8-27B51.22%80.49%
Human Expert98.78%\
Reviewer_hRDwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OrganicBench introduces a benchmark of 1,209 organic chemistry problems covering basic principles, reaction prediction, and synthetic-route completion, with image, SMILES, and molecular-graph inputs. Evaluations of frontier and chemistry-specialized LLMs reveal substantial limitations in complex chemical reasoning. Further diagnostic experiments analyze failures in molecular perception, knowledge application, structure transformation, and answer generation.

Critical strengths
  1. Expert-curated and validated benchmark. OrganicBench contains 1,209 challenging problems focused on a well-defined subdomain of chemistry—organic chemistry—rather than broadly covering the entire chemistry field. The benchmark undergoes two rounds of expert review to ensure question quality and answer correctness, enhancing its reliability as an evaluation resource.

  2. Interesting and insightful analyses. The paper provides detailed error analyses and diagnostic experiments that investigate model limitations in molecular perception, knowledge application, structure transformation, and answer generation. These analyses offer useful insights into why current LLMs struggle with complex chemical reasoning.

Critical weaknesses
  1. Insufficient differentiation from existing benchmarks (Main Concern). The Introduction motivates OrganicBench by arguing that prior benchmarks emphasize shallow perception, do not reflect research complexity, and suffer from performance saturation. However, ChemBench already contains reasoning questions and open-ended items; ChemEval includes reaction outcome prediction, retrosynthesis, and mechanism analysis; and SUPERChem is already a challenging multimodal, expert-level reasoning benchmark with process-level evaluation. The paper provides only general comparisons without clearly demonstrating what makes OrganicBench unique. Its distinctive contributions and novelty over existing benchmarks remain insufficiently justified.

  2. Limitations in process-level analysis. The diagnostic experiments do not fully isolate different reasoning capabilities. For example, multiple-choice knowledge probes provide additional cues, while reaction descriptions are generated using ground-truth information. These factors may confound the conclusions about the underlying sources of model failures.

Questions for authors
  1. What is the unique contribution of OrganicBench compared with existing chemistry benchmarks?Although the paper highlights the limitations of prior benchmarks, it remains unclear what fundamentally distinguishes OrganicBench from existing benchmarks such as ChemEval, SUPERChem, and RxnBench. Could the authors further clarify and substantiate the unique value and necessity of OrganicBench, beyond differences in task formats and difficulty? A more concrete comparison with existing benchmarks could help strengthen this justification.

  2. How reliable are the process-level diagnostic conclusions? The experiments may not fully isolate different reasoning capabilities, as knowledge probes provide additional cues and reaction descriptions are generated using ground-truth information. Could the authors clarify how these factors affect the conclusions and provide further evidence to support the claimed disentanglement of reasoning failures?

7 comments

Dear Reviewer hRDw:

Thank you very much for taking the time to review OrganicBench and for your valuable feedback. In your initial comments, you noted concerns regarding insufficient differentiation from existing chemistry benchmarks (W#1, Q#1) and potential confounding cues in our diagnostic experiments (W#2, Q#2). During the rebuttal period, we conducted targeted validation experiments and expanded our comparative analysis to address both points. Specifically, we have added the following updates and experiments:

  • Compared OrganicBench directly with ChemBench, ChemEval, SUPERChem, and ChemCoTBench in terms of task design, structural verification, and stereochemical evaluation.
  • Distinguished OrganicBench from RxnBench and FukuyamaBench, sharpened Section 2, and added an extended comparative analysis in Appendix I.
  • Converted all 546 suitable knowledge probe questions into a free response format and had human chemistry experts verify the outputs of three representative models.
  • Evaluated 64 challenging problems using general knowledge hints generated solely from original problem statements without reference solutions.
  • Updated Section 5.3.2 and included the full experimental details and validation tables in Appendices F.2 and F.3.
  • Constructed a 50-problem stratified subset evaluated independently by two experts from the Chinese Chemistry Olympiad National Training Squad to establish a human baseline (Section 5.1 and Appendix D.5).
  • Evaluated the same human experts on 41 intermediate transformation steps (from 9 problems) alongside models under fully specified instructions to verify diagnostic task clarity (Section 5.3.3 and Appendix F.4).

Here are our detailed responses to each of your questions and suggestions.↓

AuthorsOpenReview ↗

The author's detailed response [1/5]

W#1: Insufficient differentiation from existing benchmarks (Main Concern). ... Its distinctive contributions and novelty over existing benchmarks remain insufficiently justified. Q#1: What is the unique contribution of OrganicBench compared with existing chemistry benchmarks? ... A more concrete comparison with existing benchmarks could help strengthen this justification.

Re: Thank you very much for raising this thoughtful and important concern. Your deep familiarity with existing chemistry benchmarks is clear, and we genuinely appreciate you pointing out the need for a sharper, more concrete differentiation from ChemBench, ChemEval, ChemCoTBench, SUPERChem, RxnBench, and FukuyamaBench beyond broad task labels.

First, ChemBench and ChemEval differ from OrganicBench in what their answers actually test and how their outputs are verified. Although ChemBench includes 335 reasoning tagged items and 35 open-ended items in its organic chemistry subset, all 35 open-ended items require only numerical answers, such as counting isomers or 13C NMR peaks, while the remaining 394 items are multiple-choice questions focused on reactivity rankings, chemical properties, or molecular formulas. Because ChemBench evaluates only these outputs, it leaves exact bond connectivity, reactive sites, and stereochemical configurations completely untested and therefore cannot accurately assess whether a model can solve real organic reactions. ChemEval includes mechanism analysis, reaction prediction, and synthesis planning, yet its mechanism questions ask only for general intermediate classes such as carbocation, radical anion, or enolate ion evaluated by an LLM judge, without requiring the exact intermediate structure or stereochemical outcome. Its multi-step synthesis tasks likewise ask for free-text route descriptions graded on a 1 to 5 scale by an LLM judge, leaving the exact molecular structures along the route unchecked. Because ChemEval relies on such coarse evaluations, it assesses only a broad understanding of reactions and cannot accurately reflect whether a model can deduce real chemical transformations. OrganicBench avoids these limitations by grounding all routes in verified literature and requiring exact molecular structures at every unknown step.

Second, SUPERChem and ChemCoTBench differ from OrganicBench because their formats allow models to bypass genuine chemical deduction. While SUPERChem offers challenging multimodal problems and evaluates reasoning traces via RPF, all 500 items end in a single multiple-choice selection. Across its 129 items involving organic reactions, 62.8% ask only about molecular properties, counts, or local structural features, and among the 33 synthesis items, 84.8% rely on property statements or options combining figures and properties. Because properties such as ring counts, functional groups, or acid sensitivity describe only partial aspects of a molecule, the candidate options point models directly to the specific reactive site or structural feature needed to choose the correct option. For example, in a problem containing two separate reaction schemes, confirming a single lactam ring in the second scheme is enough to identify the correct option without solving the first scheme. Even though RPF checks reasoning coverage against reference steps, final accuracy reflects only whether the chosen option matches a partial feature. ChemCoTBench similarly simplifies evaluation by providing explicit reaction names, operational prompts, and step templates that inform the model which transformation to execute when predicting products of elementary steps. For mechanism selection, models simply pick a plausible answer from given textual descriptions, which makes mechanistic reasoning far simpler than deducing selectivity on concrete molecules. Similarly, retrosynthesis in ChemCoTBench is limited to recovering a precursor in one step, whereas OrganicBench requires coordinating intermediates across an entire synthetic pathway. Beyond these task differences, the public ChemCoTBench evaluator removes stereochemical flags before comparing SMILES and checks only basic connectivity. OrganicBench provides no candidate options and requires models to generate the complete structure of every unknown molecule with exact connectivity and stereochemistry.

AuthorsOpenReview ↗

The author's detailed response [2/5]

Third, RxnBench differs from OrganicBench because it is designed as a literature comprehension benchmark in which the source materials are given directly to the model. Its single-figure tasks display full reaction schemes that often already depict key intermediates and reaction transformations alongside four options, while its full-document tasks pass the entire research paper PDF or text together with the options. Consequently, even though these items carry labels such as structure reasoning or retrosynthesis, models can arrive at the right answer simply by reading the given catalytic cycles, locating yield and stereochemical records in the article, and matching depictions across figures and options, which requires only multimodal comprehension and information retrieval without demanding genuine chemical reasoning. By contrast, OrganicBench removes all explanatory schemes and paper text, hides the target intermediates and products, and requires models to deduce and generate the missing structures directly from chemical principles.

Finally, while FukuyamaBench shares a similar goal in predicting products and mechanistic intermediates for complex reactions, its evaluations are concentrated strictly within isolated single reactions. OrganicBench goes beyond single step reactions by evaluating continuous transformations across multiple consecutive reaction steps in full synthetic pathways, alongside foundational chemical principles. Furthermore, OrganicBench establishes a comprehensive evaluation framework spanning three input modalities, separate connectivity and stereochemical metrics, and detailed diagnostic analyses across each stage of reasoning, thereby revealing the limitations of current models in expert level reasoning on natural science problems through the lens of organic chemistry. To make the unique value of OrganicBench crystal clear, we have sharpened the benchmark comparisons in Section 2 and added a dedicated, in-depth comparative analysis in Appendix I of the revised manuscript.

Table 1. Comprehensive comparison with existing chemistry benchmarks.

  • Task Design: Gen. = open-ended structure generation; Lit. = literature reaction inference without source solutions; Route = multistep route completion.
  • Evaluation: Stereo. = strict stereochemical verification; Modal. = aligned image, SMILES, and molecular-graph inputs; Diag. = process-level diagnostics.
  • Qualifications: Partial = limited coverage; (M) = multiple-choice output; (T) = free-text routes; (1) = single-step retrosynthesis; (D) = supplied source documents or diagrams.
BenchmarkGen.Lit.RouteStereo.Modal.Diag.
ChemBench✗✗✗✗✗✗
ChemEvalPartial✗✗ (T)Partial✗✗
SUPERChem✗ (M)✓✗ (M)Partial (M)✗✓
ChemCoTBench✓✗✗ (1)✗✗✗
RxnBench✗ (M)✗ (D)✗✗✗✗
FukuyamaBench✓✓✗✓✗Partial
OrganicBench (Ours)✓✓✓✓✓✓
Reviewer_s9LVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OrganicBench evaluates organic chemistry reasoning using 1,209 expert-curated problems from research papers, competitions, and textbooks. It covers chemical principles, reaction prediction, and synthetic-route completion, with aligned image, SMILES, and molecular-graph inputs.

Critical strengths
  1. The benchmark checks the source, accepts alternative answers, and excludes unresolved items. It also checks molecules' bonding and stereochemistry.

  2. The paper compares image, SMILES, and graph inputs. It also shows cases where a model describes the right chemical change but produces the wrong molecular structure.

Critical weaknesses
  1. The knowledge questions test only selected chemical principles. Answering them correctly does not show that the model knows everything needed to predict the reaction.

  2. The single-step test asks for exact structures while leaving some bond and stereochemical details unstated. Errors may therefore reflect uncertainty about the intended change, rather than an inability to carry it out.

  3. The paper does not report results separately by question source or compare models with chemists answering the same questions. This makes its claims about expert research ability difficult to judge.

Questions for authors
  1. Can chemists determine a single correct structure from the starting molecule and short instruction used in the single-step test?

  2. Do models still make these errors when the bond changes and stereochemistry are fully specified?

  3. Can you report accuracy separately for research, competition, and textbook questions, and compare it with chemists answering the same questions?

8 comments

Dear Reviewer s9LV,

Thank you very much for taking the time to review our paper and for your valuable feedback. In your initial comments, you noted three key concerns: whether the knowledge probe QAs reflect the principles needed for reaction prediction (W#1), the lack of accuracy results by question source and direct comparisons with human chemists (W#3, Q#3), and potential ambiguity in the transformation test on individual steps due to unstated bond and stereochemical details (W#2, Q#1, Q#2). During the rebuttal period, we conducted additional experiments and human expert evaluations, striving to verify the effectiveness of our knowledge probes, establish a human chemist baseline across all question sources, and examine task clarity and model errors under fully specified instructions. Specifically, we have added the following updates (highlighted in orange in the revised manuscript):

  • Clarified our core intent in Section 5.3.2 regarding the gap between possessing knowledge and autonomously applying it during reasoning.
  • Invited human chemistry experts to annotate 203 intermediate reaction steps across 60 sampled problems to verify the coverage of our knowledge QAs.
  • Supplied the QA pairs of these 60 problems as context statements alongside the reaction problems to test the utility of the probed knowledge.
  • Reported accuracy separately by question source (Textbook, Competition, and Research Literature) across all 1,209 problems in Section 5.1 and Appendix D.5.
  • Constructed a stratified subset of 50 problems evaluated independently by two experts from the Chinese Chemistry Olympiad National Training Squad to establish a human baseline.
  • Evaluated 41 intermediate transformation steps from 9 problems with the same human experts using the original prompts to assess task clarity (Q#1).
  • Evaluated models on the same 41 steps using fully specified instructions that explicitly state all bond and stereochemical changes (Q#2).
  • Expanded our evaluation across all 1209 problems to include six additional models (Kimi-K3.1-Pro, DeepSeek-V4.1-Flash, DeepSeek-V4-Pro, GLM-5.3, ChemSpark, and ether0) and the ChemCrow agentic framework across three backbones (Section 4.1, Section 5.1, and Appendix D.5).

Here are our detailed responses to each of your questions and suggestions.↓

AuthorsOpenReview ↗

The author's detailed response [1/4]

W#1: The knowledge questions test only selected chemical principles. Answering them correctly does not show that the model knows everything needed to predict the reaction.

Re: Thank you for raising this insightful point. We apologize if our previous wording caused any misunderstanding. We fully agree with your observation and would like to carefully clarify our core intent in Section 5.3.2. It was never our intention to suggest that answering the knowledge probe QAs correctly proves a model possesses every piece of knowledge required to predict a reaction. Rather, our goal is to demonstrate a significant gap between the knowledge a model possesses (i.e., knowledge it can explicitly recall when queried, evidenced by achieving >80% accuracy on our probes) and its ability to autonomously recall and apply that same knowledge during complex reasoning (i.e., knowledge that, when explicitly supplied in the context, enables the model to solve problems it otherwise fails).

Although the knowledge QAs test selected chemical principles, they are directly constructed from the intermediate reasoning steps required to solve the reaction prediction problems and are thus strongly tied to the core knowledge needed for the task. To rigorously verify whether these probes align with essential reaction mechanisms and whether the probed knowledge is genuinely useful for solving the final prediction task, we conducted two additional experiments on a sampled subset of 60 reaction prediction problems containing 203 intermediate reaction steps:

  1. Relevance and Alignment with Reaction Pathways: We invited human chemistry experts to annotate whether each of the 203 intermediate steps is covered by our generated knowledge QAs. We found that 180 out of 203 steps (88.67%) are directly covered by the 183 knowledge QA pairs associated with these 60 problems. This confirms that our knowledge probes are strongly correlated with the essential mechanistic steps required for reaction prediction.
  2. Utility of the Probed Knowledge: Beyond evaluating whether models can answer these QAs directly, we reformulated the exact content of the QA pairs (questions and ground-truth answers) for each of the 60 problems into natural-language statements and supplied them as context alongside the original reaction problems. As shown in the table below, providing this QA-derived knowledge yields substantial accuracy gains across both frontier and smaller open-source models, demonstrating that the probed knowledge is directly helpful for reaction prediction:

Table 1. Reaction prediction accuracy (%) on 60 problems, without and with auxiliary knowledge statements.

ModelOriginal AccuracyWith QA-Derived Knowledge
GPT-5.6 Sol61.67%74.44% (+12.77%)
Qwen3.8-Flash-Next47.22%62.22% (+15.00%)
Qwen3.8-27B37.22%60.56% (+23.34%)

Taken together, these findings establish two key facts: (1) models can recall the probed chemical principles with high accuracy (with even the weakest model exceeding 80% in Table 2(b)), and (2) when the exact knowledge from these QAs is explicitly present in the reasoning context, both frontier and smaller open models can effectively utilize it to reach the correct product. While we acknowledge that prediction failures can stem from multiple factors, the substantial performance boost gained simply by injecting knowledge the models already know (as verified by the QA probes) demonstrates that the barrier between possessing knowledge and autonomously retrieving and applying it during reasoning is a significant bottleneck. In future work, we hope to explore mechanistic interpretability techniques to move beyond behavioral observations and investigate this gap at the level of internal model representations.

We sincerely appreciate you pointing out this potential misunderstanding. In the revised manuscript, we have clarified this rationale in Section 5.3.2 and incorporated both validation experiments and Table 1 into Section 5.3.2 and Appendix F.2.

AuthorsOpenReview ↗

The author's detailed response [2/4]

W#3: The paper does not report results separately by question source or compare models with chemists answering the same questions. This makes its claims about expert research ability difficult to judge. Q#3: Can you report accuracy separately for research, competition, and textbook questions, and compare it with chemists answering the same questions?

Re: Thank you very much for this constructive comment and question. You raise a very valid point. Reporting accuracy separately for each problem source helps show the limits of current models across different types of problems, and comparing models directly with human chemists on the same problems helps confirm the validity of our benchmark while revealing where models still fall short. To address this thoroughly, we report results grouped by problem source across the full benchmark (with the full breakdown for all evaluated models added to the revised manuscript), followed by a direct comparison with human experts on a representative subset of problems.

First, on the full 1209 problems, frontier proprietary models achieve moderate accuracy on textbook and competition questions, while experiencing a steep drop on problems derived from published research literature, as summarized below: Table 2. Accuracy (%) of proprietary models across the full 1209 problems grouped by question source.

ModelTextbook (349)Competition (402)Research Literature (458)
GPT-5.6 Sol51.86%55.64%7.93%
Gemini 3.1 Pro Preview57.69%58.13%12.01%
Claude Opus 551.67%57.79%14.34%

To establish a direct human baseline, we constructed a 50-problem subset via stratified sampling from the 1209 problems, consisting of 13 textbook, 20 competition, and 17 research literature problems. Two chemistry experts from the Chinese Chemistry Olympiad National Training Squad (top 50 nationwide) completed all 50 problems independently. The comparison across question sources is presented below: Table 3. Accuracy comparison between frontier models and human chemistry experts on the representative subset of 50 problems grouped by question source.

SubjectTextbookCompetitionResearch LiteratureOverall
GPT-5.6 Sol89.74%66.67%1.96%50.67%
Gemini 3.1 Pro Preview79.49%66.67%11.76%51.33%
Claude Opus 574.36%70.00%9.80%50.67%
Human Expert80.77%75.00%29.41%61.00%

Human experts perform comparably to frontier models on textbook and competition questions of moderate difficulty, yet substantially outperform all models on research literature problems. Because literature problems require integrating existing chemical knowledge to reason through unfamiliar reaction environments, this disparity highlights the clear advantage of human chemists in applying knowledge to complex scientific reasoning.

Breaking down the same 50-problem subset by task category reveals another informative pattern: Table 4. Accuracy comparison (%) between frontier models and human chemistry experts on the representative subset of 50 problems grouped by task category.

SubjectBasic PrinciplesReaction PredictionRoute Completion
GPT-5.6 Sol75.76%48.28%30.00%
Gemini 3.1 Pro Preview81.82%49.43%23.33%
Claude Opus 572.73%47.13%36.67%
Human Expert68.18%56.90%65.00%

Notably, human experts score slightly lower than frontier models on Basic Principles, yet decisively outperform every model on Reaction Prediction and achieve roughly double their accuracy on Route Completion. Together, these findings strongly confirm our original conclusions: while frontier LLMs possess extensive foundational chemical knowledge, a substantial gap remains between their knowledge recall and the reliable reasoning required for authentic reactions and multistep synthesis. We have incorporated the source-level breakdown and the human expert study into Section 5.1 of the revised manuscript, with the complete source-level results across all evaluated models and detailed human evaluation protocols provided in Appendix D.5.

Comments on the paper

1 comment
Area_Chair_xtf8OpenReview ↗

Reminder to Reviewers: Please Engage in the Discussion

Dear Reviewers,

As the authors have now submitted their rebuttal, I would kindly ask you to read their responses and engage in the discussion if you have not already done so.

Thank you!

Best regards,
AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.