- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes OrganicBench, a benchmark for evaluating the organic chemistry capabilities of LLMs. OrganicBench consists of three main tasks (Basic Principles, Reaction Prediction, and Route Completion) and supports three input modalities (Images, SMILES, and Molecular Graphs).
Critical strengths
I share the authors' concern that previous benchmarks have primarily focused on evaluating relatively simple perception-level capabilities, and I agree with the motivation behind this work.
-
The paper is clearly and well-written, and it includes most of the essential elements expected of a benchmark paper. The Related Work section clearly distinguishes this work from prior studies, and the benchmark covers major tasks in organic chemistry.
-
The paper provides comprehensive analyses of the benchmark results from multiple perspectives.
Critical weaknesses
-
The number and diversity of the evaluated open-source and chemistry-specialized models seem somewhat limited. The evaluation mainly focuses on the Qwen3 series, and it would be beneficial to include results from a wider variety of models (e.g., GLM, Kimi, etc.). I would also like to see more diverse chemistry-specialized models included, even if they do not support all input modalities. If feasible, adding recent agentic frameworks (e.g., ChemCrow) would further strengthen the evaluation.
-
It is unclear how the Reaction Prediction task substantially differs from conventional forward reaction prediction tasks. At first glance, it does not appear significantly different from widely used USPTO-based benchmarks. Furthermore, tasks (a) and (c) seem to have definitions similar to those in existing organic chemistry benchmarks (ChemCoTBench, RxnBench, and FukuyamaBench), even though these prior benchmarks may have limitations in their quality and depth of analysis.
Questions for authors
-
Could the authors include evaluation results from a wider range of LLMs with different architectures, sizes, and purposes?
-
Could the authors provide more detailed comparisons between OrganicBench and existing benchmarks, particularly regarding their task definitions?
-
In organic chemistry, when reaction conditions are not specified in sufficient detail, multiple chemically plausible products may exist rather than a single correct answer. Could the benchmark account for such cases where multiple answers may be valid?
Since I share the authors' concerns about the limitations of existing benchmarks, I would be willing to revise my score upward if the issues raised above are adequately addressed.
12 comments
The author's detailed response [1/5]
W#1: The number and diversity of the evaluated open-source and chemistry-specialized models seem somewhat limited. ... Q#1: Could the authors include evaluation results from a wider range of LLMs with different architectures, sizes, and purposes?
Re: Thank you very much for this constructive suggestion. We fully agree that expanding the diversity of model families, chemistry-specialized models, and agentic frameworks broadens the empirical scope of OrganicBench and substantially strengthens our evaluation. Following your recommendation, we evaluated additional general-purpose models across the Kimi, DeepSeek, and GLM families, two additional chemistry-specialized models (ChemSpark and ether0), and a reproduction of the ChemCrow agentic framework across three diverse backbones on all 1209 problems.
First, to ensure fair comparison, every newly added model was evaluated across all input modalities it natively supports (image, SMILES, and molecular graph for Kimi-K3.1-Pro; SMILES and molecular graph for DeepSeek-V4-Pro and DeepSeek-V4.1-Flash; and SMILES for GLM-5.3, ChemSpark, and ether0). Due to rebuttal time constraints, GLM-5.3 has currently completed one full evaluation run, primarily because it frequently produces excessively lengthy reasoning chains that reach the maximum token limit and require extended inference time. We will complete the remaining runs for GLM-5.3 in the revised version. The overall and task-level accuracy are summarized below:
Table 1. Evaluation results (%) of additional general-purpose and chemistry-specialized models on OrganicBench across supported input modalities.
| Model | Input Modality | Basic Principles | Reaction Prediction | Route Completion | Overall Acc. |
|---|---|---|---|---|---|
| Kimi-K3.1-Pro | Image | 71.48% | 30.81% | 26.61% | 37.14% |
| Kimi-K3.1-Pro | SMILES | 70.98% | 32.13% | 31.99% | 38.57% |
| Kimi-K3.1-Pro | Graph | 71.64% | 29.83% | 27.69% | 36.56% |
| DeepSeek-V4.1-Flash | SMILES | 70.98% | 28.81% | 33.87% | 36.34% |
| DeepSeek-V4.1-Flash | Graph | 61.69% | 16.63% | 20.70% | 24.54% |
| DeepSeek-V4-Pro | SMILES | 64.18% | 26.21% | 25.27% | 32.42% |
| DeepSeek-V4-Pro | Graph | 64.51% | 24.96% | 18.28% | 30.85% |
| GLM-5.3 | SMILES | 68.16% | 22.51% | 20.16% | 29.86% |
| ChemSpark | SMILES | 22.72% | 0.26% | 0.00% | 3.97% |
| ether0 | SMILES | 0.66% | 1.24% | 0.00% | 1.02% |
Two key findings emerge from these results. First, general-purpose models across diverse architectures show the same pattern observed in Section 5.1, performing well on Basic Principles while dropping sharply on Reaction Prediction and Route Completion. Second, the newly evaluated chemistry-specialized models still perform poorly. ChemSpark achieves only 3.97% overall accuracy, and ether0 frequently fails to follow output formatting instructions and reaches only 1.02% even after format correction. This confirms that current chemistry-specialized models still lack expert-level chemical reasoning capability.
Second, following the official ChemCrow codebase, we reproduced the ChemCrow pipeline across all 1209 problems under the SMILES modality using three backbone models (Gemini 3.1 Pro Preview, DeepSeek-V4.1-Flash, and Qwen3.8-27B), equipping them with the five standard ChemCrow tools (MolSimilarity, SMILES2Weight, FunctionalGroups, ReactionPredict, and ReactionRetrosynthesis). The comparison against the original SMILES baselines is shown below. Table 2. Performance comparison (%) of three backbone models with and without the ChemCrow agentic framework under the SMILES modality.
| Backbone | Basic Principles | Reaction Prediction | Route Completion | Overall Acc. |
|---|---|---|---|---|
| Gemini 3.1 Pro Preview | 73.80% | 38.70% | 31.18% | 43.77% |
| +ChemCrow | 70.15% (↓3.65%) | 33.94% (↓4.75%) | 26.61% (↓4.57%) | 39.21% (↓4.55%) |
| DeepSeek-V4.1-Flash | 70.98% | 28.81% | 33.87% | 36.34% |
| +ChemCrow | 65.17% (↓5.81%) | 25.00% (↓3.81%) | 30.65% (↓3.22%) | 32.26% (↓4.08%) |
| Qwen3.8-27B | 62.19% | 16.70% | 15.59% | 24.15% |
| +ChemCrow | 44.78% (↓17.41%) | 10.29% (↓6.41%) | 4.03% (↓11.56%) | 15.38% (↓8.77%) |
The author's detailed response [2/5]
Across the 1209 problems, all three backbones actively invoke the provided tools, reaching tool usage rates of 67.99% (1301 total calls) for Gemini 3.1 Pro Preview, 86.60% (9560 calls) for DeepSeek-V4.1-Flash, and 49.63% (2317 calls) for Qwen3.8-27B. Nevertheless, overall accuracy drops across every backbone by 4.08% to 8.77% relative to the SMILES baseline. A primary cause is that the underlying reaction tools (ReactionPredict and ReactionRetrosynthesis) struggle on complex literature reactions and frequently return inaccurate predictions that mislead the reasoning process. This shows that building capable chemistry agents requires stronger chemical reasoning alongside much more reliable domain tools. In the revised manuscript, we have added the new general-purpose and chemistry-specialized models to Section 4.1 and Table 1 (Section 5.1), and included the ChemCrow agent evaluation and analysis in Section 5.1 and Appendix D.5.
Finally, to provide a direct human reference alongside these expanded model results, two experts from the Chinese Chemistry Olympiad National Training Squad independently completed a stratified 50-problem subset (13 textbook, 20 competition, and 17 research literature), as reported in Tables 3-4, as well as 41 intermediate transformation steps from 9 annotated problems in this subset to examine single-step structural execution, as shown in Table 5. Human experts outperform all frontier models on Research Literature and Route Completion, and achieve 98.78% accuracy on the single-step prompts, highlighting models' persistent gap in complex reaction reasoning and description-grounded structural transformation (Section 5.1, Section 5.3.3, Appendix D.5, and Appendix F.4).
Table 3. Accuracy comparison between frontier models and human chemistry experts on the representative subset of 50 problems grouped by question source.
| Subject | Textbook | Competition | Research Literature | Overall |
|---|---|---|---|---|
| GPT-5.6 Sol | 89.74% | 66.67% | 1.96% | 50.67% |
| Gemini 3.1 Pro Preview | 79.49% | 66.67% | 11.76% | 51.33% |
| Claude Opus 5 | 74.36% | 70.00% | 9.80% | 50.67% |
| Human Expert | 80.77% | 75.00% | 29.41% | 61.00% |
Table 4. Accuracy comparison (%) between frontier models and human chemistry experts on the representative subset of 50 problems grouped by task category.
| Subject | Basic Principles | Reaction Prediction | Route Completion |
|---|---|---|---|
| GPT-5.6 Sol | 75.76% | 48.28% | 30.00% |
| Gemini 3.1 Pro Preview | 81.82% | 49.43% | 23.33% |
| Claude Opus 5 | 72.73% | 47.13% | 36.67% |
| Human Expert | 68.18% | 56.90% | 65.00% |
Table 5. Intermediate prediction accuracy on 41 transformation steps from 9 problems.
| Subject | Original Single-Step Prompt | Fully Specified Instruction |
|---|---|---|
| GPT-5.6 Sol | 68.29% | 100.00% |
| Qwen3.8-Flash-Next | 60.98% | 90.24% |
| Qwen3.8-27B | 51.22% | 80.49% |
| Human Expert | 98.78% | \ |
Dear Reviewer V5s6,
Thank you very much for taking the time to review OrganicBench and for your valuable feedback. In your comments, you noted the limited diversity of evaluated models(W#1, Q#1), the need for clearer comparisons of task definitions with existing benchmarks (W#2, Q#2), and the handling of reactions with multiple valid products (Q#3). During the rebuttal period, we conducted additional experiments across all 1209 problems and revised our manuscript to address each point. We hope these additional experiments and clarifications fully address your concerns regarding OrganicBench. Specifically, we have made the following updates:
Kimi-K3.1-Pro,DeepSeek-V4.1-Flash,DeepSeek-V4-Pro, andGLM-5.3) across all supported input modalities (Section 4.1 and Table 1).ChemCoTBench,ChemBench,ChemEval,RxnBench,SUPERChem, andFukuyamaBench(Section 2 and Appendix I).Here are our detailed responses to each of your questions and suggestions.↓