- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper introduces an evaluation environment and dataset that support the complete process of unknown chemical identification by LLMs. It conducts extensive experiments to evaluate the performance of frontier LLMs, yielding valuable observations and analyses. Although limitations remain in realism and coverage, and the setting is still far removed from real-world chemical research and production, the work helps fill an important gap in the field and represents a worthwhile initial effort.
Critical strengths
- The work establishes a stable, reproducible, and scalable evaluation environment for LLMs, offering useful insights and a valuable reference for evaluating LLM capabilities across a broader range of chemical experiments.
- The paper systematically evaluates LLM performance on unknown chemical identification, a comprehensive task involving chemical knowledge and reasoning, experimental design, and analysis of experimental results, and provides thorough analyses of the findings.
Critical weaknesses
- Using majority voting to determine whether records are retained during human validation may not be sufficiently rigorous. Objections raised by a minority of experts may reflect issues that the majority have overlooked; they should not be dismissed as false alarms merely because fewer experts identified them. This is particularly relevant in a highly specialized domain, where an individual expert’s expertise is unlikely to cover the entire field.
- The paper does not clearly specify the qualifications or levels of expertise of those involved in human validation. Are they undergraduate students, PhD students, researchers, or professors?
- The paper neither explains nor demonstrates how experimental designs outside the anticipated scope are handled. In particular, the chemical substance descriptions provided in the appendix appear to specify only positive reaction outcomes. For example, they describe the formation of a precipitate when sodium chloride solution is mixed with silver nitrate solution, but do not state that no precipitate forms when it is mixed with calcium chloride solution. In such cases, does LabAgent assume that no noticeable change occurs, or does the model predict the outcome using its own knowledge? If the latter, how are potential hallucinations handled?
- Both the experimental operations supported by LabAgent and the chemical substances included in the identification dataset appear relatively basic and simple. If my understanding is incorrect, I would welcome clarification and concrete examples. Given that current models still achieve relatively low identification accuracy and struggle to complete the task, the present setting may be sufficient for now. However, it imposes clear limitations on future extensions. In particular, as molecules and reactions become more complex, the issue raised in Weakness 3 may become even more difficult to address.
- The budget settings do not seem entirely appropriate. First, the experimental results suggest that would be a better setting, as it would better mitigate cases in which success is determined by the interaction cap. Second, given differences in cost and latency across models, the number of interaction rounds does not seem to provide a particularly fair basis for imposing a budget. API-call cost or runtime would appear to be more reasonable and practical alternatives.
Questions for authors
1/ What constitutes the laboratory state ? Does it consist solely of the state of the simulated laboratory, or does it include additional information? If it includes additional information, how is that information updated across time steps? Furthermore, how is the simulated laboratory state reflected in the information available to the agent?
6 comments
Response to Reviewer a28G(2)
Reply to W3: Experimental Outcomes Without Direct Knowledge-Base Records
Thank you for raising this important question!
The paper neither explains nor demonstrates how experimental designs outside the anticipated scope are handled. In particular, the chemical substance descriptions provided in the appendix appear to specify only positive reaction outcomes. For example, they describe the formation of a precipitate when sodium chloride solution is mixed with silver nitrate solution, but do not state that no precipitate forms when it is mixed with calcium chloride solution. In such cases, does LabAgent assume that no noticeable change occurs, or does the model predict the outcome using its own knowledge? If the latter, how are potential hallucinations handled?
We acknowledge that the Knowledge Base does not exhaustively cover all possible reaction combinations, including both positive and negative outcomes. This is partly an inherent limitation of knowledge-base construction, and we do not intend LabAgent to become a tool that primarily relies on matching experimental requests against predefined reaction records. Instead, the Knowledge Base provides foundational chemical information, while LabAgent is designed to make use of external resources and computational tools to handle experimental scenarios beyond its existing records. In the longer term, we aim to further reduce its dependence on a predefined Knowledge Base and explore whether chemical identification can be accomplished primarily through SKILL-based tools. Therefore, the absence of a matching record does not imply that no observable reaction occurs.
To avoid unsupported outcome generation, LabAgent employs three complementary mechanisms:
-
Knowledge-Base Retrieval: When a matching record exists, LabAgent retrieves the corresponding expert-validated experimental outcome.
-
External Resource and Tool Support: Otherwise, LabAgent consults external chemical resources (e.g., PubChem) and computational tools (e.g., RDKit) to obtain relevant chemical information and determine the outcome when sufficient information is available.
-
Evidence-Grounded LLM Inference: When the above sources cannot directly determine the outcome, LabAgent retrieves relevant physicochemical information about the reactants and provides it, together with experimental conditions and the current laboratory state, to the LLM Controller. This grounds the inference in explicit chemical evidence and experimental context, rather than relying solely on the LLM's internal knowledge. If the available evidence is insufficient to support a reliable inference, LabAgent abstains from generating an outcome rather than producing an unsupported prediction.
Furthermore, we conducted a controller sensitivity experiment (Section 4.4), evaluating three Experimenter models with three different LabAgent Controllers: DeepSeek-V4-Pro, StepFun-3.7-Flash, and Qwen3.8-Flash. The relative ranking of the Experimenter models remained unchanged across all three configurations, with accuracy variations within 5.0 percentage points for each model, providing additional evidence that the overall evaluation is not unduly dependent on a particular controller.
We will clarify these design principles, outcome-generation mechanisms, and corresponding experimental results in the revised manuscript.
Response to Reviewer a28G(3)
Reply to W4: Diversity and Difficulty of Chemical Tasks
Thank you for raising this important concern!
Both the experimental operations supported by LabAgent and the chemical substances included in the identification dataset appear relatively basic and simple. If my understanding is incorrect, I would welcome clarification and concrete examples. Given that current models still achieve relatively low identification accuracy and struggle to complete the task, the present setting may be sufficient for now. However, it imposes clear limitations on future extensions. In particular, as molecules and reactions become more complex, the issue raised in Weakness 3 may become even more difficult to address.
We would like to clarify that ChemDeduce covers a range of chemical categories and identification difficulties, rather than only simple chemical substances. Our dataset contains 200 real chemical substances (102 solids and 98 liquids), including halogenated compounds, amines/amides, phenols, esters, alcohols, and hydrocarbons. Appendix Table A2 further shows substantial differences in identification difficulty across categories: the average accuracy across evaluated models is 25.6% for amines/amides, compared with 59.0% for hydrocarbons. We also include chemically similar substances that require discriminative experiments. For example, in Appendix E.1, a model incorrectly identified resorcinol as phenol despite receiving multiple experimental observations.
Regarding experimental operations, LabAgent supports five categories of procedures, comprising 30 chemical actions, 70 pieces of equipment, and 31 experimental scenes. These resources support sequential experimentation and evidence integration across multiple interactions, rather than isolated reaction matching.
We acknowledge that the current benchmark has a defined scope and does not cover the full complexity of advanced chemical systems. Extending ChemDeduce to more complex molecules, reactions, and experimental procedures is an important direction for future work.
Reply to W5: Interaction Budget and Evaluation Metrics
Thank you for raising these important concerns!
The budget settings do not seem entirely appropriate. First, the experimental results suggest that 20 rounds would be a better setting, as it would better mitigate cases in which success is determined by the interaction cap. Second, given differences in cost and latency across models, the number of interaction rounds does not seem to provide a particularly fair basis for imposing a budget. API-call cost or runtime would appear to be more reasonable and practical alternatives.
We would like to clarify that interaction rounds and computational cost measure different aspects of efficiency. Each interaction round represents an experimental opportunity for the model to acquire evidence and make decisions. Applying the same round limit therefore provides a standardized basis for comparing experimental decision-making, although it does not equalize computational costs across models. We use ten rounds as a controlled resource-constrained setting, rather than claiming it is universally optimal.
To examine budget sensitivity, we additionally evaluated Kimi-K3, Grok-4.3, and MiniMax-M3 under 10-, 20-, and 30-round limits. Increasing the budget from 10 to 20 rounds improved accuracy and reduced budget exhaustion, while further increasing it to 30 rounds yielded relatively limited additional gains (Section 4.5).
We also recorded total token consumption, execution latency, and API cost during the main experiments. These metrics provide complementary measures of computational efficiency, allowing readers to assess the practical resource requirements of different models. We therefore retain interaction rounds as the primary experimental budget metric while reporting computational costs separately.
The detailed results are presented in Table 2.
Table 2. Computational efficiency across different Experimenter models.
| Model | Total Tokens ↓ | Latency (s) ↓ | API Cost / Task ($) ↓ |
|---|---|---|---|
| GPT-5.6-Sol | 93.7k | 606 | 0.396 |
| Qwen3.8-Max | 93.6k | 1,241 | 0.159 |
| Gemini-3.7-Flash | 89.2k | 581 | 0.070 |
| GLM-5.3 | 110.9k | 962 | 0.160 |
| GLM-5.2 | 118.3k | 664 | 0.171 |
| MiniMax-M3 | 112.3k | 640 | 0.035 |
| Grok-4.3 | 111.5k | 596 | 0.141 |
| Kimi-K3 | 123.3k | 719 | 0.392 |
Response to Reviewer a28G(1)
We sincerely thank the reviewer for the positive assessment of ChemDeduce and for recognizing its value. We also appreciate the thoughtful questions and constructive suggestions, which provide us with an opportunity to further explain several aspects of our methodology.
Reply to W1 and W2: Expert Validation and Evaluator Qualifications
Thank you for raising these important concerns!
We would like to clarify that dataset validation was conducted by three professors: two specializing in chemistry and one in the interdisciplinary field of biology and AI. All three reviewed the chemical correctness, experimental plausibility, and consistency of the substance records. The evaluation was completed over two days, with $10 provided for each substance record reviewed.
Importantly, record inclusion was not determined solely by majority voting. When disagreements arose, all three experts jointly revisited the relevant chemical information and supporting evidence before reaching a final decision. Among the 86 disputed records, 13 (15.1%) were retained without modification, 48 (55.8%) were revised and retained, and 25 (29.1%) were excluded. Thus, 61 records (70.9%) were ultimately retained based on collective expert assessment and supporting chemical evidence.
We will clarify the evaluators' qualifications and the disagreement-resolution procedure in the revised manuscript.
The detailed results are presented in Table 1.
Table 1. Final handling of records involving expert disagreement (86 records).