acceptodds
Under review as a conference paper at ICLR 2027

ChemDeduce: Benchmarking LLMs for Unknown Chemical Identification through Interactive Simulated Experiments

Abstract

Large language models (LLMs) have shown strong performance in chemical knowledge and reasoning, yet it remains unclear whether they can solve problems whose answers must be discovered through experimentation. We investigate this question through Unknown Chemical Identification, where a model must actively design experiments, interpret observations, update hypotheses, and ultimately identify a hidden substance. We introduce ChemDeduce, an interactive evaluation framework for assessing LLMs through simulated experiments. ChemDeduce formulates chemical identification as a sequential decision process and provides a stateful simulated laboratory with explicit instrument and operational constraints, enabling scalable and reproducible evaluation of iterative experimental reasoning. We benchmark eight frontier LLMs on 200 chemicals, yielding model–target pairs and episodes over three independent repetitions, and evaluate identification accuracy, operational compliance, interaction efficiency, and trajectory-level reasoning quality. Despite generating seemingly plausible experimental plans, current models achieve an average identification accuracy ranging from 23.83% to 67.33%, revealing substantial limitations in experiment selection, evidence interpretation, and sequential hypothesis refinement.

Primary area:
datasets and benchmarks
Keywords:
LLM evaluation, Interactive benchmarking, Chemical identification, Simulated laboratory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_a28GOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper introduces an evaluation environment and dataset that support the complete process of unknown chemical identification by LLMs. It conducts extensive experiments to evaluate the performance of frontier LLMs, yielding valuable observations and analyses. Although limitations remain in realism and coverage, and the setting is still far removed from real-world chemical research and production, the work helps fill an important gap in the field and represents a worthwhile initial effort.

Critical strengths
  1. The work establishes a stable, reproducible, and scalable evaluation environment for LLMs, offering useful insights and a valuable reference for evaluating LLM capabilities across a broader range of chemical experiments.
  2. The paper systematically evaluates LLM performance on unknown chemical identification, a comprehensive task involving chemical knowledge and reasoning, experimental design, and analysis of experimental results, and provides thorough analyses of the findings.
Critical weaknesses
  1. Using majority voting to determine whether records are retained during human validation may not be sufficiently rigorous. Objections raised by a minority of experts may reflect issues that the majority have overlooked; they should not be dismissed as false alarms merely because fewer experts identified them. This is particularly relevant in a highly specialized domain, where an individual expert’s expertise is unlikely to cover the entire field.
  2. The paper does not clearly specify the qualifications or levels of expertise of those involved in human validation. Are they undergraduate students, PhD students, researchers, or professors?
  3. The paper neither explains nor demonstrates how experimental designs outside the anticipated scope are handled. In particular, the chemical substance descriptions provided in the appendix appear to specify only positive reaction outcomes. For example, they describe the formation of a precipitate when sodium chloride solution is mixed with silver nitrate solution, but do not state that no precipitate forms when it is mixed with calcium chloride solution. In such cases, does LabAgent assume that no noticeable change occurs, or does the model predict the outcome using its own knowledge? If the latter, how are potential hallucinations handled?
  4. Both the experimental operations supported by LabAgent and the chemical substances included in the identification dataset appear relatively basic and simple. If my understanding is incorrect, I would welcome clarification and concrete examples. Given that current models still achieve relatively low identification accuracy and struggle to complete the task, the present setting may be sufficient for now. However, it imposes clear limitations on future extensions. In particular, as molecules and reactions become more complex, the issue raised in Weakness 3 may become even more difficult to address.
  5. The budget settings do not seem entirely appropriate. First, the experimental results suggest that T=20T=20 would be a better setting, as it would better mitigate cases in which success is determined by the interaction cap. Second, given differences in cost and latency across models, the number of interaction rounds does not seem to provide a particularly fair basis for imposing a budget. API-call cost or runtime would appear to be more reasonable and practical alternatives.
Questions for authors

1/ What constitutes the laboratory state ss? Does it consist solely of the state of the simulated laboratory, or does it include additional information? If it includes additional information, how is that information updated across time steps? Furthermore, how is the simulated laboratory state reflected in the information available to the agent?

6 comments
AuthorsOpenReview ↗

Response to Reviewer a28G(1)

We sincerely thank the reviewer for the positive assessment of ChemDeduce and for recognizing its value. We also appreciate the thoughtful questions and constructive suggestions, which provide us with an opportunity to further explain several aspects of our methodology.

Reply to W1 and W2: Expert Validation and Evaluator Qualifications

Thank you for raising these important concerns!

Using majority voting to determine whether records are retained during human validation may not be sufficiently rigorous. Objections raised by a minority of experts may reflect issues that the majority have overlooked; they should not be dismissed as false alarms merely because fewer experts identified them. This is particularly relevant in a highly specialized domain, where an individual expert’s expertise is unlikely to cover the entire field.

The paper does not clearly specify the qualifications or levels of expertise of those involved in human validation. Are they undergraduate students, PhD students, researchers, or professors?

We would like to clarify that dataset validation was conducted by three professors: two specializing in chemistry and one in the interdisciplinary field of biology and AI. All three reviewed the chemical correctness, experimental plausibility, and consistency of the substance records. The evaluation was completed over two days, with $10 provided for each substance record reviewed.

Importantly, record inclusion was not determined solely by majority voting. When disagreements arose, all three experts jointly revisited the relevant chemical information and supporting evidence before reaching a final decision. Among the 86 disputed records, 13 (15.1%) were retained without modification, 48 (55.8%) were revised and retained, and 25 (29.1%) were excluded. Thus, 61 records (70.9%) were ultimately retained based on collective expert assessment and supporting chemical evidence.

We will clarify the evaluators' qualifications and the disagreement-resolution procedure in the revised manuscript.

The detailed results are presented in Table 1.

Table 1. Final handling of records involving expert disagreement (86 records).

Final handlingNumberPercentageFinal decision
Retained without modification1315.1%Included
Revised and subsequently retained4855.8%Included after revision
Excluded2529.1%Excluded
Total86100%—
Total retained6170.9%Included
AuthorsOpenReview ↗

Response to Reviewer a28G(2)

Reply to W3: Experimental Outcomes Without Direct Knowledge-Base Records

Thank you for raising this important question!

The paper neither explains nor demonstrates how experimental designs outside the anticipated scope are handled. In particular, the chemical substance descriptions provided in the appendix appear to specify only positive reaction outcomes. For example, they describe the formation of a precipitate when sodium chloride solution is mixed with silver nitrate solution, but do not state that no precipitate forms when it is mixed with calcium chloride solution. In such cases, does LabAgent assume that no noticeable change occurs, or does the model predict the outcome using its own knowledge? If the latter, how are potential hallucinations handled?

We acknowledge that the Knowledge Base does not exhaustively cover all possible reaction combinations, including both positive and negative outcomes. This is partly an inherent limitation of knowledge-base construction, and we do not intend LabAgent to become a tool that primarily relies on matching experimental requests against predefined reaction records. Instead, the Knowledge Base provides foundational chemical information, while LabAgent is designed to make use of external resources and computational tools to handle experimental scenarios beyond its existing records. In the longer term, we aim to further reduce its dependence on a predefined Knowledge Base and explore whether chemical identification can be accomplished primarily through SKILL-based tools. Therefore, the absence of a matching record does not imply that no observable reaction occurs.

To avoid unsupported outcome generation, LabAgent employs three complementary mechanisms:

  1. Knowledge-Base Retrieval: When a matching record exists, LabAgent retrieves the corresponding expert-validated experimental outcome.

  2. External Resource and Tool Support: Otherwise, LabAgent consults external chemical resources (e.g., PubChem) and computational tools (e.g., RDKit) to obtain relevant chemical information and determine the outcome when sufficient information is available.

  3. Evidence-Grounded LLM Inference: When the above sources cannot directly determine the outcome, LabAgent retrieves relevant physicochemical information about the reactants and provides it, together with experimental conditions and the current laboratory state, to the LLM Controller. This grounds the inference in explicit chemical evidence and experimental context, rather than relying solely on the LLM's internal knowledge. If the available evidence is insufficient to support a reliable inference, LabAgent abstains from generating an outcome rather than producing an unsupported prediction.

Furthermore, we conducted a controller sensitivity experiment (Section 4.4), evaluating three Experimenter models with three different LabAgent Controllers: DeepSeek-V4-Pro, StepFun-3.7-Flash, and Qwen3.8-Flash. The relative ranking of the Experimenter models remained unchanged across all three configurations, with accuracy variations within 5.0 percentage points for each model, providing additional evidence that the overall evaluation is not unduly dependent on a particular controller.

We will clarify these design principles, outcome-generation mechanisms, and corresponding experimental results in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer a28G(3)

Reply to W4: Diversity and Difficulty of Chemical Tasks

Thank you for raising this important concern!

Both the experimental operations supported by LabAgent and the chemical substances included in the identification dataset appear relatively basic and simple. If my understanding is incorrect, I would welcome clarification and concrete examples. Given that current models still achieve relatively low identification accuracy and struggle to complete the task, the present setting may be sufficient for now. However, it imposes clear limitations on future extensions. In particular, as molecules and reactions become more complex, the issue raised in Weakness 3 may become even more difficult to address.

We would like to clarify that ChemDeduce covers a range of chemical categories and identification difficulties, rather than only simple chemical substances. Our dataset contains 200 real chemical substances (102 solids and 98 liquids), including halogenated compounds, amines/amides, phenols, esters, alcohols, and hydrocarbons. Appendix Table A2 further shows substantial differences in identification difficulty across categories: the average accuracy across evaluated models is 25.6% for amines/amides, compared with 59.0% for hydrocarbons. We also include chemically similar substances that require discriminative experiments. For example, in Appendix E.1, a model incorrectly identified resorcinol as phenol despite receiving multiple experimental observations.

Regarding experimental operations, LabAgent supports five categories of procedures, comprising 30 chemical actions, 70 pieces of equipment, and 31 experimental scenes. These resources support sequential experimentation and evidence integration across multiple interactions, rather than isolated reaction matching.

We acknowledge that the current benchmark has a defined scope and does not cover the full complexity of advanced chemical systems. Extending ChemDeduce to more complex molecules, reactions, and experimental procedures is an important direction for future work.


Reply to W5: Interaction Budget and Evaluation Metrics

Thank you for raising these important concerns!

The budget settings do not seem entirely appropriate. First, the experimental results suggest that 20 rounds would be a better setting, as it would better mitigate cases in which success is determined by the interaction cap. Second, given differences in cost and latency across models, the number of interaction rounds does not seem to provide a particularly fair basis for imposing a budget. API-call cost or runtime would appear to be more reasonable and practical alternatives.

We would like to clarify that interaction rounds and computational cost measure different aspects of efficiency. Each interaction round represents an experimental opportunity for the model to acquire evidence and make decisions. Applying the same round limit therefore provides a standardized basis for comparing experimental decision-making, although it does not equalize computational costs across models. We use ten rounds as a controlled resource-constrained setting, rather than claiming it is universally optimal.

To examine budget sensitivity, we additionally evaluated Kimi-K3, Grok-4.3, and MiniMax-M3 under 10-, 20-, and 30-round limits. Increasing the budget from 10 to 20 rounds improved accuracy and reduced budget exhaustion, while further increasing it to 30 rounds yielded relatively limited additional gains (Section 4.5).

We also recorded total token consumption, execution latency, and API cost during the main experiments. These metrics provide complementary measures of computational efficiency, allowing readers to assess the practical resource requirements of different models. We therefore retain interaction rounds as the primary experimental budget metric while reporting computational costs separately.

The detailed results are presented in Table 2.

Table 2. Computational efficiency across different Experimenter models.

ModelTotal Tokens ↓Latency (s) ↓API Cost / Task ($) ↓
GPT-5.6-Sol93.7k6060.396
Qwen3.8-Max93.6k1,2410.159
Gemini-3.7-Flash89.2k5810.070
GLM-5.3110.9k9620.160
GLM-5.2118.3k6640.171
MiniMax-M3112.3k6400.035
Grok-4.3111.5k5960.141
Kimi-K3123.3k7190.392
Reviewer_16gQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ChemDeduce evaluates unknown chemical identification as an interactive process: an LLM selects experiments, receives state-dependent simulated observations, and identifies a hidden substance. The dataset contains 200 chemicals, including 102 solids and 98 liquids. LabAgent combines a controller, chemical records, skills, operational validation and a stateful virtual laboratory built on LabUtopia. Eight models are evaluated over three repetitions, totaling 4,800 episodes. The paper reports identification accuracy, operational compliance, successful observation acquisition, rounds, budget exhaustion and a five-component trajectory deduction score (TDS). Accuracy ranges from 23.83% to 67.33%. Additional experiments examine controller choice and interaction budget, and an expert review checks 40 simulated trajectories.

Critical strengths
  1. A meaningful task formulation with supervision tailored to the data.

SPE component selection is a practically relevant analytical chemistry problem. Rather than assigning a single answer to each pollutant, the authors retain multiple reported components and mask missing ratio annotations. This accommodates the multiple alternatives and partial annotations present in literature-derived data and constitutes the paper’s most substantive contribution.

  1. Informative and transparent analysis of decoding behavior and limitations.

The paper examines the trade-offs between predicted set size, precision, recall, and exact match. The fixed-ranking cardinality diagnostic identifies output size as an important bottleneck. The authors also acknowledge the loss of joint assignments, the effect of missing ratios on strict evaluation, and the stronger baseline performance on some metrics. These analyses help establish an appropriately bounded interpretation of the method.

Critical weaknesses
  1. The current comparisons do not adequately establish the advantage of the proposed method itself.

Baselines use a fixed threshold of 0.5, while SPEGen receives per-label threshold calibration and architecture search; the best-performing variant additionally averages eight models. The paper does not provide comparisons with matched calibration, search budgets, or outer-loop ensemble sizes. It also lacks a simple multi-task control matched in preprocessing, capacity, and decoding.

This directly affects the main conclusion: the observed improvements may partly reflect more extensive optimization and ensembling rather than distinctive framework components. The solvent F1 and ratio MAE gains are relatively small, and baselines remain strongest on the two stricter component metrics.

  1. The task is meaningful, but methodological novelty and evidence of practical recommendation validity remain limited.

The framework primarily combines established multi-task networks, imbalance-aware losses, rank-based transformations, threshold calibration, and bagging. Its value therefore depends substantially on whether the task and data contributions support reliable and generalizable recommendations.

Pollutant-level aggregation discards the original joint relationships between cartridges, solvents, and ratios. Triplet only checks separate membership in aggregated label sets, rather than whether the components occur together in a source method or are compatible. The random split does not rule out scaffold or publication overlap, and no laboratory extraction validation is reported. Although these limitations are acknowledged, the evidence does not yet establish the system’s practical value beyond retrospective recovery of literature labels.

Questions for authors
  1. Could the strongest baselines be evaluated with the same validation-based per-label threshold calibration and step gating? Do the main gains persist when search budgets and outer-loop ensemble sizes are reasonably matched?

  2. Is there a simple multi-task MLP control matched in Rank-Gauss preprocessing, model capacity, and decoding, but without step guidance and the auxiliary training terms? This would help determine whether the task-specific components provide independent benefits.

  3. Using the original method records, what fraction of predicted cartridge–step–solvent triples actually co-occur within a single source method? Would this method-level evaluation change the model ranking relative to the current marginal-membership evaluation?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 16gQ(1)

We sincerely thank the reviewer for the positive assessment of ChemDeduce and for the accurate summary of our work. However, we may be misunderstanding some of the detailed comments, as several methodological terms and assumptions appear to describe a different supervised prediction framework rather than the task and methodology presented in ChemDeduce. We clarify these points below.

Reply to Critical Weakness 1 and Question 1: Baseline Comparisons and Evaluation Fairness

Thank you for raising these concerns!

The current comparisons do not adequately establish the advantage of the proposed method itself.

Baselines use a fixed threshold of 0.5, while SPEGen receives per-label threshold calibration and architecture search; the best-performing variant additionally averages eight models. The paper does not provide comparisons with matched calibration, search budgets, or outer-loop ensemble sizes. It also lacks a simple multi-task control matched in preprocessing, capacity, and decoding.

This directly affects the main conclusion: the observed improvements may partly reflect more extensive optimization and ensembling rather than distinctive framework components. The solvent F1 and ratio MAE gains are relatively small, and baselines remain strongest on the two stricter component metrics.

Could the strongest baselines be evaluated with the same validation-based per-label threshold calibration and step gating? Do the main gains persist when search budgets and outer-loop ensemble sizes are reasonably matched?

We would like to clarify that ChemDeduce evaluates eight frontier LLMs independently rather than training or combining supervised prediction models. As described in Section 4.1, all models are evaluated on the same 200 chemicals with three independent repetitions (4,800 episodes in total), following a unified protocol with standardized task instructions, laboratory resources, action formats, interaction budgets, and inference configurations (temperature 0.3 and maximum output length of 8,192 tokens, subject to API availability). No model-specific prompt or decoding optimization is performed. LabAgent's operational compliance check only validates experimental requests against laboratory constraints; it is not a prediction-stage step-gating mechanism.

ChemDeduce is an interactive benchmark for sequential experimental reasoning, not a supervised prediction framework. Therefore, per-label threshold calibration, architecture search, outer-loop ensembling, and a matched multi-task MLP are not directly applicable to our methodology. Similarly, solvent F1 and ratio MAE are not evaluation metrics used in ChemDeduce. We believe these concerns may stem from a different experimental framework.

AuthorsOpenReview ↗

Response to Reviewer 16gQ(2)

Reply to Critical Weakness 2 and Question 2: Methodological Novelty, Data Structure, and Practical Validation

Thank you for raising these concerns!

The task is meaningful, but methodological novelty and evidence of practical recommendation validity remain limited.

The framework primarily combines established multi-task networks, imbalance-aware losses, rank-based transformations, threshold calibration, and bagging. Its value therefore depends substantially on whether the task and data contributions support reliable and generalizable recommendations.

Pollutant-level aggregation discards the original joint relationships between cartridges, solvents, and ratios. Triplet only checks separate membership in aggregated label sets, rather than whether the components occur together in a source method or are compatible. The random split does not rule out scaffold or publication overlap, and no laboratory extraction validation is reported. Although these limitations are acknowledged, the evidence does not yet establish the system’s practical value beyond retrospective recovery of literature labels.

Is there a simple multi-task MLP control matched in Rank-Gauss preprocessing, model capacity, and decoding, but without step guidance and the auxiliary training terms? This would help determine whether the task-specific components provide independent benefits?

We would like to clarify that ChemDeduce's core contribution is an interactive benchmark and stateful simulated laboratory for evaluating LLMs' ability to actively acquire experimental evidence and iteratively update hypotheses, rather than a new supervised prediction architecture. Its dataset consists of 200 individual chemical substances and their associated physicochemical properties, qualitative reactions, and experimental resources. The pollutant–cartridge–solvent–ratio structures, Rank-Gauss transformations, auxiliary training losses, and bagging described in the review are not part of our framework. Likewise, our evaluation covers all 200 target chemicals with three independent repetitions, rather than using a conventional random train/test split of a literature-derived prediction dataset (Sections 3.1 and 4.1). For the same reason, the proposed multi-task MLP is not a directly comparable baseline without reformulating the task.

Regarding practical validation, our expert evaluation specifically examines the reliability of the simulated laboratory, not real-world wet-lab performance. Two chemistry professors reviewed complete experimental trajectories for 40 target substances, assessing request–video consistency, simulation outcome correctness, and macroscopic observation accuracy (Appendix G). We acknowledge that this does not constitute real wet-lab validation and do not claim otherwise.


Reply to Question 3: Triplet Co-occurrence in Source Methods

Thank you for this question!

Using the original method records, what fraction of predicted cartridge–step–solvent triples actually co-occur within a single source method? Would this method-level evaluation change the model ranking relative to the current marginal-membership evaluation?

This evaluation is not applicable to ChemDeduce because our dataset does not contain cartridge–step–solvent entities or corresponding source-method records. The basic evaluation unit is a model–target episode, consisting of sequential experimental requests, state-dependent observations, and a final chemical identification (Section 3.2). Consequently, there are no corresponding source-method triples or marginal-membership rankings to evaluate.

AuthorsOpenReview ↗

Reminder to Reviewer 16gQ

Dear Reviewer 16gQ,

Thank you once again for your valuable comments on our submission. As the discussion phase is approaching its end, we would like to kindly confirm whether we have sufficiently addressed all of your concerns (or at least part of them). Should there be any remaining questions or areas requiring further clarification, please do not hesitate to let us know. If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.

We sincerely look forward to your further feedback.

Reviewer_GQH4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ChemDeduce identifies 200 hidden chemicals through sequential, text-mediated experiments. The contribution is an interactive evaluation environment for gathering experimental evidence.

Critical strengths
  1. The benchmark hides the chemical’s identity and makes models choose experiments. Their next choices depend on earlier observations, so the task tests how they gather evidence and narrow down possible answers.

  2. The simulator uses chemical tools and records checked by experts. Two chemistry professors reviewed complete experiment sequences for 40 chemicals.

Critical weaknesses
  1. A model can choose useful experiments but misinterpret the results and name the wrong chemical. Because wrong or missing identification limits two reasoning scores, those scores do not clearly distinguish poor experiment choices from mistaken conclusions.

  2. The scoring instructions both forbid and allow combining independent tests in one round. The paper does not compare automated scores with human scores. Expert checks cover the simulator instead.

  3. Models are not told the ten-round limit. Three tested models improve accuracy by about ten percentage points with twenty rounds. The error analysis excludes runs without final answers, leaving many failures outside its categories.

Questions for authors
  1. Did the scoring actually allow several independent tests in one round?

  2. Do the same reasoning weaknesses appear when humans score the experiments without automatically lowering scores for wrong or missing final answers?

  3. How well do models and chemists identify samples when both know the round limit and can give a final answer when it is reached? Please include runs without final answers.

6 comments
AuthorsOpenReview ↗

Response to Reviewer GQH4(1)

We sincerely thank the reviewer for the positive assessment of ChemDeduce and for the constructive comments on trajectory evaluation, experimental constraints, and human comparison. We address each concern below.

Reply to CW1: Distinguishing Experimental Process Quality from Final Identification

Thank you for raising this important concern!

A model can choose useful experiments but misinterpret the results and name the wrong chemical. Because wrong or missing identification limits two reasoning scores, those scores do not clearly distinguish poor experiment choices from mistaken conclusions.

We would like to clarify that DED and OIF independently evaluate experimental design and observation interpretation, respectively. DED assesses the diagnostic value of selected experiments, while OIF evaluates whether the model correctly interprets experimental observations. Importantly, CHM, DED, and OIF are scored independently of the final identification outcome, regardless of whether the answer is correct or provided.

To further examine this distinction, we analyzed all 4,800 evaluation episodes, including those without final declarations, and further categorized the 988 incorrectly identified episodes according to the five failure types in Appendix F.3. Two observations are noteworthy:

  1. Observation interpretation: Misread episodes (N = 378) received the lowest mean OIF score (7.33/15), compared with 8.33 for Premature and Off-track, and 9.44 for Near-neighbor. This is consistent with OIF capturing errors in interpreting experimental observations.

  2. Experimental design: DED scores vary across failure categories, providing dimension-level information about experimental design quality. These categories may involve multiple reasoning aspects and are not expected to correspond exclusively to individual TDS dimensions.

The detailed results are presented in Tables 1 and 2.

Table 1. TDS scores across different episode outcomes (4,800 episodes).

Episode outcomeNCHM (/15) ↑DED (/20) ↑OIF (/15) ↑ESU (/30) ↑EGDE (/20) ↑TDS (/100) ↑
Correct2,14411.214.911.523.716.578.0
Incorrect9888.210.38.18.74.740.1
No final declaration1,6688.810.19.42.20.030.5

Table 2. Dimension-level scores across five failure categories of incorrect identifications (988 episodes).

Failure typeNMean DED (/20) ↑Mean OIF (/15) ↑Mean CHM (/15) ↑
Misread37810.027.337.38
Premature4329.928.338.27
Near-neighbor6812.189.4410.06
Off-track9410.778.339.41
Other1615.5011.6212.56
All wrong98810.288.088.23
AuthorsOpenReview ↗

Response to Reviewer GQH4(2)

Reply to Q2: Human Evaluation and Alignment with LLM-Judge Scores

Thank you for raising this important question!

Do the same reasoning weaknesses appear when humans score the experiments without automatically lowering scores for wrong or missing final answers?

To investigate this, we conducted an independent human evaluation involving 120 experimental trajectories, covering 40 chemicals and three LLM Experimenter models. Three evaluators participated: two PhD students specializing in chemistry and one professor working at the intersection of biology and AI. Importantly, CHM, DED, and OIF were evaluated directly based on the recorded experimental processes, independently of final identification outcomes and without applying outcome-dependent score reductions.

The human evaluation scores closely align with the original LLM-Judge scores across all three dimensions and all three models. The mean absolute difference across the nine model–dimension comparisons is only 0.60 points, with the largest difference being 1.44 points. This agreement provides complementary evidence that TDS captures meaningful aspects of experimental trajectory quality and that its process-level assessments are consistent with independent human judgments, rather than being merely artifacts of outcome-dependent scoring constraints.

We will incorporate this human evaluation and the corresponding comparison with LLM-Judge into the main text of the revised manuscript to provide further validation of TDS.

The detailed comparison is presented in Table 3.

Table 3. Comparison of human evaluation and LLM-Judge scores across three process dimensions.

Experimenter modelEvaluationCHM (/15) ↑DED (/20) ↑OIF (/15) ↑
GPT-5.6-SolHuman10.0513.0010.80
GPT-5.6-SolLLM-Judge9.1313.3311.26
Gemini-3.7-FlashHuman9.5312.2011.32
Gemini-3.7-FlashLLM-Judge9.3712.209.88
Qwen3.8-MaxHuman10.1212.2011.55
Qwen3.8-MaxLLM-Judge10.5213.3511.04
AuthorsOpenReview ↗

Response to Reviewer GQH4(3)

Reply to CW2 and Q1: Clarifying the Single-Round Experimental Constraint

Thank you for raising this important concern!

The scoring instructions both forbid and allow combining independent tests in one round. The paper does not compare automated scores with human scores. Expert checks cover the simulator instead.

Did the scoring actually allow several independent tests in one round?

We would like to clarify that a single round can include multiple basic operations or routine measurements within an experimental procedure, such as measuring both pH and conductivity of the same sample. The restriction applies to conditional experiments that depend on observations not yet obtained. For example, after mixing two solutions, the Experimenter must first observe whether a reaction or precipitation occurs before determining the appropriate follow-up experiment. It cannot execute different follow-up procedures based on hypothetical outcomes within the same round.

Thus, the restriction is on premature conditional experimentation, rather than the number of basic operations performed in one round. This ensures that subsequent experiments are determined by actual observations rather than hypothetical assumptions.


Reply to CW3 and Q3: Budget Awareness, Human Baselines, and Episodes without Final Answers

Thank you for raising these important concerns!

Models are not told the ten-round limit. Three tested models improve accuracy by about ten percentage points with twenty rounds. The error analysis excludes runs without final answers, leaving many failures outside its categories.

How well do models and chemists identify samples when both know the round limit and can give a final answer when it is reached? Please include runs without final answers.

We would like to clarify that not explicitly informing the Experimenter of the ten-round limit is an intentional design choice, allowing us to evaluate autonomous experimental planning, evidence acquisition, and stopping decisions. The twenty-round experiment examines the effect of additional interaction opportunities, a different aspect of experimental reasoning.

To further examine budget awareness, we evaluated all eight models on the same 200 chemicals under an explicit ten-round limit, requiring a final identification (1,600 evaluations in total). All eight models improved in accuracy, with the largest gains observed for GLM-5.2 (+17.0 pp), GLM-5.3 (+14.0 pp), and Kimi-K3 (+13.0 pp). Execution-related metrics, however, showed varying changes across models, while mean interaction rounds remained relatively stable.

We also conducted a matched human–LLM comparison using the same three evaluators from our human TDS study (PhD 1, PhD 2, and Professor 1). Each evaluated the same 40 chemicals using LabAgent under the same ten-round limit. The human average accuracy was 78.3% (94/120), compared with 70.0% for the best-performing LLM, GPT-5.6-Sol.

Separately, all 4,800 episodes in the original evaluation were included: 2,144 correct identifications, 988 incorrect identifications, and 1,668 episodes (34.8%) without a valid final declaration. The latter are separately accounted for by BER and reported as an independent outcome category in Table 1. The five failure categories in Appendix F.3 apply only to the 988 incorrect identifications.

The detailed results are presented in Tables 4 and 5.

Table 4. Model performance comparison between the original and budget-aware settings.

ModelOriginal Acc. ↑Original BER ↓Budget-aware Acc. ↑Δ Acc. (pp) ↑Δ Mean Rounds ↓Δ Op. Comp. ↑Δ PSR ↑
GPT-5.6-Sol64.0%20.0%68.5%+4.5+0.49−0.06−0.92
Qwen3.8-Max62.5%18.0%65.0%+2.5−0.21−0.42−1.59
Gemini-3.7-Flash55.5%12.0%58.0%+2.5+0.07−2.29−2.25
GLM-5.347.0%40.0%61.0%+14.0−0.12+0.15−0.69
GLM-5.239.0%50.0%56.0%+17.0−0.32−0.28−1.30
MiniMax-M334.5%43.0%37.5%+3.0−0.44+8.58+6.21
Grok-4.327.0%39.5%33.5%+6.5−0.29−0.52+0.98
Kimi-K325.0%63.5%38.0%+13.0−0.21+3.80+2.34

Table 5. Human–LLM comparison under matched experimental conditions (40 chemicals, ten-round limit).

ExperimenterCorrect identificationsAccuracy ↑Δ vs. Human Average (pp)
PhD 131/4077.5%−0.8
PhD 231/4077.5%−0.8
Professor 132/4080.0%+1.7
Human average94/12078.3%0.0
GPT-5.6-Sol28/4070.0%−8.3
Qwen3.8-Max26/4065.0%−13.3
Gemini-3.7-Flash24/4060.0%−18.3
GLM-5.327/4067.5%−10.8
GLM-5.221/4052.5%−25.8
MiniMax-M318/4045.0%−33.3
Grok-4.319/4047.5%−30.8
Kimi-K317/4042.5%−35.8

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response Summary to Area Chair

We sincerely appreciate the constructive feedback from all three reviewers. During the rebuttal phase, we provided additional explanations, quantitative analyses, and supplementary experiments to further clarify our methodology and validate our experimental design. We summarize the main points below.

1. TDS Validity and Human Evaluation

Reviewer GQH4 questioned whether TDS can distinguish experimental process quality from final identification correctness. We clarified that CHM, DED, and OIF independently assess experimental processes regardless of final outcomes. To further validate this design, we analyzed all 4,800 episodes and categorized the 988 incorrect identifications into five failure types. Misread episodes received the lowest mean OIF score (7.33/15), supporting its ability to capture observation interpretation errors (Tables 1 and 2). Additionally, we conducted human evaluations of 120 trajectories by two chemistry PhD students and one biology–AI professor. Human scores closely aligned with LLM-Judge scores, with a mean absolute difference of only 0.60 points across nine comparisons (Table 3), providing independent evidence of TDS reliability.

2. Experimental Constraints and Budget Evaluation

Reviewer GQH4 questioned the single-round constraint, ten-round budget, and the absence of human comparisons. We clarified that multiple routine operations are permitted within one round, while conditional experiments must depend on actual observations, preserving sequential experimental reasoning. The original ten-round setting intentionally evaluates autonomous planning and stopping behavior. To further examine budget awareness, we evaluated all eight models on the same 200 chemicals with an explicit ten-round limit (1,600 evaluations). All models improved in accuracy, with gains of 17.0, 14.0, and 13.0 percentage points for GLM-5.2, GLM-5.3, and Kimi-K3, respectively (Table 4). We also conducted a matched human–LLM comparison using the same three evaluators and 40 chemicals: humans achieved 78.3% average accuracy, compared with 70.0% for the best LLM (Table 5). All 4,800 original episodes, including 1,668 without final declarations, remain included and are accounted for through BER.

3. Task Formulation and Methodological Contributions

Reviewer 16gQ raised concerns about baseline fairness, methodological novelty, data structure, and practical validation. We clarified that ChemDeduce is an interactive benchmark for sequential experimental reasoning, not a supervised chemical prediction framework. Its core mechanism involves active experiment selection, state-dependent observations, iterative hypothesis updates, and final identification. Therefore, per-label threshold calibration, architecture search, bagging, and pollutant–cartridge–solvent structures mentioned in the review are not components of our framework and are not directly applicable. We also clarified that our expert evaluation assesses the chemical plausibility and reliability of the simulated laboratory, rather than claiming real-world wet-lab validation.

4. Expert Validation, LabAgent Reliability, and Computational Efficiency

Reviewer a28G questioned expert qualifications, Knowledge-Base coverage, task complexity, and computational cost. We clarified that dataset validation involved three professors (two in chemistry and one in biology–AI), completed over two days, with $10 per reviewed substance record. Among 86 disputed records, 61 were ultimately retained after expert discussion and evidence-based revision, while 25 were excluded (Table 7). LabAgent combines curated knowledge retrieval, external resources (PubChem), computational tools (RDKit), and evidence-grounded LLM inference, abstaining when evidence is insufficient. Controller sensitivity experiments further showed unchanged model rankings across three controllers, with accuracy variations within 5.0 percentage points. We also highlighted the benchmark's 200 chemicals, five procedure categories, 30 chemical actions, 70 equipment types, and 31 experimental scenes, with varying identification difficulty across chemical categories. Finally, we reported token consumption, latency, and API costs for all eight models to provide a complementary assessment of computational efficiency (Table 6).

Table 1. TDS scores across different episode outcomes (4,800 episodes).

Episode outcomeNCHM (/15) ↑DED (/20) ↑OIF (/15) ↑ESU (/30) ↑EGDE (/20) ↑TDS (/100) ↑
Correct2,14411.214.911.523.716.578.0
Incorrect9888.210.38.18.74.740.1
No final declaration1,6688.810.19.42.20.030.5

Table 2. Dimension-level scores across five failure categories of incorrect identifications (988 episodes).

Failure typeNMean DED (/20) ↑Mean OIF (/15) ↑Mean CHM (/15) ↑
Misread37810.027.337.38
Premature4329.928.338.27
Near-neighbor6812.189.4410.06
Off-track9410.778.339.41
Other1615.5011.6212.56
All wrong98810.288.088.23

Table 3. Comparison of human evaluation and LLM-Judge scores across three process dimensions (120 trajectories).

Experimenter modelEvaluationCHM (/15) ↑DED (/20) ↑OIF (/15) ↑
GPT-5.6-SolHuman10.0513.0010.80
LLM-Judge9.1313.3311.26
Gemini-3.7-FlashHuman9.5312.2011.32
LLM-Judge9.3712.209.88
Qwen3.8-MaxHuman10.1212.2011.55
LLM-Judge10.5213.3511.04

Table 4. Model performance comparison between original and budget-aware settings.

ModelOriginal Acc. ↑Original BER ↓Budget-aware Acc. ↑Δ Acc. (pp) ↑Δ Mean RoundsΔ Op. Comp.Δ PSR
GPT-5.6-Sol64.0%20.0%68.5%+4.5+0.49−0.06−0.92
Qwen3.8-Max62.5%18.0%65.0%+2.5−0.21−0.42−1.59
Gemini-3.7-Flash55.5%12.0%58.0%+2.5+0.07−2.29−2.25
GLM-5.347.0%40.0%61.0%+14.0−0.12+0.15−0.69
GLM-5.239.0%50.0%56.0%+17.0−0.32−0.28−1.30
MiniMax-M334.5%43.0%37.5%+3.0−0.44+8.58+6.21
Grok-4.327.0%39.5%33.5%+6.5−0.29−0.52+0.98
Kimi-K325.0%63.5%38.0%+13.0−0.21+3.80+2.34

Table 5. Human–LLM comparison under matched experimental conditions (40 chemicals, ten-round limit).

ExperimenterCorrect identificationsAccuracy ↑Δ vs. Human Average (pp)
PhD 131/4077.5%−0.8
PhD 231/4077.5%−0.8
Professor 132/4080.0%+1.7
Human average94/12078.3%0.0
GPT-5.6-Sol28/4070.0%−8.3
Qwen3.8-Max26/4065.0%−13.3
Gemini-3.7-Flash24/4060.0%−18.3
GLM-5.327/4067.5%−10.8
GLM-5.221/4052.5%−25.8
MiniMax-M318/4045.0%−33.3
Grok-4.319/4047.5%−30.8
Kimi-K317/4042.5%−35.8

Table 6. Computational efficiency across eight Experimenter models.

ModelTotal Tokens ↓Latency (s) ↓API Cost / Task ($) ↓
GPT-5.6-Sol93.7k6060.396
Qwen3.8-Max93.6k1,2410.159
Gemini-3.7-Flash89.2k5810.070
GLM-5.3110.9k9620.160
GLM-5.2118.3k6640.171
MiniMax-M3112.3k6400.035
Grok-4.3111.5k5960.141
Kimi-K3123.3k7190.392

Table 7. Final handling of records involving expert disagreement (86 records).

Final handlingNumberPercentageFinal decision
Retained without modification1315.1%Included
Revised and subsequently retained4855.8%Included after revision
Excluded2529.1%Excluded
Total86100%—
Total retained6170.9%Included
Area_Chair_YkFeOpenReview ↗

Reminder to Reviewers: Please Engage in the Discussion

Dear Reviewers,

As the authors have now submitted their rebuttal, I would kindly ask you to read their responses and engage in the discussion.

Thank you!

Best regards,
AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.