acceptodds
Under review as a conference paper at ICLR 2027

Can LLMs Design Experiments? Benchmarking Experiment Design for Autonomous Labs

Abstract

Large language models (LLMs) have demonstrated strong capabilities in chemical reasoning and reaction prediction. However, knowing what reaction to perform does not necessarily imply knowing how to perform it in the laboratory. Experimental design requires translating scientific knowledge into concrete decisions about materials, operations, conditions, equipment, and execution order. Yet this capability remains largely underexplored in current LLMs. We introduce LabProtocol, a dataset and benchmark for learning and evaluating experimental design. LabProtocol aligns chemical reactions with real experimental procedures and represents each procedure as a sequence of structured, parameterized laboratory operations. These operations capture the materials, conditions, equipment, and temporal dependencies required to perform an experiment. Using LabProtocol, we evaluate leading LLMs on tasks ranging from complete procedure generation to single-operation completion. We find that current LLMs struggle to generate complete experimental procedures. More surprisingly, they still perform poorly when asked to predict only a single missing operation, suggesting that the limitation goes beyond long-horizon planning and reflects a lack of experimental procedural knowledge. We further fine-tune Qwen on LabProtocol and observe modest improvements in complete protocol generation but little improvement in single-operation completion. This contrast suggests that learning global protocol patterns does not necessarily yield fine-grained procedural understanding. LabProtocol provides a foundation for developing autonomous laboratory agents that translate scientific objectives into executable experiments.

Primary area:
datasets and benchmarks
Keywords:
Executable Experimental Procedures, Autonomous Laboratories, Scientific Agents

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_L6BgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces LabProtocol, a dataset of 18,729 curated synthesis procedures represented using 42 laboratory operation types, with associated materials, parameters, and equipment. It evaluates six LLMs on generating operation sequences from target products and completing a single missing operation under different context settings. The benchmark provides a resource for studying procedural knowledge, while its metrics measure agreement with reported workflows rather than experimental success.

Critical strengths
  1. The motivation is compelling. Translating scientific knowledge into laboratory procedures is an important next step for AI for Science.

  2. LabProtocol integrates diverse sources into a structured representation, preserving experimental details and their provenance.

  3. Comparing complete generation with single-operation completion provides a useful perspective on global planning and local procedural knowledge.

Critical weaknesses
  1. The framing is broader than the evaluation supports. The title and abstract should specify chemical and materials synthesis. Predicting action labels alone also does not establish experimental executability.

  2. The distinction from existing resources is insufficiently explained. Direct comparisons with Smiles2Actions, Orgsyn, and BioPlanner should clarify what LabProtocol adds in task design, annotations, and evaluation [1–3].

  3. Related work needs greater depth. Prior studies of autonomous experimentation, structured protocol generation, and procedural evaluation provide important context [4–6]. The paper should explain how its scope and evaluation differ from these efforts.

  4. The experiments are too limited to establish benchmark validity. Six general-purpose models and one trained variant provide limited coverage without specialized procedure-generation models, retrieval baselines, or expert evaluation.

  5. Reference matching can penalize valid alternative procedures. Given the limited input and multiple possible synthesis routes, low scores do not necessarily demonstrate a lack of experimental knowledge.

  6. Some figures feel overly decorative and formulaic.

References

[1] Vaucher et al. Inferring experimental procedures from text-based representations of chemical reactions. Nature Communications, 2021.

[2] Zhang et al. ProcedureT5: adaptive experimental procedure prediction with data-augmented pre-training and multi-source data integration. Reaction Chemistry & Engineering, 2026.

[3] O’Donoghue et al. BioPlanner: Automatic Evaluation of LLMs on Protocol Planning in Biology. EMNLP, 2023.

[4] Boiko et al. Autonomous chemical research with large language models. Nature, 2023.

[5] Ruan et al. An automatic end-to-end chemical synthesis development platform powered by large language models. Nature Communications, 2024.

[6] Sun et al. Unleashing Scientific Reasoning for Bio-experimental Protocol Generation via Structured Component-based Reward Mechanism. ICLR, 2026.

Questions for authors

see Weaknesses.

Reviewer_U9fiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

A dataset of chemical reactions and their realized experimental protocols is presented. This dataset is used to define two benchmark tasks: generating complete protocols and filling-in masked steps in a protocol. Current LLMs are evaluated on these tasks, characterizing their weaknesses and demonstrating full protocol generation is not yet possible. A model is trained on the former formulation, improving performance, but fails to generalize to the latter formulation.

Critical strengths
  • Experiment protocol generation is formulated as a well-defined natural language generation task and thoroughly evaluated across several LLMs
  • The evaluation task formulations offer insight into where current models struggle
  • The benchmark construction is amenable to RL, which the authors demonstrate.
Critical weaknesses
  • I am unsure of the novelty of this work. In particular, we can compare to the OMG paper (Kim et al) which is used as a source dataset here. Both papers build a structured representation of synthesis experiment protocols and evaluate LLMs' ability to generate protocols. I think the task formulation in this work is slightly different and novel (e.g. masking a single step vs OMG's LLM-as-judge eval).
  • I have a hard time interpreting the Complete Protocol Generation setting, in particular the conclusion 'Current LLMs remain limited'. The authors note reaction-to-protocol is a one-to-many mapping, making it hard to evaluate what constitutes 'good' in these metrics.
Questions for authors
  1. How do you know your benchmark protocols are not already stated in the scientific literature and included in model pretraining corpora? Can you stratify results by publication date? Using extremely recent protocols from e.g. chemrxiv and showing the metrics do not move substantially would provide confidence
  2. Per my point above about the Complete Protocol Generation metrics not capturing the one-to-many task, I would ask for one of two supporting studies to bolster this result: either a human baseline. Or, for the studied open models, computing pLLM(Π∣r)p_{LLM}(\Pi|r) and showing positive correlation between the conditional probability and any of reported metrics.
  3. How do you determine when a model-generated step is equivalent to the ground truth? You note that you preserve incomplete reporting - if the reference did not report ambient pressure but the model did, is it penalized?
  4. I don't understand Seq. Mask vs Full Mask from the given explanation and appendix B5. What exactly is the additional experimental record provided in Full Mask and masked in Seq. Mask?
  5. The GRPO section could use more detail - e.g. what is the reward structure?
Reviewer_G4u6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces LabProtocol, a dataset designed to evaluate the ability of large language models (LLMs) to plan experimental workflows. It comprises 18,729 experimental records and 42 categories of standardized operations. The authors define two tasks, generating complete experimental workflows and completing individual procedural steps, and evaluate six different LLMs on these tasks. Additionally, they explore fine-tuning the Qwen model using GRPO. The experiments reveal that current models perform poorly on both tasks; while providing more context improves performance on single-step completion, fine-tuning yields only limited gains.

Critical strengths
  1. Integrating data sources from various chemistry and materials experiments, while standardizing the representation of experimental procedures and preserving details such as equipment, parameters, and experimental conditions, yields data resources of significant value.
  2. Furthermore, by establishing tasks for both full-process generation and single-step operation completion, this approach provides a starting point for analyzing predictive capabilities across different levels of experimental workflows.
Critical weaknesses
  1. Insufficient innovation and a lack of thorough comparison with existing work. Vaucher et al. (2021) have already explored the generation of experimental procedure sequences from chemical reactions, providing large-scale datasets, specialized models, and expert evaluations. ProcedureT5 (2026) further addresses experimental parameters and post-processing steps. Although the paper cites these studies, it fails to provide a direct comparison or clearly articulate its own substantive contributions. The experiments are limited to general-purpose LLMs, omitting comparisons with existing specialized models, retrieval-based methods, or simple statistical baselines. Consequently, it is difficult to assess the value of LabProtocol relative to prior research.

  2. The evaluation design is inadequate to support the core conclusions. While the paper emphasizes experimental design and executability, its evaluation is restricted to sequence prediction across 42 categories of operational labels—ignoring critical factors such as materials, parameters, equipment, and conditions that determine whether an experiment can actually be performed. Generating a full procedure based solely on a target product does not yield a unique experimental route, and relying on matching metrics against a single reference sequence is insufficient to determine whether a proposed protocol is scientifically sound. Furthermore, the single-step completion task lacks simple baselines, and the "Full Mask" masking strategy is not clearly defined. The current experimental setup fails to disentangle factors such as limited model capability, missing input information, and flawed evaluation methodology; thus, it cannot substantiate the paper’s primary conclusion regarding a lack of procedural knowledge.

  • Vaucher et al., Inferring experimental procedures from text-based representations of chemical reactions, Nature Communications, 2021
  • Zhang et al., ProcedureT5: Adaptive Experimental Procedure Prediction with Data-Augmented Pre-training and Multi-Source Data Integration, Reaction Chemistry & Engineering, 2026
Questions for authors
  1. Compared to Vaucher et al. (2021), BioPlanner (2023), and ProcedureT5 (2026), what are the substantive innovations of this paper in terms of data, methodology, and evaluation? Why were no direct comparisons made with these approaches?
  2. Why does the full-procedure generation output only the target product, rather than the complete reaction information? How are cases handled where a single product can be synthesized via multiple valid routes?
  3. Could you provide results for simple baselines, such as retrieval-based methods, operation frequency analysis, n-grams, or Markov models? Do the current single-step completion results significantly outperform these baselines?
  4. Does the "Full Mask" approach completely remove the original text, parameters, and other associated information corresponding to the predicted operations? How is label leakage prevented?
  5. Are there any expert evaluations or experimental constraint checks to demonstrate that the current automated metrics accurately reflect the scientific validity and feasibility of the experimental procedures?
  • Vaucher et al., Inferring experimental procedures from text-based representations of chemical reactions, Nature Communications, 2021
  • O'Donoghue et al., BioPlanner: Automatic Evaluation of LLMs on Protocol Planning in Biology, EMNLP, 2023
  • Zhang et al., ProcedureT5: Adaptive Experimental Procedure Prediction with Data-Augmented Pre-training and Multi-Source Data Integration, Reaction Chemistry & Engineering, 2026

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.