- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents LabCompiler, a prompting pipeline that takes a written lab protocol and turns it into a workflow split between robots/experimental devices and people. It works in three steps. First, an LLM acts as a "firewall" and looks for ambiguity in the protocol and fixes it. Second, the fixed protocol is converted into a graph of standard lab actions. Then, each group of actions is either turned into code for one of 45 lab devices or given to a person as written instructions. This work also introduces a benchmark of protocols with errors in them: 2,000 synthetic ones and 259 real ones taken from iGEM team protocols, each with an expert's correction. The generated device code is checked statically and in simulators but never run on real hardware, and about 63% of the generated code passes those checks.
Critical strengths
-
The problem is important, and the benchmark includes real-world errors. Translating written lab protocols into partially automated experiments is a practical challenge. The hybrid approach, where unsupported steps are assigned to humans, also makes sense for real labs. The 259 iGEM errors with expert corrections are a valuable contribution.
-
The pipeline improves the underlying model's performance. Compared to directly prompting DeepSeek, LabCompiler improves the repair score from 35.9 to 46.9, while execution metrics improve by roughly 30 points. The improvements are particularly noticeable for errors involving multiple steps, where the structured approach seems most useful. Achieving performance comparable to frontier models with a cheaper model is also interesting.
Critical weaknesses
-
It is unclear how much the method actually contributes and at what cost. The main comparison evaluates LabCompiler using DeepSeek against stronger models with direct prompting, but LabCompiler is never evaluated with those stronger models (even on a smaller benchmark subset, this would be helpful). The paper also does not report inference costs, even though LabCompiler can make multiple model calls per protocol, while the baselines only make one. Another concern is that the largest improvements come from notes learned on 400 synthetic examples generated using the same process as the test set, while the baselines do not have access to these notes. Although the train/test split separates protocol IDs, similar protocols might still appear in both sets, and no similarity analysis is provided.
-
The claim of "verifiable execution" is not fully supported. The firewall is responsible for correcting protocols before execution, but it is only evaluated on protocols containing errors (App. E.1; Table E.12). This leaves open the question of whether it might incorrectly modify protocols that are already correct. More importantly, much of the protocol verification relies on another LLM rather than an independent symbolic or programmatic verifier (App. B.1). The generated code is also never executed on real hardware (Sec. 5.4). Only 2 of the 45 devices have official simulators, and 8 are checked only for valid parameter ranges (Tables D.8 and D.10). While the paper provides some static and simulation-based checks, I think the evidence is insufficient to support the broader claim of verifiable execution.
Questions for authors
- How much does each protocol cost to process (tokens, API cost, and number of api call) for both LabCompiler and each baseline?
- How well do frontier models perform when given access to the same 400 examples?
- How does LabCompiler perform when using GPT-5.6 or Claude?
- Does LabCompiler's advantage hold after removing near-duplicate protocols between the examples used to generate notes and the test set?
- When the firewall processes protocols that are already correct, how often does it incorrectly modify them?
2 comments
Dear Reviewer PDhn,
Thank you once again for your time in reviewing our paper. If any concerns remain, please don't hesitate to inform us.
Best regards,
Authors
Thank you for reviewing our paper. Below, we provide detailed responses to your comments.
Compared with direct prompting, adding the LabCompiler framework improves classification accuracy by 6.7–16.1 percentage points and repair scores by 6.6–11.0 points across all four backbones. Using the same GPT-5.6 Sol backbone, LabCompiler also outperforms Codex by 6.5 percentage points in accuracy and 4.4 points in repair. Appendix F.5 provides details. The last four rows further show that LabCompiler benefits from stronger backbone models.
The latency and token usage for protocol repair and verification are shown below:
The latency and token usage for code generation are shown below:
We agree that different protocol IDs do not guarantee dissimilar protocols. We therefore tested whether the same memory improves direct LLM prompting and helps LabCompiler handle unseen error types and natural errors in real-world protocols.
(1) We additionally evaluated MiniMax-M3, GPT-5.6 Sol, and Claude Opus 5 with access to the same self-evolved troubleshooting memory. For each of the three LLMs, direct prompting with memory achieves higher classification accuracy and repair scores than direct prompting without memory.
Values are reported as without / with memory.
(2) We evaluated step reordering and incorrect step insertion (500 instances each), neither used for memory construction, with memory frozen and the classification options expanded to six categories. Adding memory improves accuracy by 7.0 and 3.4 percentage points and repair scores by 8.4 and 2.8 points, respectively. LabCompiler* achieves the highest scores on both metrics for both types (Appendix F.2).
(3) On the real-world error subset, which was not used to build the memory, adding memory improves accuracy from 54.4% to 76.8% and the repair score from 45.8 to 55.5 (Fig. 3c,d).
(1) Screening on correct and faulty protocols.
We constructed a new set of 100 correct and 100 faulty protocols. We added an “already correct” option to the screening prompt and kept the remaining inputs and workflow unchanged.
(2) Preservation of correct steps during repair.
Each protocol in the original benchmark contains a faulty step. We recruited four biology PhD students to check all outputs, checking whether any originally correct step was incorrectly modified, omitted, or deleted.
Protocols preserving all correct steps:
To demonstrate that generated code can be validated through execution on real hardware, we present two experiments.
(1) Fluorescein serial dilution. LabCompiler-generated code prepared three replicate six-point tenfold dilution series with PBS blanks on Opentrons Flex, and Cytation 1 measured fluorescence. Details are provided in Appendix K.1.
(2) ADH kinetics. LabCompiler-generated code prepared ADH reactions across seven substrate concentrations on Opentrons Flex. Synergy H1 monitored absorbance at 340 nm. Details are provided in Appendix K.2.