acceptodds
Under review as a conference paper at ICLR 2027

LabCompiler: A Framework for Verifiable Human-Machine Laboratory Execution from Imperfect Protocols

Abstract

Closing the loop of AI-driven science requires reliably translating natural-language protocols into executable laboratory workflows despite protocol defects and heterogeneous hardware. Existing approaches often couple scientific interpretation with device-specific code generation, making errors difficult to localize and limiting reuse across laboratory platforms. We introduce LabCompiler, a robust protocol-to-execution framework that transforms imperfect natural-language protocols into verifiable human-machine workflows. Its protocol firewall diagnoses and repairs defects before compilation while preserving unaffected content, guided by troubleshooting knowledge accumulated through perturbation-guided self-evolution. A two-level, device-agnostic graph representation separates experimental semantics from hardware-specific implementation and supports capability-based device assignment with coordinated human execution. We further introduce a protocol recovery benchmark covering four controlled perturbation types and naturally occurring errors collected from protocols written by iGEM participants. Experiments show that LabCompiler, outperforms frontier LLMs in protocol error correction while enabling coordinated human-machine execution across heterogeneous, partially automated laboratory settings.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Self-driving laboratory, Coding agent

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_PDhnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents LabCompiler, a prompting pipeline that takes a written lab protocol and turns it into a workflow split between robots/experimental devices and people. It works in three steps. First, an LLM acts as a "firewall" and looks for ambiguity in the protocol and fixes it. Second, the fixed protocol is converted into a graph of standard lab actions. Then, each group of actions is either turned into code for one of 45 lab devices or given to a person as written instructions. This work also introduces a benchmark of protocols with errors in them: 2,000 synthetic ones and 259 real ones taken from iGEM team protocols, each with an expert's correction. The generated device code is checked statically and in simulators but never run on real hardware, and about 63% of the generated code passes those checks.

Critical strengths
  1. The problem is important, and the benchmark includes real-world errors. Translating written lab protocols into partially automated experiments is a practical challenge. The hybrid approach, where unsupported steps are assigned to humans, also makes sense for real labs. The 259 iGEM errors with expert corrections are a valuable contribution.

  2. The pipeline improves the underlying model's performance. Compared to directly prompting DeepSeek, LabCompiler improves the repair score from 35.9 to 46.9, while execution metrics improve by roughly 30 points. The improvements are particularly noticeable for errors involving multiple steps, where the structured approach seems most useful. Achieving performance comparable to frontier models with a cheaper model is also interesting.

Critical weaknesses
  1. It is unclear how much the method actually contributes and at what cost. The main comparison evaluates LabCompiler using DeepSeek against stronger models with direct prompting, but LabCompiler is never evaluated with those stronger models (even on a smaller benchmark subset, this would be helpful). The paper also does not report inference costs, even though LabCompiler can make multiple model calls per protocol, while the baselines only make one. Another concern is that the largest improvements come from notes learned on 400 synthetic examples generated using the same process as the test set, while the baselines do not have access to these notes. Although the train/test split separates protocol IDs, similar protocols might still appear in both sets, and no similarity analysis is provided.

  2. The claim of "verifiable execution" is not fully supported. The firewall is responsible for correcting protocols before execution, but it is only evaluated on protocols containing errors (App. E.1; Table E.12). This leaves open the question of whether it might incorrectly modify protocols that are already correct. More importantly, much of the protocol verification relies on another LLM rather than an independent symbolic or programmatic verifier (App. B.1). The generated code is also never executed on real hardware (Sec. 5.4). Only 2 of the 45 devices have official simulators, and 8 are checked only for valid parameter ranges (Tables D.8 and D.10). While the paper provides some static and simulation-based checks, I think the evidence is insufficient to support the broader claim of verifiable execution.

Questions for authors
  1. How much does each protocol cost to process (tokens, API cost, and number of api call) for both LabCompiler and each baseline?
  2. How well do frontier models perform when given access to the same 400 examples?
  3. How does LabCompiler perform when using GPT-5.6 or Claude?
  4. Does LabCompiler's advantage hold after removing near-duplicate protocols between the examples used to generate notes and the test set?
  5. When the firewall processes protocols that are already correct, how often does it incorrectly modify them?
2 comments

Thank you for reviewing our paper. Below, we provide detailed responses to your comments.

  1. Performance based on stronger models.

Compared with direct prompting, adding the LabCompiler framework improves classification accuracy by 6.7–16.1 percentage points and repair scores by 6.6–11.0 points across all four backbones. Using the same GPT-5.6 Sol backbone, LabCompiler also outperforms Codex by 6.5 percentage points in accuracy and 4.4 points in repair. Appendix F.5 provides details. The last four rows further show that LabCompiler benefits from stronger backbone models.

MethodControlled Accuracy (%) ↑Controlled Repair ↑
MiniMax-M344.527.5
DeepSeek-V4-Pro50.835.9
GPT-5.6 Sol62.243.4
Claude Opus 564.945.4
Codex (GPT-5.6 Sol)64.246.8
LabCompiler (MiniMax-M3)51.235.5
LabCompiler (DeepSeek-V4-Pro)66.946.9
LabCompiler (GPT-5.6 Sol)70.751.2
LabCompiler (Opus 5)73.852.0
  1. Inference costs.

The latency and token usage for protocol repair and verification are shown below:

MetricLabCompiler*LabCompilerCodex-GPT-5.6 SolGPT-5.6 SolClaude Opus 5
Avg. Total Tokens11,880.895,432.0013,533.005,733.855,379.85
Avg. Latency (s)34.330.420.6614.0735.74
Avg. API Cost (USD)0.1129280.0538880.3659900.1670160.134496

The latency and token usage for code generation are shown below:

MetricLabCompilerGPT-5.6 SolClaude Opus 5
Avg. Total Tokens23,312.4214,290.8015,322.20
Avg. Latency (s)89.5392.9841.44
Avg. API Cost (USD)0.2248120.4137240.362055
  1. Analysis of self-evolved memory.

We agree that different protocol IDs do not guarantee dissimilar protocols. We therefore tested whether the same memory improves direct LLM prompting and helps LabCompiler handle unseen error types and natural errors in real-world protocols.

(1) We additionally evaluated MiniMax-M3, GPT-5.6 Sol, and Claude Opus 5 with access to the same self-evolved troubleshooting memory. For each of the three LLMs, direct prompting with memory achieves higher classification accuracy and repair scores than direct prompting without memory.

Values are reported as without / with memory.

MethodClass. acc. (%) ↑Repair score ↑
MiniMax-M344.5 / 47.227.5 / 30.1
GPT-5.6 Sol62.2 / 65.843.4 / 46.6
Claude Opus 564.9 / 69.345.4 / 49.1

(2) We evaluated step reordering and incorrect step insertion (500 instances each), neither used for memory construction, with memory frozen and the classification options expanded to six categories. Adding memory improves accuracy by 7.0 and 3.4 percentage points and repair scores by 8.4 and 2.8 points, respectively. LabCompiler* achieves the highest scores on both metrics for both types (Appendix F.2).

(3) On the real-world error subset, which was not used to build the memory, adding memory improves accuracy from 54.4% to 76.8% and the repair score from 45.8 to 55.5 (Fig. 3c,d).

  1. Analysis for the false negatives.

(1) Screening on correct and faulty protocols.

We constructed a new set of 100 correct and 100 faulty protocols. We added an “already correct” option to the screening prompt and kept the remaining inputs and workflow unchanged.

MetricResult
Correct protocols: wrongly modified0/100 (0%)
Faulty protocols: flagged for repair100/100 (100%)
Faulty protocols: correct error type73/100 (73%)

(2) Preservation of correct steps during repair.

Each protocol in the original benchmark contains a faulty step. We recruited four biology PhD students to check all outputs, checking whether any originally correct step was incorrectly modified, omitted, or deleted.

Protocols preserving all correct steps:

SubsetPreserved / Total (%)
Controlled1,991/2,000 (99.55%)
Real-world253/259 (97.68%)
Overall2,244/2,259 (99.34%)
  1. Results of the real-world execution

To demonstrate that generated code can be validated through execution on real hardware, we present two experiments.

(1) Fluorescein serial dilution. LabCompiler-generated code prepared three replicate six-point tenfold dilution series with PBS blanks on Opentrons Flex, and Cytation 1 measured fluorescence. Details are provided in Appendix K.1.

(2) ADH kinetics. LabCompiler-generated code prepared ADH reactions across seven substrate concentrations on Opentrons Flex. Synergy H1 monitored absorbance at 340 nm. Details are provided in Appendix K.2.

Dear Reviewer PDhn,

Thank you once again for your time in reviewing our paper. If any concerns remain, please don't hesitate to inform us.

Best regards,

Authors

Reviewer_fUfBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a methodological framework to robustify text-to-protocol AI assisted pipelines for both scientific discovery and industrial application. The framework proposed a text-to-graph translation that confines linguistic variability to well-defined and objective procedural requirements. The graph is then used to validate the procedural correctness, implement patches and standardize deployment.

Critical strengths

The paper proposes a procedural framework of substantial value to the optimization and automation of AI assisted discovery and production. The importance of the objective makes the paper an important and timely ideological contribution.

Critical weaknesses

The paper implements a text-to-protocol translational layer using LLMs and neural networks instead of formal checkers. Consequently, they introduce in their framework the same bias that they aim to solve at the translation layer, creating a potentially unsafe AI-on-AI loop control. The lack of formal details on the text-to-graph translation layer, propagation and check substantially weakens the solution to an otherwise very interesting problem.

Questions for authors
  • Can you provide a detailed account of the scientific methodology of the entire pipeline text-graph-check-procedure in a formally unambiguous manner?
  • Can you introduce formal checkers in the procedure and thus sidestep the implicit AI-on-AI control bias? If the implementation is infeasible under review deadlines, provide a careful analysis of how the issue might be mitigated in future works.
  • Results heavily rely on the number of faulty procedural artifact the proposed method is able to correct. Of equal importance is the number of false negative, i.e. the number of correct procedural steps that are nevertheless altered by the framework. Please provide a detailed account of such instances in all the experimental cases.
2 comments

Thank you for reviewing our paper. Below, we provide detailed responses to your comments.

  1. Implementation details of each stage

LabCompiler is not a purely AI-on-AI control loop. It incorporates rule-based checks throughout the pipeline to validate model outputs before they proceed to execution. It combines LLM-based reasoning and generation with rule-based transformations and checks across its three stages:

  • Protocol firewall. The LLM first interprets the scientific context to diagnose and repair protocol defects. It then assesses whether the revised protocol resolves the identified defects while preserving unaffected content. Rule-based checks primarily verify the presence of required elements and ID consistency.
  • Semantic abstraction. The LLM constructs the Semantic Experiment Graph (GsemG_{\mathrm{sem}}), capturing operations, materials, parameters, and dependencies. Deterministic rules transform GsemG_{\mathrm{sem}} into the Canonical Action Graph (GcanG_{\mathrm{can}}) by mapping operations to the 26 canonical actions and normalizing parameters and units.
  • Execution planning and routing. The LLM proposes and reviews execution units based on experimental objectives, material continuity, and dependencies. Rule-based checks validate structural and capability constraints. After physical-resource binding, deterministic adapters compile automated units into device-specific commands or code.

The prompt templates for the LLM-based steps in each stage are shown in the Appendix J. The pipeline therefore requires more than agreement between model-generated proposals and reviews. Every automated realization must also satisfy independently enforced constraints.

  1. Example from Protocol to Device Code.

Here, we show a case of the entire pipeline of text-graph-check procedure. Consider

Incubate the reaction mixture for 20 min at 20 °C.

Since the duration and temperature have been specified, the protocol diagnosis passes. The transformations in the subsequent steps are

Semantic graph
op-002, incubate, OBJ-mixture
duration="20 min", temperature="20 °C"
predecessor=op-001

Canonical graph
norm-op-002, incubation, sample=OBJ-mixture
duration=(20, minute), temperature=(20, celsius)
predecessor=norm-op-001, source=op-002

Execution unit
node-002, actions=[norm-op-002], predecessor=node-001

Resource binding
OBJ-mixture → compatible Corning 96-well plate/A1
plate → OT-2 Temperature Module GEN2, slot 3

Checks require positive duration, a supported temperature and compatible labware. With the configuration validated and loaded, the incubation commands are

module_001.set_temperature(20)
protocol.delay(seconds=1200)

The generated code must pass OT-2 backend validation before acceptance. At runtime, set_temperature blocks until the module reaches its setpoint, after which the 1200-second delay begins. Appendices C and D describe the IR schemas and execution-grounding procedure with illustrative examples; Appendix J provides core prompt excerpts.

  1. Analysis for the false negatives

(1) Screening on correct and faulty protocols.

We constructed a new set of 100 correct and 100 faulty protocols. We added an “already correct” option to the screening prompt, kept the remaining inputs and workflow unchanged.

MetricResult
Correct protocols wrongly modified0/100 (0%)
Faulty protocols flagged for repair100/100 (100%)
Faulty protocols assigned the correct error type73/100 (73%)

(2) Preservation of correct steps during repair.

Each protocol in the original benchmark contains a faulty step. We recruited four biology PhD students to check all outputs, checking whether any originally correct step was incorrectly modified, omitted, or deleted.

SubsetControlled subsetReal-world subsetOverall
Protocols preserving all correct steps1,991/2,000 (99.55%)253/259 (97.68%)2,244/2,259 (99.34%)

Dear Reviewer fUfB,

Thank you once again for your time in reviewing our paper. If any concerns remain, please don't hesitate to inform us.

Best regards,

Authors

Reviewer_WfQcOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors have introduced LabCompiler which is an LLM pipeline that converts imperfect protocols written in natural text into code and set of executable instructions. The core idea is to avoid human subjectivity and let the translation done by a defined pipeline to minimise errors. An LLM (Deepseek-v4-pro) reads the protocol and finds the problematic steps e.g. wrong temperature or missing steps and rewrites the protocol through successive refinement. A memory component is built from ~400 training protocols with curated answers and the LLM reads it as context for unseen cases. The goal is to convert the fixed protocol into standard lab actions using a graph that also has dependencies on earlier steps. The actions are then grouped and forwarded on a supported device where generated code is checked in the simulator or against the device specifications and unspported device steps are given to a human. The benchmark compares the performance of this pipeline with different LLMs (claude and GPT) on protocols with injected errors and 259 real iGEM protocols.

Critical strengths
  • To stop LLMs from rewriting the complete protocol and fix only certain steps, the pipeline compares the rewrite with the original imperfect protocol and logs the evidence and the fixes.
  • Benchmark of real iGEM protocols is a nice addition by the authors as this provides good evidence on real messy text that this pipeline outperforms the baselines.
  • The paper also shows on ~30% of model outputs the LLM judges are within 2 points (repair score) of 8 biologists who didn't know which of the outputs were generated by which LLM .
  • Compared to earlier approaches, that write code for a robot and if one step fails it cannot continue, the authors divide the protocol into groups. Each of them goes to a device that can do all the steps in it, otherwise the group is given to a human with a written instruction. Furthermore, the pipeline marks the vague steps for clarification.
Critical weaknesses
  • In my opinion there is barely any novelty in the suggested approach as the whole pipeline is a chain of prompts around an baseline DeepSeek model. The repair step is a prompt, the experiment graph is a JSON schema and the device routing is a hand written table.
  • I think the performance gains come from the memory of ~400 prompts which are built using the same perturbation rules as the test set. Moreover, the paper does not describe how the memory entries are retrieved or given to the model for a new protocol.
  • For multi-round repair and verify loop the latency, cost and token usage is not reported anywhere.
  • There is no analysis/interpretation of what the model is actually doing as the paper reports merely the aggregated scores. There are no failure cases shown, generalisability or how often does the loop remain unresolved.
Questions for authors
  • Since the pipeline does not change the model, can you run it on top of other closed/opensource models instead of DeepSeek? This would help us to see if the pipeline itself adds anything over a stronger model prompted directly.
  • Can you also report the average number of iterations, tokens and latency per protocol for the repair and verify loop.
  • Can you show failure cases for each error category and report how often the loop ends unresolved?
2 comments

Thank you for your valuable comments. We hope our responses below can help address your concerns on this paper.

  1. The novelty of LabCompiler.

LabCompiler is an execution harness integrating protocol recovery, device-independent representation, and execution planning for verifiable human–machine workflows, rather than DeepSeek-specific prompting.

(1) Protocol assessment and iterative repair. Before code generation, the firewall iteratively diagnoses, repairs, and verifies protocols, preserving valid content and using knowledge distilled from reference-corrected cases. With the same backbone, iterative screening raises the repair score from 40.9 to 47.0 over one-pass repair (Table F.13).

(2) Device-independent experiment graphs. The two-level IR standardizes varied protocols into actions with hardware-independent experimental logic. Graph nodes encode operations; directed edges encode execution order and material dependencies. JSON is one encoding. The same graph can be compiled via adapters for compatible devices without protocol reinterpretation. Over direct protocol-to-code generation, gains are 25.1/22.1 percentage points in route accuracy/device executability and 15.1 points in code fidelity (Table 3(a)).

(3) Coordinated human–machine execution. LabCompiler groups actions by material flow and dependencies. Each execution unit is routed to a device able to execute it fully within operating and resource constraints; unsupported operations remain human steps. With the same device catalog and checks, execution-unit routing improves routing accuracy (61.2%→72.9%) and device executability (49.5%→59.6%) over action-wise routing (Table 3(b)).

  1. On memory gains and retrieval.

Memory construction and evaluation share perturbation types but no source protocols or variants, using a source-protocol split.

On unseen step reordering/incorrect step insertion (500 each), the same frozen evolved memory adds 7.0/3.4 percentage points to classification accuracy and 8.4/2.8 points to repair scores, respectively; LabCompiler* scores highest on both metrics for both types (Appendix F.2). On the Real-World Error Subset (259 protocols with natural errors), memory raises accuracy (54.4%→76.8%) and repair score (45.8→55.5; Fig. 3c,d). This suggests transfer beyond memory-construction perturbations.

At inference, screening prompts list only knowledge-base names. The agent chooses whether and which to read; contents enter its reasoning context only via a knowledge-reading tool call.

  1. Computational cost and latency of iterative repair.

On 200 test protocols, evolved memory increases tokens, API calls, latency, and cost. Both variants cost less than all three baselines, run slower than Codex and direct GPT-5.6 Sol but faster than Claude Opus 5. Costs reflect cache hits; calls include network retries.

AverageLabCompiler*LabCompilerCodex-GPT-5.6 SolGPT-5.6 SolClaude Opus 5
Total tokens11,880.895,432.0013,533.005,733.855,379.85
Latency (s)34.330.420.6614.0735.74
API cost (USD)0.1129280.0538880.3659900.1670160.134496
API calls1.671.491.921.161.28

Under tested models and billing conditions, LabCompiler's protocol-to-code generation uses more tokens but costs less than direct GPT-5.6 Sol and Claude Opus 5, with latency close to GPT-5.6 Sol but above Claude Opus 5.

AverageLabCompilerGPT-5.6 SolClaude Opus 5
Total tokens23,312.4214,290.8015,322.20
Latency (s)89.5392.9841.44
API cost (USD)0.2248120.4137240.362055
API calls5.131.241.65
  1. Analysis of failure cases and generalisability.

We expanded the analysis. With the same backbone, iterative screening raises repair scores (40.9→47.0) over one-pass repair (Table F.13), showing that verification feedback helps refine protocols and correct remaining errors. Appendix F.1.1 gives full success/failure traces: proposed edits, verification feedback, and final outcomes.

At screening, unresolved means blocking ambiguities or failure after 12 iterations. Rates cover both benchmark subsets and separate protocol-to-code inputs.

Controlled perturbation subsetReal-world error subsetExecution inputs
Unresolved rates0%0.77%3%

Self-evolved memory from errors in the controlled perturbation subset is frozen and tested on the real-world error subset (259 iGEM protocols). Repair scores rise (45.8→55.5; Fig. 3), demonstrating transfer to natural errors without test-set adaptation.

  1. Performance across different backbones

We also tested LabCompiler with MiniMax-M3, GPT-5.6 Sol, and Claude Opus 5. Repair scores improve over direct prompting of each backbone by 8.0/6.3, 7.8/8.4, and 6.6/11.0 points on the controlled perturbation/real-world error subsets, respectively. Gains hold for all tested backbones, including stronger GPT and Claude (Appendix F.5).

Dear Reviewer WfQc,

Thank you once again for your time in reviewing our paper. If any concerns remain, please don't hesitate to inform us.

Best regards,

Authors

Reviewer_mbG2OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses issues in natural-language experiment protocols, such as wording errors and ambiguity, as well as information loss and strong device dependence that can occur when converting protocols to lab equipment. To tackle these problems, the authors propose LabCompiler, which consists of three modules: protocol correction, device-agnostic representation, and capability-grounded execution. They also validate the effectiveness of their architecture through three corresponding sets of experiments, demonstrating practical value for human-machine collaboration in automated labs.

Critical strengths
  1. Reliable conversion of natural-language experimental protocols into machine-executable workflows is an important part in automated laboratories.
  2. The three stages of LabCompiler (protocol correction, device-agnostic representation, and device adaptation) connect with each other in a sensible way. The artifacts are traceable, and they also provide a basis for transfer across different devices.
  3. The iGEM subset of the proposed benchmark uses real experimental protocols with naturally occurring errors, making evaluation more realistic and going beyond error detection with predefined options.
  4. The authors carry out thorough multi-level experiments and ablations, so the soundness of the framework is validated from several angles.
Critical weaknesses

1.Appendix A compares related systems qualitatively, but LabCompiler's unique contribution relative to BioProAgent's Design–Verify–Rectify loop and ProtoPilot's staged generation remains unclear. In particular, how the protocol firewall, the two-layer graph generation, and machine-code conversion are implemented is not sufficiently mentioned, e.g., whether they rely on the LLM or on hard-coded rules. Adding more details can help to assess the innovation of these designs. 2.The experiments mainly compare LabCompiler with direct LLMs and Codex. But the paper does not sufficiently clarify the settings for these baselines. It is therefore difficult to determine how much of LabCompiler’s gains come from more effective prompts, additional revision rounds, or access to more useful tools. 3.The results in execution experiments seem to be limited, indicating that a bottleneck still exists in going from protocols to reliable device execution. Appendix D outlines some manual takeover approaches, but it lacks an explanation of how action units are implemented as device-specific code, as well as analyses of failure cases and the possible recovery methods. Adding this content would help assess the system's practical usability. 4.Evolved experience improves error-correction ability, but the experience library appears to be frozen before deployment. Extending the mechanism to support evolution during deployment could make the system more useful in practice.

Questions for authors

1.Please clarify which steps of LabCompiler use an LLM and which use deterministic rules. For the LLM-based steps, please include the prompt templates in the appendix. Comparing LabCompiler with BioProAgent or ProtoPilot on applicable subsets of the paper’s evaluation tasks would better distinguish its relative strengths. 2.Please detail the settings of the direct LLM and Codex baselines, particularly which task information, tools, and prompts were shared with LabCompiler and which differed. 3.What are the main sources of error behind device executability and code fidelity? A worked example tracing actions to device-specific code, particularly a case where device routing is correct but the code does not implement the protocol, would help clarify the cause and possible recovery.

3 comments

We appreciate the reviewer's constructive comments.

  1. On clarity of the contributions.

(1) Implementation details of each stage

All three stages combine LLM reasoning/generation with rule-based transformations and checks (prompt excerpts in Appendix J):

  • Protocol firewall. The LLM diagnoses/repairs defects in scientific context, then assesses resolution and preservation of unaffected content. Rules mainly check required elements/ID consistency.
  • Semantic abstraction. The LLM builds SEG (Semantic Experiment Graph) with operations/materials/parameters/dependencies. Deterministic rules map it to CAG (Canonical Action Graph): 26 canonical actions, normalized parameters/units.
  • Execution planning and routing. The LLM proposes/reviews units by objectives/material continuity/dependencies. Rules check structural/capability constraints. After physical-resource binding, deterministic adapters compile automated units to device code/commands.

(2) Novelty of LabCompiler

BioProAgent uses a design–verify–rectify FSM; ProtoPilot progressively refines protocols/SOPs into code.

  • Content-preserving recovery: the firewall repairs errors before compilation while preserving valid content (App. B).
  • Device-agnostic IR: compatible backends reuse CAG via bindings/adapters, separating protocol semantics from hardware. Full vs. direct-to-code: +25.1/+22.1 pp in route accuracy/device executability; +15.1 code-fidelity points.
Compilation pathRoute acc. (%) ↑Device exec. (%) ↑Code fidelity ↑
Protocol→Code49.641.238.7
Protocol→CAG→Code63.552.748.9
Protocol→SEG→CAG→Code74.763.353.8
  1. Baseline settings and attribution of gains.

(1) Baseline configurations

All methods share per-task goals, protocols, instructions, output requirements, and device catalogs (categories/supported actions). Direct LLMs use one pass without tools. Codex (GPT-5.6 Sol) has all available tools except web search. LabCompiler (DeepSeek-V4-Pro) uses custom tools for diagnosis/repair, graph building, capability-based routing, and device-code generation/validation.

(2) Attribution of gains

All variants use DeepSeek-V4-Pro on the controlled perturbation subset. Self-evolved memory (400 training examples; Sec. 3.2) is frozen before evaluation and shared by memory-enabled methods. LabCompiler′ keeps tools/skills with memory/iterative revision off. Mean results:

MethodMemoryIterative revisionClass. acc. (%) ↑Correction score ↑
DeepSeek-V4-ProNoNo50.835.9
DeepSeek-V4-Pro + MemoryYesNo54.939.4
LabCompiler′NoNo59.243.7
LabCompilerNoYes66.946.9
LabCompiler*YesYes77.654.0

Framework contribution. Without memory/iterative revision, the harness gains 8.4 pp in accuracy and 7.8 correction points over direct prompting.

Iterative revision. Feedback-based revision (memory off; other settings fixed) raises accuracy 59.2%→66.9% and correction 43.7→46.9.

Self-evolved memory. Gains: 10.7 pp in accuracy/7.1 correction points for LabCompiler; 4.1 pp/3.5 points for the direct LLM.

  1. Device-specific code generation and failure recovery.

Appendix D.5 presents a worked example linking the protocol, IR, resource bindings, and device code.

(1) Protocol → IR → device code

Protocol: After DNA binding, capture beads magnetically for 5 min, then remove 100 uL supernatant to waste.

IR: magnetic_separation(sample,5 min) → liquid_removal(sample,waste,100 uL).

Code: Opentrons Flex is chosen for action support. Validated bindings:

protocol.move_labware(labware_002, new_location=module_001, use_gripper=True)  # module_001=Magnetic Block
protocol.delay(seconds=300)
pipette.pick_up_tip(tip_rack_001['A3'])  # tip
pipette.aspirate(100, labware_002['A1'])  # sample well
pipette.dispense(100, labware_003['A1'])  # waste well
pipette.drop_tip()

(2) Failure detection and repair

Protocol: Add 20 µL/well of 1:1 CellTiter-Blue/HBSS; incubate at 37°C for 4 h; read fluorescence (Ex560/Em590 nm) at 1, 2, 3, 4 h during incubation.

Initial code:

module_001.set_temperature(37)
protocol.delay(seconds=14400)
module_001.deactivate()
# One read afterward

Repair: Simulation passed; the log showed one read_plate call, missing the first three timepoints. SEG captured all four; CAG kept them only as text, so unit construction serialized incubation/reading. New temporal check at CAG review: executable schedules must preserve read count/timepoints and reading during incubation.

Corrected code:

t=monotonic()
for h in (1,2,3,4):
    await sleep(max(0,t+3600*h-monotonic()))
    await r.read_fluorescence(560,590,z)
await b.stop_heating_or_cooling()
  1. Evolution during deployment.

We froze experience to test unseen-protocol transfer without test feedback. Online experience updates can help: classification accuracy rose 77.6%→79.2% and repair score 54.0→57.3.

Thank you for the detailed response and revisions. The ablation study and implementation details address many of my concerns. A comparison with other iterative agent harnesses, such as BioProAgent and ProtoPilot, under matched tool access and comparable iteration budgets would further clarify LabCompiler's relative contribution. I maintain my original rating.

Thank you for your thoughtful feedback and insightful comments.

Comments on the paper

1 comment

We thank all reviewers for their constructive comments. We have revised the manuscript with the following key modifications:

New Results:

  1. Added a compilation ablation within LabCompiler comparing direct protocol-to-code generation, canonical-only compilation, and the full two-level representation on 100 protocols (Section 5.5, Table 3(a)).

  2. Added generalization tests on two error types excluded from memory construction: step reordering and incorrect step insertion, with 500 test instances per type (Appendix F.2, Table F.14).

  3. Added evaluations using MiniMax-M3, GPT-5.6 Sol, and Claude Opus 5 as alternative backbones, including overall and per-category comparisons (Appendix F.5, Tables F.16–F.17).

  4. Added real-world execution experiments for fluorescein serial dilution and ADH enzyme kinetics, including generated code excerpts and experimental measurements (Appendix K, Figures K.2–K.3).

Other Revisions:

  1. Moved the existing execution-unit grouping ablation into the main text and expanded its discussion (Section 5.5, Table 3(b)).

  2. Added a worked example tracing magnetic separation and supernatant removal through intermediate representation, device binding, and generated code (Appendix D.5).

  3. Added successful and unsuccessful screening-loop trajectories illustrating candidate revisions and validation feedback (Appendix F.1.1).

  4. Added four representative protocol-recovery failure cases, one for each controlled error category (Appendix I).

  5. Provided core prompt excerpts and input/output specifications for protocol screening, semantic abstraction, and execution planning and routing (Appendix J).

We appreciate the time and effort of the AC and reviewers and welcome any further comments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.