acceptodds
Under review as a conference paper at ICLR 2027

BioLab: Benchmarking Robotic Manipulation in Biology Laboratory from Individual Skills to Experiment SOPs

Abstract

Robotic automation of biology experiment requires more than executing isolated manipulations. A robot must follow a standard operating procedure (SOP) across multiple instruments while adapting to varied layouts and states produced by earlier steps. Existing laboratory benchmarks assess valuable skills and multi-step tasks, but offer limited evidence about complete SOP-level manipulation. We introduce BioLab, a framework for studying how laboratory skills carry through a full workflow. We develop BioLab-Sim with configurable unified platform and hierarchical annotation & checking for scripted demonstration collection and policy evaluation. Building on this framework, we construct BioLab-Bench by decomposing an SOP into basic laboratory skills (L1), context-dependent operation sequences (L2), and complete biology experiment SOPs (L3). Occurrence-aligned demonstrations and ordered physical checks support action learning and stage-level progress evaluation, while held-out layouts test spatial generalization. We evaluate six vision-language-action policies and find limited reliability even on L1 skills. To study procedural control on longer tasks, we further propose BioLab-HVLA, which combines execution memory and visual progress decisions with leaf-conditioned action control. This hierarchy supports longer workflow execution, although physical checks reveal early manipulation failures and discrepancies between predicted progress and actual completion. We demonstrate that BioLab provides a unified setting for learning, evaluating, and diagnosing the transition from laboratory skills to SOP-level biological manipulation.

Primary area:
datasets and benchmarks
Keywords:
laboratory robotics, robotic manipulation benchmarks, long-horizon manipulation, vision-language-action models, hierarchical control

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Bgo8OpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work proposes a laboratory robotics benchmark that organizes tasks into L1 single skills, L2 context-dependent operation sequences, and L3 complete biological experiment SOPs.

It also introduces the BioLab-HVLA method, which uses execution history and visual progress assessments to select subtasks, which are then executed by the VLA.

Critical strengths

The problem is valuable. Laboratory procedures is quite complex; even humans require some training.

Well-defined task and appropriate data setup are particularly important in this setting. The three-tiered task structure is well motivated and reasonable

Critical weaknesses

The baseline VLAs coverage is somewhat limited for comparison, in fact, a considerable number of VLAs with some memory capacity have recently emerged.

Some experiments use too few trials to reliably assess performance. Although the tasks are challenging and experiments is costly, fifteen trials per real-world task provide limited statistical evidence to rule out the influence of random variables

Questions for authors

The method involves many components, but the discussion of each component’s contribution is incomplete. For example, how much improvement comes from memory and from sub-instruction training?

Automated task generation and checker construction form a relatively complex pipeline that have a significant impact on the overall system. A clearer description of this pipeline would strengthen the paper.

Reviewer_nqEdOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BioLab, a benchmark framework for robotic manipulation in biological laboratories, aiming to bridge the gap between isolated manipulation skills and complete experimental standard operating procedures (SOPs).

BioLab consists of three main components: (1) BioLab-Sim, a unified multi-instrument simulation environment supporting layout randomization, automated task-tree construction, and physical progress checking; (2) BioLab-Bench, a hierarchical benchmark covering 14 individual laboratory skills (L1), eight context-dependent operation groups (L2), and three complete biological SOP workflows (L3); and (3) BioLab-HVLA, a hierarchical vision-language-action framework that integrates execution memory, visual stage prediction, and low-level action generation.

The authors collect 2,843 training demonstrations spanning approximately 42.5 hours and evaluate six VLA policies on L1 tasks. Experiments on L2 demonstrate that BioLab-HVLA improves task success rates over direct π0.5 control, while L3 experiments reveal substantial difficulties in executing complete SOPs. Additional evaluations investigate generalization to unseen layouts and performance on real robotic hardware.

The paper's primary contribution is a unified benchmark and evaluation infrastructure for studying how individual laboratory manipulation skills transfer to longer procedural workflows.

Critical strengths
  1. A meaningful benchmark contribution for long-horizon laboratory manipulation.

The paper addresses an important gap between isolated manipulation skills and complete laboratory procedures. BioLab provides a unified multi-instrument environment in which tasks share physical context and earlier operations affect subsequent states. Its hierarchical L1–L3 organization and occurrence-aligned annotations enable systematic evaluation of skill composition and procedural execution.

I consider this a substantial contribution because it establishes a useful evaluation setting for studying challenges that are difficult to capture with existing short-horizon laboratory benchmarks.

  1. Fine-grained physical evaluation and broad empirical analysis.

The combination of hierarchical task trees and physical progress checkers is particularly valuable. Rather than reporting only final success, BioLab provides stage-level progress information that helps identify where manipulation policies fail.

The experiments include six VLA baselines, held-out layout evaluations, hierarchical control comparisons, and real-robot demonstrations. The detailed failure statistics provide useful insights into current limitations and offer a foundation for future methodological development.

Critical weaknesses
  1. Some experimental comparisons would benefit from stronger controls.

The evaluation protocols are not entirely straightforward to compare. Appendix G.2 describes revisions to physical success criteria and handoff rules for the later hierarchical and oracle evaluations, whereas the root-instruction baseline retains its original outcomes. Although the authors provide detailed documentation, further clarification or matched evaluation would strengthen confidence in the reported improvements.

Furthermore, the real-robot comparison uses different executor initialization conditions: direct π0.5 starts from its pretrained base model, while BioLab-HVLA benefits from a simulation-fine-tuned executor. This makes it difficult to isolate the contribution of hierarchical planning from the effects of additional pretraining.

These concerns primarily affect the interpretation of baseline improvements rather than the validity of the benchmark itself.

  1. Complete SOP-level validation remains limited.

Although BioLab includes three SOP families, the reported autonomous L3 evaluation focuses on plasmid transformation with 18 evaluation episodes. None completes the procedure, and the rollouts do not reach the later workflow phases. Consequently, the evaluation provides limited evidence about the execution and diagnosis of dependencies across an entire SOP.

Additionally, some physical success criteria measure manipulation-level proxies rather than actual biological outcomes, such as transferred liquid volume or mixing quality. This distinction should be emphasized when interpreting what successful SOP execution means.

Nevertheless, these limitations are understandable given the difficulty of the task and represent opportunities for extending the benchmark rather than fundamental flaws in its design.

Questions for authors

Q1. Evaluation consistency.

Could the authors clarify whether the root-instruction, hierarchical, and oracle methods are evaluated under identical physical success criteria and execution rules? If there are differences between evaluation campaigns, could the authors provide matched results or explain their quantitative impact?

Q2. Contribution of hierarchical planning.

In the real-robot experiments, BioLab-HVLA uses a simulation-fine-tuned executor, whereas direct π0.5 starts from its pretrained base model. Could the authors provide a comparison with matched executor initialization? An ablation separating hierarchical memory, visual stage prediction, and leaf-conditioned action control would also help clarify which components contribute most to the improvements.

Q3. Validation of complete SOP execution.

Since all reported L3 rollouts terminate within the early stages, could the authors provide additional evidence that later SOP stages and their physical checkers are reliable, for example through scripted or oracle execution? It would also be helpful to clarify how manipulation-level completion criteria relate to the actual biological objectives.

Reviewer_d4zkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BioLab, a benchmark for evaluating robotic manipulation in biological laboratories, covering individual skills (L1), context-dependent operation sequences (L2), and complete standard operating procedures (L3). It provides a unified simulation platform, hierarchical task annotations, physical progress checkers, and aligned demonstrations. The authors also propose BioLab-HVLA, a hierarchical baseline that combines execution memory, visual progress estimation, and VLA-based action execution. Experiments evaluate six VLA policies, investigate layout generalization, and include real-robot studies. The results highlight the difficulty of reliable long-horizon laboratory automation and the limitations of current VLA policies.

Critical strengths
  1. The benchmark evaluates robotic manipulation from individual laboratory skills to complete SOP workflows, addressing an important gap in existing laboratory robotics benchmarks.
  2. The hierarchical task structure and physical progress checkers enable fine-grained evaluation and interpretable diagnosis of execution failures.
  3. The evaluation covers six VLA policies, layout generalization, and real-robot experiments, providing a useful assessment of current capabilities.
  4. The results reveal substantial limitations in existing VLA policies and demonstrate the benefits of hierarchical planning for context-dependent manipulation tasks.
Critical weaknesses
  1. The technical novelty of BioLab-HVLA is limited, as it primarily combines existing components such as hierarchical memory, VLM-based stage selection, and VLA-based action execution.
  2. The absolute success rates remain low, with only 8.14% in-domain success on L2 tasks and no successful complete L3 workflows among 18 trials.
  3. The real-robot evaluation covers only two task types with 15 trials each, providing limited evidence of robust performance in practical laboratory settings.
  4. The OOD evaluation focuses primarily on layout variations, leaving generalization to unseen SOPs, instruments, and manipulation procedures insufficiently explored.
  5. The benchmark covers only three SOP families, and the scalability of its task-generation and physical-checking pipeline to more diverse biological experiments remains unclear.
Questions for authors
  1. Can the authors clarify the key technical innovations of BioLab-HVLA beyond combining hierarchical memory, VLM-based stage selection, and VLA-based action execution?
  2. What are the dominant failure modes responsible for the low L2 success rates and the absence of successful L3 workflows, and can additional ablations quantify the contribution of each component?
  3. Can the authors provide further real-robot evaluations or statistical uncertainty estimates to better establish the reliability of the reported improvements?
  4. Can the authors evaluate generalization to unseen SOPs or instruments rather than only held-out layouts?
  5. Could the authors clarify the differences between the episode counts reported in the main paper and the split-specific dataset statistics in the appendix?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.