- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces MicroBio-Bench, a VLA simulation and evaluation benchmark for microbiology laboratory tasks. The benchmark includes 18 laboratory assets and 60 tasks organized into three hierarchical levels, along with an automated scene generation pipeline that constructs laboratory environments satisfying predefined constraints. Compared with prior work, this paper introduces contact-dependent contamination constraints, requiring robots to modify their original manipulation plans and discard contaminated items when contamination occurs.The benchmark is evaluated in both simulation and real-world settings using three models: π0.5, RDT, and π0. The evaluations examine contamination response, normal task execution capabilities, and generalization to unseen contamination scenarios. Experimental results show that existing models perform relatively well on atomic tasks but exhibit significant limitations in compositional tasks and generalization to unseen contamination scenarios.
Critical strengths
- The paper introduces VLA-based manipulation into microbiology laboratory environments and extends the scope of investigation to aseptic operation protocols. Compared with conventional constraints based solely on geometric positions or safety zones, contamination-induced tool invalidation rules better reflect the safety and procedural compliance requirements of robotic laboratory operations.
- The paper develops high-fidelity simulation models by leveraging professional CAD models and constructing dedicated collision meshes. This approach offers advantages over conventional simulation models built through direct asset importing or standard digital-twin construction pipelines.
Critical weaknesses
-
The definition of the contamination mechanism is overly simplified, and its scientific significance and research value are insufficiently justified.
(1) Contamination-aware evaluation is one of the central contributions of this work. However, the paper models contamination merely as a binary state triggered by a "prohibited contact" event, followed by a predefined cleanup action. Such event-driven state transitions and conditional safety rules are not unique to microbiology laboratories and can also be evaluated in general robotic manipulation scenarios. More importantly, the paper does not adequately justify what fundamentally new challenges arise from this domain-specific formulation, or whether these challenges constitute sufficiently distinct and meaningful research problems.
(2) The core contamination-response experiments involve only two types of tools and two groups of L1 manipulation tasks, and contamination events are deliberately injected through scripts during the predefined setup stage. Corresponding evaluations are absent from the real-world experiments. These limitations further weaken the empirical support for the paper's central contribution.
(3) The contamination-response experiments do not adequately disentangle safety-related decision-making from low-level manipulation and generalization capabilities. Failures at unseen contact locations do not necessarily indicate a lack of contamination-aware reasoning; they may instead result from limited execution capabilities when the model encounters unfamiliar spatial configurations. This ambiguity further weakens the paper's claim that the benchmark reveals a distinctive and scientifically meaningful limitation of existing VLA models.
-
Although the paper introduces a new application scenario in microbiology laboratories, its overall novelty remains limited: The proposed benchmark lacks clear differentiation from existing laboratory benchmarks. While Table 1 highlights advantages in terms of task quantity and supported features, the overall task diversity appears limited. Although the paper claims to provide 60 tasks, these tasks are organized into three hierarchical levels, where L2 tasks are compositions of L1 tasks, while L0 tasks consist only of simple atomic operations. This raises concerns about the effective number of genuinely distinct tasks. Furthermore, bimanual manipulation appears in only two L1 tasks (L1-21 and L1-23), which essentially share the same underlying manipulation logic, differing only in the indirectly manipulated object (container). Consequently, the paper's claimed support for bimanual manipulation is not sufficiently substantiated.
Questions for authors
- Could the authors further clarify what additional capabilities or advantages MicroBio-Bench offers over existing laboratory benchmarks, beyond its domain-specific assets and contamination-response tasks, particularly regarding task diversity and complexity?
- Among the 60 tasks, how many genuinely represent distinct manipulation skills?
- What fundamentally distinguishes the proposed "contamination problem" from a general robotic safety constraint that requires a robot to terminate its original action and execute an alternative action after contacting a particular region?
- Why is the contamination-response evaluation limited to only two types of tools? Can the authors provide evidence that such contamination scenarios occur frequently enough in realistic laboratory workflows to justify their broader significance?
- Does the performance degradation observed in unseen scenarios reflect a failure of safety-related reasoning, or is it primarily attributable to limitations in spatial generalization capabilities?
3 comments
Response to Reviewer AWMj (2/2)
Q4 and W1. Why evaluate these two tools at fixed stages, and how common are such contacts?
We chose the loop and spreader because both are discarded at the end of their normal workflow. This lets us compare the same disposal skill at the scheduled endpoint and immediately after prohibited contact.
- Loop (L1-06/07/08 in L2-03). Contact is injected after dipping, before streaking. The response study ends at tool disposal.
- Spreader (L1-25/26/27 in L2-10). Contact is injected after pickup, before the first agar contact.
We inject contact at a fixed stage so that the next normal operation stays the same as the contact site varies (§3.3 and Appendix E). The risk of contact with nonsterile surfaces is described in Aseptic Laboratory Techniques: Plating Methods. 2012 . We have not measured how often these contacts occur in routine laboratory work.
Q5. Is the drop at held-out sites due to response selection or spatial execution?
The G5 results do not separate errors in response selection from errors in disposal execution. In the supplemental annotations, 261 of 272 failures (96.0%) were labeled as incomplete disposal, including 124/131 for the loop and 137/141 for the spreader. The other 11 failures involved continued or attempted normal use (Appendix C, Table 18). Figure 16 shows the robot moving the loop toward the bin and releasing it beside the opening. In Figure 17, it holds the spreader away from the agar until the rollout ends (Appendix E.1).
G4 and G5 use the same response-augmented checkpoints and instructions, with the tool, workflow stage, and required response held fixed. Disposal succeeded in 284/300 trials at seen sites and 28/300 at held-out sites (Table 17). Changing the contact site can affect both the cue for switching and the starting pose for disposal.
W2. Why focus the hardware study on local operations, and are post-contact results available?
We did not evaluate post-contact disposal on hardware because the simulation protocol is difficult to reproduce on a physical robot. In simulation, a script induces the prohibited contact and immediately hands control back to the policy at the contact pose (Appendix E). It is difficult to reproduce both the controlled contact and the immediate handoff on hardware. We therefore focused the hardware study on laboratory operations with high success rates in simulation. Each model ran 50 trials per operation for pipette immersion, lid removal, spreader grasping, and streaking (§5.5, Table 8, and Appendix F).
W3. What bimanual coordination does the benchmark test?
Aspiration (L1-21) and dispensing (L1-23) are the only tasks in the real-world workflows represented in the benchmark that require both arms. Both use the same coordination pattern: one arm stabilizes the pipette while the other depresses its button by at least 5 mm. Across the three policies, success was 24–46% for aspiration and 8–28% for dispensing (Appendix A, Table 11 and Figures 11–12, and Appendix B, Table 13).
We have added the new experiments to the manuscript and highlighted the clarifications of task structure, contamination response, and hardware evaluation.
Follow-up on Our Response to Reviewer AWMj
Dear Reviewer,
Thank you again for your detailed review. Your questions about task diversity and complexity, the contamination-response formulation, and the interpretation of held-out-site results helped us clarify the benchmark's evaluation scope and experimental design. We would like to draw your attention to the following additional analyses and clarifications:
Task diversity: After merging shared operations among the 32 L1 task instances, we identify 23 distinct operation templates. We further describe the seven families of completion predicates they cover and their relationship to L0 atomic actions and the 12 L2 workflows (Section 3.1 and Appendix A).
Continuous workflows and cross-level evaluation: We explain how each stage inherits the grasp and placement states produced by preceding operations, and use shared completion predicates to compare the same checkpoint on isolated operations and their corresponding workflows. This allows us to track the same operation across execution contexts and characterize the composition requirements of continuous workflows (Sections 3.1 and 5.2, and Appendix C, Tables 20 and 22).
Contamination-response formulation and experimental design: We distinguish sample or agar contacts required by the workflow from prohibited contacts that invalidate the tool. Both the loop and spreader workflows normally end in disposal; prohibited contact instead requires earlier disposal before any further sample or agar contact. We selected these tools to compare disposal at the normal workflow endpoint with earlier disposal following prohibited contact. We fix the contact-injection stage so that the next normal operation remains the same as the contact site varies (Sections 3.2–3.3, Appendix E, and our responses to Q3–Q4).
Failure analysis at held-out contact sites: We provide behavioral outcome breakdowns by tool and policy. Of the 272 G5 failures, 261 (96.0%) were labeled as incomplete disposal, while 11 involved continued or attempted normal tool use. Together with the seen-versus-held-out comparisons and trajectory examples, these analyses inform our discussion of response selection and spatial execution in interpreting the results (Appendix C, Tables 17–18, and Appendix E.1).
We would be grateful to hear whether these additional analyses and clarifications address your main concerns about task diversity, workflow complexity, and contamination-response evaluation, or whether any points would benefit from further clarification. Thank you again for the time you have devoted to reviewing our work.
Response to Reviewer AWMj (1/2)
Thank you for your questions about task coverage and the contamination-response experiments.
MicroBio-Bench adds two kinds of complexity to microbiology laboratory benchmarks: composing local operations into continuous workflows and recovering from contamination during a workflow. Contamination recovery is a task in its own right: after prohibited contact, the robot must interrupt the nominal sequence and discard the tool before any further sample or agar contact. Both workflows normally end in disposal, so the challenge is not just to execute disposal but to do so earlier when the contact history calls for it (§§3.2–3.3).
MicroBio-Bench links local operations to 12 continuous workflows, using the same completion predicates to score them in both settings. The tasks require precise placement, tool actuation, sustained contact with agar, and transfer between instruments. In L2-04, for example, the robot must open the incubator, transfer the plate, and close the incubator. We score these steps in order, with each operation starting from the grasp and placement produced by the preceding one (§3.1 and Appendix A, Table 9).
In the cross-level experiment, we test the same checkpoint on an operation in isolation and within its workflow. The RDT L2 checkpoint transferred a plate into the incubator in 42% of isolated trials and completed the full L2-04 workflow in 88%. Transfer therefore succeeded in at least 88% of workflow trials. The instructions and starting conditions differ between these evaluations, and transfer in the workflow uses the state produced by earlier operations (§5.2 and Appendix C, Tables 20 and 22).
AutoBio groups tasks into three difficulty tiers and tests two-step lid operations (AutoBio: A Simulation and Benchmark for Robotic Automation in Digital Biology Laboratory). Our nesting map and shared predicates let us track the same operation across isolated and workflow evaluations using the same checkpoint. The contamination-recovery task tests a complementary capability: whether the policy can depart from the nominal sequence when continuing would risk cross-contamination.
We count 23 distinct operation templates among the 32 L1 instances after merging nine pairs shared by the loop and spreader workflows. These pairs cover five setup operations (L1-01–05 and L1-15–19), plate closure (L1-09 and L1-29), and three incubation operations (L1-10–12 and L1-30–32). We keep separate task IDs for the paired instances because the single-arm and dual-arm workflows have different action spaces (§3.1 and Appendix A, Tables 9–11).
The L1 tasks use seven predicate families covering precision placement, removal and staging, alignment and closure, positioning a held tool, actuation, contact trajectories, and disposal. The 60 task instances also include 16 L0 instances across grasping, transport, alignment, and release, plus 12 L2 workflows (§3.1 and Appendix A).
The task uses a temporal contact rule, with the permitted contacts and required response defined by the microbiology workflow. Sample contact during loop dipping and agar contact during streaking or spreading are required steps. If the working end touches the workbench or plate-support exterior, the tool must be discarded and cannot be used again during the rollout (§3.2). At the same pose, a tool may be used before prohibited contact but must be discarded afterward.
Both workflows normally end in disposal. A prohibited contact requires the policy to discard the tool earlier, before any further sample or agar contact (§§3.2–3.3). SafeManip also tracks cross-contamination over time and scores avoidance of clean contact until sanitization (SafeManip: A Property-Driven Benchmark for Temporal Safety Evaluation in Robotic Manipulation).