acceptodds
Under review as a conference paper at ICLR 2027

MicroBio-Bench: Can Vision Language Action Models Work in Microbiology Labs?

Abstract

Vision–language–action (VLA) models offer a way to automate laboratory procedures that require precise manipulation and decisions about when a tool can still be used. In microbiology, a disposable tool whose working end touches a prohibited surface must be discarded before further sample contact, even if it remains correctly positioned for use. We introduce MicroBio-Bench, a simulation environment and benchmark for evaluating how VLA models compose laboratory operations and respond when contamination changes the required action. The suite connects 58 tasks across atomic actions, local operations, and multi-operation tasks, and tracks tool contacts to determine when disposal is required. A contamination-response study tests whether policies learn disposal, retain normal tool use, and generalize to new contact sites. We evaluate three models over 12,750 simulation episodes, complemented by hardware tests. Atomic-action success averages 88.8–97.3%, yet mean multi-operation task success peaks at 26.5%. Response demonstrations yield 100% disposal success for loops and 86–92% for spreaders at seen sites, while unseen-site success falls to 0–20% and 0–18%, respectively. These findings identify workflow composition and generalization of contamination response as key challenges for VLA models in microbiology labs.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Vision-Language-Action Models, Robotic Manipulation Benchmarks, Laboratory Automation, Long-Horizon Manipulation, Contamination Response

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_AWMjOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MicroBio-Bench, a VLA simulation and evaluation benchmark for microbiology laboratory tasks. The benchmark includes 18 laboratory assets and 60 tasks organized into three hierarchical levels, along with an automated scene generation pipeline that constructs laboratory environments satisfying predefined constraints. Compared with prior work, this paper introduces contact-dependent contamination constraints, requiring robots to modify their original manipulation plans and discard contaminated items when contamination occurs.The benchmark is evaluated in both simulation and real-world settings using three models: π0.5, RDT, and π0. The evaluations examine contamination response, normal task execution capabilities, and generalization to unseen contamination scenarios. Experimental results show that existing models perform relatively well on atomic tasks but exhibit significant limitations in compositional tasks and generalization to unseen contamination scenarios.

Critical strengths
  1. The paper introduces VLA-based manipulation into microbiology laboratory environments and extends the scope of investigation to aseptic operation protocols. Compared with conventional constraints based solely on geometric positions or safety zones, contamination-induced tool invalidation rules better reflect the safety and procedural compliance requirements of robotic laboratory operations.
  2. The paper develops high-fidelity simulation models by leveraging professional CAD models and constructing dedicated collision meshes. This approach offers advantages over conventional simulation models built through direct asset importing or standard digital-twin construction pipelines.
Critical weaknesses
  1. The definition of the contamination mechanism is overly simplified, and its scientific significance and research value are insufficiently justified.

    (1) Contamination-aware evaluation is one of the central contributions of this work. However, the paper models contamination merely as a binary state triggered by a "prohibited contact" event, followed by a predefined cleanup action. Such event-driven state transitions and conditional safety rules are not unique to microbiology laboratories and can also be evaluated in general robotic manipulation scenarios. More importantly, the paper does not adequately justify what fundamentally new challenges arise from this domain-specific formulation, or whether these challenges constitute sufficiently distinct and meaningful research problems.

    (2) The core contamination-response experiments involve only two types of tools and two groups of L1 manipulation tasks, and contamination events are deliberately injected through scripts during the predefined setup stage. Corresponding evaluations are absent from the real-world experiments. These limitations further weaken the empirical support for the paper's central contribution.

    (3) The contamination-response experiments do not adequately disentangle safety-related decision-making from low-level manipulation and generalization capabilities. Failures at unseen contact locations do not necessarily indicate a lack of contamination-aware reasoning; they may instead result from limited execution capabilities when the model encounters unfamiliar spatial configurations. This ambiguity further weakens the paper's claim that the benchmark reveals a distinctive and scientifically meaningful limitation of existing VLA models.

  2. Although the paper introduces a new application scenario in microbiology laboratories, its overall novelty remains limited: The proposed benchmark lacks clear differentiation from existing laboratory benchmarks. While Table 1 highlights advantages in terms of task quantity and supported features, the overall task diversity appears limited. Although the paper claims to provide 60 tasks, these tasks are organized into three hierarchical levels, where L2 tasks are compositions of L1 tasks, while L0 tasks consist only of simple atomic operations. This raises concerns about the effective number of genuinely distinct tasks. Furthermore, bimanual manipulation appears in only two L1 tasks (L1-21 and L1-23), which essentially share the same underlying manipulation logic, differing only in the indirectly manipulated object (container). Consequently, the paper's claimed support for bimanual manipulation is not sufficiently substantiated.

Questions for authors
  1. Could the authors further clarify what additional capabilities or advantages MicroBio-Bench offers over existing laboratory benchmarks, beyond its domain-specific assets and contamination-response tasks, particularly regarding task diversity and complexity?
  2. Among the 60 tasks, how many genuinely represent distinct manipulation skills?
  3. What fundamentally distinguishes the proposed "contamination problem" from a general robotic safety constraint that requires a robot to terminate its original action and execute an alternative action after contacting a particular region?
  4. Why is the contamination-response evaluation limited to only two types of tools? Can the authors provide evidence that such contamination scenarios occur frequently enough in realistic laboratory workflows to justify their broader significance?
  5. Does the performance degradation observed in unseen scenarios reflect a failure of safety-related reasoning, or is it primarily attributable to limitations in spatial generalization capabilities?
3 comments
AuthorsOpenReview ↗

Response to Reviewer AWMj (1/2)

Thank you for your questions about task coverage and the contamination-response experiments.

Q1. What task complexity does MicroBio-Bench add to existing laboratory benchmarks?

MicroBio-Bench adds two kinds of complexity to microbiology laboratory benchmarks: composing local operations into continuous workflows and recovering from contamination during a workflow. Contamination recovery is a task in its own right: after prohibited contact, the robot must interrupt the nominal sequence and discard the tool before any further sample or agar contact. Both workflows normally end in disposal, so the challenge is not just to execute disposal but to do so earlier when the contact history calls for it (§§3.2–3.3).

MicroBio-Bench links local operations to 12 continuous workflows, using the same completion predicates to score them in both settings. The tasks require precise placement, tool actuation, sustained contact with agar, and transfer between instruments. In L2-04, for example, the robot must open the incubator, transfer the plate, and close the incubator. We score these steps in order, with each operation starting from the grasp and placement produced by the preceding one (§3.1 and Appendix A, Table 9).

In the cross-level experiment, we test the same checkpoint on an operation in isolation and within its workflow. The RDT L2 checkpoint transferred a plate into the incubator in 42% of isolated trials and completed the full L2-04 workflow in 88%. Transfer therefore succeeded in at least 88% of workflow trials. The instructions and starting conditions differ between these evaluations, and transfer in the workflow uses the state produced by earlier operations (§5.2 and Appendix C, Tables 20 and 22).

AutoBio groups tasks into three difficulty tiers and tests two-step lid operations (AutoBio: A Simulation and Benchmark for Robotic Automation in Digital Biology Laboratory). Our nesting map and shared predicates let us track the same operation across isolated and workflow evaluations using the same checkpoint. The contamination-recovery task tests a complementary capability: whether the policy can depart from the nominal sequence when continuing would risk cross-contamination.

Q2. How many distinct manipulation operations are represented?

We count 23 distinct operation templates among the 32 L1 instances after merging nine pairs shared by the loop and spreader workflows. These pairs cover five setup operations (L1-01–05 and L1-15–19), plate closure (L1-09 and L1-29), and three incubation operations (L1-10–12 and L1-30–32). We keep separate task IDs for the paired instances because the single-arm and dual-arm workflows have different action spaces (§3.1 and Appendix A, Tables 9–11).

The L1 tasks use seven predicate families covering precision placement, removal and staging, alignment and closure, positioning a held tool, actuation, contact trajectories, and disposal. The 60 task instances also include 16 L0 instances across grasping, transport, alignment, and release, plus 12 L2 workflows (§3.1 and Appendix A).

Q3. What distinguishes this contamination task from a generic contact-triggered safety rule?

The task uses a temporal contact rule, with the permitted contacts and required response defined by the microbiology workflow. Sample contact during loop dipping and agar contact during streaking or spreading are required steps. If the working end touches the workbench or plate-support exterior, the tool must be discarded and cannot be used again during the rollout (§3.2). At the same pose, a tool may be used before prohibited contact but must be discarded afterward.

Both workflows normally end in disposal. A prohibited contact requires the policy to discard the tool earlier, before any further sample or agar contact (§§3.2–3.3). SafeManip also tracks cross-contamination over time and scores avoidance of clean contact until sanitization (SafeManip: A Property-Driven Benchmark for Temporal Safety Evaluation in Robotic Manipulation).

AuthorsOpenReview ↗

Response to Reviewer AWMj (2/2)

Q4 and W1. Why evaluate these two tools at fixed stages, and how common are such contacts?

We chose the loop and spreader because both are discarded at the end of their normal workflow. This lets us compare the same disposal skill at the scheduled endpoint and immediately after prohibited contact.

  • Loop (L1-06/07/08 in L2-03). Contact is injected after dipping, before streaking. The response study ends at tool disposal.
  • Spreader (L1-25/26/27 in L2-10). Contact is injected after pickup, before the first agar contact.

We inject contact at a fixed stage so that the next normal operation stays the same as the contact site varies (§3.3 and Appendix E). The risk of contact with nonsterile surfaces is described in Aseptic Laboratory Techniques: Plating Methods. 2012 . We have not measured how often these contacts occur in routine laboratory work.

Q5. Is the drop at held-out sites due to response selection or spatial execution?

The G5 results do not separate errors in response selection from errors in disposal execution. In the supplemental annotations, 261 of 272 failures (96.0%) were labeled as incomplete disposal, including 124/131 for the loop and 137/141 for the spreader. The other 11 failures involved continued or attempted normal use (Appendix C, Table 18). Figure 16 shows the robot moving the loop toward the bin and releasing it beside the opening. In Figure 17, it holds the spreader away from the agar until the rollout ends (Appendix E.1).

G4 and G5 use the same response-augmented checkpoints and instructions, with the tool, workflow stage, and required response held fixed. Disposal succeeded in 284/300 trials at seen sites and 28/300 at held-out sites (Table 17). Changing the contact site can affect both the cue for switching and the starting pose for disposal.

W2. Why focus the hardware study on local operations, and are post-contact results available?

We did not evaluate post-contact disposal on hardware because the simulation protocol is difficult to reproduce on a physical robot. In simulation, a script induces the prohibited contact and immediately hands control back to the policy at the contact pose (Appendix E). It is difficult to reproduce both the controlled contact and the immediate handoff on hardware. We therefore focused the hardware study on laboratory operations with high success rates in simulation. Each model ran 50 trials per operation for pipette immersion, lid removal, spreader grasping, and streaking (§5.5, Table 8, and Appendix F).

W3. What bimanual coordination does the benchmark test?

Aspiration (L1-21) and dispensing (L1-23) are the only tasks in the real-world workflows represented in the benchmark that require both arms. Both use the same coordination pattern: one arm stabilizes the pipette while the other depresses its button by at least 5 mm. Across the three policies, success was 24–46% for aspiration and 8–28% for dispensing (Appendix A, Table 11 and Figures 11–12, and Appendix B, Table 13).

We have added the new experiments to the manuscript and highlighted the clarifications of task structure, contamination response, and hardware evaluation.

AuthorsOpenReview ↗

Follow-up on Our Response to Reviewer AWMj

Dear Reviewer,

Thank you again for your detailed review. Your questions about task diversity and complexity, the contamination-response formulation, and the interpretation of held-out-site results helped us clarify the benchmark's evaluation scope and experimental design. We would like to draw your attention to the following additional analyses and clarifications:

Task diversity: After merging shared operations among the 32 L1 task instances, we identify 23 distinct operation templates. We further describe the seven families of completion predicates they cover and their relationship to L0 atomic actions and the 12 L2 workflows (Section 3.1 and Appendix A).

Continuous workflows and cross-level evaluation: We explain how each stage inherits the grasp and placement states produced by preceding operations, and use shared completion predicates to compare the same checkpoint on isolated operations and their corresponding workflows. This allows us to track the same operation across execution contexts and characterize the composition requirements of continuous workflows (Sections 3.1 and 5.2, and Appendix C, Tables 20 and 22).

Contamination-response formulation and experimental design: We distinguish sample or agar contacts required by the workflow from prohibited contacts that invalidate the tool. Both the loop and spreader workflows normally end in disposal; prohibited contact instead requires earlier disposal before any further sample or agar contact. We selected these tools to compare disposal at the normal workflow endpoint with earlier disposal following prohibited contact. We fix the contact-injection stage so that the next normal operation remains the same as the contact site varies (Sections 3.2–3.3, Appendix E, and our responses to Q3–Q4).

Failure analysis at held-out contact sites: We provide behavioral outcome breakdowns by tool and policy. Of the 272 G5 failures, 261 (96.0%) were labeled as incomplete disposal, while 11 involved continued or attempted normal tool use. Together with the seen-versus-held-out comparisons and trajectory examples, these analyses inform our discussion of response selection and spatial execution in interpreting the results (Appendix C, Tables 17–18, and Appendix E.1).

We would be grateful to hear whether these additional analyses and clarifications address your main concerns about task diversity, workflow complexity, and contamination-response evaluation, or whether any points would benefit from further clarification. Thank you again for the time you have devoted to reviewing our work.

Reviewer_diQmOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether VLA models can compose precise operations into reliable microbiology experiment, and whether they change their actions when contamination makes tool use invalid. MicroBio-Bench provides a lot of assets and nested tasks across atomic actions, together with an automated scene-construction pipeline. The paper assesses the crucial operational constraints in microbiological laboratories, the aseptic constraints. A contamination-response study compares policies trained with and without response demonstrations. Evaluating π0.5, RDT and π0, the authors find that atomic-action success is high but multi-operation task success is low, and that response demonstrations enable disposal at seen contact sites but generalize poorly to held-out sites and may impair normal performance.

Critical strengths
  1. Microbiology assets for the community. The paper contributes expert-built MJCF assets for microbiology scenes, covering tools, vessels and instruments, with separate visual and collision meshes and working-surface annotations. If the full benchmark is released, it would be a useful resource for the community.
  2. An important aseptic constraint. The paper considers a constraint that is important to microbiology laboratories: after prohibited working-end contact, a disposable tool must be discarded before further contact. Existing laboratory benchmarks leave aseptic tool use untested . This constraint is important for bringing robots into microbiology laboratories.
  3. Clear presentation. The paper is clearly written, and its figures and tables are well designed and easy to read.
Critical weaknesses
  1. Limited novelty, and the contamination setting is not rigorous. Compared with AutoBio, which already models biological instruments in a laboratory benchmark, the paper offers new assets and tasks but little methodological novelty. For the most important aseptic setting, it moves the working end to a predefined contamination point and then returns control to the model, while π0 and π0.5 receive images only from the current timestep. Training with additional response demonstrations therefore seems to test how an imitation policy generalizes to new states, rather than whether it understands that a tool becomes ineligible after contact.
  2. Response training degrades normal execution, without analysis. After response training, normal-workflow SR drops, for example from 94% to 24% for RDT on the spreader and from 10% to 0% for π0 on the loop. The paper does not say whether these failures are caused by disposal being triggered without contamination, which makes the results hard to interpret as response learning.
  3. Fixed evaluation scenes. The paper does not describe any randomization of initial object poses. It is therefore unclear where the variation across trials comes from, which directly affects the meaning of the success rates and of the conclusions about generalization.
  4. The automated scene construction is not evaluated. The automated scene-construction pipeline is listed as a contribution, but the paper reports no generation success rate and no comparison with manually built scenes, and the validated scenes then remain fixed for evaluation. This does not match the claimed contribution.
  5. Lack of baselines. Only three VLA models, π0.5, RDT and π0, are evaluated . There are no imitation-learning baselines such as Diffusion Policy and ACT, and no policies with memory or hierarchical structure. (This type of model is better at handling the response issues after instrument contamination.)
Questions for authors
  1. Are the initial states randomized during evaluation, and where does the variation across the trials of each task come from? If not randomized, the benchmark may mainly measure how well the policies fit the demonstrations rather than their capabilities.
  2. Can the authors add Diffusion Policy, ACT and policies with memory of past observations as baselines? For the aseptic task in particular, whether a policy with memory can solve it seems to me the more important question to test.
  3. The current contamination setting seems to test how imitation policies generalize to new states rather than in reflecting the actual requirement of a microbiology laboratory. Can a policy with memory handle the case that better matches the motivation, in which the tool returns to its normal position after contact before control is handed back?
6 comments
AuthorsOpenReview ↗

Response to Reviewer diQm (1/2)

Thank you for your constructive suggestions on baseline comparisons and scene validation.

Q1. Are initial states randomized, and what varies between trials?

Yes. At each standard L1 reset, we sample task-relevant object positions within a 20 mm × 20 mm region around the nominal layout. Each L2 rollout samples its initial layout once and carries the physical state across stages. Policy action sampling and the contact site in response groups also vary between trials. Section 4.2 describes how the layouts are validated.

Q2. How do ACT, Diffusion Policy, and a policy with visual history perform?

We added ACT and the hierarchical Agentic π0.5\pi_{0.5} baseline with Temporal Visual Memory, described in W4. At held-out contact sites, ACT had 0% disposal SR for both tools. Agentic π0.5\pi_{0.5} reached 26% for the loop and 0% for the spreader. At seen sites, both methods reached 92–100%. All values below are disposal SR for the three contact conditions in Table 2.

PolicyToolG2G4 (seen)G5 (held out)
ACTLoop44%92%0%
ACTSpreader0%100%0%
Agentic π0.5\pi_{0.5}Loop24%100%26%
Agentic π0.5\pi_{0.5}Spreader0%92%0%
Standalone π0.5\pi_{0.5}Loop24%100%20%
Standalone π0.5\pi_{0.5}Spreader0%92%0%

G2 uses standard-only checkpoints, and G4–G5 use response-augmented checkpoints. Success requires disposal after prohibited contact and before any further sample or agar contact. G2 and G4 use the same four sites. The ACT and agentic results are reported in Appendix C, Table 19, and the standalone π0.5\pi_{0.5} results in Table 17.

Our diffusion-based VLA baseline, RDT, uses a two-frame image history (Appendix H). We have not evaluated a separate Diffusion Policy baseline.

Q3. Does visual history help when the tool returns to a normal pose before handoff?

We have not evaluated a return to the normal pose before handoff. Control resumes at the contact pose under the protocol in Appendix E. The agentic supervisor receives current and past images, as described in Appendix C (“Additional baselines”).

W1. What does MicroBio-Bench add to workflow evaluation beyond AutoBio?

We compare the same operation in isolation and within a workflow. MicroBio-Bench's nesting map links local operations to 12 workflows, with shared completion predicates and ordered progress scoring (§3.1). The cross-level experiment evaluates L1- and L2-trained checkpoints on 14 isolated operations and three L2 workflows (Appendix C, Tables 20–21). AutoBio organizes tasks into three difficulty tiers and tests two-step lid operations by switching prompts after the first operation (AutoBio: A Simulation and Benchmark for Robotic Automation in Digital Biology Laboratory).

For example, the same RDT L2 checkpoint transferred a plate into the incubator in 42% of isolated trials and completed the full L2-04 workflow in 88% of trials. Since the full workflow includes transfer, its transfer success rate was at least 88%. The instructions and starting conditions differ between these evaluations, and transfer in the workflow uses the state produced by earlier operations (§5.2 and Appendix C, Tables 20 and 22).

W2. How many normal-workflow failures were caused by premature disposal?

The stage-level annotations do not identify premature disposal separately, so we cannot give a reliable count. G1 and G3 both use the standard instruction for normal execution, with G3 adding response demonstrations during training. Each row compares 50 trials per group (Table 2 and Appendix C, Tables 17–18).

Policy and toolG1 successG3 successSupplemental G3 failure annotations
RDT, spreader94%24%All 38 failures were labeled as spreading failures
π0\pi_0, loop10%0%48 of 50 failures missed sample dipping. The other two were labeled as streaking or disposal failures
AuthorsOpenReview ↗

Response to Reviewer diQm (2/2)

W3. How does automated scene construction compare with manual construction, and why fix the validated scenes?

In the new scene-construction experiment, automated construction passed 170/180 attempts (94.4%), compared with 100/180 (55.6%) for manual construction on the same 60 scene specifications. It was also faster on average at all three levels (Appendix G.5, Figure 20).

Three operators familiar with the editor each independently built one scene for every specification, giving 180 manual attempts. They selected assets, adjusted object poses and support relationships, and exported the scene. The automated pipeline ran three times per specification, also giving 180 attempts. Both methods used the same scene instructions, asset library, and validation checks. A scene passed only if it satisfied both the physical layout and robot feasibility checks. We timed manual construction from reading the instruction through scene export and validation.

LevelSpecificationsManual time (s)Automated time (s)Manual passesAutomated passes
L016253.9 ± 52.936.22 ± 8.5330/48 (62.5%)48/48 (100%)
L132194.3 ± 29.145.96 ± 12.1362/96 (64.6%)92/96 (95.8%)
L212280.7 ± 34.453.36 ± 11.298/36 (22.2%)30/36 (83.3%)

Times are mean ± SD. For automated construction, we first average the three runs for each specification, including failed attempts, then compute the mean and SD across specifications. Pass rates count individual construction attempts. Each failed manual attempt was assigned one primary cause. Robot feasibility accounted for 66 of 80 failures (82.5%), including 54 unreachable targets and 12 cases of insufficient approach clearance. Improper support and object interpenetration accounted for seven failures each (Figure 21).

We keep the validated manifests fixed so that every policy uses the same nominal layout, camera configuration, and scoring predicates. Object positions still vary within that layout at standard L1 resets. Section 4.2 and Figure 2 describe the validation checks.

W4. How does the hierarchical policy switch to disposal, and how well does it perform?

The hierarchical baseline pairs a visual supervisor (Qwen3.8-Max-0902) with a frozen π0.5\pi_{0.5} controller. At a policy query about once per simulated second, the supervisor receives the current image and eight past frames sampled uniformly from the available history. It returns KEEP_TASK to retain the original instruction or DISCARD_TOOL when it detects prohibited contact. Once selected, the disposal instruction remains active until the rollout ends. The frozen controller receives the active instruction, current camera images, and robot state, and generates both normal and disposal actions.

For the loop, disposal SR at held-out sites was 26% with supervision and 20% without it, a difference of 6 percentage points. Both controllers had 0% SR for the spreader at held-out sites. At seen sites, both reached 100% for the loop and 92% for the spreader (Appendix C, Tables 17 and 19). Q2 gives the results for all three contact conditions.

We have added the new experiments to the manuscript and highlighted the clarifications of scene validation, the five-group protocol, and cross-level evaluation.

AuthorsOpenReview ↗

Follow-up to Reviewer diQm (Q2): Visual history in RDT

We compared two visual-history settings for RDT on the inoculating-loop task. The baseline uses two consecutive frames 0.04 s apart. We trained and evaluated the new version with four frames spaced 0.24 s apart, covering 0.72 s. Each cell below gives the success rate over 50 trials under the conditions in Table 2.

RDT visual historyG1 (normal)G2 (contact)G3 (normal)G4 (seen contact)G5 (held-out contact)
Two frames (Table 17)60%56%58%100%18%
Four frames (new)82%64%56%100%0%

With standard-only training, normal-workflow success rose by 22 percentage points (G1), and disposal success rose by 8 points (G2). After response training, normal-workflow success was similar (G3). Both models completed disposal in all 50 trials at seen sites (G4), but at held-out sites (G5), successes fell from 9/50 to 0/50. In this experiment, the longer-window RDT performed better on the standard workflow after training on standard demonstrations alone, but generalized more poorly to held-out contact sites after response training.

Reviewer_M8NpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces a microbiology manipulation benchmark comprising 18 laboratory assets and 60 tasks organized into atomic actions, local operations, and multi-operation workflows. It evaluates three VLA models and introduces a contamination-response protocol in which prohibited contact requires a disposable tool to be discarded before further sample contact.

The most compelling contribution is making tool eligibility depend on contact history rather than geometric configuration alone. The benchmark also offers useful infrastructure for studying the transition from isolated operations to continuous workflows. However, the contamination experiments currently conflate several distinct capabilities: observing contamination, remembering contact history, selecting the appropriate response, and physically completing disposal. Consequently, the results establish substantial difficulty under the proposed protocol more convincingly than they identify the underlying cause of that difficulty.

Critical strengths

Meaningful domain-specific task formulation. Contact-dependent tool eligibility adds a substantive requirement beyond conventional pick-and-place manipulation. Distinguishing the working end from the shaft makes the constraints concrete.

Useful hierarchical evaluation structure. Shared operations across L0, L1, and L2, together with fixed-checkpoint cross-level evaluations, provide more diagnostic information than end-to-end success rates alone.

Detailed reporting. The paper provides per-task results and Wilson intervals. Reporting medians and the concentration of successful L2 episodes in a small subset of plate-handling tasks appropriately qualifies the aggregate results.

Evaluation of normal-workflow retention. The contamination study examines whether response training damages ordinary execution. The reduction in RDT’s spreader-workflow success from 94% to 24% is particularly informative.

Physical experiments. Four hardware operations, with 50 trials per model–task pair, provide useful evidence that the manipulation challenges are relevant beyond simulation.

Critical weaknesses

Section 3.2 explicitly allows contamination to change tool eligibility without changing its appearance. Appendix H states that π₀ and π₀.₅ receive current images and robot state, whereas RDT receives only a two-frame image history. Contaminating contact is introduced by a script, after which control returns to the policy. This raises a fundamental question: what evidence of the contact is actually available when the policy must act? If contaminated and uncontaminated tools can have identical current images and robot states, a memoryless policy cannot reliably distinguish them. Conversely, if control is returned while the tool remains at a distinctive contamination location, a policy may learn a location-conditioned disposal behavior without representing contact history.

Figures 15 and 16 provide direct examples of this ambiguity. In the held-out loop example, the robot moves toward the waste container but releases the tool to the right of its opening. In the spreader example, the robot stops using the tool on the agar but retains it outside the container.

The five-group design leaves instruction and spatial shortcuts insufficiently controlled.

The authors appropriately acknowledge that L1 checkpoints receive full L2 instructions that were unseen during L1 training. Consequently, zero L1-to-L2 success jointly reflects instruction generalization, stage selection, and operation composition.

Questions for authors

Which frames are available to each policy during scripted contamination and at control handoff?

What fraction of G5 failures occur after normal tool use has already stopped?

How well do L1 policies compose when given oracle stage instructions?

What training-seed, scene-randomization, and demonstration-sampling variations were evaluated?

5 comments
AuthorsOpenReview ↗

Response to Reviewer M8Np (1/2)

Thank you for your valuable questions about the handoff protocol and workflow composition.

Q1. What does each policy observe during injection and at handoff?

A script controls the robot during contact injection and returns control to the policy at the contact pose. At the first policy query, π0\pi_0 and π0.5\pi_{0.5} receive current camera images, while RDT receives a two-frame image history. We have not verified whether RDT's earlier frame was captured before or after contact. All three policies also receive the current robot state and language instruction. The loop scene uses an overview and one wrist view, and the spreader scene uses an overview and both wrist views (Appendices E and H).

In the added Agentic π0.5\pi_{0.5} baseline, the frozen controller uses the same inputs as standalone π0.5\pi_{0.5}. Its visual supervisor receives the current image and eight past frames at each check (Appendix C, “Additional baselines”). We sample these past frames uniformly from the available history.

W1. How do the five groups handle instruction and pose cues?

The policy can use the contact pose as a cue at handoff. G1 and G3 use the standard instruction for normal execution. G2, G4, and G5 add the same conditional disposal rule to that instruction. The five groups in Table 2 allow the following comparisons.

ComparisonHeld fixedVaried
G1 vs. G2Standard-only checkpointNormal execution and standard instruction vs. scripted contact and an added conditional disposal rule
G1 vs. G3Normal execution and standard instructionAddition of response demonstrations
G2 vs. G4Same four contact sites and conditional instructionAddition of response demonstrations
G4 vs. G5Response-augmented checkpoint and conditional instructionSeen vs. held-out contact sites

Within each backbone, both training conditions start from the same pretrained weights and standard demonstrations, and we evaluate both checkpoints at 30k steps (Appendix E). A new contact site can change both the visible cue and the starting pose for disposal, so these comparisons do not isolate the role of contact history.

Q2. How many G5 failures occurred after normal tool use stopped?

Of the 272 G5 failures, 261 (96.0%) were labeled as incomplete disposal. These outcome labels provide an estimate of how often normal tool use stopped or remained interrupted after handoff. The other 11 failures involved continued or attempted normal use.

ToolPolicyContinued or attempted useIncomplete disposalSuccessful disposalIncomplete disposal / failures
Loopπ0.5\pi_{0.5}1391039/40 (97.5%)
LoopRDT635935/41 (85.4%)
Loopπ0\pi_0050050/50 (100%)
Spreaderπ0.5\pi_{0.5}446046/50 (92.0%)
SpreaderRDT041941/41 (100%)
Spreaderπ0\pi_0050050/50 (100%)
Both toolsAll three1126128261/272 (96.0%)

There were 50 G5 trials per model and tool. The counts above are converted from the outcome percentages in Appendix C, Table 18. The first three outcome columns include all trials, and the last column uses failed trials as its denominator. Disposal SRs are also reported in Table 17.

AuthorsOpenReview ↗

Response to Reviewer M8Np (2/2)

Q3. How do L1 checkpoints compose workflows with oracle stage instructions?

None of the three L1 checkpoints completed L2-02, L2-03, or L2-04 under either full-workflow or oracle stage instructions (0% SR). The submitted baseline keeps the full L2 instruction fixed throughout the rollout. The new oracle experiment uses the same checkpoints and provides the L1 instruction for the current stage. The instruction switches when the stage's completion predicate is satisfied, and the physical state carries over between stages.

L2 taskπ0.5\pi_{0.5} PG, full → oracleRDT PG, full → oracleπ0\pi_0 PG, full → oracle
L2-02 Transfer and open plate3% → 14%34% → 36%49% → 46%
L2-03 Streak inoculation0% → 24.5%0% → 0%0% → 6%
L2-04 Incubation handoff0% → 11.33%19.3% → 22.67%0% → 1.33%

Ordered progress (PG) is the mean percentage of consecutive milestones completed from the start (§3.1). The milestones are listed in Appendix A, Table 9. The full-instruction results come from 50 trials per checkpoint and task (Appendix C, Table 21). Figure 14 reports the oracle stage-instruction results.

Oracle instructions increased π0.5\pi_{0.5}'s progress by 11–24.5 percentage points. On L2-02, transfer completion rose from 6% to 28% for π0.5\pi_{0.5} and from 68% to 72% for RDT. It fell from 98% to 92% for π0\pi_0. In this two-stage task, transfer completion equals twice PG minus the full-workflow success rate (Table 5b and Appendix C, Eq. 4). Every oracle trial that completed transfer then failed at lid removal.

Q4. What varies across evaluation trials, training seeds, and demonstration pools?

  • Evaluation. At each standard L1 reset, we sample task-relevant object positions within a 20 mm × 20 mm region around the validated layout. Each L2 rollout samples the initial layout once and carries the physical state across stages. Policy action sampling and the contact site in response groups also vary between trials. The submission reports 50 episodes per model and task in the standard evaluations and 50 per model, tool, and group in the response study (§5.1).
  • Demonstrations. Within each backbone, we add 100 response demonstrations per tool sequence to the same standard training pool (§3.3 and Appendix E). Contact sites in those demonstrations are sampled at the four plate-rack corners and on a uniform grid within the corner regions.
  • Seeds. We report one checkpoint per backbone and condition, with distinct seeds across backbones. The Wilson intervals quantify uncertainty in the episode-level success rates of those checkpoints (Appendix B, Eq. 3). We have not measured sensitivity to training seeds or resampled demonstration pools.

We have added the new experiments to the manuscript and highlighted the clarifications of policy observations, the contact-response protocol, and episode-level uncertainty.

AuthorsOpenReview ↗

Follow-up on Our Response to Reviewer M8Np

Dear Reviewer M8Np,

Thank you again for your detailed review. Your questions about policy observations at handoff, failure outcomes after handoff, and the role of stage instructions in workflow composition helped us further develop our experimental analysis. We would like to draw your attention to the following additions and clarifications:

Policy observations and experimental controls: We specify the camera views and image history available to each policy at handoff, and summarize the instruction, training, and contact-site controls across G1–G5 (Table 2 and Appendices E and H).

G5 failure analysis: We provide a breakdown by tool and policy. Of the 272 failures, 261 (96.0%) were labeled as incomplete disposal, while 11 involved continued or attempted normal tool use, directly addressing your question about outcomes after normal tool use stops (Appendix C, Table 18).

Oracle stage instructions: We evaluate the same L1 checkpoints with the correct instruction supplied at each stage. For π0.5\pi_{0.5}, ordered progress increases by 11–24.5 percentage points, while full-workflow success remains 0% across the three policies and three tested workflows. These results further characterize the effect of stage instructions on workflow execution (Appendix C, Figure 14).

Evaluation and training details: We clarify initial-position randomization, demonstration sampling, checkpoint selection, and the statistical interpretation of the reported Wilson intervals (our response to Q4 and Appendix B).

We would be grateful to hear whether these additions and clarifications address your main concerns, or whether any points would benefit from further clarification. Thank you again for the time you have devoted to reviewing our work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Request for Reviewer Engagement and Summary of Rebuttal Updates

Dear Area Chair,

Thank you for overseeing the review of MicroBio-Bench. As of this message, none of the three reviewers has provided follow-up feedback on our rebuttal. We would appreciate your help in encouraging them to engage with our responses and share any remaining concerns during the discussion period.

To facilitate this discussion, we would like to briefly highlight the additions and clarifications addressing each reviewer's comments:

Regarding M8Np: We clarified the policy observations at handoff and the controls across the five experimental groups. We added a detailed G5 failure breakdown: 261 of 272 failures (96.0%) were labeled as incomplete disposal, directly addressing the reviewer's question about outcomes after normal tool use stops. We also conducted the requested oracle stage-instruction experiment. Providing the correct stage instructions increased π0.5\pi_{0.5}'s ordered progress by 11–24.5 percentage points, while full-workflow success remained 0% across the three policies and three tested workflows (Appendix C, Table 18 and Figure 14).

Regarding diQm: We added ACT and Agentic π0.5\pi_{0.5} with visual history to address the request for additional baselines (Appendix C, Table 19). We also evaluated automated versus manual scene construction on the same 60 specifications. Automated construction passed 170/180 attempts (94.4%), compared with 100/180 (55.6%) for manual construction, and was faster at all three task levels (Appendix G.5, Figures 20–21). Our response further clarifies the initial-state randomization and the comparisons between isolated operations and continuous workflows.

Regarding AWMj: We clarified task diversity by identifying 23 distinct operation templates among the 32 L1 instances, and explained how these operations are composed into 12 continuous workflows. We also addressed the rationale for the contamination protocol, the scope of the bimanual and hardware evaluations, and the interpretation of held-out failures. In particular, our response distinguishes the observed performance drop from attributing it specifically to contamination-aware reasoning.

We would be grateful if you could encourage the reviewers to indicate whether these additions address their concerns and identify any points that still require clarification. We hope to use the discussion period to resolve outstanding questions and remain available to provide further explanations.

Thank you again for your time and support.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.