acceptodds
Withdrawn from ICLR 2027

Beyond Motion Execution: Benchmarking Embodied Agents on Wet-Lab Operations and Procedures

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Completing a manipulation sequence does not guarantee wet-lab task success: an agent may still spill a reagent, miss a target mass, or overshoot a titration endpoint. We therefore introduce a 20-task dual-arm simulation benchmark, called WET-OP Bench, which couples manipulation with evolving experimental states. Tasks are organized around laboratory operations composed of shared atomic manipulation skills, spanning basic operations and experimental procedures of varying horizons. Task-specific demonstrations are provided for fourteen tasks, while six are held out to assess zero-shot generalization to new operation compositions and experimental conditions. For demonstrated tasks, controlled variations in task configuration, background, and instruction enable systematic evaluation of within-task generalization. The simulator couples manipulation with dynamic liquid transfer, quantitative instrument feedback, and procedure-conditioned appearance changes; the same underlying states drive observable feedback and success checks for containment, quantitative targets, and reaction outcomes. Reusable interaction graphs implement these state–action couplings across tasks, while Lab2Sim, an agentic asset-compilation pipeline, constructs specification-grounded, validated manipulation-ready assets. Finally, we conduct a capability-oriented evaluation of representative vision-language-action models and harness-based agents under controlled within-task and zero-shot settings, combining task success rates with skill-aligned stage scores for partial-progress tracking and failure diagnosis. The project website is available at https://beyond-motion-execution.pages.dev.

Primary area:
datasets and benchmarks
Keywords:
Wet-Lab Manipulation, Dual-Arm Manipulation, Simulation Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_5zmEOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces WET-OP Bench, a simulated bimanual wet-lab benchmark with 20 tasks. The tasks are grouped into four operation families: Handling & Placement (HP), Transfer & Mixing (TM), Instrument Operation (IO), and Metering & Dosing (MD). They are also split into two scopes, basic operations and full experimental workflows. Each task is annotated with its horizon, precision requirement, whether it has a quantitative target (Q), and whether it requires observing a reaction (R). 14 tasks come with 500 demonstrations each, and the remaining 6 are held out for zero-shot evaluation.

The benchmark is built on a set of reusable interaction graphs implemented in OmniGraph. These track physical state such as liquid transfer and spillage (via PBD particles), event-triggered appearance changes, and balance readings, and the same state is used both for policy observations and for success checking. Assets are generated with Lab2Sim, an agentic asset-generation workflow.

The paper evaluates two systems: a multi-task fine-tuned π0.5, and a harness agent in which GPT-6-Astra handles planning and calls the same frozen π0.5. In the Mixed setting, both reach a success rate of about 25% with stage scores above 55. On the zero-shot tasks, π0.5 succeeds in only 1 of 120 episodes.

Critical strengths

This paper brings several wet-lab capabilities together in a single evaluation suite. These include bimanual coordination, instrument readings the policy can observe, simulated liquid behavior, and task termination that depends on the current reaction state. Combining them in one benchmark is a useful contribution to the community.

Critical weaknesses

Weakness 1: The evaluation is thin. It covers only one policy family: the fine-tuned π0.5, and a harness agent that wraps the same π0.5. In addition, zero-shot results are reported only for π0.5. The harness agent is left out.

Weakness 2: The validity and feasibility of the tasks themselves are questionable.The construction details is not enough. For example, it does not explain how the demonstrations were generated. It is also unclear whether R-type tasks genuinely require observing the reaction.

Questions for authors
  1. How were the demonstrations generated(need more details)
  2. What exactly are the scoring criteria? How are stage scores defined and weighted,...etc
  3. Could the authors state the paper's positioning more clearly? A side-by-side comparison table against LabUtopia, AutoBio, and Labimus may help.
Reviewer_c1HbOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces WET-OP Bench, a simulation benchmark designed to evaluate both final experimental outcomes and procedural completeness for dual-arm robotic manipulation in chemistry and biology laboratories. The benchmark comprises 20 dual-arm tasks. Among them, 14 tasks are accompanied by 500 expert demonstrations each to evaluate in-task generalization across variations in layout, background, and language instructions. The remaining 6 tasks assess zero-shot transfer to novel procedural compositions and unseen experimental conditions without task-specific demonstrations.

The simulation environment realistically models liquid transfer and conservation, digital scale mass readings, and procedural visual changes in reagent appearance, utilizing identical underlying states for both policy observations and evaluation rubrics. Performance is measured using both final success rates and weighted step completion scores that capture intermediate progress.

The authors evaluate two systems: a fine-tuned π0.5\pi_{0.5} baseline and a harness-based system where GPT-6-Astra orchestrates the same π0.5\pi_{0.5} policy. Under in-task generalization conditions, both models achieve step completion scores above 55/100, yet their final success rates remain low at 25.0% and 26.4%, respectively. Furthermore, on the 6 out-of-distribution (OOD) tasks without demonstrations, π0.5\pi_{0.5} succeeds only once across 120 evaluation trials. These empirical results reveal a substantial gap between partial step execution and full task completion, while underscoring the limited transfer capabilities of current VLA systems in OOD laboratory settings.

Critical strengths

Outcome-grounded and physics-driven evaluation: The benchmark accurately models realistic physical phenomena, such as liquid transfer and conservation, quantitative scale readouts, and visual transitions resulting from chemical reactions. Importantly, by applying the same physical ground truth to both policy observations and evaluation criteria, the benchmark effectively distinguishes whether a robot is merely mimicking plausible motions or genuinely achieving the intended physical outcomes (e.g., target mass or chemical reactions). Given that valid experimental results take precedence over superficial motion execution in wet-lab automation, building such an outcome- and procedure-grounded evaluation environment is a significant and practical contribution. *(However, as detailed in the weaknesses below, the lack of fine-grained diagnostic metrics to fully evaluate these simulation stages remains a limitation.)

Critical weaknesses

Inability to isolate failure modes and bottlenecks: Relying solely on the proposed final success rates and aggregate step scores makes it difficult to diagnose which specific capability deficiencies cause task failures. For instance, in target mass measurement tasks, average step scores and motion-level success rates fail to isolate whether the bottleneck stems from visual digit recognition (OCR), procedural reasoning, or fine-grained liquid dispensing control. Without such diagnostic granularity, the practical utility of designing detailed intermediate simulation stages is diminished.

Questionable interpretation of OOD transfer and limited model diversity: While the paper claims to provide a benchmark for capability-wise diagnosis and OOD transfer analysis, the empirical evidence is insufficient to fully support these claims:

  1. Conflating in-distribution failure with OOD transfer failure: Only a single quantitative task is included in the demonstrated set, and the model struggles even on the in-distribution target value (0.20g). Consequently, attributing the failure on the novel 0.12g target to a lack of OOD generalization is unconvincing; it is more accurately characterized as an in-distribution execution failure.
  2. Limited model coverage: The evaluation is confined to only two systems that share the exact same low-level action policy (π0.5\pi_{0.5}). This shared dependency significantly narrows the scope of the conclusions regarding broader model capabilities.

Overall, the empirical evaluation in its current form does not sufficiently substantiate the paper's core claimed contributions regarding granular capability diagnosis and cross-task transfer analysis.

Questions for authors

[Clarification on Task Setup] What was the exact, full instruction provided to the model during the evaluation of Task 06? While the instruction in Figure 5 only specifies funnel placement, the evaluation rubric explicitly includes a requirement to transfer at least 80% of the liquid. How was this liquid transfer condition conveyed in the actual model input, and how are the final success criteria and intermediate step scores formally defined for this task?

[Empirical Evidence for Failure Diagnosis] Could the authors provide step-by-step completion rates and conditional success rates (i.e., success conditioned on reaching the preceding step) for Task 14? Specifically, I am interested in whether logged scale readings and dispensing/stopping actions empirically support a specific bottleneck—such as visual digit recognition, tare/zero-point calibration, or fine-grained dispensing control. Furthermore, please clarify how the aggregated motion-level results in Figure 4 directly link back to the primary failure causes in this specific task.

Reviewer_atyTOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces WET-OP Bench, a dual-arm robotic manipulation benchmark consisting of 20 wet-lab tasks. The benchmark incorporates evolving experimental states, including liquid transfer, quantitative instrument feedback, and reaction-dependent visual changes. The authors also introduce Lab2Sim, an automated pipeline for generating and validating simulation assets, and provide stage-wise evaluation metrics to assess task progress and failure modes. Experiments with a fine-tuned VLA model and a harness-based agent demonstrate substantial difficulties in completing wet-lab procedures, particularly under zero-shot generalization.

Critical strengths

1. Meaningful application domain.

The paper addresses robotic manipulation in wet-lab environments, where task success depends not only on object manipulation but also on experimental outcomes. Incorporating liquid transfer, quantitative measurements, and reaction-dependent feedback is a potentially useful extension of conventional manipulation benchmarks.

2. Structured task organization and evaluation.

The benchmark provides a systematic organization of laboratory operations, controlled generalization settings, and stage-wise evaluation metrics. These components could provide useful diagnostic information about the capabilities and limitations of embodied agents in laboratory environments.

Critical weaknesses

1. Insufficient justification of the benchmark's novelty and necessity.

My primary concern is whether the proposed benchmark provides sufficiently distinct evaluation capabilities compared with existing general-purpose robotic manipulation benchmarks such as RoboCasa365 and RoboTwin 2.0.

These benchmarks already support diverse manipulation tasks, long-horizon execution, task composition, and generalization evaluation. If the primary objective is to evaluate robotic manipulation capabilities, it is unclear why training or evaluating agents on a smaller collection of laboratory-specific tasks would provide additional insights beyond those obtained from existing large-scale benchmarks.

The most distinctive aspect of WET-OP appears to be its incorporation of experimental states, such as liquid volume, spill quantities, and reaction-dependent feedback. However, the paper does not sufficiently demonstrate whether these components measure capabilities that cannot be captured by conventional manipulation evaluation.

In particular, it remains unclear whether failures arise from insufficient general manipulation capabilities or from difficulties in understanding and responding to evolving experimental states. A controlled comparison between manipulation-only and experimental-state-aware evaluation would help establish the necessity of the proposed benchmark.

2. Insufficient validation of robotic embodiment and simulation fidelity.

The paper emphasizes complex wet-lab manipulation, but provides limited discussion of whether the selected robotic hardware, particularly the end-effector, is suitable for the proposed tasks.

Laboratory operations such as powder scooping, precise liquid transfer, and instrument manipulation can be highly sensitive to gripper geometry, dexterity, and contact dynamics. The paper should more clearly specify and justify the end-effector configuration and distinguish failures caused by policy limitations from those arising from hardware constraints.

More importantly, the experimental-state simulation is not quantitatively validated against real-world physical processes. Although the simulator models liquid transfer, spill quantities, and reaction-dependent appearance changes, it is unclear whether these simulated outcomes accurately reflect real laboratory operations.

Without physical validation, it is difficult to assess whether performance on the benchmark is meaningfully related to real-world wet-lab manipulation or sim-to-real transfer.

The experimental evaluation is also limited. Only one fine-tuned low-level VLA policy and a harness-based extension are compared. Furthermore, the zero-shot evaluation is reported only for the VLA policy, which succeeds in just one of 120 episodes. This makes it difficult to determine whether the benchmark can reliably distinguish different levels of procedural generalization capability.

Questions for authors

Q1. Differentiation from existing manipulation benchmarks

What specific robotic capabilities can WET-OP evaluate that cannot be adequately assessed using existing benchmarks such as RoboCasa365 and RoboTwin 2.0? Could the authors provide a controlled experiment demonstrating the additional value of experimental-state-aware evaluation?

Q2. Connection between experimental states and robotic control

How do the simulated quantitative variables, such as transferred liquid volume and spill quantities, contribute to evaluating closed-loop robotic manipulation? Can the authors distinguish failures in conventional manipulation from failures in interpreting and responding to experimental feedback?

Q3. End-effector suitability and physical validation

What end-effector configuration is used, and how was its suitability for the proposed laboratory tasks established? Furthermore, are there quantitative comparisons between the simulated and real-world outcomes of liquid transfer, spill behavior, or instrument measurements?

Q4. Generalization and baseline evaluation

Why is zero-shot evaluation reported only for the fine-tuned VLA policy, rather than also including the harness-based agent? Given the extremely low zero-shot success rate, can the authors provide additional evidence that the benchmark meaningfully differentiates procedural reasoning capabilities across agents?

Reviewer_bzVYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces WET-OP Bench, a dual-arm simulation benchmark containing 20 laboratory tasks. Fourteen tasks provide 500 training demonstrations each, and six are held out for transfer to new operation compositions or experimental conditions. The simulator combines manipulation with particle-based liquid transfer, event-conditioned reaction appearance, and quantitative instrument feedback. Lab2Sim constructs and validates laboratory assets, while reusable interaction graphs maintain the experimental states used for observations and evaluation. Binary success is complemented by weighted stage scores. Experiments compare a multitask π0.5 policy with a GPT-6-Astra harness using the same frozen policy, and evaluate π0.5 under physical, visual, linguistic, and held-out-task shifts.

Critical strengths

1,A relevant benchmark objective that goes beyond completing robot motions. Containment, target mass, and reaction-dependent stopping conditions are meaningful objectives that can fail despite substantial procedural progress. Coupling observable consequences and success checks to the same maintained simulator states is a useful design choice for evaluating closed-loop laboratory manipulation (Sections 3.4 and 4.2).

2,A structured task suite with explicit generalization regimes and diagnostic rubrics. The separation of demonstrated tasks from six held-out tasks, combined with isolated configuration/background/instruction variations and stage scores, provides a practical framework for studying transfer and locating failures. The appendix supplies task-level rubrics rather than leaving all evaluation at an aggregate success rate.

Critical weaknesses

1,The connection between stated laboratory objectives and implemented evaluation is not sufficiently established. In Appendix D, the instruction for Funnel-Assisted Liquid Transfer asks only for funnel placement, while the rubric also requires liquid transfer and returning the cylinder upright. Conversely, Centrifuge Tube Loading specifies opposite rotor slots, but the listed insertion criteria do not explicitly check that relationship. Some reaction- or stirring-related tasks also use geometric milestones without an explicit corresponding outcome check in the listed rubric. These may be documentation omissions rather than implementation errors, but that distinction is central to a benchmark whose main claim is experimental validity. Appendix B describes the mechanisms qualitatively, without quantitative calibration or sensitivity evidence for the liquid, mass, and reaction thresholds. Internal consistency of rendering and evaluation is helpful, but does not itself validate those task semantics.

2,The empirical design is too narrow to support several capability-level interpretations. Both evaluated agents share the same low-level π0.5 policy, and the main comparison uses only ten episodes per task. The 26.4% versus 25.0% aggregate success difference is not accompanied by uncertainty estimates. The quantitative-target group contains one demonstrated task, while reaction-observation tasks coincide with the procedure group in Table 3, so these attributes are not disentangled. Most importantly, no feedback-removal control establishes that reaction or instrument feedback is actually necessary for success; low performance alone does not identify failure to use that feedback. These limitations weaken claims about the benchmark's ability to diagnose distinct reasoning and control capabilities.

Questions for authors

1,Clarification affecting benchmark validity: What are the exact binary terminal predicates, as distinct from the weighted stage rubrics? Do they enforce opposite centrifuge slots, actual stirring/reaction requirements, and all instruction constraints? Please reconcile the funnel-task instruction with its scoring requirements.

2,Evidence affecting the central claim: On titration and target-mass tasks, how does performance change when reaction/readout feedback is hidden or held constant while initial conditions vary? A control of this kind would establish whether success requires the intended state-dependent response rather than a learned timing or motion template.

3,Clarification affecting the comparison: Does the harness receive the same action/step budget as standalone π0.5, including retries? What additional information is in its annotated demonstration, and can uncertainty for the paired agent comparison be reported?

4,Clarification affecting reproducibility: How were the 500 demonstrations per task generated and validated, what are the numeric train/test variation ranges, and how were Lab2Sim asset admission and repair success measured? The supplied appendix does not make these details sufficiently concrete.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.