- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces WET-OP Bench, a simulated bimanual wet-lab benchmark with 20 tasks. The tasks are grouped into four operation families: Handling & Placement (HP), Transfer & Mixing (TM), Instrument Operation (IO), and Metering & Dosing (MD). They are also split into two scopes, basic operations and full experimental workflows. Each task is annotated with its horizon, precision requirement, whether it has a quantitative target (Q), and whether it requires observing a reaction (R). 14 tasks come with 500 demonstrations each, and the remaining 6 are held out for zero-shot evaluation.
The benchmark is built on a set of reusable interaction graphs implemented in OmniGraph. These track physical state such as liquid transfer and spillage (via PBD particles), event-triggered appearance changes, and balance readings, and the same state is used both for policy observations and for success checking. Assets are generated with Lab2Sim, an agentic asset-generation workflow.
The paper evaluates two systems: a multi-task fine-tuned π0.5, and a harness agent in which GPT-6-Astra handles planning and calls the same frozen π0.5. In the Mixed setting, both reach a success rate of about 25% with stage scores above 55. On the zero-shot tasks, π0.5 succeeds in only 1 of 120 episodes.
Critical strengths
This paper brings several wet-lab capabilities together in a single evaluation suite. These include bimanual coordination, instrument readings the policy can observe, simulated liquid behavior, and task termination that depends on the current reaction state. Combining them in one benchmark is a useful contribution to the community.
Critical weaknesses
Weakness 1: The evaluation is thin. It covers only one policy family: the fine-tuned π0.5, and a harness agent that wraps the same π0.5. In addition, zero-shot results are reported only for π0.5. The harness agent is left out.
Weakness 2: The validity and feasibility of the tasks themselves are questionable.The construction details is not enough. For example, it does not explain how the demonstrations were generated. It is also unclear whether R-type tasks genuinely require observing the reaction.
Questions for authors
- How were the demonstrations generated(need more details)
- What exactly are the scoring criteria? How are stage scores defined and weighted,...etc
- Could the authors state the paper's positioning more clearly? A side-by-side comparison table against LabUtopia, AutoBio, and Labimus may help.